What is always missing for me in these lists is a tutorial for knocking together a small lambda calculus + dependently typed interpreter in C. Not ML, Not Ocaml, not, Haskell, not any language that has _already_ got lambdas in it!
Always when I read these I see, "get your Haskell compiler and..." and I'm like, if I knew Haskell well enough to do that I wouldn't be doing this tutorial in the first place.
Case in point, article says "particularly for programming pracitioners who didn’t learn it at school." -- Correct! I taught myself Basic, I taught myself a bit of x86 assembler, most importantly, I taught myself C. My working mental model of a computer is the C machine model. And even now, now that I know Ruby with its lambdas, I know that Ruby is written in C. And the Linux kernel. And Nginx (C++ maybe? Dunno). And on, and on.
Show me the tutorial where I can build, _in C_ a bare bones working (tail recursion) implementation of the lambda calculus with type inference and dependent (algebraic?) types and I'll shake your hand and call you a champion.
Closest I've seen is Make A Lisp - https://github.com/kanaka/mal but I don't think it has the sweet types and tail recursion and other good stuff. PS: bonus points if Regexen are native type :)
> knocking together a small lambda calculus + dependently typed interpreter in C. Not ML, Not Ocaml, not, Haskell, not any language that is _already_ got lambdas in it!
This wouldn't be any more educational and just plain painful. "Research" programming languages tend to work very well for building compilers, that's why they are used in many toy programming language projects.
Dealing with tree and graph structures in a language like C would be a lot of work, a lot of effort going into memory management which is not at all useful towards the goal of learning how to implement a language.
To give a practical example: last time I was working on a toy compiler using Haskell, I needed to calculate the strongly connected components in a directed graph to perform dependency analysis (rearranging "letrecs") that is required to do type inference. Had I been writing the compiler in C, it would have required a week's worth of engineering to get done. With Haskell, I could grab Data.Graph.SCC and be done in a few hours. Arguably I could have picked up a graph library in C, but then I'd have to spend time understanding how memory management works in the library and so on.
I understand the appeal in using C or just an imperative programming language but quite frankly, that would really hurt the pedagogic aspects of a project like that.
The flip side of this is that if you're writing something in C, you actually do have to understand how details like closures are implemented on real hardware. These are not trivial problems, yet many programming language design papers and articles effectively brush the whole issue of actually implementing these things under the carpet.
Go ahead and search for that particular example. There is almost nothing on the Web about practical, real world techniques for implementing that sort of programming language feature, despite its ubiquity in functional languages and the increasing presence of -- or emphasis on -- higher order functions and related tools in mainstream imperative languages. The same could be said of many other non-trivial language features, say laziness, or various concurrency models.
I think this is regrettable, because it creates a real barrier for anyone who's interested in learning about these features and maybe writing their own languages one day, but who doesn't come from an academic background where this kind of material was taught. It also makes it unnecessarily difficult for someone from an imperative programming background -- which is probably still the overwhelming majority of programmers -- to understand the real costs and performance implications of using these kinds of features.
This is not quite true. There is SPJ's famous "The implementation
of functional programming languages" [1]. Then there is lots of
material on various abstract machines, like the SECD machine, the
CAM, the many variants of the Krivine machine, the G-machine and
plenty of others whose name I forget.
All that said, I kind-of agree with you. It would be nice to have some
easy to grasp explanations of how to compile higher-order functions
to actual machines, x86, ARM, MIPS ... This material can be found
e.g. in Appel's book [2], but I don't think it's available
freely (and legally) on the web. I think this is in parts because
the compilation process is not so easy: you need to understand the compilation of normal languages, in particular stack layout, and then add more pointers. You also need to think about how to do memory management if a closure lives longer than its creating context. This is quite rich material, and not even usually covered in undergraduate compilers courses.
The situation with concurrency is much worse, but that only
reflects the fact that this is far from a solved problem on many
levels. To cite a recent paper [3]:
"Despite decades of research, we do not have [in 2015] a satisfactory concurrency semantics for any general-purpose programming language that aims to support concurrent systems code [...] Disturbingly, 40+ years after the first relaxed-memory hardware was introduced (the IBM 370/158MP), the field still does not have a credible proposal for the concurrency semantics of any general-purpose high-level language that includes high-performance shared-memory concurrency primitives. This is a major open problem for programming language semantics."
And memory models are but one issue to deal with in
concurrency.
> The flip side of this is that if you're writing something in C, you actually do have to understand how details like closures are implemented on real hardware.
If you're writing a compiler targeting real hardware and not a high level virtual machine, you'll need to learn how to implement closures regardless of the "host" language you write your compiler in.
But you're correct, when implementing an interpreter, it's so much easier if you have a garbage collector and an object system of some kind at hand. You'll have to implement this yourself if you're working with C.
If you're writing a compiler targeting real hardware and not a high level virtual machine, you'll need to learn how to implement closures regardless of the "host" language you write your compiler in.
Right, but a lot of the tutorial material in this area doesn't really consider that issue, even though it's rather fundamental to practical applications. Instead the explanation often settles for some sort of indirect dependency on the implementation language's existing version of a similar feature. I think this is unfortunate, not least because it reduces us to a model where most people implementing these ideas depend on some sort of magic run-time environment instead of learning how to actually write that run-time.
Thats because tutorial material is going to keep things simple. If you really want to implement a garbage collector on your own then you should go looking for an advanced compiler book instead of a tutorial.
Of course. But it's still reasonable to ask how you'd write a compiler that implements such features in a free-standing way, without depending on either the built-in tools of the language you're writing an interpreter in or a black box run-time environment/VM that does all the tricky stuff for you by magic. With a relatively low-level language like C, you don't have any choice but to confront such issues head-on, while many tutorials manage to side-step them using higher-level languages. Of course it's easier to write almost anything in a suitable higher-level language, but for educational purposes that isn't really the point IMHO.
You're confusing levels. If you are compiling to C or assembly, you have to understand how the constructs you are building work. That places no requirement on the language of implementation of the compiler.
Yes, but IME many tutorial presentations on these subjects never get as far as actually building a complete, functional compiler to demonstrate the concepts. Often they use some form of interpreter instead, which means they can use structures in the interpreter's implementation language that somewhat reflect the run-time structure of the implemented language using the analogous language features.
Well, it makes sense for tutorials to use what they can to simplify the bits they're not focusing on, but certainly implementing a construct by mapping it into an identical construct is not going to be terribly informative as to the workings of that construct. That may be totally appropriate if the focus of the tutorial is on the parsing or such, but it sounds like that sort of tutorial (of which there are many, and which are perfectly useful to other purposes) are not what you are looking for.
I'm not sure if it satisfies all your needs,
but Scheme 9 from Empty Space is a full R4RS Scheme implementation written from scratch in pure C (and Scheme).
It has a heavily commented source code and a book, which is supposed to be quite good.
Here are some key points from the website which the book addresses:
- How does tail call elimination work?
- How is macro expansion implemented?
- How is arbitrary precision arithmetics implemented?
Show me the tutorial where I can build, _in C_ a bare bones working (tail recursion) implementation of the lambda calculus with type inference and dependent (algebraic?) types
Why would you want to do that?
Type inference and dependent types don't go well together (already System F doesn't have type-inference), although it depends on the details of what you mean by dependent types.
Tail recursion is a concept of compilation and does not show up in an interpreter.
Note also that C has higher-order functions (via function pointers). The only relevant difference from lambda-calculus is that C functions must be named, cannot be nested and must exist at compile time. But conceptually these restrictions make little difference to the understanding of what lambda does: it creates a function.
Finally writing something like that (to the extent that the requirements make sense) is gonna be really painful in C. If you understand the lambda-calculus well enough to know what dependent types are, the implementation language should not make much of a difference.
I think the easiest way to learn the lambda-calculus
program in a functional language. The second easiest way
is to read the theory, after all the lambda-calculus as a calculus is incredibly simple, and you can hand-wave away the problems of bound-variable renaming. A slow-paced textbook like Hankin's
"An Introduction to Lambda Calculi for Computer Scientists", and doing the exercises in the first few chapters should do the job.
I don't think the grandparent wants all of these features at once necessarily in a single tutorial, but an introduction to them in a language that they already know.
Also, the things you say that function pointers don't have are exactly the things that make lambdas interesting. Lambdas are anonymous and, crucially, can be nested. It's the very nesting that is what makes them difficult to get right, as they close over their containing scope. Variables mentioned in lambda bodies that are not parameters get their values from the definition scope, not the scope where they are used. In C function pointers, this issue doesn't even come up.
Indeed, you have put it more succinctly and cogently that I ever could.
I would like an introduction to these advanced topics in the language that I already now. For instance, for a Haskell module I was studying we used a teeny tiny interpreter that had been created just for the task of learning the basics of Haskell. This interpreter was written in Lua. It was small, and I could follow the logic of it. It lacked type inference though, and if I had seen a tutorial _in C_ about how to bolt on a Hindley Milner type inference engine to a lambda calc core I bet I could have made a stab at improving this toy project.
Even with Ruby now I bet I code Ruby as if it were a really dynamic C. I hardly ever use map() or collect(), for instance. I just don't have it in my mental toolbox. I'm not total loser though, I do like my {|o| blocks!}
I need to emphasize this. I am fluent in C, and can think in it. When I open Pierce TaPL and see immediately ML my heart sinks.
When I open Pierce TaPL and see immediately ML my heart sinks.
I strongly recommend that you ignore this feeling and learn a modicum of ML coding. If you want to learn TAPL, just learn a bit of Ocaml. You don't need much. If you already speak C, a bit of Haskell and Ruby, you should be able to pick up enough Ocaml in a few hours/days. Programming in Ocaml is way easier than in Haskell or C. The only two things that are a bit difficult in Ocaml are modules and the object system. You can completely ignore objects. Nobody ever uses them, and for TAPL you can also ignore modules (although you occasionally have to open a module, which is trival, something like: open Support.Pervasive).
I feel like C fits my brain, know what I mean? I don't even think that it was because I learnt it early on. It's a simple language, I feel.
Why is programming way easier in Ocaml than in C? Setting aside that for me programming is way easier in C than in Ocaml because I know C and don't know Ocaml, what is it about Ocaml that is way easier than C?
Anyway, my original point is that it is _cheating_ to explain how to build a Lisp by saying, "First reach for your nearest functional programming language". Useless car analogy. Today I'm going to learn how to build a car. First, take the engine from my Toyata... oh... um...
I've written vast amounts of Ocaml and C/C++, certainly over 100K LoCs in each. Programming in ML dialects is so much easier than C/C++, it's not funny. I'd estimate it takes me about 10 times longer to implement the same functionality in C/C++ than in an ML dialect. The three key features enabling this difference are the following.
* Garbage collection, removes all memory bugs, avoids tons of ugly and distracting (de)allocation commands and keeps your head free to think about more important stuff.
* Pattern matching, enables you to code up complex decision making code (and most functinality needs this) in a readable and maintainable way.
* higher-order functions allow you to parameterise code with behaviour in a neat way.
(There is one caveat, which is code for extremely high performance, I'd probably write that in C/C++, but this is not an issue concerning a learner.)
As to useless car analogies, I think implementing lambda in C is like saying: Today I'm going to learn how to build a car. First, let's do this while riding a uni-cycle ... oh... um...
Are you sure C is simpler then ML? If you really get into the nitty gritty details (which is what you do when interacting with programming language theory), the definition of the lambda calculus fits in a single page and an ML-lite that is powerful enough for real programming is just a little larger than that.
On the other hand, C needs a manual with hundreds of pages and has lots of dark corners and undefined behaviour. IMO, if you really want to keep things simple you need to stick to assembly language instead of C.
You think that assembly needs less of a manual than C? You think assembly has less dark corners? You think it's going to make things simpler? My mind boggles.
Oh, and also, which assembler? 68000? x86? ARM? (One of these is not like the others in terms of complexity...)
Maybe talking about undefined behavior wasn't the best approach I could take :) But if you stick to simple machine languages then it shouldn't be a big problem. Something that is closer to Knuth's MIX than to x86 won't have much undefined behavior.
The thing that I was thinking about was that many operations that are defined in assembly language, such as integer overflow are undefined in C (and this undefined behavior is often abused by optimizing compilers) and C also abstracts over a lot of control flow conventions.
> Today I'm going to learn how to build a car. First, take the engine from my Toyata... oh... um...
To the extent that the analogy works, it doesn't help your case: nobody these days builds or learns how to build a car from scratch, without access to an existing car. It's not only that you're typically making a modified version of an existing model: you also use a car to get to the factory, you use existing vehicles to transport raw materials and tools to the factory, etc.
In any case, as others have pointed out, it's not like you need to use a full {Lisp, Haskell, ML} compiler to implement (say) tail recursion. It's that you bootstrap: as soon as you have a minimal viable {Lisp, Haskell, ML} compiler, it makes sense to use that instead of C to implement other features.
I agree with you it'd be useful to see how you might implement lambdas in a language like C. But after that first example, implementing other more complicated language constructs in C, instead of learning ML/haskell is going to be more time fighting C, and less time learning anything new. The lack of side-effects and equational reasoning in functional languages let you get a lot of details out of your way.
That being said, this is if you're interested in the theory. If you're interested in building compilers, this list won't help too much.
The lack of side-effects and equational reasoning in functional languages let you get a lot of details out of your way.
Actually ... You (kind-of/sort-of) need side effects when implementing a language with higher-order functions. You at least need it for generating fresh variables to avoid free name capture. Yes, I know you can do this with a monad, but this is not going to help a beginner. It's much easier to maintain a global counter that you increment every time you want a fresh variable name.
The way I did it was just recursively pass down an accumulator argument to generate De Bruijn index-like variable names, rather than keeping a global counter
Point taken though that it's a bit simpler with global state
> I would like an introduction to these advanced topics in the language that I already now.
The problem with this may be best expressed by Whorf[1]:
"Language shapes the way we think, and determines what we can think about."
A philosophical school of thought concerned with this is known as Linguistic Relativity[2]. But I digress.
Part of internalizing foreign concepts is approaching them with minimal prior prejudices. Trying to relate concepts inexpressible in C to C idioms is extremely difficult, if not impossible, before being able to _translate_ those concepts into what can be represented in C. It's the classic "chicken and egg" problem.
The best thing I can think to recommend is to pick a language which is generally considered capable of supporting the concepts you wish to learn and approach it as if you have never seen a programming language before.
> When I open Pierce TaPL and see immediately ML my heart sinks.
IMHO, this reinforces the fact that tools which you have used before (likely to great success) are influencing your view of those tools which you are not as familiar. IOW, you learned BASIC, x86 assembler, and C... Why not ML?
Best of luck and hopefully the path you walk is fun!
It's actually a lot easier to learn ML and then read/write the interpreter in ML than to read/write the interpreter in C. An interpreter written in C would be too convoluted.
I did point out the nesting and the requirement to name in C.
The scoping follows immediately from beta-reduction
(lambda x.M)N --> M[N/x]
which I find very intuitive. I never had trouble understanding this, but maybe I'm overgeneralising from my own learning process. The real difficulty, or rather the real subtlety, is not nesting but name binding and the automatic renaming of bound variables where necessary. Having taught lambda-calculus to generations of students, that causes confusion with some students.
And I think implementing a type-inferencer in C is going to be confusing because beauty of e.g. the W type-inference algorithm is drowned by orthogonal issues that C forces on you like memory management. Interpreters are best written in languages with pattern matching. So I prefer to bootstrap one's learning: get the basics of lambda-calculus, then implement a lambda-calculus interpreter and type-inferencer in a modern FP language. (As an aside, that's what SICP also does.)
The real difficulty, or rather the real subtlety, is not nesting but name binding and the automatic renaming of bound variables where necessary.
If you're trying to understand what a closure does, perhaps that's true.
If you're trying to understand how it does it, and you come from an imperative programming background where you know the ABIs for your languages like the back of your hand and are used to the kind of function set-up that always allocates space for local variables on the stack, there is a huge question of how all that lovely, neat theory is actually implemented in practice.
Some of this discussion reminds me of the jokes about professors who prove that a solution to some elegantly expressed problem must exist, but then lead it to the grad students to actually find it...
That's a _bit_ extreme though. Let's start with C, the portable assembly language. Having said that, now that WebAssembly is on the cards (boo, hiss), we'll have many more of these from-the-ground-up affairs.
re: reddit thread: that Kragen, name is familiar...
That "C is portable assembly language" mantra bugs me a bit. There are lots of things that are dirt simple in real assembly language but impossible to do in (standard) C: tail call optimization, computed gotos, arithmetic overflow detection, SIMD, control over registers and local variable allocations, etc...
If you want to make things really low level, then make a compiler that targets assembly language and then bootstrap it. Coding the first version of the compiler in assembly or C instead of something more suitable for the task is just unnecessary pain.
I don't know how field-famous he is, but googling his name (he has a ML named after him) was full of fun crazy hacks idea. Like writing binaries under MS-DOS (just the few commands it provides like dir, copy etc) by using codepage keycodes `COPY CON > MY.COM`. And then expanding possibilities.
I agree it's a bit much to answer your question, but the spirit is there. And I agree, although learning lisp in lisp was very valuable on the intellectual level, the ignored primitives (cons, gc, interned symbols, IO) are worth knowing too. I found this book (warning: french) http://www.decitre.fr/livres/la-programmation-applicative-97... at a library, discussing a scheme system up until the io primitives. It was a great read. Sidenote: they called macros doubly-recursive functions :)
There is no point in doing such things in C, since C is just too low level.
What is normally done is a very simple and stupid, but extensible language bootstrapped with C (or whatever else), and then grown up to a level when it's convenient to implement a functional language compiler (i.e., at least higher order functions and some pattern matching).
And a great many fine suggestions have been made. To add one more, you might take a look at CHICKEN Scheme. It's written in C, and compiles to C.
The CHICKEN website and user manual contain a wealth of info re: theory and implementation of the compiler. The development team has always been friendly and willing to answer questions.
The first language I learned was C. I don't claim to be an expert, but I'm comfortable enough with it that I can hack on an OS kernel with C.
I have also learnt how to write compilers for simple languages (mostly from Appel's book). Through reading the book, I picked up ML. At first it was an uncomfortable transition from C, but I eventually loved writing ML code. Strong typing eliminates a lot of common errors, and the compiler really feels like a friend that's watching over you.
Later on, I decided to pick up Haskell. Frankly it was a lot harder to get used to it (I still can't say I understand Haskell) because of how side effects in Haskell are made much more explicit than either C or ML. But after about a year and a half, I can now code comfortably in Haskell. And if someone let me choose a language that I can only code in for the rest of my life, it might actually just be Haskell.
But ML and Haskell are not without their problems.
C and functional languages like ML/Haskell have different problem domains. Haskell, being (almost) pure, is really good at modeling side-effect free computations.
The compiling process is really a translation process (thus a computation), and it's easier to model it in Haskell. It is quite easy to mess things up in a language like C where side-effects show up everywhere, although if you're disciplined enough you could probably translate a Haskell program into C with some modifications.
But Haskell/ML is definitely not for everything. I would not write a kernel in Haskell (although I do know people who have done it). With the runtime problem aside, it doesn't give me enough control over the bare metal it's running over (or it's probably I just don't know it well enough yet). I can't layout things exactly the way I want with Haskell, but with C, I can build structs, cast integers into pointers (might not be a good idea, but sometimes necessary when you're dealing with a kernel) and so on. With C it feels like you're the creator, taking care of everything, hence, you have much more responsibility on your shoulder. But with Haskell, the compiler takes care a lot of things for you if you're in the right problem domain, and it lets you focus on abstractions and the problem itself.
Anyway, just a little thought. I love Haskell and C equally, and they're both absolutely wonderful languages. But try to keep yourself open and just try a little bit of new things at a time. Pretty soon you might find yourself designing compiler for your own language in Haskell (or C)!
Always when I read these I see, "get your Haskell compiler and..." and I'm like, if I knew Haskell well enough to do that I wouldn't be doing this tutorial in the first place.
Case in point, article says "particularly for programming pracitioners who didn’t learn it at school." -- Correct! I taught myself Basic, I taught myself a bit of x86 assembler, most importantly, I taught myself C. My working mental model of a computer is the C machine model. And even now, now that I know Ruby with its lambdas, I know that Ruby is written in C. And the Linux kernel. And Nginx (C++ maybe? Dunno). And on, and on.
Show me the tutorial where I can build, _in C_ a bare bones working (tail recursion) implementation of the lambda calculus with type inference and dependent (algebraic?) types and I'll shake your hand and call you a champion.
Closest I've seen is Make A Lisp - https://github.com/kanaka/mal but I don't think it has the sweet types and tail recursion and other good stuff. PS: bonus points if Regexen are native type :)
(Open to suggestions (I am))