There are ways to be more clever than just mmap() + madvise() to improve performance in cases that might block (e.g. mincore() and some other system interfaces) but the further down that road you go the less portable and more complicated the implementation becomes.

In short, mmap() and friends were designed for a world where context-switching (e.g. multithread or multiprocess) is a great idea. Unfortunately, context-switching has become extraordinarily expensive for a lot of server software which makes multithreading a less palatable option.

These days, you want (1) native async I/O, (2) strict control of cache replacement behavior, (3) strict control of I/O scheduling, (4) minimal context-switching, and (5) memory locality control. On Linux, this means DIRECT_IO, io_submit, managing your own physical cache RAM, and locking one thread to every core (ignoring hyperthreads for the moment). This is more complicated to implement because there is not a simple, portable interface like mmap() widely available but it is also much more efficient and performant when done correctly. To make matters worse, some useful interfaces (like io_submit) are poorly documented.

But yes, if you build your own memory and swapping subsystem directly on top of the native interfaces, and use threads as an abstraction of a core rather than a swappable context, you can build very efficient server engines that stall minimally on I/O. (Note: even using io_submit and IO_DIRECT for async disk access, there are conditions that can cause blocking. They are just much rarer and easier to manage than mmap().)

I'll just repost my famous comment from two years ago:

If you want your program to take page faults as PHK suggests, it has to be multithreaded. If you choose event-driven concurrency you can't afford to take page faults in mmap() or read(). When you make the threads vs. events decision you're implicitly making a bunch of related decisions about I/O and scheduling as well; a hybrid approach (like using events and mmap) won't work well. http://news.ycombinator.com/item?id=1760642

Varnish is heavily threaded. It maintains queues where connections are put and worker threads pull them out. It is expected that a single connection gets a dedicated thread.

Since each thread is an actual kernel thread, this will limit the concurrent connections to the maximum number of threads a kernel can handle which isn't that high.

Not as such. The connection itself can be accepted and put into a queue on a different thread than the one that serves the request. This means that only the actual number of requests concurrently being fulfilled (cached value is being retrieved from ram or storage, or is being written to the socket) is limited by the amount of kernel threads. With that in place that number doesn't really even have to be that high to handle lots of concurrent load.

Linux can create over 250,000 threads, but that may have been on a 32-bit system. On 64-bit it should be limited only by RAM.

The overhead of context switching becomes pretty high. Some say that context switching has become cheap, but you still at the very least need to update the tlb, and schedule the next pthread.

At least the performance of context switching should scale with the number of cores, which seems to be the main direction of increased performance in hardware looking into the future.