> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
You've made very good points about smart clients, but at some point one has to ponder if it's worth it or one should just not use kafka in the first place.
I've seen databases used as messaging queues and if it was up to me, I'd never do that. It's usually "but we already have kafka + db, why burden ourselves with another messaging technology?", which is fair.
> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
That is very nice -- certainly seems better than just batch processing of kafka messages, but you're still just kicking the can down the road.
How large do allow the buffer to become and what do you do when it's getting too large?
You probably use a DLQ.
Don't get me wrong, I think the buffer idea probably works most of the time.
Completely agree. Kafka was another team's decision, not mine, so I had to figure it out. RabbitMQ is very convenient in that you don't need to read a couple of books on reliable data integration patterns to get something working simply and intuitively.
I am fond of Kafka now that I understand it, but I was also an assembly language programmer in a past life so my opinion is probably in the minority.
Regarding the buffer size: you need to implement back pressure, especially if you are CPU and not IO bound; it's another thing that's easy to get wrong with Kafka.
> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
You are quite correct - you absolutely can use auto offset commits in that case. In my scenario, though, I have a lot of messages and a low recovery time objective on service restart so I find it cleaner to skip messages I know I won't need. Also reduces noise on the service logs, makes for easier debugging etc.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.