Kafka vs Pulsar - why we should be skeptical about benchmarks
- 254 words
- 2 min
The Real Aim of Benchmarks
Comparison benchmarks in IT industry it's not the only way to prove some metric supremacy of some tech/frameworks, but also a way to promote your technology. The last point is often interested in vendors and companies providing consulting services. By myself, from time to time I check Web Framework Benchmarks to see how freaking fast Rust frameworks like Actix in comparison to more adopted web frameworks writen on other languages.
Kafka supremacy
As usually I wasn't surprised to see some benchmark of Kafka that's lead over competitors in real-time streaming, especially if this article was provided by Confluent: Benchmarking Apache Kafka, Apache Pulsar, and RabbitMQ: Which is the Fastest?
The Empire Pulsar Strikes Back
I was pleasantly surprised by the guys who are developing Pulsar and who gave a decent answer: Benchmarking Pulsar and Kafka - A More Accurate Perspective on Pulsar’s Performance
Why you should be sceptical about Benchmarks
- Methodology - each benchmark use own methodology that often doesn't represent all side of compared technologies that put technologies into non-competitive environment.
- Expertise in one technology and incompetence in others - as a result, we are able to see benchmarks, where compared competitive technology, could be not correctly set up/ not properly used in some cases.
- Wishful thinking - results of benchmarks could be interpreted wrongly with desire put one technology into better position to other.