sijie commented on issue #1334: Experimental settings to achieve 5 ms latency URL: https://github.com/apache/incubator-pulsar/issues/1334#issuecomment-408508215 @ratcashdev > Kafka (and also RabbitMQ) can achieve sub 1 ms latency, while using persistence. `persistence` means different things when different people talk about it. when most of the projects talking about disk persistence, they mean data are written to filesystem. However when data is written to filesystem doesn't really mean data is persistent. Kafka only writes to filesystem without fsyncs. the "persistence" here is a `filesystem` persistence, but it still faces data-loss possibility. In relational databases, people talk about "ACID" - "D" for "Durability", when data is persistent to disk, it means writing to filesystem and also *fsync* to disk to ensure data is not lost when machine crashes. this disk-level persistence has much stronger data integrity than filesystem-level persistence. The DZone blog post is benchmarking kafka with filesystem-level-persistence. since most of the data is in filesystem page cache, which even not touch disks. so the latency is expected to be low, since those are effectively "memory" latency. The gigaom report is benchmarking pulsar and kafka with disk-level-persistence (However due to kafka's design, they can not really achieve disk-level-persistence as what Pulsar can achieve. The benchmark was using a close-enough settings to simulate same disk persistence behavior to do an apple-to-apple comparison). so the latency measure there were real disk persistence latency. keep "filesystem-persistence" and "disk-persistence" in the mind when you read the blog posts, they will help you understand those numbers. Pulsar recently added a flag to disable `fsync` behavior. so frankly speaking, pulsar is able to achieve both `filesystem-persistence` and `disk-persistence`, people can decide to choose which one based on their tradeoffs. However I don't think there is any benchmark results regarding pulsar `filesystem-persistence`. All the benchmark results or claims in the blog posts about 5ms latency are about `disk-persistence`. for replicating the claims, you can try out [open-messaging-benchmark](http://openmessaging.cloud/docs/benchmarks/). open messaging is a linux foundation project on building a vendor neutral messaging standard and benchmark. Hope this helps.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
