Re: AWS EFS
On Wed, Jun 29, 2016 at 01:30:34PM +0100, Tom Crayford wrote: > I think you'll be far better off using EBS and Kafka's inbuilt distribution > than NFS mounts. Kafka's designed for distributing data natively, and not > for NFS style mounts. > I did some initial benchmarking of amazon EFS and agree with your assesment. I think you'll be better off with EBS as you can have slightly more control over the performance of the disk subsystem as opposed to EFS which is based on NFSv4. And while the performance of EFS on my FreeBSD test systems was OK, I did find that it was trivial to saturate the network interface the EFS volume is mounted on. I am personally evaluating EFS for usecases where I need to share data for content delivery or as a scratch space for PostgreSQL WAL logs for for example. Cheers, -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: [DISCUSS] Java 8 as a minimum requirement
On Fri, Jun 17, 2016 at 12:02:17AM +0200, jan.o...@wooga.net wrote: > > Hi Ismael, > > Unfortunately Java 8 doesn't play nice with FreeBSD. We have seen a lot of > JVM crashes running our 0.9 brokers on Java 8... Java 7 on the other hand is > totally stable. > > Until these issues have been addressed, this would cause some serious issues > for us. > I can second this on the FreeBSD front. -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Introducing Dory
On Mon, Jun 13, 2016 at 11:45:13AM -0600, Jason J. W. Williams wrote: > Hi Dave, > > Dory sounds very exciting. Without persistence its less useful for clients > connected over a WAN, since if the WAN goes wonky you could build up quite > a queue until it comes back. > I was thinking the same thing. My first thought was to see what the LOE would be to implement some sort of spill-over process, where if the preallocated memory segment is exhausted it could spool data to disk. When connectivity to the brokers is back it could then de-spool data and produce it back to the cluster. I could see this being something worth persuing if you are handling data that is critical for financial reasons (as opposed to data used for non-financial reporting or metrics). -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Using lz4 compression
On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote: More like getting a feel from the community about using lz4 for compression? Has anyone used in the kafka setup. I am aware that gzip and snappy are more older implementation and regressed. Given that lz4 has better compression/decompression cycles (though slightly less compression ratio), was thinking to leveraging the same. Regards,Virendra i use the lz4 compression algorithm quite extensively in conjunction with ZFS (ZFS can configure a filesystem to do transparent compression) and have not had any issues with it under load. i've also found that it does a better job than snappy with negligible overhead that i've been able to observe. i tend to avoid gzip in production as i have measured more overhead using this algorithm, and generally speaking i've found lz4 to compact data better. i am not super familiar with the lz4 code as implemented in kafka, but i would assume the java implementation is pretty solid. hope this helps, -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: LVM overhead
There will be some slight lvm overhead, depending on your configuration, but in my experience it will be negligible. I would suggest avoiding creating *any* snapshots when using LVM though as that will decrease performance pretty quickly. For kafka it is unlikely you will want to create any snapshots of your datastores - so you will probably OK. In my environment we use an HBA with a baterybacked cache, then create RAID-10 volumes in hardware. Then we overlay XFS on top of that volume. This seems to be a pretty optimal configuration in our environment as we can obtain greater IOPS from a RAID-10 volume as opposed to what you would get in a single disk (and the battery helps reduce the likelihood of data corruption in a power failure event). LVM will give you bit better perf in theory as well (assuming you create a stripe of disks) although there will be greater overhead managing the stripe in software as opposed to hardware. Hope this helps! -pete On 02/11/16 07:24, yazgoo wrote: > Hi everyone, > > I have multiple disks on my broker. > Do you know if there's a noticeable overhead using LVM versus multiple > log.dirs ? > > Thanks > -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Kafka and Btrfs
On 10/27/15 12:34, Elias Levy wrote: > Anyone using Kafka with Brtfs successfully? Any recommendations against > taking that path? > I would suggest avoiding BTRFS for any production systems, especially if you have high data integrity requirements. I know Linux distro's are starting to adopt it, but IMHO (and personal experience) I would not deem it production ready. If you are looking for a filesystem that supports many of the features (and more) of BTRFS I would take a close look at ZFS. It is a much more mature (and stable) codebase, and offers some features that could be beneficial for kafka (transparent filesystem compression, ability to cache read and write datasets on SSD storage, etc.). I ran some tests locally of a kafka broker using CentOS-6.x and XFS (which is what I run in production and consider to be a very stable filesystem) against FreeBSD-10.1-RELEASE and ZFS. Our ZFS system tended to outperform the XFS system in my local kafka consumer/producer tests - and performed better in synthetic filesystem benchmarks. Both systems were using OpenJDK-8.x, and were on the same hardware. If you do not want to live way outside of the sandbox there is ZFS support for Linux here: http://zfsonlinux.org/ http://open-zfs.org/wiki/Main_Page Hope this helps - and doesn't start a huge bikeshed discussion :) Cheers, -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
noticing blocking of kafka-network-thread
Hi There, while investigating higher than expected CPU utilization on one of our Kafka brokers we noticed multiple instances of the kafka-network-thread running in a BLOCKED state, all of whom are waiting for a single thread to release a lock. Here is an example from a stack trace: (blocked thread) kafka-network-thread-9092-10 - Thread t@50 java.lang.Thread.State: BLOCKED at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) - waiting to lock 2bb6fb92 (a java.lang.Object) owned by kafka-network-thread-9092-13 t@53 at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) (thread 53) kafka-network-thread-9092-13 - Thread t@53 java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.size0(Native Method) at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:84) at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309) - locked 2bb6fb92 (a java.lang.Object) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None What is interesting is that: 1) we have seen the broker with high CPU utilization float around our cluster. I.e node 0 will have high load for a time, then node 2, etc. 2) when we do a thread dump on normally operating brokers we find no BLOCKING threads 3) all nodes we capture thread dumps on seem to all get blocked by one thread. Has anyone else seen this, and if not - is there an easy way to determine which object these threads are seeing contention on? Thanks in advance! -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Design questions related to kafka
On 04/15/15 09:31, Manoj Khangaonkar wrote: # I looked the documents of kafka and I see that there is no way a consume instance can read specific messages from partition. With Kafka you read messages from the beginning multiple times. Since you say later that you do not have many messages per topic, you can iterate over the message and read the ones that you need. Of course this might not the be the most efficient. couldn't you also store the offset of the last read message, then resume reading messages after that offset to ensure your consumer does not consume the same message twice? Cheers, -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Updates To cwiki For Producer
Hi Gwen - thanks for sending this along. I'll patch my local checkout and take a look at this. Cheers, -pete On 03/20/15 21:16, Gwen Shapira wrote: We have a patch with examples: https://issues.apache.org/jira/browse/KAFKA-1982 Unfortunately, its not committed yet. Gwen On Fri, Mar 20, 2015 at 11:24 AM, Pete Wright pwri...@rubiconproject.com wrote: Thanks that's helpful. I am working on an example producer using the new API, if I have any helpful notes or examples I'll share that. I was basically trying to be lazy and poach some example code as a starting point for our internal tests :) Cheers, -pete On 03/20/15 10:59, Guozhang Wang wrote: For the new java producer, its java doc can be found here: http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html We can update the wiki if there are some examples that are still missing from this java doc. Guozhang On Thu, Mar 19, 2015 at 4:37 PM, Pete Wright pwri...@rubiconproject.com wrote: Hi, Is there a plan to update the producer documentation on the wiki located here: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example This would be helpful for people working on implementing the new producer class deployed in 0.8.2.x. If there are any patches available for this or other docs available for reference that would be helpful as well. Thanks! -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298 -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298 -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Updates To cwiki For Producer
Thanks that's helpful. I am working on an example producer using the new API, if I have any helpful notes or examples I'll share that. I was basically trying to be lazy and poach some example code as a starting point for our internal tests :) Cheers, -pete On 03/20/15 10:59, Guozhang Wang wrote: For the new java producer, its java doc can be found here: http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html We can update the wiki if there are some examples that are still missing from this java doc. Guozhang On Thu, Mar 19, 2015 at 4:37 PM, Pete Wright pwri...@rubiconproject.com wrote: Hi, Is there a plan to update the producer documentation on the wiki located here: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example This would be helpful for people working on implementing the new producer class deployed in 0.8.2.x. If there are any patches available for this or other docs available for reference that would be helpful as well. Thanks! -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298 -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Updates To cwiki For Producer
Hi, Is there a plan to update the producer documentation on the wiki located here: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example This would be helpful for people working on implementing the new producer class deployed in 0.8.2.x. If there are any patches available for this or other docs available for reference that would be helpful as well. Thanks! -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Sizing Cluster
On 10/21/14 21:13, István wrote: Hi Pete, Yes you are right, both nodes has all of the data. I was just wondering what is the scenario for losing one node, in production it might not fly. If this is for testing only, you are good. Answering your question, I think retention policy (log.retention.hours) is for controlling the disk utilization. I think disk IO (log.flush.* section) and network IO (num.network.threads, etc.) saturation you might want to measure during tests and spec it based on that. Here is a link with examples for the full list of relevant settings, with more description: https://kafka.apache.org/08/ops.html. I guess the most important question is, how many clients do you want to support. You could work out how much space you need based on that, assuming few things. For more complete documentation refer to: https://kafka.apache.org/08/configuration.html Thanks Istvan - this is helpful. Cheers, -pete -- Pete Wright Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Sizing Cluster
Hi There, I have a question regarding sizing disk for kafka brokers. Let's say I have systems capable of providing 10TB of storage, and they act as Kafka brokers. If I were to deploy two of these nodes, and enable replication in Kafka, would I actually have 10TB available for my producers to write to? Is there any overhead I should be concerned with? I guess I am just wanting to make sure that there are not any major pitfalls in deploying a two-node cluster, versus say a 3-node cluster. Any advice or best-practices would be very helpful! Thanks in advance, -pete -- Pete Wright Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
Re: Sizing Cluster
Thanks Istvan - I think I understand what you are say here - although I was under the impression that if I ensured each topic was being replicated N+1 times a two node cluster would ensure each node has a copy of the entire contents of the message bus at any given time. I agree with your assessment though that having 3 nodes is a more durable configuration, but was hoping others could explain how they calculate capacity and scaling issues on their storage subsystems. Cheers, -pete On 10/21/14 11:28, István wrote: One thing that you have to keep in mind is that moving 10T between nodes takes long time. If you have a node failure and you need to rebuild (resync) the data your system is going to be vulnerable against the second node failure. You could mitigate this with using raid. I think generally speaking 3 node clusters are better for production purposes. I. On Tue, Oct 21, 2014 at 11:12 AM, Pete Wright pwri...@rubiconproject.com wrote: Hi There, I have a question regarding sizing disk for kafka brokers. Let's say I have systems capable of providing 10TB of storage, and they act as Kafka brokers. If I were to deploy two of these nodes, and enable replication in Kafka, would I actually have 10TB available for my producers to write to? Is there any overhead I should be concerned with? I guess I am just wanting to make sure that there are not any major pitfalls in deploying a two-node cluster, versus say a 3-node cluster. Any advice or best-practices would be very helpful! Thanks in advance, -pete -- Pete Wright Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298 -- Pete Wright Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
