Regarding issue - https://lists.apache.org/thread.html/1f2ffc93483cbe71167fa47875c5ecda8dbcd5275d3d41b5af3220d9@%3Cusers.kafka.apache.org%3E

Won't setting STATE_CLEANUP_DELAY_MS_CONFIG to a large int make the size of store very big with time. For me it was 34MB in 7 hours. Won't it keep increasing? This could cause OOM on the machine. This is in Kafka Streams 2.0.0

kafka stream latency

Hi Guozhang, Here is the very simple kafka producer/consumer/stream app, using the latest version and just create 2 topics input and output all component are just running on localhost. Thanks, Nan -Original Message- From: Nan Xu [mailto:nanxu1...@gmail.com] Sent: Friday,

RE: kafka stream latency

Hi Guozhang, Here is the very simple kafka producer/consumer/stream app, using the latest version and just create 2 topics input and output all component are just running on localhost. Sorry, git hub is not allowed from work, nor java extension file Thanks, Nan -Original

Re: kafka stream latency

Looks really promising but after upgrade, still show the same result. I will post the program soon. Maybe you can see where the problem could be. Nan On Thu, Aug 23, 2018, 7:34 PM Guozhang Wang wrote: > Hello Nan, > > Kafka does not tie up the processing thread to do disk flushing. However, >

Re: Performance Impact with Apache Kafka Security

Hi Eric, I have SSL configured for Inter-broker communication as well but I left the acks at 1. So I will not wait for Guaranteed Replicated Delivery. My process just goes on once I get acknowledgement about delivery from leader and not all ISRs. So do you really think that will cause the

Re: Performance Impact with Apache Kafka Security

I saw a similar 30-40% performance hit when testing a move from Plaintext to SASL_SSL. It seemed to be due to the additional traffic generated by replication between brokers. Enabling SSL only between the client and brokers and leaving inter-broker traffic on Plaintext only introduced ~10%

Re: Maintain "consumer has all the messages for a key" delivery guarantee with two topics?

The default partition assignment for the consumer should guarantee this as long as the number of partitions in each topic is the same. The term used for this in the Streams doc is co-partitioning. The link below is referring specifically to the KStreams Join method, but the concept is similar to

Kafka Upgrade Issue

Hi, We are using Kafka 2.10_0.9.0.1 and recently trying to upgrade it to latest Kafka version(i.e. Kafka 2.11_2.0.0).But we are having Kafka Consumer in our Legacy Code which is using JDK 1.7. Now, we are facing the issue of version compatibility as Legacy Code requires JDK 1.7 and Consumer

subscription

-- Big Data Architect / Consultant chez Orange https://github.com/lsroudi

Re: Certification

CONFLUENT BUT NOT YET On Fri, Aug 24, 2018 at 7:12 PM Jacob Sheck wrote: > https://www.confluent.io/certification/ > > On Fri, Aug 24, 2018 at 12:09 PM chinchu chinchu > wrote: > > > Does any companies provide kafka certification ? > > > -- Big Data Architect / Consultant chez Orange

Re: Certification

https://www.confluent.io/certification/ On Fri, Aug 24, 2018 at 12:09 PM chinchu chinchu wrote: > Does any companies provide kafka certification ? >

Certification

Does any companies provide kafka certification ?

Re: Kafka Upgrade Issue

You can use Apache Kafka 1.1.1 if you need Java 7 support. Ismael On Fri, Aug 24, 2018 at 7:23 AM Nitya Mehrotra wrote: > Hi, > > > We are using Kafka 2.10_0.9.0.1 and recently trying to upgrade it to > latest Kafka version(i.e. Kafka 2.11_2.0.0).But we are having Kafka > Consumer in our

Re: Is Kafka broker 1.1.0 backward compatible with Java Kafka Client 0.10.1.1?

Yes. Ismael On Fri, Aug 24, 2018 at 12:20 AM Debraj Manna wrote: > Hi > > Is Kafka broker 1.1.0 backward compatible with Java Kafka Client ? It is > yes as per my understanding after reading this page > > but > I want to

Re: Performance Impact with Apache Kafka Security

Hi Harsha, Given below are all the details. We are using Kafka On CDH. Do you have any suggestion based on the below statistics. CDK - 2.2.0 - 0.10.2.0+kafka2.2.0+110 Apache Kafka - 0.10.2. java version "1.8.0_151" Tried Using Java 9 with not much difference. We need to make a small change

Re: Exposing Kafka on WAN

Thanks Ryanne. That's one of the options we had considered. I was hoping to keep solution simple and efficient. With HTTP proxy, we would have to worry about configurations, scalability, and operation. This is probably true with proxy solution as well, but at least my thinking was that deploying

Re: Maintain "consumer has all the messages for a key" delivery guarantee with two topics?

Instead of using two topics, consider adding a priority field to your records, and then use a priority queue in the consumers. For example, each consumer can have two queues, one for high and one for low priority records, and can process the low priority queue only when the high priority queue is

Offset for consumer group set to Null in __consumer_offsets, leads to data loss

Hi everyone, *Short Version:* We had an unusual situation where after restart of our spark job, rather than reading from the last known offset of the consumer group, it started consuming from the latest offset, thus leading to data loss. *Long Version:* We have a spark job which crashed last

Re: Exposing Kafka on WAN

Can you use a Kafka HTTP proxy instead of using the Kafka protocol directly? Ryanne On Thu, Aug 23, 2018, 7:29 PM Jack S wrote: > Hello, > > We have a requirement for opening Kafka on WAN where external producers and > consumers need to be able to talk to Kafka. I was able to get Zookeeper and

Kafka Upgrade Issue

Hi, We are using Kafka 2.10_0.9.0.1 and recently trying to upgrade it to latest Kafka version(i.e. Kafka 2.11_2.0.0).But we are having Kafka Consumer in our Legacy Code which is using JDK 1.7. Now, we are facing the issue of version compatibility as Legacy Code requires JDK 1.7 and Consumer

How to limit the number of topics allow to be created

Hi, Is it possible to set a limit on the number of topics allowed to be auto created by kafka? We’re worried about run away topic creation from a bad actor and would like to define a limit/ We’ve tried using the “create.topic.policy.class.name” property to modify the validate() function to

Maintain "consumer has all the messages for a key" delivery guarantee with two topics?

Hello, I need to parallel-process messages with per key ordering guarantees so I have kafka topic with keyed messages and multiple partitions. The need has arised to process some messages with higher priority, so I thought of adding a second topic where the messages with higher priority will

Is Kafka broker 1.1.0 backward compatible with Java Kafka Client 0.10.1.1?

Hi Is Kafka broker 1.1.0 backward compatible with Java Kafka Client ? It is yes as per my understanding after reading this page but I want to confirm. Thanks,