Clarification on producer onCompletion callback

2019-10-16 Thread Brian Putt
Hello, I was hoping someone could help clarify what the javadocs say. http://kafka.apache.org/23/javadoc/org/apache/kafka/clients/producer/Callback.html Are there cases where metadata has -1 values (which indicate an error) AND have exception set to null? We're trying to tighten our ingest laye

Kafka Streams processing.guarantee behavior

2019-10-16 Thread Alex Leung (BLOOMBERG/ SAN FRAN)
We're using the Kafka Streams processor API and directly performing get() and put() on two different state stores (KeyValueStore) from a single processor. If two puts are performed from one processor, e.g.: 1. store1.put(...) 2. store2.put(...) my understanding is that if processing.guarantee="

Re: Tumbling Time Window

2019-10-16 Thread Navneeth Krishnan
Thanks a lot Matthias. I looked at the aggregation and I'm fine with aggregating data before forwarding it to downstream but the KStreamWindowAggregateProcessor::process uses the key to determine whether the data has to be aggregated and forwarded or not. My worry is if I have a tumbling window for

Re: Running a stand alone Mirror Maker 2.0 cluster and Issues

2019-10-16 Thread Ryanne Dolan
Vishal, you should be able to override the properties passed to the internal workers using properties like A->B.consumer.auto.offset.reset or A.consumer.auto.offset.reset in the mm2.properties file. Certain top-level properties like tasks.max are honored without the A->B or A prefix, but auto.offse

Re: Very low bytesout for internal repartition topic, is it related to RocksDB?

2019-10-16 Thread Xiyuan Hu
Hi Sophie, A follow up questions, I set the cache to 0 and after 12 hours running, some nodes turn dead with error messages: task [1_0] Abort sending since an error caught with a previous record (key 333283323209294164cf16bb333c14a3506001b8fec3XXX3\x00\x00\x01m\xD5\x155\x80 value [B@33574f49 times

Closing socket - IllegalArgumentException - Invalid version for API key OFFSET_FOR_LEADER_EPOCH: 2

2019-10-16 Thread Jose Manuel Vega Monroy
Hi there, During rolling upgrade to allow Kafka to listen SSL communications via new port we noticed this: [2019-10-16 14:28:48,900] ERROR Closing socket for 10.191.68.155:9092-10.191.68.161:33320 because of error (kafka.network.Processor) org.apache.kafka.common.errors.InvalidRequestException

Re: Running a stand alone Mirror Maker 2.0 cluster and Issues

2019-10-16 Thread Vishal Santoshi
Hey Ryanne, How do I override auto.offset.reset = latest for consumers through mm2.properties. I have tried straight up . auto.offset.reset and consumer. auto.offset.reset but it defaults to earliest.. I do have a query in another thread but though you might know off hand.. I would imagine

custom authentication; inter-broker issues?

2019-10-16 Thread Joris Peeters
Hello, I am trying to set up Kafka with custom authentication & authorisation. The authentication is PLAIN, i.e. user/pass. On a single broker, this worked fine (i.e. users were able to authenticate correctly), but in a multi-broker set-up I am struggling to get the interbroker communication to w

Kafka2.2.0 with SASL-SCRAM

2019-10-16 Thread Venkata MR
Hi All, Error: SSL peer is not authenticated, returning ANONYMOUS instead while connecting as a producer on port 9094 (which is sasl-ssl with SCRAM mechanism) I have kafka 2.2.0 in windows systems with SSL enabled, where the kafka broker with plaintext is running on 9092 and SSL on 9093. On to