kafka broker is dropping the messages after acknowledging librdkafka

Hi Kafka Team, I'm using kafka (kafka_2.11-0.9.0.1) with librdkafka (0.8.1) API for producer During a run of 2hrs, I notice the total number of messaged ack'd by librdkafka delivery report is greater than the maxoffset of a partition in kafka broker. I'm running kafka broker with replication facto

Re: Verify log compaction

For some reason, I cannot find log-cleaner.log anywhere. On Fri, Jul 29, 2016 at 8:20 PM John Holland < john.holl...@objectpartners.com> wrote: > Check the log-cleaner.log file on the server. When the thread runs you'll > see output for every partition it compacts and the compaction ratio it > a

Re: Kafka Connect questions

Maybe you need a different KafkaConnect source? I think this one may fit your needs better: https://github.com/jcustenborder/kafka-connect-spooldir It was built to copy data from files in directory into Kafka... On Wed, Aug 10, 2016 at 8:56 AM, Jeyhun Karimov wrote: > Hi community, > > I am us

Re: [ANNOUCE] Apache Kafka 0.10.0.1 Released

Woohoo! Thank you, Ismael! You make a great release manager :) On Wed, Aug 10, 2016 at 5:01 PM, Ismael Juma wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.1. > This is a bug fix release that fixes 53 issues in 0.10.0.0. > > All of the changes in

Consumer skip/ignore uncommitted message

Hi, We have come up with Kafka consumer and we only have 1 consumer in the group. We are using Kafka 0.9. We are able to consume messages as soon as we start polling, however for some reason our consumer gets crashed after processing the message but before manually committing it. When we restart

[ANNOUCE] Apache Kafka 0.10.0.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.0.1. This is a bug fix release that fixes 53 issues in 0.10.0.0. All of the changes in this release can be found in the release notes: *https://archive.apache.org/dist/kafka/0.10.0.1/RELEASE_NOTES.html

[ANNOUCE] Apache Kafka 0.10.0.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.0.1. This is a bug fix release that fixes 53 issues in 0.10.0.0. All of the changes in this release can be found in the release notes: *https://archive.apache.org/dist/kafka/0.10.0.1/RELEASE_NOTES.html

Re: Streams 'External source topic not found' while using groupBy/aggregate

Hi Guozhang, Yes, it does seem to be fixed in trunk. Thanks. I should have tried that, but I assumed that the recently released 0.10.0.1 would be pretty close to trunk. I can see where that was mistaken, since 0.10.0 is quite divergent from trunk. Mathieu On Wed, Aug 10, 2016 at 2:39 PM, Guo

Re: Streams 'External source topic not found' while using groupBy/aggregate

Hello Mathieu, I think this issue is fixed in trunk but may get missed in the 0.10.0 branch, could you try running your program from trunk to verify if it is the case? If yes we can consider backportting the hotfix from trunk to 0.10.0 and have another bug fix release. Guozhang On Wed, Aug 10,

RE: Kafka consumer getting duplicate message

Can anyone please check this one? Thanks Achintya -Original Message- From: Ghosh, Achintya (Contractor) Sent: Monday, August 08, 2016 9:44 AM To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject: RE: Kafka consumer getting duplicate message Thank you , Ewen for your response. Ac

Re: Kafka behind NAT

Marcin, The DNS seems to be your friend. /etc/hosts should be sufficient but it might be an operational hassle. – Best regards, Radek Gruchalski ra...@gruchalski.com On August 10, 2016 at 10:03:16 PM, Marcin (kan...@o2.pl) wrote: We have kafka behind NAT with *only one broker*. Let say we have

Kafka behind NAT

We have kafka behind NAT with *only one broker*. Let say we have internal (A) and external (B) network. When we try to reach the broker from external network (we use bootstrap.servers parameter set to B address) then what is obvious the broker responds with internal network's address (A) which is

Kafka 0.9.0.1 allows offset commits for offsets already committed

I ran in to a strange scenario today that after looking closer I am seeing that Kafka doesn't fail on several invalid scenarios of committing offsets. I have two partitions and two processes in the same consumer group, so each process has one partition. Process 1 retrieved a set of records from

Kafka Connect questions

Hi community, I am using Kafka-Connect to import text files to kafka. Because I have to give exact file name (like ./data.txt) and not the parent directory (./*) as an input to Kafka-Connect, how can I import files in particular directory as they are created in runtime? The solutions that come to

Streams 'External source topic not found' while using groupBy/aggregate

Hey there, Kafka Users, I'm trying to join two topics with Kafka Streams. The first topic is a changelog of one object, and the second is a changelog of a related object. In order to join these tables, I'm grouping the second table by a piece of data in it that indicates what record it is relate

Console consumer with SSL

Hi, I'm trying a simple console producer & consumer test on a broker with SSL enabled. listeners=PLAINTEXT://192.168.50.4:9092,SSL://192.168.50.4:9093 Producer was started and is publishing using SSL interface. bin/kafka-console-producer.sh --broker-list 192.168.50.4:9093 --topic test-topic -

Re: simple test fails on OSX

Thanks Manu for taking a look. I updated the test to run against 0.10.0.0, and it appears to work. I'm now working my way back to my original issue in our application to see if it is still broken. Testing is slowed by the fact that I don't use a Mac, and I'm very new to kafka. Hopefully you don

Re: Deleting by writing null payload not working in compacted logs

I'd possibly check the log segments in question, by using the DumpLogSegments tool. Note that Kafka keeps a deleted tombstone for 24h by default after a deletion. Are you checking for the key and the value being present during this testing? Sample code for the producer messages would be useful as

Re: simple test fails on OSX

Hi Eric, I can reproduce your problem although I have no idea yet what is going wrong. Do you have any updates ? Regards, Manu Zhang On Mon, Aug 8, 2016 at 11:03 AM Eric Newton wrote: > I wrote a little unit test and it works as I expect on linux (ubuntu, > redhat, openjdk 1.8). > > It fails r

Deleting by writing null payload not working in compacted logs

Hi all, I am trying to set up a minimal example understanding log compaction behaviour using kafka-clients-0.10.0.0.jar. I got the compaction behaviour working fine, however, when trying to delete a message explicitly by writing a null value, the message seems to not be deleted. These are my sett