[jira] [Commented] (KAFKA-4368) Unclean shutdown breaks Kafka cluster

[ https://issues.apache.org/jira/browse/KAFKA-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631715#comment-15631715 ] huxi commented on KAFKA-4368: - Could you paste the entire stack trace for both client and serv

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

I think the bump isn't strictly required, but if the client is KIP-35 aware, it can proactively choose a compatible SASL mechanism based on its initial ApiVersionRequest and avoid an extra connection round trip when there are client/broker version differences. Without this, a newer client would hav

[jira] [Commented] (KAFKA-4368) Unclean shutdown breaks Kafka cluster

[ https://issues.apache.org/jira/browse/KAFKA-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631475#comment-15631475 ] Anukool Rattana commented on KAFKA-4368: Hi [~huxi_2b], Yes, although the broken b

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631474#comment-15631474 ] Json Tu commented on KAFKA-4360: I'm sorry, I rollback one comment commit, and https://gi

[GitHub] kafka pull request #2094: KAFKA-4360：Controller may deadLock when autoLead...

GitHub user xiguantiaozhan opened a pull request: https://github.com/apache/kafka/pull/2094 KAFKA-4360：Controller may deadLock when autoLeaderRebalance encounter zk expired You can merge this pull request into a Git repository by running: $ git pull https://github.com/xigua

[GitHub] kafka pull request #2085: KAFKA-4360：Controller may deadLock when autoLead...

Github user xiguantiaozhan closed the pull request at: https://github.com/apache/kafka/pull/2085 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (KAFKA-4368) Unclean shutdown breaks Kafka cluster

[ https://issues.apache.org/jira/browse/KAFKA-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631259#comment-15631259 ] huxi commented on KAFKA-4368: - Do you mean the producer still cannot be able to send messages

[jira] [Commented] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[ https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631227#comment-15631227 ] huxi commented on KAFKA-4370: - yes, compacted topics no longer accept messages without key and

Re: why cant SslTransportLayer be muted before handshake completion?

HI Radai, One main reason is to keep the handshake details away from the application layer. i.e Kafka network layer which is sending Kafka protocols doesn't need to worry about the handshake details, all it needs is a validation that the connection is completed and it can start sendin

[jira] [Commented] (KAFKA-4367) MirrorMaker shuts down gracefully without being stopped

[ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631208#comment-15631208 ] huxi commented on KAFKA-4367: - Did you close the terminal where runs this command ? Based on t

[jira] [Resolved] (KAFKA-4348) On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server

[ https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxi resolved KAFKA-4348. - Resolution: Duplicate Assignee: huxi > On Mac OS, KafkaConsumer.poll returns 0 when there are still messages

Re: why cant SslTransportLayer be muted before handshake completion?

Sriharsha can validate this, but I think the reason is that if we allow muting/unmuting at will (via those public APIs) that can completely mess up the handshake itself. It should be possible to pause/resume the handshake if that's what you'r elooking for but I'm not sure it is worth it for the pur

Re: [DISCUSS] KIP-82 - Add Record Headers

my biggest issues with a "standard" wrapper format: 1. _ALL_ client _CODE_ (as opposed to kafka lib version) must be updated to know about the container, because any old naive code trying to directly deserialize its own payload would keel over and die (it needs to know to deserialize a container,

Re: [DISCUSS] KIP-82 - Add Record Headers

> On Nov 2, 2016, at 2:33 AM, Michael Pearce wrote: > > Thanks James for taking the time out. > > My comments per solution below you commented about. (I note you didn’t > comment on the 3rd at all , which is the current proposal in the kip) > 1) > a. This forces all clients to have distin

why cant SslTransportLayer be muted before handshake completion?

Hi, as part of testing my code for KIP-72 (broker memory control), i ran into the following code snippet in SslTransportLayer: public void removeInterestOps(int ops) { if (!key.isValid()) throw new CancelledKeyException(); else if (!handshakeComplete) t

Build failed in Jenkins: kafka-trunk-jdk8 #1017

See Changes: [cshapi] MINOR: Fix NPE when Connect offset contains non-primitive type -- [...truncated 12422 lines...] (topicPartition, new ListOffsetResponse.PartitionData(Errors.forE

[jira] [Commented] (KAFKA-4369) ZkClient is not closed upon streams shutdown

[ https://issues.apache.org/jira/browse/KAFKA-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630582#comment-15630582 ] Andy Chambers commented on KAFKA-4369: -- I found this while developing a test fixture

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

Both the Confluent Avro Converter and the Confluent Avro Serializer use the Schema Registry. The reason is, as Tommy Becker mentioned below, to avoid storing the entire schema in each record (which the JSON serializer in Apache Kafka does). It has few other benefits schema validation and such. If

Build failed in Jenkins: kafka-trunk-jdk7 #1668

See Changes: [cshapi] MINOR: Fix NPE when Connect offset contains non-primitive type -- [...truncated 14343 lines...] org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest >

[GitHub] kafka-site pull request #28: Add Becket to the committers page

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/28 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Updated] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[ https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Pfannenschmidt updated KAFKA-4370: --- Priority: Trivial (was: Minor) > CorruptRecordException when ProducerRecord construct

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congrats! Eno > On 1 Nov 2016, at 05:57, Harsha Chintalapani wrote: > > Congrats Becket! > -Harsha > > On Mon, Oct 31, 2016 at 2:13 PM Rajini Sivaram > wrote: > >> Congratulations, Becket! >> >> On Mon, Oct 31, 2016 at 8:38 PM, Matthias J. Sax >> wrote: >> >>> -BEGIN PGP SIGNED MESSAGE

[jira] [Commented] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[ https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630350#comment-15630350 ] Lars Pfannenschmidt commented on KAFKA-4370: Ah... compaction was enabled on t

[jira] [Commented] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[ https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630310#comment-15630310 ] Ismael Juma commented on KAFKA-4370: This should work, are there errors in the server

Re: [DISCUSS] KIP-81: Max in-flight fetches

Thanks for all the feedback. I agree, throttling the requests sent will most likely result in a loss of throughput -> BAD ! As suggested, selectively reading from the socket should enable to control the memory usage without impacting performance. I've had look at that today and I can see how that

[jira] [Updated] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[ https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Pfannenschmidt updated KAFKA-4370: --- Priority: Minor (was: Major) > CorruptRecordException when ProducerRecord constructed

[jira] [Created] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

Lars Pfannenschmidt created KAFKA-4370: -- Summary: CorruptRecordException when ProducerRecord constructed without key nor partition and send Key: KAFKA-4370 URL: https://issues.apache.org/jira/browse/KAFKA-437

Re: [DISCUSS] KIP-81: Max in-flight fetches

Hey Radai, I think there are a couple discussions here. The first is about what is the interface to the user. The other is about what is exposed in the protocol, and implementation details of reading requests. I strongly agree with giving the user a simple "use X MB of memory" config and we calcul

Re: [DISCUSS] KIP-81: Max in-flight fetches

+1 On Wed, Nov 2, 2016 at 10:34 AM, radai wrote: > In my opinion a lot of kafka configuration options were added using the > "minimal diff" approach, which results in very nuanced and complicated > configs required to indirectly achieve some goal. case in point - timeouts. > > The goal here is t

Re: [DISCUSS] KIP-81: Max in-flight fetches

In my opinion a lot of kafka configuration options were added using the "minimal diff" approach, which results in very nuanced and complicated configs required to indirectly achieve some goal. case in point - timeouts. The goal here is to control the memory requirement. the 1st config was max size

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

Although I can't speak to details of the Confluent packaging, anytime you're using Avro you need the schemas for the records you're working with. In an Avro data file the schema is included in the file itself. But when you're encoding individual records like in Kafka, most people instead encode

[GitHub] kafka pull request #2087: MINOR: Fix NPE when Connect offset contains non-pr...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2087 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka-site issue #28: Add Becket to the committers page

Github user becketqin commented on the issue: https://github.com/apache/kafka-site/pull/28 I'll try to merge this myself :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[GitHub] kafka pull request #2093: MINOR: Add description of how consumer wakeup acts...

GitHub user srdo opened a pull request: https://github.com/apache/kafka/pull/2093 MINOR: Add description of how consumer wakeup acts if no threads are awakened I think the Javadoc should describe what happens if wakeup is called and no other thread is currently blocking. This may b

[GitHub] kafka-site pull request #28: Add Becket to the committers page

GitHub user becketqin opened a pull request: https://github.com/apache/kafka-site/pull/28 Add Becket to the committers page You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka-site asf-site Alternatively you can revi

[jira] [Created] (KAFKA-4369) ZkClient is not closed upon streams shutdown

Ryan P created KAFKA-4369: - Summary: ZkClient is not closed upon streams shutdown Key: KAFKA-4369 URL: https://issues.apache.org/jira/browse/KAFKA-4369 Project: Kafka Issue Type: Bug Compon

Re: 0.10.1.0 - KafkaConsumer.poll() blocks background heartbeat thread causing consumer to be considered dead?

Thanks Ismael. Just checked, that one doesn't look like it's the same issue, but could be a similar one. In that JIRA it looks like the issue was probably addressed for the commitSync call. However, in this specific instance the KafkaConsumer.poll(...) itself leads to locking the object monitor

Re: 0.10.1.0 - KafkaConsumer.poll() blocks background heartbeat thread causing consumer to be considered dead?

Maybe https://issues.apache.org/jira/browse/KAFKA-4303? On 2 Nov 2016 10:15 am, "Jaikiran Pai" wrote: > We have been trying to narrow down an issue in 0.10.1 of Kafka in our > setups where our consumers are marked as dead very frequently causing > rebalances almost every few seconds. The consume

0.10.1.0 - KafkaConsumer.poll() blocks background heartbeat thread causing consumer to be considered dead?

We have been trying to narrow down an issue in 0.10.1 of Kafka in our setups where our consumers are marked as dead very frequently causing rebalances almost every few seconds. The consumer (Java new API) then starts seeing exceptions like: org.apache.kafka.clients.consumer.CommitFailedExcepti

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628871#comment-15628871 ] Andrew Olson commented on KAFKA-4362: - Is there documentation that advises for a COORD

[jira] [Commented] (KAFKA-4365) In case async producer closes the TCP connection to Kafka broker, last sent messages might be lost.

[ https://issues.apache.org/jira/browse/KAFKA-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628696#comment-15628696 ] Rajini Sivaram commented on KAFKA-4365: --- [PR #1836|https://github.com/apache/kafka/p

[jira] [Updated] (KAFKA-3703) Handle close gracefully for consumers and producers with acks=0

[ https://issues.apache.org/jira/browse/KAFKA-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-3703: -- Summary: Handle close gracefully for consumers and producers with acks=0 (was: Selector.close()

[jira] [Created] (KAFKA-4368) Unclean shutdown breaks Kafka cluster

Anukool Rattana created KAFKA-4368: -- Summary: Unclean shutdown breaks Kafka cluster Key: KAFKA-4368 URL: https://issues.apache.org/jira/browse/KAFKA-4368 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-4367) MirrorMaker shuts down gracefully without being stopped

[ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex updated KAFKA-4367: Description: Start: bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.proper

[jira] [Updated] (KAFKA-4367) MirrorMaker shuts down gracefully without being stopped

[ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex updated KAFKA-4367: Description: Start: bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.proper

[jira] [Updated] (KAFKA-4367) MirrorMaker shuts down gracefully without being stopped

[ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex updated KAFKA-4367: Description: Start: bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.proper

[jira] [Created] (KAFKA-4367) MirrorMaker shuts down gracefully without being stopped

Alex created KAFKA-4367: --- Summary: MirrorMaker shuts down gracefully without being stopped Key: KAFKA-4367 URL: https://issues.apache.org/jira/browse/KAFKA-4367 Project: Kafka Issue Type: Bug

[GitHub] kafka pull request #2092: MINOR: remove commented out code and System.out.pr...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2092 MINOR: remove commented out code and System.out.println Remove commented out code and System.out.println from KTableKTableJoinIntegrationTest You can merge this pull request into a Git repository by r

[jira] [Work started] (KAFKA-4366) KafkaStreams.close() blocks indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4366 started by Damian Guy. - > KafkaStreams.close() blocks indefinitely > >

[jira] [Assigned] (KAFKA-4366) KafkaStreams.close() blocks indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy reassigned KAFKA-4366: - Assignee: Damian Guy (was: Guozhang Wang) > KafkaStreams.close() blocks indefinitely > -

[jira] [Commented] (KAFKA-4348) On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server

[ https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628551#comment-15628551 ] Yiquan Zhou commented on KAFKA-4348: Yes it's very likely the same issue as KAFKA-3135

[GitHub] kafka pull request #2091: MINOR: Bug fixed

GitHub user himani1 opened a pull request: https://github.com/apache/kafka/pull/2091 MINOR: Bug fixed You can merge this pull request into a Git repository by running: $ git pull https://github.com/himani1/kafka minor_fix Alternatively you can review and apply these changes a

[jira] [Commented] (KAFKA-4355) StreamThread intermittently dies with "Topic not found during partition assignment" when broker restarted

[ https://issues.apache.org/jira/browse/KAFKA-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628442#comment-15628442 ] Michal Borowiecki commented on KAFKA-4355: -- KAFKA-4366 created for the KafkaSteam

[jira] [Created] (KAFKA-4366) KafkaStreams.close() blocks indefinitely

Michal Borowiecki created KAFKA-4366: Summary: KafkaStreams.close() blocks indefinitely Key: KAFKA-4366 URL: https://issues.apache.org/jira/browse/KAFKA-4366 Project: Kafka Issue Type: Bu

[jira] [Created] (KAFKA-4365) In case async producer closes the TCP connection to Kafka broker, last sent messages might be lost.

Ciprian Pascu created KAFKA-4365: Summary: In case async producer closes the TCP connection to Kafka broker, last sent messages might be lost. Key: KAFKA-4365 URL: https://issues.apache.org/jira/browse/KAFKA-4365

Kafka Connect key.converter and value.converter properties for Avro encoding

I am using Kafka Connect in source mode i.e. using it to send events to Kafka topics. With the key.converter and value.converter properties set to org.apache.kafka.connect.storage.StringConverter I can attach a consumer to the topics and see the events in a readable form. This is helpful and r

Re: [DISCUSS] KIP-82 - Add Record Headers

Thanks James for taking the time out. My comments per solution below you commented about. (I note you didn’t comment on the 3rd at all , which is the current proposal in the kip) 1) a. This forces all clients to have distinct knowledge of platform level implementation detail b. enforces single

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Joel , et al. Any comments on the below idea to handle roll out / compatibility of this feature, using a configuration? Does it make sense/clear? Does it add value? Do we want to enforce flag by default, or value by default, or both? Cheers Mike On 10/27/16, 4:47 PM, "Michael Pearce" wro

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi James, We would send currently our message headers wrapper so we can have some platform/audit data even on the delete available. We find having a custom wrapper means some of our internal platform pieces we currently cannot open source or share but would love to (see kip-82) This data stil

[jira] [Commented] (KAFKA-4348) On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server

[ https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628112#comment-15628112 ] huxi commented on KAFKA-4348: - I tried a larger SO_RCVBUF setting and it does improve the perf