Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Ismael Agreed this would make sense, as long as all those kips are approved before hand. I would have reservations holding back a feature if the others are not approved before the Jan release else we run the risk of those kips staying in their current position of spinning wheels. eg I'd

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Michael, Yes, something along those lines. Although I'd probably have the public constructor take a `boolean` and have a private constructor for the `Boolean` (this way, we can retire the config at some point without affecting the API exposed by the class). Also, I would just hardcode it in

Re: [DISCUSS] KIP-100 - Relax Type constraints in Kafka Streams API

Hi everyone, Ismael made a good point in the voting thread to expand the scope of this KIP to make functions covariant in their result types as well, so I did. Please read the updated KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API and

Re: [VOTE] KIP-100 - Relax Type constraints in Kafka Streams API

Ismael made a good point so I updated KIP-100 and expanded its scope to include covariant result types for functions applied to streams. I will update the discussion thread accordingly. On Tue, Dec 13, 2016 at 12:13 AM Ismael Juma wrote: > Hi Xavier, > > Thanks for the KIP.

Re: Brokers cashing with OOME Map failed

Brokers failed repeatedly leaving behind page-cache in memory, which caused broker restarts to fail with OOM every time. After manually cleaning up page-cache, I was able to restart the broker. However, still wondering what could have caused this state in the first place. Any ideas? -Zakee

Jenkins build is back to normal : kafka-trunk-jdk8 #1110

See

Re: [DISCUSS] KIP-102 - Add close with timeout for consumers

I think this is great. Sounds like one implication is that existing code that called close() and hit the timeout would now hang indefinitely. We saw this kind of thing a lot in automated testing scenarios where people don't correctly sequence their shutdown of client and server. I think this is

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Yeah good point. I relent! -jay On Fri, Dec 16, 2016 at 1:46 PM Jason Gustafson wrote: > Jay/Ismael, > > > > I agree that lazy initialization of metadata seems unavoidable. Ideally, we > > could follow the same pattern for transactions, but remember that in the > >

Build failed in Jenkins: kafka-trunk-jdk7 #1762

See Changes: [ismael] KAFKA-4552; README.md has org.gradle.project.maxParallelForms instead of -- [...truncated 17379 lines...]

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

I've added the 3 new metrics/sensors i've implemented to the KIP. at this point I would need to re-validate the functionality (which i expect to do early january). code reviews welcome ;-) On Mon, Nov 28, 2016 at 10:37 AM, radai wrote: > will do (only added a

[jira] [Commented] (KAFKA-3841) MirrorMaker topic renaming

[ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755909#comment-15755909 ] James Cheng commented on KAFKA-3841: [~yangguo1220], should we mark this JIRA as "resolved"? >

[jira] [Commented] (KAFKA-4187) Adding a flag to prefix topics with mirror maker

[ https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755905#comment-15755905 ] James Cheng commented on KAFKA-4187: [~m...@vrischmann.me], should we close this JIRA? > Adding a

[jira] [Resolved] (KAFKA-4552) README.md has org.gradle.project.maxParallelForms instead of maxParallelForks

[ https://issues.apache.org/jira/browse/KAFKA-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4552. Resolution: Fixed Fix Version/s: 0.10.2.0 > README.md has

[jira] [Comment Edited] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755625#comment-15755625 ] Raghu Baddam edited comment on KAFKA-3687 at 12/16/16 11:11 PM:

[GitHub] kafka pull request #2270: KAFKA-4552. README.md has org.gradle.project.maxPa...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2270 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-4552) README.md has org.gradle.project.maxParallelForms instead of maxParallelForks

[ https://issues.apache.org/jira/browse/KAFKA-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755746#comment-15755746 ] ASF GitHub Bot commented on KAFKA-4552: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #2270: KAFKA-4552. README.md has org.gradle.project.maxPa...

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/2270 KAFKA-4552. README.md has org.gradle.project.maxParallelForms instead… … of maxParallelForks You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-4552) README.md has org.gradle.project.maxParallelForms instead of maxParallelForks

[ https://issues.apache.org/jira/browse/KAFKA-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755665#comment-15755665 ] ASF GitHub Bot commented on KAFKA-4552: --- GitHub user cmccabe opened a pull request:

[jira] [Work started] (KAFKA-4552) README.md has org.gradle.project.maxParallelForms instead of maxParallelForks

[ https://issues.apache.org/jira/browse/KAFKA-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4552 started by Colin P. McCabe. -- > README.md has org.gradle.project.maxParallelForms instead of

[jira] [Created] (KAFKA-4552) README.md has org.gradle.project.maxParallelForms instead of maxParallelForks

Colin P. McCabe created KAFKA-4552: -- Summary: README.md has org.gradle.project.maxParallelForms instead of maxParallelForks Key: KAFKA-4552 URL: https://issues.apache.org/jira/browse/KAFKA-4552

[jira] [Comment Edited] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755625#comment-15755625 ] Raghu Baddam edited comment on KAFKA-3687 at 12/16/16 10:02 PM:

[jira] [Commented] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755625#comment-15755625 ] Raghu Baddam commented on KAFKA-3687: - [~rsivaram]: I am about to send a mail to you :), you are

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Ismael So I understand what is being suggested, is we allow on the ProducerRecord at time of construction to be nullable aka big B, Boolean, where old constructors set it to null. /** * Creates a record with a specified timestamp to be sent to a specified topic and partition

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Jay/Ismael, I agree that lazy initialization of metadata seems unavoidable. Ideally, we could follow the same pattern for transactions, but remember that in the consumer+producer use case, the initialization needs to be completed prior to setting the consumer's position. Otherwise we risk reading

[jira] [Commented] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755587#comment-15755587 ] Rajini Sivaram commented on KAFKA-3687: --- [~raghu98...@gmail.com] You need to set

Build failed in Jenkins: kafka-trunk-jdk8 #1109

See Changes: [jason] MINOR: Replace deepIterator/shallowIterator with -- [...truncated 32530 lines...] org.apache.kafka.streams.KafkaStreamsTest >

[jira] [Comment Edited] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755523#comment-15755523 ] Raghu Baddam edited comment on KAFKA-3687 at 12/16/16 9:17 PM: --- [~ijuma]: I

[jira] [Commented] (KAFKA-3687) Internal Authorization Failure at startup with ACLs and security.inter.broker.protocol=PLAINTEXT

[ https://issues.apache.org/jira/browse/KAFKA-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755523#comment-15755523 ] Raghu Baddam commented on KAFKA-3687: - [~ijuma]: I am using SSL port but still I am getting the above

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Michael, I am not sure about using a new policy. One other option (suggested by Gwen) is that the behaviour of the producer with regards to tombstones when you pass a `null` value and don't use the constructor that takes a tombstone can be influenced by a producer config. Not ideal, but would

[GitHub] kafka pull request #2269: fixup typo

GitHub user smferguson opened a pull request: https://github.com/apache/kafka/pull/2269 fixup typo pretty boring, "no" -> "not" You can merge this pull request into a Git repository by running: $ git pull https://github.com/smferguson/kafka fixup_typo_in_add_remove_topics

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael, Sorry if it's frustrating. Definitely not a Confluent thing, I think you've seen both sides from different Confluent folks. I'm just trying to understand the compatibility implications to get on board myself. Given you're saying there is no compatibility implication I think maybe I'm

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Jay I disagree here that we are breaking any compatibility, we went through this on the discussion thread in fact with the help of that thread is how the kip came to the solution. Also on the supported combinations front you mention, we are not taking anything away afaik. Currently

Re: [DISCUSS] KIP 88: OffsetFetch Protocol Update

Thanks Vahid. To clarify the impact of this issue, since we have no way to send an error code in the OffsetFetchResponse when requesting all offsets, we cannot detect when the coordinator has moved to another broker or when it is still in the process of loading the offsets. This means we cannot

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hey Michael, I do think it might have been better had we started with a separate concept of null vs delete. Given that we are where we are, I'm not sure that the possible cures we explored so far are better than the disease. I apologize for coming to this late, but I didn't really understand the

[GitHub] kafka pull request #2261: MINOR: Replace deepIterator/shallowIterator with d...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2261 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Multiple Consumer Group In Single Topic

Hi Guys, Good day! I have question regarding how to consume a specific message belongs to that consumer group? Here's the scenario: Publish message "A" to topic "X" Consume by Consumer Group A the message "A" from topic "X" Publish message "B" to topic "X" Consume by Consumer Group B the

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755064#comment-15755064 ] Michael Andre Pearce (IG) commented on KAFKA-4477: -- Agreed, this one indeed does seem

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

a few more points: 1. "we can make that efficient by storing only the 'start offsets' of transactions and then replaying the log once we hit the corresponding commit/abort markers" this is efficient in terms of memory consumption on the consumer. it is definitely not efficient when i consider

[GitHub] kafka pull request #2268: MINOR: Replace TopicAndPartition with TopicPartiti...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/2268 MINOR: Replace TopicAndPartition with TopicPartition in `Log` and `ReplicaManager` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Chaps, Can we either get one more +1 binding (we have 2 already) on the existing? Or have some response on the below possible alternative. We are keen to get working on this, so we make next feature release. Cheers Mike On 13/12/2016, 16:32, "Michael Pearce" wrote:

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Apurva, here's an outline of what I had in mind: 1. TXs in progress are written "sideways" on the partition leader. this can be (in order of increasing reliability and decreasing performance): 1.1 an im-mem buffer on the partition leader (TX lost on broker crash, memory limits need to be

[jira] [Created] (KAFKA-4551) StreamsSmokeTest.test_streams intermittent failure

Roger Hoover created KAFKA-4551: --- Summary: StreamsSmokeTest.test_streams intermittent failure Key: KAFKA-4551 URL: https://issues.apache.org/jira/browse/KAFKA-4551 Project: Kafka Issue Type:

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754984#comment-15754984 ] Roger Hoover commented on KAFKA-4166: - Happened again here:

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

[ https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754980#comment-15754980 ] Roger Hoover commented on KAFKA-4527: - Happened again:

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754982#comment-15754982 ] Roger Hoover commented on KAFKA-4526: - Happened again here:

[jira] [Comment Edited] (KAFKA-3468) Java clients never fail when bootstrap uses incorrect port

[ https://issues.apache.org/jira/browse/KAFKA-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753998#comment-15753998 ] Marco Buccini edited comment on KAFKA-3468 at 12/16/16 10:20 AM: - I bumped

log.flush.interval.messages setting of Kafka 0.9.0.0

Hi all, we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 replicas, and send it with ack -1, our sending latency is avg 7ms. I prepare to optimize performance of cluster through adjusting some params. we find our brokers has set config item as below,

[jira] [Commented] (KAFKA-3468) Java clients never fail when bootstrap uses incorrect port

[ https://issues.apache.org/jira/browse/KAFKA-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753998#comment-15753998 ] Marco Buccini commented on KAFKA-3468: -- I bumped into the same issue yesterday, and I tried to figure

[jira] [Commented] (KAFKA-4514) Add Codec for ZStandard Compression

[ https://issues.apache.org/jira/browse/KAFKA-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753970#comment-15753970 ] ASF GitHub Bot commented on KAFKA-4514: --- GitHub user dongjinleekr opened a pull request:

[GitHub] kafka pull request #2267: KAFKA-4514: Add Codec for ZStandard Compression

GitHub user dongjinleekr opened a pull request: https://github.com/apache/kafka/pull/2267 KAFKA-4514: Add Codec for ZStandard Compression Hello. This PR resolves [KAFKA-4514: Add Codec for ZStandard Compression](https://issues.apache.org/jira/browse/KAFKA-4514). Please have a look

Re: Kafka ACL's with SSL Protocol is not working

You need to set ssl.client.auth="required" in server.properties. Regards, Rajini On Wed, Dec 14, 2016 at 12:12 AM, Raghu B wrote: > Hi All, > > I am trying to enable ACL's in my Kafka cluster with along with SSL > Protocol. > > I tried with each and every parameters but

[jira] [Commented] (KAFKA-4540) Suspended tasks that are not assigned to the StreamThread need to be closed before new active and standby tasks are created

[ https://issues.apache.org/jira/browse/KAFKA-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753862#comment-15753862 ] ASF GitHub Bot commented on KAFKA-4540: --- GitHub user dguy opened a pull request:

[GitHub] kafka pull request #2266: KAFKA-4540: Suspended tasks that are not assigned ...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2266 KAFKA-4540: Suspended tasks that are not assigned to the StreamThread need to be closed before new active and standby tasks are created During `onPartitionsAssigned` first close, and remove, any

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753806#comment-15753806 ] Juan Chorro commented on KAFKA-4474: Hi [~enothereska], We have set up 6 independent physical

[jira] [Updated] (KAFKA-4474) Poor kafka-streams throughput

[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Chorro updated KAFKA-4474: --- Attachment: Performance test results.png > Poor kafka-streams throughput >

[jira] [Commented] (KAFKA-4015) Change cleanup.policy config to accept a list of valid policies

[ https://issues.apache.org/jira/browse/KAFKA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753796#comment-15753796 ] huxi commented on KAFKA-4015: - [~damianguy]I did not find any occurrence in the doc or source code for

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Radai, Thanks for your email. You raise some good points. Your larger point centers on your --correct!-- observation that persisting messages belonging to aborted transactions causes complications for consumers. You are right in observing that the current proposal does require consumers to