Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi All, I've updated the KIP by adding the dedicated endpoints for controller connections, and pinning threads for controller requests. Also I've updated the title of this KIP. Please take a look and let me know your feedback. Thanks a lot for your time! Lucas On Tue, Jul 24, 2018 at 10:19 AM,

[jira] [Created] (KAFKA-7207) Make rate & total metrics documentation consistent

Kevin Lu created KAFKA-7207: --- Summary: Make rate & total metrics documentation consistent Key: KAFKA-7207 URL: https://issues.apache.org/jira/browse/KAFKA-7207 Project: Kafka Issue Type:

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

> The voting has passed. Dong Lin (binding) is a committer. Thanks for the kind reminder. Let me correct the voting result (see below): +3 (binding) 1) Jason Gustafson 2) Gwen Shapira 3) Dong Lin +3 (non-binding) 1) Colin McCabe 2) Manikumar 3) Vahid S Hashemian KIP-338 has passed. Thank you

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

+1 On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep wrote: > Hello, > > I'm calling a vote for KIP-321: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-321%3A+Update+TopologyDescription+to+better+represent+Source+and+Sink+Nodes > > Best, > Nishanth Pradeep >

[Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

Hello, I'm calling a vote for KIP-321: https://cwiki.apache.org/confluence/display/KAFKA/KIP-321%3A+Update+TopologyDescription+to+better+represent+Source+and+Sink+Nodes Best, Nishanth Pradeep

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Hi, KIP-268 (rebalance meatadata) is finished and included in AK 2.0 release. Thus, I want to pick up this KIP again to get the RocksDB upgrade done for 2.1. I updated the KIP accordingly and also have a "prove of concept" PR ready (for "in place" upgrade only):

Build failed in Jenkins: kafka-2.0-jdk8 #91

See Changes: [jason] MINOR: Add thread dumps if broker node cannot be stopped (#5373) -- [...truncated 2.48 MB...] org.apache.kafka.streams.integration.ResetIntegrationTest

[jira] [Resolved] (KAFKA-6014) new consumer mirror maker halts after committing offsets to a deleted topic

[ https://issues.apache.org/jira/browse/KAFKA-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6014. Resolution: Resolved We changed the behavior of the consumer in KAFKA-6829 so that unknown

[jira] [Created] (KAFKA-7206) Enable batching in FindCoordinator

Yishun Guan created KAFKA-7206: -- Summary: Enable batching in FindCoordinator Key: KAFKA-7206 URL: https://issues.apache.org/jira/browse/KAFKA-7206 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi Vahid, I was thinking we'd only use the old API version if we had to. That is, only if the user has explicitly configured "" as the group.id. Otherwise, we'd just use the new one. Another option is to just drop support in the client for the empty group id, but usually we allow a deprecation

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi All, I have made some updates to the KIP. As many of you know, a side project of mine has been specifying the Kafka replication protocol in TLA. You can check out the code here if you are interested: https://github.com/hachikuji/kafka-specification. In addition to uncovering a couple unknown

[jira] [Created] (KAFKA-7205) KafkaConsumer / KafkaProducer should allow Reconfiguration of SSL Configuration

Magnus Jungsbluth created KAFKA-7205: Summary: KafkaConsumer / KafkaProducer should allow Reconfiguration of SSL Configuration Key: KAFKA-7205 URL: https://issues.apache.org/jira/browse/KAFKA-7205

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi Jason, Thanks for additional clarification. So the next version of the OffsetCommit API will return an INVALID_GROUP_ID error for empty group ids; but on the client side we call the older version of the client until the next major release. The table below should summarize this.

[jira] [Created] (KAFKA-7204) MockConsumer poll clears all records in poll(), including records for subscriptions that are paused.

Subodh Bhattacharjya created KAFKA-7204: --- Summary: MockConsumer poll clears all records in poll(), including records for subscriptions that are paused. Key: KAFKA-7204 URL:

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

For the cleaner thread specifically, I do not think respawning will help at all because we are more than likely to run into the same issue again which would end up crashing the cleaner. Retrying makes sense for transient errors or when you believe some part of the system could have healed itself,

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hi Rajini, I updated the KIP. Please check if the clarification is okay On Wed, Jul 25, 2018 at 10:49 AM Rajini Sivaram wrote: > Hi Stanislav, > > 1. Can you clarify the following line in the KIP in the 'Public Interfaces' > section? When you are reading the KIP for the first time, it sounds

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

Chia-Ping Tsai, The voting has passed. Dong Lin (binding) is a committer. On Wed, Jul 25, 2018 at 8:24 AM Chia-Ping Tsai wrote: > Current voting result is shown below. > > +2 (binding) > 1) Jason Gustafson > 2) Gwen Shapira > > +4 (non-binding) > 1) Colin McCabe > 2) Manikumar > 3) Dong Lin >

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

<< wrote: > On Mon, Jul 23, 2018, at 23:20, James Cheng wrote: > > Hi Stanislav! Thanks for this KIP! > > > > I agree that it would be good if the LogCleaner were more tolerant of > > errors. Currently, as you said, once it dies, it stays dead. > > > > Things are better now than they used to be.

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hi Stanislav, 1. Can you clarify the following line in the KIP in the 'Public Interfaces' section? When you are reading the KIP for the first time, it sounds like we adding a new Kafka config. But we are adding JAAS config options with a prefix that can be used with the default unsecured bearer

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

+1 (Non-binding). Thanks for the KIP and the PR, Stanislav. Ron On Wed, Jul 25, 2018 at 1:04 PM Stanislav Kozlovski wrote: > Hey everbody, > > I'd like to start a vote thread for KIP-342 Add support for custom SASL > extensions in OAuthBearer authentication > < >

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

On Mon, Jul 23, 2018, at 23:20, James Cheng wrote: > Hi Stanislav! Thanks for this KIP! > > I agree that it would be good if the LogCleaner were more tolerant of > errors. Currently, as you said, once it dies, it stays dead. > > Things are better now than they used to be. We have the metric >

Re: [VOTE] KIP-344: The auto-generated client id should be passed to MetricsReporter

+1 On Wed, Jul 25, 2018 at 9:49 AM Kevin Lu wrote: > Hi All, > > I am calling a vote for KIP-344 > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-344%3A+The+auto-generated+client+id+should+be+passed+to+MetricsReporter > . > > If any committer feels that this minor change does not need

[jira] [Resolved] (KAFKA-5799) New KafkaSpoutConfig(Scheme)-ByteArrayKeyValueScheme

[ https://issues.apache.org/jira/browse/KAFKA-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5799. -- Resolution: Auto Closed Closing Apache Storm - Kafka Spout related query.  If this still issue, please

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Thanks Nishanth, I've taken a look at the updated KIP and it looks good to me. I think you can start a new VOTE thread on the current proposal. Guozhang On Tue, Jul 24, 2018 at 5:56 PM, Nishanth Pradeep wrote: > I have updated the KIP >

Re: Request wiki access to create KIP

Hello Shaobo, Thanks for your interest in contributing. I've granted you the permission. Cheers, Guozhang On Wed, Jul 25, 2018 at 7:20 AM, lambdaliu(刘少波) wrote: > Hi, > > Can someone give me the permissions for creating a new KIP? > My username is *lambdaliu*. > > Thanks. > > --

[VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

Hey everbody, I'd like to start a vote thread for KIP-342 Add support for custom SASL extensions in OAuthBearer authentication -- Best, Stanislav

Re: [VOTE] 2.0.0 RC3

+1 (non-binding) Built from source and exercised the new SASL/OAUTHBEARER functionality with unsecured tokens. Thanks, Rajini -- apologies for KAFKA-7182. Ron On Tue, Jul 24, 2018 at 5:10 PM Vahid S Hashemian wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully

[VOTE] KIP-344: The auto-generated client id should be passed to MetricsReporter

Hi All, I am calling a vote for KIP-344 https://cwiki.apache.org/confluence/display/KAFKA/KIP-344%3A+The+auto-generated+client+id+should+be+passed+to+MetricsReporter . If any committer feels that this minor change does not need to go through the KIP process, then feel free to leave a code review

Re: Didn't receive video stream data into spark through kafka

Hi, Thank you for your support! everything is working except can't figure out how to pass the video frame( serialize in a pickle file) to spark. My problem is during loading the pickle file stream got an EOF error into spark stream context. I'm suspecting that due to large file size TCP broke

Request wiki access to create KIP

Hi, Can someone give me the permissions for creating a new KIP? My username is *lambdaliu*. Thanks. -- Lambdaliu(Shaobo Liu)

[jira] [Resolved] (KAFKA-2396) I have config host.name and advertised.host.name but still have a link connet to 0.0.0.0 with random port

[ https://issues.apache.org/jira/browse/KAFKA-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2396. -- Resolution: Not A Problem extra port is related to JMX. See  KAFKA-1869 > I have config host.name and

Build failed in Jenkins: kafka-trunk-jdk8 #2839

See Changes: [wangguoz] KAFKA-7144: Fix task assignment to be even (#5390) -- [...truncated 108.95 KB...] kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

[jira] [Resolved] (KAFKA-2395) Linked to a interfaces even though the host.name had config

[ https://issues.apache.org/jira/browse/KAFKA-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2395. -- Resolution: Not A Problem extra port is related to KAFKA-1869 > Linked to a interfaces even though

Re: Didn't receive video stream data into spark through kafka

Hi Biswajit, Can you please provide more information: - What other symptoms do you see? Are all your Kafka brokers up and running? - What replication did you set for offsets.topic.replication.factor (i.e. replication factor of __consumer_offsets) in your Kafka broker's config ? What is set for

[jira] [Resolved] (KAFKA-7159) mark configuration files in confluent-kafka RPM SPEC file

[ https://issues.apache.org/jira/browse/KAFKA-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari resolved KAFKA-7159. --- Resolution: Won't Fix > mark configuration files in confluent-kafka RPM SPEC file >

Build failed in Jenkins: kafka-trunk-jdk10 #315

See Changes: [wangguoz] KAFKA-7144: Fix task assignment to be even (#5390) -- [...truncated 1.98 MB...]

Metric names

Hi there, I am having problems aggregating metrics for our monitoring system, because some JMX beans use dots in its names (e.g. kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec, "topic" just uses the plain topic name as is). Other JMX beans replace dots by underscores, like the one

***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hey Vahid, Sorry for the confusion. I think we all agree that going forward, we shouldn't support the empty group id, so the question is just around compatibility. I think we have to bump the OffsetCommit API version so that old clients which are unknowingly depending on the default empty group

Build failed in Jenkins: kafka-trunk-jdk8 #2838

See Changes: [github] MINOR: internal config objects should not be logged (#5389) -- [...truncated 883.98 KB...] kafka.utils.SchedulerTest >