Re: [ANNOUNCE] Apache Kafka 3.1.0

[Gwen Shapira]

Exciting! Thanks for driving the release, David.

On Mon, Jan 24, 2022 at 9:04 AM David Jacot  wrote:
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.1.0.
>
> It is a major release that includes many new features, including:
>
> * Apache Kafka supports Java 17
> * The FetchRequest supports Topic IDs (KIP-516)
> * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> * Add broker count metrics (KIP-748)
> * Differentiate consistently metric latency measured in millis and
> nanos (KIP-773)
> * The eager rebalance protocol is deprecated (KAFKA-13439)
> * Add TaskId field to StreamsException (KIP-783)
> * Custom partitioners in foreign-key joins (KIP-775)
> * Fetch/findSessions queries with open endpoints for
> SessionStore/WindowStore (KIP-766)
> * Range queries with open endpoints (KIP-763)
> * Add total blocked time metric to Streams (KIP-761)
> * Add additional configuration to control MirrorMaker2 internal topics
> naming convention (KIP-690)
>
> You may read a more detailed list of features in the 3.1.0 blog post:
> https://blogs.apache.org/kafka/
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#3.1.0
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 114 contributors to this release!
>
> A. Sophie Blee-Goldman, Alexander Iskuskov, Alexander Stohr, Almog
> Gavra, Andras Katona, Andrew Patterson, Andy Chambers, Andy Lapidas,
> Anna Sophie Blee-Goldman, Antony Stubbs, Arjun Satish, Bill Bejeck,
> Boyang Chen, Bruno Cadonna, CHUN-HAO TANG, Cheng Tan, Chia-Ping Tsai,
> Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel
> Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo
> Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez,
> Ismael Juma, Israel Ekpo, Ivan Ponomarev, Jakub Scholz, James Galasyn,
> Jason Gustafson, Jeff Kim, Jim Galasyn, JoeCqupt, Joel Hamill, John
> Gray, John Roesler, Jongho Jeon, Jorge Esteban Quilcate Otoya, Jose
> Sancio, Josep Prat, José Armando García Sancio, Jun Rao, Justine
> Olshan, Kalpesh Patel, Kamal Chandraprakash, Kevin Zhang, Kirk True,
> Konstantine Karantasis, Kowshik Prakasam, Leah Thomas, Lee Dongjin,
> Lucas Bradstreet, Luke Chen, Manikumar Reddy, Matthew Wong, Matthias
> J. Sax, Michael Carter, Mickael Maison, Nigel Liang, Niket, Niket
> Goel, Oliver Hutchison, Omnia G H Ibrahim, Patrick Stuedi, Phil
> Hardwick, Prateek Agarwal, Rajini Sivaram, Randall Hauch, René Kerner,
> Richard Yu, Rohan, Ron Dagostino, Ryan Dielhenn, Sanjana Kaundinya,
> Satish Duggana, Sergio Peña, Sherzod Mamadaliev, Stanislav Vodetskyi,
> Ted Yu, Tom Bentley, Tomas Forsman, Tomer Wizman, Uwe Eisele, Victoria
> Xia, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Weisheng
> Yang, Xavier Léauté, Yanwen(Jason) Lin, Yi Ding, Zara Lim, andy0x01,
> dengziming, feyman2016, ik, ik.lim, jem, jiangyuan, kpatelatwork,
> leah, loboya~, lujiefsi, sebbASF, singingMan, vamossagar12,
> wenbingshen
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> David

[ANNOUNCE] New Kafka PMC member: David Jacot

[Gwen Shapira]

Hi everyone,

David Jacot has been an Apache Kafka committer since Oct 2020 and has been 
contributing to the community consistently this entire time - especially 
notable the fact that he reviewed around 150 PRs in the last year. It is my 
pleasure to announce that David agreed to join the Kafka PMC.

Congratulations, David!

Gwen Shapira, on behalf of Apache Kafka PMC

Re: [ANNOUNCE] Apache Kafka 2.7.0

[Gwen Shapira]

woooh!!!

Great job on the release Bill and everyone!

On Mon, Dec 21, 2020 at 8:01 AM Bill Bejeck  wrote:
>
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.7.0
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.7.0/RELEASE_NOTES.html
>
> You can read about some of the more prominent changes in the Apache Kafka
> blog:
> https://blogs.apache.org/kafka/entry/what-s-new-in-apache4
>
> You can download the source and binary release (Scala 2.12, 2.13) from:
> https://kafka.apache.org/downloads#2.7.0
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 117 contributors to this release!
>
> A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Adem Efe Gencer,
> albert02lowis, Alex Diachenko, Andras Katona, Andre Araujo, Andrew Choi,
> Andrew Egelhofer, Andy Coates, Ankit Kumar, Anna Povzner, Antony Stubbs,
> Arjun Satish, Ashish Roy, Auston, Badai Aqrandista, Benoit Maggi, bill,
> Bill Bejeck, Bob Barrett, Boyang Chen, Brian Byrne, Bruno Cadonna, Can
> Cecen, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> David Arthur, David Jacot, David Mao, Dhruvil Shah, Dima Reznik, Edoardo
> Comar, Ego, Evelyn Bayes, feyman2016, Gal Margalit, gnkoshelev, Gokul
> Srinivas, Gonzalo Muñoz, Greg Harris, Guozhang Wang, high.lee, huangyiming,
> huxi, Igor Soarez, Ismael Juma, Ivan Yurchenko, Jason Gustafson, Jeff Kim,
> jeff kim, Jesse Gorzinski, jiameixie, Jim Galasyn, JoelWee, John Roesler,
> John Thomas, Jorge Esteban Quilcate Otoya, Julien Jean Paul Sirocchi,
> Justine Olshan, khairy, Konstantine Karantasis, Kowshik Prakasam, leah, Lee
> Dongjin, Leonard Ge, Levani Kokhreidze, Lucas Bradstreet, Lucent-Wong, Luke
> Chen, Mandar Tillu, manijndl7, Manikumar Reddy, Mario Molina, Matthias J.
> Sax, Micah Paul Ramos, Michael Bingham, Mickael Maison, Navina Ramesh,
> Nikhil Bhatia, Nikolay, Nikolay Izhikov, Ning Zhang, Nitesh Mor, Noa
> Resare, Rajini Sivaram, Raman Verma, Randall Hauch, Rens Groothuijsen,
> Richard Fussenegger, Rob Meng, Rohan, Ron Dagostino, Sanjana Kaundinya,
> Sasaki Toru, sbellapu, serjchebotarev, Shaik Zakir Hussain, Shailesh
> Panwar, Sharath Bhat, showuon, Stanislav Kozlovski, Thorsten Hake, Tom
> Bentley, tswstarplanet, vamossagar12, Vikas Singh, vinoth chandar, Vito
> Jeng, voffcheg109, xakassi, Xavier Léauté, Yuriy Badalyantc, Zach Zhang
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> Bill Bejeck



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.7.0 RC6

[Gwen Shapira]

+1 (binding)

Thank you for the release, Bill!
I validated signatures, built from source package and ran the perf
producer/consumer to validate.

On Wed, Dec 16, 2020 at 6:53 AM Bill Bejeck  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the seventh candidate for release of Apache Kafka 2.7.0.
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> *** Please download, test and vote by Monday, December 21, 12 PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc6
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-2.7-jdk8/detail/kafka-2.7-jdk8/81/
>
> Thanks,
> Bill



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.7.0 RC3

[Gwen Shapira]

+1 (binding) - assuming we get a successful Jenkins build for the branch.

I built from sources, tested resulting binaries locally, verified
signature and checksums.

Thank you for the release, Bill.

On Wed, Nov 25, 2020 at 7:31 AM Bill Bejeck  wrote:
>
> This is the fourth candidate for the release of Apache Kafka 2.7.0.
>
> This is a major release that includes many new features, including:
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 84
> bug fixes.
>
> Release notes for the 2.7.0 release:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday, December 2, 12PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc3
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests: (link to follow)
> System tests: (link to follow)
>
> Thanks,
> Bill



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

[Gwen Shapira]

Congratulations, Sophie!

On Mon, Oct 19, 2020 at 9:41 AM Matthias J. Sax  wrote:
>
> Hi all,
>
> I am excited to announce that A. Sophie Blee-Goldman has accepted her
> invitation to become an Apache Kafka committer.
>
> Sophie is actively contributing to Kafka since Feb 2019 and has
> accumulated 140 commits. She authored 4 KIPs in the lead
>
>  - KIP-453: Add close() method to RocksDBConfigSetter
>  - KIP-445: In-memory Session Store
>  - KIP-428: Add in-memory window store
>  - KIP-613: Add end-to-end latency metrics to Streams
>
> and helped to implement two critical KIPs, 429 (incremental rebalancing)
> and 441 (smooth auto-scaling; not just implementation but also design).
>
> In addition, she participates in basically every Kafka Streams related
> KIP discussion, reviewed 142 PRs, and is active on the user mailing list.
>
> Thanks for all the contributions, Sophie!
>
>
> Please join me to congratulate her!
>  -Matthias
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[ANNOUNCE] New committer: David Jacot

[Gwen Shapira]

The PMC for Apache Kafka has invited David Jacot as a committer, and
we are excited to say that he accepted!

David Jacot has been contributing to Apache Kafka since July 2015 (!)
and has been very active since August 2019. He contributed several
notable KIPs:

KIP-511: Collect and Expose Client Name and Version in Brokers
KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies:
KIP-570: Add leader epoch in StopReplicaReques
KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations
KIP-496 Added an API for the deletion of consumer offsets

In addition, David Jacot reviewed many community contributions and
showed great technical and architectural taste. Great reviews are hard
and often thankless work - but this is what makes Kafka a great
product and helps us grow our community.

Thanks for all the contributions, David! Looking forward to more
collaboration in the Apache Kafka community.

-- 
Gwen Shapira

Re: [VOTE] 2.6.0 RC2

[Gwen Shapira]

Thank you, Randall for driving this release.

+1 (binding) after verifying signatures and hashes, building from sources,
running unit/integration tests and some manual tests with the 2.13 build.

Two minor things:
1. There were two sitedoc files - 2.12 and 2.13, we don't really need two
sitedocs generated. Not a big deal, but maybe worth tracking and fixing.
2. I got one test failure locally:

org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus failed, log
available in
/Users/gwenshap/releases/2.6.0-rc2/kafka-2.6.0-src/tools/build/reports/testOutput/org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus.test.stdout

org.apache.kafka.trogdor.agent.AgentTest > testAgentGetStatus FAILED
java.lang.RuntimeException:
at
org.apache.kafka.trogdor.rest.RestExceptionMapper.toException(RestExceptionMapper.java:69)
at
org.apache.kafka.trogdor.rest.JsonRestServer$HttpResponse.body(JsonRestServer.java:285)
at
org.apache.kafka.trogdor.agent.AgentClient.status(AgentClient.java:130)
at
org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus(AgentTest.java:115)

Gwen

On Tue, Jul 28, 2020 at 2:50 PM Randall Hauch  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 2.6.0. This is a
> major release that includes many new features, including:
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes a few other features, 74 improvements, 175 bug
> fixes, plus other fixes.
>
> Release notes for the 2.6.0 release:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, August 3, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> https://github.com/apache/kafka/releases/tag/2.6.0-rc2
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/101/
> System tests: (link to follow)
>
>
> Thanks,
> Randall Hauch
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC1

[Gwen Shapira]

Oh wow, I love this checklist. I don't think we'll have time to create one for 
this release, but will be great to track this via JIRA and see if we can get 
all those contributed before 2.6...

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Mon, Mar 16, 2020 at 3:02 PM, Israel Ekpo < israele...@gmail.com > wrote:

> 
> 
> 
> - Download artifacts successfully
> - Verified signatures successfully
> - All tests have passed so far for Scala 2.12. Have not run it on 2.13 yet
> 
> 
> 
> 
> +1 (non-binding) for the release
> 
> 
> 
> I do have some feedback so I think we should include in the RC
> announcement a link for how the community should test and include
> information like:
> 
> 
> 
> - How to set up test environment for unit and functional tests
> - Java version(s) needed for the tests
> - Scala version(s) needed for the tests
> - Gradle version needed
> - Sample script for running sanity checks and unit tests
> - Sample Helm Charts for running all the basic components on a Kubernetes
> - Sample Ansible Script for running all the basic components on Virtual
> Machines
> 
> 
> 
> It takes a bit of time for newcomers to investigate why the tests are not
> running successfully in the beginning and providing guidance for these
> categories of contributors will be great. If I did not know where to look
> (kafka-2.5.0-src/gradle/dependencies.gradle) it would take longer to
> figure out why the tests are not working/running
> 
> 
> 
> Thanks.
> 
> 
> 
> On Thu, Mar 12, 2020 at 11:21 AM Bill Bejeck < bbejeck@ gmail. com (
> bbej...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi David,
>> 
>> 
>> 
>> 1. Scanned the Javadoc, looks good
>> 2. Downloaded kafka_2.12-2.5.0 and ran the quickstart and streams
>> quickstart
>> 3. Verified the signatures
>> 
>> 
>> 
>> +1 (non-binding)
>> 
>> 
>> 
>> Thanks for running the release David!
>> 
>> 
>> 
>> -Bill
>> 
>> 
>> 
>> On Tue, Mar 10, 2020 at 4:01 PM David Arthur < david. arthur@ confluent. io
>> ( david.art...@confluent.io ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Thanks for the test failure reports, Tom. Tracking (and fixing) these is
>>> important and will make future release managers have an easier time :)
>>> 
>>> 
>>> 
>>> -David
>>> 
>>> 
>>> 
>>> On Tue, Mar 10, 2020 at 10:16 AM Tom Bentley < tbentley@ redhat. com (
>>> tbent...@redhat.com ) > wrote:
>>> 
>>> 
>>>> 
>>>> 
>>>> Hi David,
>>>> 
>>>> 
>>>> 
>>>> I verified signatures, built the tagged branch and ran unit and
>>>> integration
>>>> tests. I found some flaky tests, as follows:
>>>> 
>>>> 
>>>> 
>>>> https:/ / issues. apache. org/ jira/ browse/ KAFKA-9691 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9691 ) (new) https:/ / issues.
>>>> apache. org/ jira/ browse/ KAFKA-9692 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9692 ) (new) https:/ / issues.
>>>> apache. org/ jira/ browse/ KAFKA-9283 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9283 ) (already reported)
>>>> 
>>>> 
>>>> 
>>>> Many thanks,
>>>> 
>>>> 
>>>> 
>>>> Tom
>>>> 
>>>> 
>>>> 
>>>> On Tue, Mar 10, 2020 at 3:28 AM David Arthur < mumrah@ gmail. com (
>>>> mum...@gmail.com ) > wrote:
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> Hello Kafka users, developers and client-developers,
>>>>> 
>>>>> 
>>>>> 
>>>>> This is the second candidate for release of Apache Kafka 2.5.0. The
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> first
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> release candidate included an erroneous NOTICE file, so another RC was
>>>>> needed to fix that.
>>>>> 
>>>>> 
>>>>> 
>>>>> This is a major release of Kafka which includes many new features,
>>>>> improvements, and bug fixes including:
>>>>> 
>>>>> 
>>>>> 
>>>>> * TLS 1.3 support (1.2 is now the

[ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Gwen Shapira]

Hi everyone,

I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
Reddy are now members of Apache Kafka PMC.

Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
2019. They all contributed many patches, code reviews and participated
in many KIP discussions. We appreciate their contributions and are
looking forward to many more to come.

Congrats Colin, Vahid and Manikumar!

Gwen, on behalf of Apache Kafka PMC

Re: [VOTE] 2.4.0 RC4

[Gwen Shapira]

+1 (binding)

Validated signatures, tests and ran some test workloads.

Thank you so much for driving this. Mani.

On Mon, Dec 9, 2019 at 9:32 AM Manikumar  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the fifth candidate for release of Apache Kafka 2.4.0.
>
> This release includes many new features, including:
> - Allow consumers to fetch from closest replica
> - Support for incremental cooperative rebalancing to the consumer rebalance
> protocol
> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
> engine
> - New Java authorizer Interface
> - Support for non-key joining in KTable
> - Administrative API for replica reassignment
> - Sticky partitioner
> - Return topic metadata and configs in CreateTopics response
> - Securing Internal connect REST endpoints
> - API to delete consumer offsets and expose it via the AdminClient.
>
> Release notes for the 2.4.0 release:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, December 12, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/javadoc/
>
> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
> https://github.com/apache/kafka/releases/tag/2.4.0-rc4
>
> * Documentation:
> https://kafka.apache.org/24/documentation.html
>
> * Protocol:
> https://kafka.apache.org/24/protocol.html
>
> Thanks,
> Manikumar

Re: [ANNOUNCE] New committer: Mickael Maison

[Gwen Shapira]

Congratulations Mickael! Well deserved!

On Thu, Nov 7, 2019 at 1:38 PM Jun Rao  wrote:
>
> Hi, Everyone,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer Mickael
> Maison.
>
> Mickael has been contributing to Kafka since 2016. He proposed and
> implemented multiple KIPs. He has also been propomating Kafka through blogs
> and public talks.
>
> Congratulations, Mickael!
>
> Thanks,
>
> Jun (on behalf of the Apache Kafka PMC)

Re: [VOTE] 2.3.1 RC1

[Gwen Shapira]

David,

Why do we have two site-doc packages, one for each Scala version? It
is just HTML, right? IIRC, in previous releases we only packaged the
docs once?

Gwen

On Fri, Oct 4, 2019 at 6:52 PM David Arthur  wrote:
>
> Hello all, we identified a few bugs and a dependency update we wanted to
> get fixed for 2.3.1. In particular, there was a problem with rolling
> upgrades of streams applications (KAFKA-8649).
>
> Check out the release notes for a complete list.
> https://home.apache.org/~davidarthur/kafka-2.3.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday October 9th, 9pm PST
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-2.3.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~davidarthur/kafka-2.3.1-rc1/javadoc/
>
> * Tag to be voted upon (off 2.3 branch) is the 2.3.1 tag:
> https://github.com/apache/kafka/releases/tag/2.3.1-rc1
>
> * Documentation:
> https://kafka.apache.org/23/documentation.html
>
> * Protocol:
> https://kafka.apache.org/23/protocol.html
>
> * Successful Jenkins builds for the 2.3 branch are TBD but will be located:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.3-jdk8/
>
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.3/
>
>
> Thanks!
> David Arthur

Re: [VOTE] 2.3.0 RC3

[Gwen Shapira]

+1 (binding)
Verified signatures, verified good build on jenkins, built from
sources anyway and ran quickstart on the 2.11 binary.

Looks good!

On Sun, Jun 23, 2019 at 3:06 PM Jakub Scholz  wrote:
>
> +1 (non-binding). I used the binaries and run some of my tests against them.
>
> On Thu, Jun 20, 2019 at 12:03 AM Colin McCabe  wrote:
>
> > Hi all,
> >
> > We discovered some problems with the second release candidate (RC2) of
> > 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which includes the
> > fix for this issue.
> >
> > Check out the release notes for the 2.3.0 release here:
> > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
> >
> > The vote will go until Saturday, June 22nd, or until we create another RC.
> >
> > * Kafka's KEYS file containing PGP keys we use to sign the release can be
> > found here:
> > https://kafka.apache.org/KEYS
> >
> > * The release artifacts to be voted upon (source and binary) are here:
> > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
> >
> > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.3.0-rc3
> >
> > best,
> > Colin
> >
> > C.
> >



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.2.1 RC1

[Gwen Shapira]

+1 (binding)

Validated signatures and last good test.

I also took the liberty of adding Vahid's keys to
http://www.apache.org/dist/kafka/KEYS.
The signature process
(https://www.apache.org/dev/release-signing.html#keys-policy) requires
that the keys used to sign the release will be added and in the past
the PMC received emails expressing concern about the validity of our
releases.

Gwen

On Sat, Jun 1, 2019 at 2:33 PM Vahid Hashemian
 wrote:
>
> I'm a +1 on this RC too. I compiled the source, ran quickstart and tests
> successfully.
>
> Therefore, 2.2.1 RC1 passes with the following +1 votes and no -1 or 0
> votes:
>
> Binding +1s: Harsha, Matthias, Vahid
> Non-binding +1s: Jonathan, Jakub, Victor, Andrew, Mickael, Satish
>
> Here are the vote threads:
> - https://www.mail-archive.com/dev@kafka.apache.org/msg97862.html
> - https://www.mail-archive.com/users@kafka.apache.org/msg34256.html
>
> Thanks to everyone who spent time verifying this release candidate.
>
> I'll proceed with the release process.
>
> --Vahid
>
>
> On Mon, May 13, 2019 at 8:15 PM Vahid Hashemian 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 2.2.1.
> >
> > Compared to RC0, this release candidate also fixes the following issues:
> >
> >- [KAFKA-6789] - Add retry logic in AdminClient requests
> >- [KAFKA-8348] - Document of kafkaStreams improvement
> >- [KAFKA-7633] - Kafka Connect requires permission to create internal
> >topics even if they exist
> >- [KAFKA-8240] - Source.equals() can fail with NPE
> >- [KAFKA-8335] - Log cleaner skips Transactional mark and batch
> >record, causing unlimited growth of __consumer_offsets
> >- [KAFKA-8352] - Connect System Tests are failing with 404
> >
> > Release notes for the 2.2.1 release:
> > https://home.apache.org/~vahid/kafka-2.2.1-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, May 16, 9:00 pm PT.
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~vahid/kafka-2.2.1-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~vahid/kafka-2.2.1-rc1/javadoc/
> >
> > * Tag to be voted upon (off 2.2 branch) is the 2.2.1 tag:
> > https://github.com/apache/kafka/releases/tag/2.2.1-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/22/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/22/protocol.html
> >
> > * Successful Jenkins builds for the 2.2 branch:
> > Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/115/
> >
> > Thanks!
> > --Vahid
> >
>
>
> --
>
> Thanks!
> --Vahid



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.2.0 RC2

[Gwen Shapira]

+1

Validated signatures, build from source and quickstart on the binaries.
Looks good!
Awesome release, Matthias.

On Sat, Mar 9, 2019 at 2:56 PM Matthias J. Sax 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 2.2.0.
>
>  - Added SSL support for custom principal name
>  - Allow SASL connections to periodically re-authenticate
>  - Command line tool bin/kafka-topics.sh adds AdminClient support
>  - Improved consumer group management
>- default group.id is `null` instead of empty string
>  - API improvement
>- Producer: introduce close(Duration)
>- AdminClient: introduce close(Duration)
>- Kafka Streams: new flatTransform() operator in Streams DSL
>- KafkaStreams (and other classed) now implement AutoClosable to
> support try-with-resource
>- New Serdes and default method implementations
>  - Kafka Streams exposed internal client.id via ThreadMetadata
>  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
> output `NaN` as default value
> Release notes for the 2.2.0 release:
> https://home.apache.org/~mjsax/kafka-2.2.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test, and vote by Thursday, March 14, 9am PST.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mjsax/kafka-2.2.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mjsax/kafka-2.2.0-rc2/javadoc/
>
> * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag:
> https://github.com/apache/kafka/releases/tag/2.2.0-rc2
>
> * Documentation:
> https://kafka.apache.org/22/documentation.html
>
> * Protocol:
> https://kafka.apache.org/22/protocol.html
>
> * Jenkins builds for the 2.2 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.2/
>
> /**
>
> Thanks,
>
> -Matthias
>
>

-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: [VOTE] 2.1.1 RC2

[Gwen Shapira]

+1
(Validated signatures, unit tests, quickstart)

Thanks, Colin, for managing the bugfix release.

On Fri, Feb 8, 2019 at 12:02 PM Colin McCabe  wrote:
>
> Hi all,
>
> This is the third candidate for release of Apache Kafka 2.1.1.  This release 
> includes many bug fixes for Apache Kafka 2.1.
>
> Compared to rc1, this release includes the following changes:
> * MINOR: release.py: fix some compatibility problems.
> * KAFKA-7897; Disable leader epoch cache when older message formats are used
> * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER refresh login 
> fails
> * MINOR: Fix more places where the version should be bumped from 2.1.0 -> 
> 2.1.1
> * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if the 
> hostname of the broker changes.
> * KAFKA-7873; Always seek to beginning in KafkaBasedLog
> * MINOR: Correctly set dev version in version.py
>
> Check out the release notes here:
> http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html
>
> The vote will go until Wednesday, February 13st.
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/
>
> * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag:
> https://github.com/apache/kafka/releases/tag/2.1.1-rc2
>
> * Jenkins builds for the 2.1 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/
>
> Thanks to everyone who tested the earlier RCs.
>
> cheers,
> Colin



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Gwen Shapira]

Congrats, Vahid. Thank you for all your contribution!

On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson  Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [VOTE] 2.0.1 RC0

[Gwen Shapira]

+1 (binding)

Checked signatures, build and quickstart.

Thank you for managing the release, Mani!


On Thu, Oct 25, 2018 at 7:29 PM Manikumar  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 2.0.1.
>
> This is a bug fix release closing 49 tickets:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>
> Release notes for the 2.0.1 release:
> http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by  Tuesday, October 30, end of day
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>
> * Documentation:
> http://kafka.apache.org/20/documentation.html
>
> * Protocol:
> http://kafka.apache.org/20/protocol.html
>
> * Successful Jenkins builds for the 2.0 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/177/
>
> /**
>
> Thanks,
> Manikumar



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Gwen Shapira]

Congrats Dong Lin! Well deserved!

On Mon, Aug 20, 2018, 3:55 AM Ismael Juma  wrote:

> Hi everyone,
>
> Dong Lin became a committer in March 2018. Since then, he has remained
> active in the community and contributed a number of patches, reviewed
> several pull requests and participated in numerous KIP discussions. I am
> happy to announce that Dong is now a member of the
> Apache Kafka PMC.
>
> Congratulation Dong! Looking forward to your future contributions.
>
> Ismael, on behalf of the Apache Kafka PMC
>

Re: [VOTE] 2.0.0 RC2

[Gwen Shapira]

+1 (binding)

- validated signatures
- quickstart on binary distributions
- unit-tests and packaging on src distribution

Looking awesome! Excited for this release and especially the new connect
features :)

On Tue, Jul 10, 2018 at 10:17 AM, Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new  KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
> A few notable highlights:
>
>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
>(KIP-277)
>- SASL/OAUTHBEARER implementation (KIP-255)
>- Improved quota communication and customization of quotas (KIP-219,
>KIP-257)
>- Efficient memory usage for down conversion (KIP-283)
>- Fix log divergence between leader and follower during fast leader
>failover (KIP-279)
>- Drop support for Java 7 and remove deprecated code including old scala
>clients
>- Connect REST extension plugin, support for externalizing secrets and
>improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>- Scala API for Kafka Streams and other Streams API improvements
>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.0/
> 27/
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: Details of segment deletion

[Gwen Shapira]

See below:

On Mon, Jun 11, 2018 at 3:36 AM, Simon Cooper <
simon.coo...@featurespace.co.uk> wrote:

> Hi,
>
> I've ben trying to work out the details of when exactly kafka log segments
> get deleted for to the retention period, so it would be helpful if someone
> could clarify the behaviour:
>
>
>   *   Is a segment only deleted when all messages in that segment have
> 'timed out', or are messages deleted within each segment?
>

Kafka only deletes entire segments (except for compacted topics, which are
a different story)



>   *   Does the server artificially limit the messages returned to clients
> to those within the retention period, even if they still exist in the
> segment file?
>

Older messages can be read if the segment wasn't deleted yet. You can check
the "beginning of log" offset JMX metric to see what is the oldest offset
available to consumers on each partition.


>   *   Does the segment deletion happen when a new segment is created, or
> is it done as a separate operation by the log cleaner?
>

Separate operation by log cleaner, but note that active segment is never
deleted so sometimes you are waiting for new segment to get created before
a new one is deleted.


>
> Thanks for the help!
> Simon Cooper
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Hoping to see the community at Kafka Summit SF

[Gwen Shapira]

Hello Kafka users and contributors,

Kafka Summit SF call for proposal is open until Saturday, June 16. You are
all invited to submit your talk proposals. Sharing your knowledge, stories
and experience is a great way to contribute to the community.

I consistently notice that people with great stories somehow decide that
they are not good enough. I encourage you to submit anyway and let the
conference committee decide. If you want feedback on a proposal, feel free
to email me directly and I'll be happy to help.

Even if you don't submit a talk, you should absolutely attend. The talks
will be amazing and you'll become a better Kafka expert. Most of the
committers will be there, so you will really have this opportunity to
discuss the details of Kafka, why design decisions were made - and how to
contribute more to Kafka.

And since the entire Kafka community should attend, here's the community
discount code: KS18Comm25

Looking forward to your amazing abstracts and to see you all there.

Gwen Shapira

Re: [VOTE] 1.1.0 RC4

[Gwen Shapira]

+1

Checked keys, built, ran quickstart. LGTM.

On Fri, Mar 23, 2018 at 4:37 PM, Rajini Sivaram <rajinisiva...@gmail.com>
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fifth candidate for release of Apache Kafka 1.1.0.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge
> cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
> Release notes for the 1.1.0 release:
>
> http://home.apache.org/~rsivaram/kafka-1.1.0-rc4/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Tuesday March 27th 4pm PT.
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-1.1.0-rc4/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-1.1.0-rc4/javadoc/
>
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
>
> https://github.com/apache/kafka/tree/1.1.0-rc4
>
>
>
> * Documentation:
>
> http://kafka.apache.org/11/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/11/protocol.html
>
>
>
> Thanks,
>
>
> Rajini
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Gwen Shapira]

Dear Kafka Developers, Users and Fans,

Rajini Sivaram became a committer in April 2017.  Since then, she remained
active in the community and contributed major patches, reviews and KIP
discussions. I am glad to announce that Rajini is now a member of the
Apache Kafka PMC.

Congratulations, Rajini and looking forward to your future contributions.

Gwen, on behalf of Apache Kafka PMC

Call for Paper: Kafka Summit London 2018

[Gwen Shapira]

Hi Kafka Users and Contributors,

We are having the first ever Kafka Summit that is really close to Europe
even if not quite part of Europe.

Call for papers is now open and you should all submit abstracts and tell
your colleagues to submit abstracts too! Seriously, if you use Kafka,
whether beginner or advanced, I'm sure you have stories to share and we
really want to hear all about it.

Every year some people ask me "I have a pretty small cluster, are you still
interested?". Yes! innovative use-cases are as important than huge scale.
Probably more so. We are looking for diverse agenda - different industries,
experiences, stream processing technologies, programming languages, data
stores, job descriptions, use-cases, sizes, colors, genders, ages, etc,
etc. We want the conference to have a cool talk for everyone.

Here's the important part:

Submit your proposal here
<https://kafka-summit.org/kafka-summit-london-2018/speakers/>:
*https://kafka-summit.org/kafka-summit-london-2018/speakers/
<https://kafka-summit.org/kafka-summit-london-2018/speakers/>*

Important dates to note:

   -

   Call for Papers closes: December 1, 2017
   -

   Notifications sent: December 20, 2017
   -

   Presentations due for initial review: March 19, 2018
   -

   Presentations due for final approval: April 9, 2018


I hope to see you in London! Registration will open soon!


Gwen Shapira Kafka PMC and conference enthusiast

Re: [VOTE] 0.11.0.0 RC2

[Gwen Shapira]

Hi,

One super minor issue (that can be fixed without a new RC): The big
exactly-once stuff (KIP-98) doesn't actually show up as new features in the
release notes. Most chunks appear as sub-tasks, but the new feature itself
(KAFKA-4815) is marked as 0.11.1.0 so this is missing. I get that this is
cosmetic, but having the biggest feature of the release missing from the
release notes seems like a big deal to me :)

Other than that...
Validated signatures, ran quickstart, ran tests and everything looks good.

+1 (binding).


On Mon, Jun 26, 2017 at 6:54 PM Ismael Juma  wrote:

> Hi Vahid,
>
> There are a few known issues when running Kafka on Windows. A PR with some
> fixes is: https://github.com/apache/kafka/pull/3283. The fact that the
> index cannot be accessed indicates that it may be a similar issue. I
> suggest we move this discussion to the relevant JIRAs instead of the
> release thread.
>
> Ismael
>
> On Mon, Jun 26, 2017 at 11:25 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Ismael,
> >
> > This is the output of core tests from the start until the first failed
> > test.
> >
> > kafka.admin.AdminRackAwareTest >
> testAssignmentWithRackAwareWithUnevenRacks
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest >
> testAssignmentWithRackAwareWithUnevenReplicas
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware PASSED
> >
> > kafka.admin.AdminRackAwareTest > testRackAwareExpansion PASSED
> >
> > kafka.admin.AdminRackAwareTest >
> testAssignmentWith2ReplicasRackAwareWith6Partitions
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRac
> > kAwareWith6PartitionsAnd3Brokers PASSED
> >
> > kafka.admin.AdminRackAwareTest >
> testGetRackAlternatedBrokerListAndAssignReplicasToBrokers
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks PASSED
> >
> > kafka.admin.AdminRackAwareTest > testSingleRack PASSED
> >
> > kafka.admin.AdminRackAwareTest >
> testAssignmentWithRackAwareWithRandomStartIndex
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment
> > PASSED
> >
> > kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks PASSED
> >
> > kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED
> >
> > kafka.admin.ConfigCommandTest > testScramCredentials PASSED
> >
> > kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsEntityType
> > PASSED
> >
> > kafka.admin.ConfigCommandTest > testUserClientQuotaOpts PASSED
> >
> > kafka.admin.ConfigCommandTest > shouldAddTopicConfig PASSED
> >
> > kafka.admin.ConfigCommandTest > shouldAddClientConfig PASSED
> >
> > kafka.admin.ConfigCommandTest > shouldDeleteBrokerConfig PASSED
> >
> > kafka.admin.DeleteConsumerGroupTest >
> testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup
> > PASSED
> >
> > kafka.admin.ConfigCommandTest > testQuotaConfigEntity PASSED
> >
> > kafka.admin.ConfigCommandTest >
> shouldNotUpdateBrokerConfigIfMalformedBracketConfig
> > PASSED
> >
> > kafka.admin.ConfigCommandTest > shouldFailIfUnrecognisedEntityType PASSED
> >
> > kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED
> >
> > kafka.admin.ConfigCommandTest >
> shouldNotUpdateBrokerConfigIfNonExistingConfigIsDeleted
> > PASSED
> >
> > kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED
> >
> > kafka.admin.BrokerApiVersionsCommandTest >
> checkBrokerApiVersionCommandOutput
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldRemoveThrottleReplicaListBasedOnProposedAssignment
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldFindMovingReplicasMultipleTopics
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldNotOverwriteExistingPropertiesWhenLimitIsAdded
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldFindMovingReplicasMultipleTopicsAndPartitions
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldRemoveThrottleLimitFromAllBrokers
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicas
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldFindMovingReplicasMultiplePartitions
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest > shouldSetQuotaLimit PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldFindMovingReplicasWhenProposedIsSubsetOfExisting
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest > shouldUpdateQuotaLimit PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldFindTwoMovingReplicasInSamePartition
> > PASSED
> >
> > kafka.admin.ReassignPartitionsCommandTest >
> shouldNotOverwriteEntityConfigsWhenUpdatingThrottledReplicas
> > PASSED
> >
> > kafka.admin.ConfigCommandTest >
> shouldNotUpdateBrokerConfigIfMalformedEntityName

Re: [ANNOUNCE] New committer: Damian Guy

[Gwen Shapira]

Congratulations :)

On Fri, Jun 9, 2017 at 1:49 PM Vahid S Hashemian 
wrote:

> Great news.
>
> Congrats Damian!
>
> --Vahid
>
>
>
> From:   Guozhang Wang 
> To: "d...@kafka.apache.org" ,
> "users@kafka.apache.org" ,
> "priv...@kafka.apache.org" 
> Date:   06/09/2017 01:34 PM
> Subject:[ANNOUNCE] New committer: Damian Guy
>
>
>
> Hello all,
>
>
> The PMC of Apache Kafka is pleased to announce that we have invited Damian
> Guy as a committer to the project.
>
> Damian has made tremendous contributions to Kafka. He has not only
> contributed a lot into the Streams api, but have also been involved in
> many
> other areas like the producer and consumer clients, broker-side
> coordinators (group coordinator and the ongoing transaction coordinator).
> He has contributed more than 100 patches so far, and have been driving on
> 6
> KIP contributions.
>
> More importantly, Damian has been a very prolific reviewer on open PRs and
> has been actively participating on community activities such as email
> lists
> and slack overflow questions. Through his code contributions and reviews,
> Damian has demonstrated good judgement on system design and code
> qualities,
> especially on thorough unit test coverages. We believe he will make a
> great
> addition to the committers of the community.
>
>
> Thank you for your contributions, Damian!
>
>
> -- Guozhang, on behalf of the Apache Kafka PMC
>
>
>
>
>

[VOTE] KIP-162: Enable topic deletion by default

[Gwen Shapira]

Hi,

The discussion has been quite positive, so I posted a JIRA, a PR and
updated the KIP with the latest decisions.

Lets officially vote on the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default

JIRA is here: https://issues.apache.org/jira/browse/KAFKA-5384

Gwen

Re: KIP-162: Enable topic deletion by default

[Gwen Shapira]

Thanks Vahid,

Do you mind if we leave the command-line out of scope for this?

I can see why adding confirmations, options to bypass confirmations, etc
would be an improvement. However, I've seen no complaints about the current
behavior of the command-line and the KIP doesn't change it at all. So I'd
rather address things separately.

Gwen

On Fri, May 26, 2017 at 8:10 PM Vahid S Hashemian <vahidhashem...@us.ibm.com>
wrote:

> Gwen, thanks for the KIP.
> It looks good to me.
>
> Just a minor suggestion: It would be great if the command asks for a
> confirmation (y/n) before deleting the topic (similar to how removing ACLs
> works).
>
> Thanks.
> --Vahid
>
>
>
> From:   Gwen Shapira <g...@confluent.io>
> To: "d...@kafka.apache.org" <d...@kafka.apache.org>, Users
> <users@kafka.apache.org>
> Date:   05/26/2017 07:04 AM
> Subject:KIP-162: Enable topic deletion by default
>
>
>
> Hi Kafka developers, users and friends,
>
> I've added a KIP to improve our out-of-the-box usability a bit:
> KIP-162: Enable topic deletion by default:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default
>
>
> Pretty simple :) Discussion and feedback are welcome.
>
> Gwen
>
>
>
>
>

KIP-162: Enable topic deletion by default

[Gwen Shapira]

Hi Kafka developers, users and friends,

I've added a KIP to improve our out-of-the-box usability a bit:
KIP-162: Enable topic deletion by default:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default

Pretty simple :) Discussion and feedback are welcome.

Gwen

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

[Gwen Shapira]

+1. Also not sure that adding a parameter to a CLI requires a KIP. It seems
excessive.


On Tue, May 9, 2017 at 7:57 PM Jay Kreps  wrote:

> +1
> On Tue, May 9, 2017 at 3:41 PM BigData dev 
> wrote:
>
> > Hi, Everyone,
> >
> > Since this is a relatively simple change, I would like to start the
> voting
> > process for KIP-156: Add option "dry run" to Streams application reset
> tool
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69410150
> >
> >
> > The vote will run for a minimum of 72 hours.
> >
> >
> > Thanks,
> >
> > Bharat
> >
>

[ANNOUCE] Apache Kafka 0.10.2.1 Released

[Gwen Shapira]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 0.10.2.1. This is a bug fix release that fixes 29 issues in 0.10.2.0.

All of the changes in this release can be found in the release notes:
*https://archive.apache.org/dist/kafka/0.10.2.1/RELEASE_NOTES.html
<https://archive.apache.org/dist/kafka/0.10.2.1/RELEASE_NOTES.html.>

Apache Kafka is a distributed streaming platform with four four core APIs:

** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an output
stream to one or more output topics, effectively transforming the input
streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might capture
every change to a table.three key capabilities:


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react to the
streams of data.


You can download the source release from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.1/kafka-0.10.2.1-src.tgz

and binary releases from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.1/kafka_2.10-0.10.2.1.tgz
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz
<https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz>

A big thank you for the following 25 contributors to this release!

Aaron Coburn, Apurva Mehta, Armin Braun, Ben Stopford, Bill Bejeck,
Bruce Szalwinski, Clemens Valiente, Colin P. Mccabe, Damian Guy, Dong
Lin, Eno Thereska, Ewen Cheslack-Postava, Guozhang Wang, Gwen Shapira,
Ismael Juma, Jason Gustafson, Konstantine Karantasis, Marco Ebert,
Matthias J. Sax, Michael G. Noll, Onur Karaman, Rajini Sivaram, Ryan
P, simplesteph, Vahid Hashemian

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://kafka.apache.org/


Thanks,
-- Gwen

Re: [VOTE] 0.10.2.1 RC3

[Gwen Shapira]

Quick update:
I closed the release on JIRA and bumped the versions in github. Uploaded
artifacts and released the jars in Maven.
Waiting for everything to actually show up before I update the website and
send the announcement. Expect something tonight or tomorrow morning.

Gwen

On Wed, Apr 26, 2017 at 12:16 PM, Gwen Shapira <g...@confluent.io> wrote:

>
> Vote summary:
> +1: 6 (3 binding) - Eno, Ian, Guozhang, Jun, Gwen and Shimi
> 0: 0
> -1: 0
>
> W00t! 72 hours passed and we have 3 binding +1!
>
> Thank you for playing "bugfix release". See you all at the next round :)
> I'll get our bug fixes out the door ASAP.
>
> Gwen
>
>
> On Wed, Apr 26, 2017 at 12:12 PM, Shimi Kiviti <shim...@gmail.com> wrote:
>
>> +1
>>
>> I compiled our (Rollout.io) kafka-stream project, run unit-tests and
>> end-to-end tests (against streams 0.10.2.1 and broker 0.10.1.1)
>> Everything works as expected
>>
>> On Wed, Apr 26, 2017 at 10:05 PM, Gwen Shapira <g...@confluent.io> wrote:
>>
>> > +1 (binding)
>> >
>> > Validated unit tests, quickstarts, connect, signatures
>> >
>> > On Wed, Apr 26, 2017 at 11:30 AM, Guozhang Wang <wangg...@gmail.com>
>> > wrote:
>> >
>> > > +1
>> > >
>> > > Verified unit test on source, and quick start on binary (Scala 2.12
>> > only).
>> > >
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Wed, Apr 26, 2017 at 2:43 AM, Ian Duffy <i...@ianduffy.ie> wrote:
>> > >
>> > > > +1
>> > > >
>> > > > Started using kafka client 0.10.2.1 for our streams applications,
>> seen
>> > a
>> > > > much greater improvement on retries when failures occur.
>> > > > We've been running without manual intervention for > 24 hours which
>> is
>> > > > something we haven't seen in awhile.
>> > > >
>> > > > Found it odd that the RC tag wasn't within the version on the maven
>> > > > staging repository, how do you identify different RC versions? How
>> do
>> > you
>> > > > flush clients cache? etc. Ended up digging down on the index of
>> pages
>> > and
>> > > > verifying the last modified date matched the date on this email
>> thread.
>> > > >
>> > > > Thanks,
>> > > > Ian.
>> > > >
>> > > > On 22 April 2017 at 22:45, Michal Borowiecki <
>> > > > michal.borowie...@openbet.com> wrote:
>> > > >
>> > > >> It's listed below:
>> > > >>
>> > > >> * Maven artifacts to be voted upon:https://repository.
>> > > apache.org/content/groups/staging/
>> > > >>
>> > > >>
>> > > >>
>> > > >> On 22/04/17 19:23, Shimi Kiviti wrote:
>> > > >>
>> > > >> Is there a maven repo with these jars so I can test it against our
>> > kafka
>> > > >> streams services?
>> > > >>
>> > > >> On Sat, Apr 22, 2017 at 9:05 PM, Eno Thereska <
>> eno.there...@gmail.com
>> > >
>> > > <eno.there...@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >>
>> > > >> +1 tested the usual streams tests as before.
>> > > >>
>> > > >> Thanks
>> > > >> Eno
>> > > >>
>> > > >> On 21 Apr 2017, at 17:56, Gwen Shapira <g...@confluent.io> <
>> > > g...@confluent.io> wrote:
>> > > >>
>> > > >> Hello Kafka users, developers, friends, romans, countrypersons,
>> > > >>
>> > > >> This is the fourth (!) candidate for release of Apache Kafka
>> 0.10.2.1.
>> > > >>
>> > > >> It is a bug fix release, so we have lots of bug fixes, some super
>> > > >> important.
>> > > >>
>> > > >> Release notes for the 0.10.2.1 release:http://home.apache.
>> > > org/~gwenshap/kafka-0.10.2.1-rc3/RELEASE_NOTES.html
>> > > >>
>> > > >> *** Please download, test and vote by Wednesday, April 26, 2017 ***
>> > > >>
>> > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > http://kafka.apache.org/KEYS
>> > > >>
>> 

Re: [VOTE] 0.10.2.1 RC3

[Gwen Shapira]

Vote summary:
+1: 6 (3 binding) - Eno, Ian, Guozhang, Jun, Gwen and Shimi
0: 0
-1: 0

W00t! 72 hours passed and we have 3 binding +1!

Thank you for playing "bugfix release". See you all at the next round :)
I'll get our bug fixes out the door ASAP.

Gwen


On Wed, Apr 26, 2017 at 12:12 PM, Shimi Kiviti <shim...@gmail.com> wrote:

> +1
>
> I compiled our (Rollout.io) kafka-stream project, run unit-tests and
> end-to-end tests (against streams 0.10.2.1 and broker 0.10.1.1)
> Everything works as expected
>
> On Wed, Apr 26, 2017 at 10:05 PM, Gwen Shapira <g...@confluent.io> wrote:
>
> > +1 (binding)
> >
> > Validated unit tests, quickstarts, connect, signatures
> >
> > On Wed, Apr 26, 2017 at 11:30 AM, Guozhang Wang <wangg...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Verified unit test on source, and quick start on binary (Scala 2.12
> > only).
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, Apr 26, 2017 at 2:43 AM, Ian Duffy <i...@ianduffy.ie> wrote:
> > >
> > > > +1
> > > >
> > > > Started using kafka client 0.10.2.1 for our streams applications,
> seen
> > a
> > > > much greater improvement on retries when failures occur.
> > > > We've been running without manual intervention for > 24 hours which
> is
> > > > something we haven't seen in awhile.
> > > >
> > > > Found it odd that the RC tag wasn't within the version on the maven
> > > > staging repository, how do you identify different RC versions? How do
> > you
> > > > flush clients cache? etc. Ended up digging down on the index of pages
> > and
> > > > verifying the last modified date matched the date on this email
> thread.
> > > >
> > > > Thanks,
> > > > Ian.
> > > >
> > > > On 22 April 2017 at 22:45, Michal Borowiecki <
> > > > michal.borowie...@openbet.com> wrote:
> > > >
> > > >> It's listed below:
> > > >>
> > > >> * Maven artifacts to be voted upon:https://repository.
> > > apache.org/content/groups/staging/
> > > >>
> > > >>
> > > >>
> > > >> On 22/04/17 19:23, Shimi Kiviti wrote:
> > > >>
> > > >> Is there a maven repo with these jars so I can test it against our
> > kafka
> > > >> streams services?
> > > >>
> > > >> On Sat, Apr 22, 2017 at 9:05 PM, Eno Thereska <
> eno.there...@gmail.com
> > >
> > > <eno.there...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>
> > > >> +1 tested the usual streams tests as before.
> > > >>
> > > >> Thanks
> > > >> Eno
> > > >>
> > > >> On 21 Apr 2017, at 17:56, Gwen Shapira <g...@confluent.io> <
> > > g...@confluent.io> wrote:
> > > >>
> > > >> Hello Kafka users, developers, friends, romans, countrypersons,
> > > >>
> > > >> This is the fourth (!) candidate for release of Apache Kafka
> 0.10.2.1.
> > > >>
> > > >> It is a bug fix release, so we have lots of bug fixes, some super
> > > >> important.
> > > >>
> > > >> Release notes for the 0.10.2.1 release:http://home.apache.
> > > org/~gwenshap/kafka-0.10.2.1-rc3/RELEASE_NOTES.html
> > > >>
> > > >> *** Please download, test and vote by Wednesday, April 26, 2017 ***
> > > >>
> > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > > >>
> > > >> * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/
> > > >>
> > > >> * Maven artifacts to be voted upon:https://repository.
> > > apache.org/content/groups/staging/
> > > >>
> > > >> * Javadoc:http://home.apache.org/~gwenshap/kafka-0.10.2.1-
> > rc3/javadoc/
> > > >>
> > > >> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
> > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > > >>
> > > >> 8e4f09caeaa877f06dc75c7da1af7a727e5e599f
> > > >>
> > > >> * Documentation:http://kafka.apache.org/0102/documentation.html
> &

Re: [VOTE] 0.10.2.1 RC3

[Gwen Shapira]

+1 (binding)

Validated unit tests, quickstarts, connect, signatures

On Wed, Apr 26, 2017 at 11:30 AM, Guozhang Wang <wangg...@gmail.com> wrote:

> +1
>
> Verified unit test on source, and quick start on binary (Scala 2.12 only).
>
>
> Guozhang
>
>
> On Wed, Apr 26, 2017 at 2:43 AM, Ian Duffy <i...@ianduffy.ie> wrote:
>
> > +1
> >
> > Started using kafka client 0.10.2.1 for our streams applications, seen a
> > much greater improvement on retries when failures occur.
> > We've been running without manual intervention for > 24 hours which is
> > something we haven't seen in awhile.
> >
> > Found it odd that the RC tag wasn't within the version on the maven
> > staging repository, how do you identify different RC versions? How do you
> > flush clients cache? etc. Ended up digging down on the index of pages and
> > verifying the last modified date matched the date on this email thread.
> >
> > Thanks,
> > Ian.
> >
> > On 22 April 2017 at 22:45, Michal Borowiecki <
> > michal.borowie...@openbet.com> wrote:
> >
> >> It's listed below:
> >>
> >> * Maven artifacts to be voted upon:https://repository.
> apache.org/content/groups/staging/
> >>
> >>
> >>
> >> On 22/04/17 19:23, Shimi Kiviti wrote:
> >>
> >> Is there a maven repo with these jars so I can test it against our kafka
> >> streams services?
> >>
> >> On Sat, Apr 22, 2017 at 9:05 PM, Eno Thereska <eno.there...@gmail.com>
> <eno.there...@gmail.com>
> >> wrote:
> >>
> >>
> >> +1 tested the usual streams tests as before.
> >>
> >> Thanks
> >> Eno
> >>
> >> On 21 Apr 2017, at 17:56, Gwen Shapira <g...@confluent.io> <
> g...@confluent.io> wrote:
> >>
> >> Hello Kafka users, developers, friends, romans, countrypersons,
> >>
> >> This is the fourth (!) candidate for release of Apache Kafka 0.10.2.1.
> >>
> >> It is a bug fix release, so we have lots of bug fixes, some super
> >> important.
> >>
> >> Release notes for the 0.10.2.1 release:http://home.apache.
> org/~gwenshap/kafka-0.10.2.1-rc3/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Wednesday, April 26, 2017 ***
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/
> >>
> >> * Maven artifacts to be voted upon:https://repository.
> apache.org/content/groups/staging/
> >>
> >> * Javadoc:http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/javadoc/
> >>
> >> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >>
> >> 8e4f09caeaa877f06dc75c7da1af7a727e5e599f
> >>
> >> * Documentation:http://kafka.apache.org/0102/documentation.html
> >>
> >> * Protocol:http://kafka.apache.org/0102/protocol.html
> >>
> >> /**
> >>
> >> Your help in validating this bugfix release is super valuable, so
> >> please take the time to test and vote!
> >>
> >> Suggested tests:
> >> * Grab the source archive and make sure it compiles
> >> * Grab one of the binary distros and run the quickstarts against them
> >> * Extract and verify one of the site docs jars
> >> * Build a sample against jars in the staging repo
> >> * Validate GPG signatures on at least one file
> >> * Validate the javadocs look ok
> >> * The 0.10.2 documentation was updated for this bugfix release
> >> (especially upgrade, streams and connect portions) - please make sure
> >> it looks ok: http://kafka.apache.org/documentation.html
> >>
> >> But above all, try to avoid finding new bugs - we want to get this
> >>
> >> release
> >>
> >> out the door already :P
> >>
> >>
> >> Thanks,
> >> Gwen
> >>
> >>
> >>
> >> --
> >> *Gwen Shapira*
> >> Product Manager | Confluent650.450.2760 <(650)%20450-2760> | @gwenshap
> >> Follow us: Twitter <https://twitter.com/ConfluentInc> <
> https://twitter.com/ConfluentInc> | blog<http://www.confluent.io/blog> <
> http://www.confluent.io/blog>
> >>
> >>
> >> 

Re: Kafka source connector for avro files

[Gwen Shapira]

We added a Byte Converter which essentially does no conversion. Is this
what you are looking for?

https://issues.apache.org/jira/browse/KAFKA-4783

On Tue, Apr 25, 2017 at 11:54 AM, Stas Chizhov <schiz...@gmail.com> wrote:

> Hi,
>
> I have a kafka topic with avro messages + schema registry, which is being
> backed up into s3 as a set of avro files. I need to be able to restore a
> subset of those files into a new topic in the original format with schemas
> published into a schema registry. Am I right that at the moment there is no
> way of avoiding conversion of original avro messages into kafka connect
> format and back in a source connector?
>
> Thank you,
> Stanislav.
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

[ANNOUNCE] New committer: Rajini Sivaram

[Gwen Shapira]

The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
are pleased to announce that she has accepted!

Rajini contributed 83 patches, 8 KIPs (all security and quota
improvements) and a significant number of reviews. She is also on the
conference committee for Kafka Summit, where she helped select content
for our community event. Through her contributions she's shown good
judgement, good coding skills, willingness to work with the community on
finding the best
solutions and very consistent follow through on her work.

Thank you for your contributions, Rajini! Looking forward to many more :)

Gwen, for the Apache Kafka PMC

[VOTE] 0.10.2.1 RC3

[Gwen Shapira]

Hello Kafka users, developers, friends, romans, countrypersons,

This is the fourth (!) candidate for release of Apache Kafka 0.10.2.1.

It is a bug fix release, so we have lots of bug fixes, some super
important.

Release notes for the 0.10.2.1 release:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/RELEASE_NOTES.html

*** Please download, test and vote by Wednesday, April 26, 2017 ***

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc3/javadoc/

* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=8e4f09caeaa877f06dc75c7da1af7a727e5e599f


* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

/**

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok
 * The 0.10.2 documentation was updated for this bugfix release
(especially upgrade, streams and connect portions) - please make sure
it looks ok: http://kafka.apache.org/documentation.html

But above all, try to avoid finding new bugs - we want to get this release
out the door already :P


Thanks,
Gwen



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

[VOTE] 0.10.2.1 RC2

[Gwen Shapira]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 0.10.2.1.

It is a bug fix release, so we have lots of bug fixes, some super
important.

Release notes for the 0.10.2.1 release:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Friday, 8am PST. ***

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc2/javadoc/

* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=dea3da5b31cc310974685a8bbccc34a2ec2ac5c8


* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

/**

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok
 * The 0.10.2 documentation was updated for this bugfix release
(especially upgrade, streams and connect portions) - please make sure
it looks ok: http://kafka.apache.org/documentation.html

Thanks,

Gwen

Re: [VOTE] 0.10.2.1 RC1

[Gwen Shapira]

Dear Kafka Users and Developers,

Over the weekend Becket reported a new bug that exists in 0.10.2:
https://issues.apache.org/jira/browse/KAFKA-5075
Dong Lin was kind and quick enough to resolve it.

While this issue is not a regression, the bug does cause data loss and is
therefore a serious one. I'm inclined to roll another RC for 0.10.2.1 that
includes this patch.

If there are no objections, I will roll out another RC and re-initiate the
vote. Thank you everyone for your patience. Less bugs is good for all of us
:)



On Wed, Apr 12, 2017 at 5:25 PM, Gwen Shapira <g...@confluent.io> wrote:

> Hello Kafka users, developers, client-developers, friends, romans,
> citizens, etc,
>
> This is the second candidate for release of Apache Kafka 0.10.2.1.
>
> This is a bug fix release and it includes fixes and improvements from 24
> JIRAs
> (including a few critical bugs).
>
> Release notes for the 0.10.2.1 release:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, April 17, 5:30 pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> Your help in validating this bugfix release is super valuable, so
> please take the time to test and vote!
>
> Suggested tests:
>  * Grab the source archive and make sure it compiles
>  * Grab one of the binary distros and run the quickstarts against them
>  * Extract and verify one of the site docs jars
>  * Build a sample against jars in the staging repo
>  * Validate GPG signatures on at least one file
>  * Validate the javadocs look ok
>  * The 0.10.2 documentation was updated for this bugfix release
> (especially upgrade, streams and connect portions) - please make sure
> it looks ok: http://kafka.apache.org/documentation.html
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> e133f2ca57670e77f8114cc72dbc2f91a48e3a3b
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> /**
>
> Thanks,
>
> Gwen Shapira
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: [VOTE] 0.10.2.1 RC1

[Gwen Shapira]

Verified my own signatures, ran quickstart and created few Connectors.

+1 (binding)


On Wed, Apr 12, 2017 at 5:25 PM, Gwen Shapira <g...@confluent.io> wrote:
> Hello Kafka users, developers, client-developers, friends, romans,
> citizens, etc,
>
> This is the second candidate for release of Apache Kafka 0.10.2.1.
>
> This is a bug fix release and it includes fixes and improvements from 24 JIRAs
> (including a few critical bugs).
>
> Release notes for the 0.10.2.1 release:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, April 17, 5:30 pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> Your help in validating this bugfix release is super valuable, so
> please take the time to test and vote!
>
> Suggested tests:
>  * Grab the source archive and make sure it compiles
>  * Grab one of the binary distros and run the quickstarts against them
>  * Extract and verify one of the site docs jars
>  * Build a sample against jars in the staging repo
>  * Validate GPG signatures on at least one file
>  * Validate the javadocs look ok
>  * The 0.10.2 documentation was updated for this bugfix release
> (especially upgrade, streams and connect portions) - please make sure
> it looks ok: http://kafka.apache.org/documentation.html
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e133f2ca57670e77f8114cc72dbc2f91a48e3a3b
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> /**
>
> Thanks,
>
> Gwen Shapira



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[VOTE] 0.10.2.1 RC1

[Gwen Shapira]

Hello Kafka users, developers, client-developers, friends, romans,
citizens, etc,

This is the second candidate for release of Apache Kafka 0.10.2.1.

This is a bug fix release and it includes fixes and improvements from 24 JIRAs
(including a few critical bugs).

Release notes for the 0.10.2.1 release:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Monday, April 17, 5:30 pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok
 * The 0.10.2 documentation was updated for this bugfix release
(especially upgrade, streams and connect portions) - please make sure
it looks ok: http://kafka.apache.org/documentation.html

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/javadoc/

* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e133f2ca57670e77f8114cc72dbc2f91a48e3a3b

* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

/**

Thanks,

Gwen Shapira

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Thank you for reporting Eno, and thank you for everyone who tested the
release. This is really appreciated.

I am rolling out a new RC right now. Expect a new vote thread soon :)

On Wed, Apr 12, 2017 at 10:01 AM, Eno Thereska <eno.there...@gmail.com> wrote:
> Hi Gwen,
>
> Unfortunately we found (and just fixed) a bug: 
> https://issues.apache.org/jira/browse/KAFKA-5038 
> <https://issues.apache.org/jira/browse/KAFKA-5038>. The PR just went in 
> 0.10.2 and trunk. It'd be great to do another RC since this problem is in the 
> general space of streams lock issues and we intended to fix them all for this 
> bug fix release.
>
> Thanks
> Eno
>
>> On 12 Apr 2017, at 08:37, Swen Moczarski <swen.moczar...@gmail.com> wrote:
>>
>> Hi Gwen,
>> thanks for the release candidate. Did a quick test, used the RC in my
>> recent project on client side, integration test against server version
>> 0.10.1.1 worked well.
>>
>> +1 (non binding)
>>
>> 2017-04-12 11:31 GMT+02:00 Mickael Maison <mickael.mai...@gmail.com 
>> <mailto:mickael.mai...@gmail.com>>:
>>
>>> I ran the quickstart steps against the 2.11 binary. Everything worked fine
>>> +1
>>>
>>> On Wed, Apr 12, 2017 at 8:53 AM, Michal Borowiecki <
>>> michal.borowie...@openbet.com> wrote:
>>>
>>>> FWIW, I upgraded without issue and noticed the speedup from
>>>> KAFKA-4851/KAFKA-4876.
>>>>
>>>> +1 from me (non-binding)
>>>>
>>>> On 12/04/17 02:06, Gwen Shapira wrote:
>>>>
>>>> Wrong link :)http://kafka.apache.org/documentation/#upgrade
>>>> andhttp://kafka.apache.org/documentation/streams#streams_api_changes_0102
>>>>
>>>> On Tue, Apr 11, 2017 at 5:57 PM, Gwen Shapira <g...@confluent.io 
>>>> <mailto:g...@confluent.io>> <g...@confluent.io <mailto:g...@confluent.io>> 
>>>> wrote:
>>>>
>>>> FYI: I just updated the upgrade notes with Streams 
>>>> changes:http://kafka.apache.org/documentation/#gettingStarted 
>>>> <http://kafka.apache.org/documentation/#gettingStarted>
>>>>
>>>> On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira <g...@confluent.io 
>>>> <mailto:g...@confluent.io>> <g...@confluent.io <mailto:g...@confluent.io>> 
>>>> wrote:
>>>>
>>>> Hello Kafka users, developers and client-developers,
>>>>
>>>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>>>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>>>> (including a few critical bugs). See the release notes for more details:
>>>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html 
>>>> <http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html>
>>>>
>>>> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>>>>
>>>> Your help in validating this bugfix release is super valuable, so
>>>> please take the time to test and vote!
>>>>
>>>> Few notes:
>>>> 1. There are missing "Notable Changes" in the 
>>>> docs:https://github.com/apache/kafka/pull/2824 
>>>> <https://github.com/apache/kafka/pull/2824>
>>>> I will review, merge and update the docs by Monday.
>>>> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
>>>> yet. We may need another RC if system tests fail tonight.
>>>>
>>>> Suggested tests:
>>>> * Grab the source archive and make sure it compiles
>>>> * Grab one of the binary distros and run the quickstarts against them
>>>> * Extract and verify one of the site docs jars
>>>> * Build a sample against jars in the staging repo
>>>> * Validate GPG signatures on at least one file
>>>> * Validate the javadocs look ok
>>>>
>>>> *
>>>>
>>>> Kafka's KEYS file containing PGP keys we use to sign the 
>>>> release:http://kafka.apache.org/KEYS <http://kafka.apache.org/KEYS>
>>>>
>>>> * Release artifacts to be voted upon (source and 
>>>> binary):http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/ 
>>>> <http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/>
>>>>
>>>> * Maven artifacts to be voted 
>>>> upon:https://repository.apache.org/content/groups/staging 
>>>> <https://reposi

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Wrong link :)
http://kafka.apache.org/documentation/#upgrade
and
http://kafka.apache.org/documentation/streams#streams_api_changes_0102

On Tue, Apr 11, 2017 at 5:57 PM, Gwen Shapira <g...@confluent.io> wrote:
> FYI: I just updated the upgrade notes with Streams changes:
> http://kafka.apache.org/documentation/#gettingStarted
>
> On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira <g...@confluent.io> wrote:
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>> (including a few critical bugs). See the release notes for more details:
>>
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>>
>> Your help in validating this bugfix release is super valuable, so
>> please take the time to test and vote!
>>
>> Few notes:
>> 1. There are missing "Notable Changes" in the docs:
>> https://github.com/apache/kafka/pull/2824
>> I will review, merge and update the docs by Monday.
>> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
>> yet. We may need another RC if system tests fail tonight.
>>
>> Suggested tests:
>>  * Grab the source archive and make sure it compiles
>>  * Grab one of the binary distros and run the quickstarts against them
>>  * Extract and verify one of the site docs jars
>>  * Build a sample against jars in the staging repo
>>  * Validate GPG signatures on at least one file
>>  * Validate the javadocs look ok
>>
>> *
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging
>>
>> * Javadoc:
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> Thanks,
>> Gwen Shapira
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

FYI: I just updated the upgrade notes with Streams changes:
http://kafka.apache.org/documentation/#gettingStarted

On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira <g...@confluent.io> wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
> is a bug fix release and it includes fixes and improvements from 24 JIRAs
> (including a few critical bugs). See the release notes for more details:
>
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>
> Your help in validating this bugfix release is super valuable, so
> please take the time to test and vote!
>
> Few notes:
> 1. There are missing "Notable Changes" in the docs:
> https://github.com/apache/kafka/pull/2824
> I will review, merge and update the docs by Monday.
> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
> yet. We may need another RC if system tests fail tonight.
>
> Suggested tests:
>  * Grab the source archive and make sure it compiles
>  * Grab one of the binary distros and run the quickstarts against them
>  * Extract and verify one of the site docs jars
>  * Build a sample against jars in the staging repo
>  * Validate GPG signatures on at least one file
>  * Validate the javadocs look ok
>
> *
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
>
> * Javadoc:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> Thanks,
> Gwen Shapira



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Thanks for the feedback.

I'm not super familiar with the inner workings of Apache's Maven
repos, so I can't explain why we do things the way we do. I followed
the same process on all Apache projects I was on (Kafka, Sqoop,
Flume). Do you know projects that do things the way you suggested?

Either way, may be worthwhile to start a different discussion thread
about RC releases in Maven. Perhaps more knowledgable people will see
it and jump in.

Gwen

On Tue, Apr 11, 2017 at 4:31 PM, Steven Schlansker
<sschlans...@opentable.com> wrote:
>
>> On Apr 7, 2017, at 5:12 PM, Gwen Shapira <g...@confluent.io> wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>> (including a few critical bugs). See the release notes for more details:
>
> Hi Gwen,
>
> I downloaded and tested the RC with a small Kafka Streams app and the upgrade
> seems to have gone smoothly.  (I did not upgrade any brokers though).
>
> One question about the RC process -- currently it seems that the RC is 
> uploaded
> to a staging repo with the final release version.
>
> Would it not be easier for the community if instead the RC is uploaded to the
> main repo with a "-rc" version?
>
>
> Currently, you have to convince Maven to get "0.10.2.1" from the staging repo,
> and then when the final version hits Maven would never update in case there 
> were
> any post-RC changes.
>
> Additionally, if there are further RCs, it is quite easy to confuse yourself
> and not be sure exactly which RC jar you are running at any given time, and 
> the
> problem compounds itself when multiple developers or build boxes are involved.
>
> Many other projects instead would create a "0.10.2.1-rc0" version and publish
> that to the normal Maven Central -- that way it is publicly downloadable and
> strongly tagged / versioned as the RC.
>
> Has the Kafka project given any thought to this sort of a proposal?
> As a tester / outside user it would make the process a little easier.
>
> Either way, excited for the 0.10.2.1 release, and thanks for all the work!
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Thank you for testing!!!

On Mon, Apr 10, 2017 at 7:36 AM, Mathieu Fenniak
<mathieu.fenn...@replicon.com> wrote:
> Hi Gwen,
>
> +1, looks good to me.  Tested broker upgrades, and connect & streams
> applications.
>
> Mathieu
>
>
> On Fri, Apr 7, 2017 at 6:12 PM, Gwen Shapira <g...@confluent.io> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>> (including a few critical bugs). See the release notes for more details:
>>
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>>
>> Your help in validating this bugfix release is super valuable, so
>> please take the time to test and vote!
>>
>> Few notes:
>> 1. There are missing "Notable Changes" in the docs:
>> https://github.com/apache/kafka/pull/2824
>> I will review, merge and update the docs by Monday.
>> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
>> yet. We may need another RC if system tests fail tonight.
>>
>> Suggested tests:
>>  * Grab the source archive and make sure it compiles
>>  * Grab one of the binary distros and run the quickstarts against them
>>  * Extract and verify one of the site docs jars
>>  * Build a sample against jars in the staging repo
>>  * Validate GPG signatures on at least one file
>>  * Validate the javadocs look ok
>>
>> *
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging
>>
>> * Javadoc:
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> d08115f05da0e39c7f75b45e05d6d14ad5baf71d
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> Thanks,
>> Gwen Shapira
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Hello Kafka users, developers and client-developers,

This is the first candidate for the release of Apache Kafka 0.10.2.1. This
is a bug fix release and it includes fixes and improvements from 24 JIRAs
(including a few critical bugs). See the release notes for more details:

http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, 13 April, 8am PT ***

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Few notes:
1. There are missing "Notable Changes" in the docs:
https://github.com/apache/kafka/pull/2824
I will review, merge and update the docs by Monday.
2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
yet. We may need another RC if system tests fail tonight.

Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok

*

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/

* Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d

* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

Thanks,
Gwen Shapira

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Gwen Shapira]

I looked here:
https://repository.apache.org/#nexus-search;gav~org.apache.kafka~kafka_2.12kw,versionexpand


On Wed, Feb 22, 2017 at 2:26 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11
>
> Am I missing something ?
>
> On 23 February 2017 at 9:21:08 am, Gwen Shapira (g...@confluent.io) wrote:
>
> I saw them in Maven yesterday?
>
> On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
> <steph...@simplemachines.com.au> wrote:
> > Awesome thanks a lot! When should we expect the dependencies to be
> released
> > in Maven? (including 2.12 scala)
> >
> > On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
> >
> > Thanks for driving the release, Ewen.
> >
> > Jun
> >
> > On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava <
> ewe...@apache.org>
> > wrote:
> >
> >> The Apache Kafka community is pleased to announce the release for
> Apache
> >> Kafka 0.10.2.0. This is a feature release which includes the completion
> >> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> >> requests merged.
> >>
> >> All of the changes in this release can be found in the release notes:
> >> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> >>
> >> Apache Kafka is a distributed streaming platform with four four core
> >> APIs:
> >>
> >> ** The Producer API allows an application to publish a stream records
> to
> >> one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output
> >> stream to one or more output topics, effectively transforming the input
> >> streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might
> capture
> >> every change to a table.three key capabilities:
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of
> application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react to
> >> the
> >> streams of data.
> >>
> >>
> >> You can download the source release from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka-0.10.2.0-src.tgz
> >>
> >> and binary releases from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.11-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.10-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.12-0.10.2.0.tgz
> >> (experimental 2.12 artifact)
> >>
> >> Thanks to the 101 contributors on this release!
> >>
> >> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> >> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> >> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> >> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> >> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> >> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> >> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> >> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> >> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> >> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> >> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> >> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> >> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> >> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> >> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> >> MURAKAMI Masahiko, Nikki Thean

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Gwen Shapira]

I saw them in Maven yesterday?

On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
<steph...@simplemachines.com.au> wrote:
> Awesome thanks a lot! When should we expect the dependencies to be released
> in Maven? (including 2.12 scala)
>
> On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
>
> Thanks for driving the release, Ewen.
>
> Jun
>
> On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava <ewe...@apache.org>
> wrote:
>
>> The Apache Kafka community is pleased to announce the release for Apache
>> Kafka 0.10.2.0. This is a feature release which includes the completion
>> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
>> requests merged.
>>
>> All of the changes in this release can be found in the release notes:
>> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
>>
>> Apache Kafka is a distributed streaming platform with four four core
>> APIs:
>>
>> ** The Producer API allows an application to publish a stream records to
>> one or more Kafka topics.
>>
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>>
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an
>> output
>> stream to one or more output topics, effectively transforming the input
>> streams to output streams.
>>
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might capture
>> every change to a table.three key capabilities:
>>
>>
>> With these APIs, Kafka can be used for two broad classes of application:
>>
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>>
>> ** Building real-time streaming applications that transform or react to
>> the
>> streams of data.
>>
>>
>> You can download the source release from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka-0.10.2.0-src.tgz
>>
>> and binary releases from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.11-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.10-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.12-0.10.2.0.tgz
>> (experimental 2.12 artifact)
>>
>> Thanks to the 101 contributors on this release!
>>
>> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
>> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
>> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
>> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
>> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
>> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
>> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
>> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
>> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
>> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
>> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
>> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
>> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
>> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
>> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
>> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
>> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
>> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
>> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
>> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
>> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
>> Yang Wei, yaojuncn, Yuto Kawamura
>>
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at
>> http://kafka.apache.org/
>>
>> Thanks,
>> Ewen
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KIP-121 [VOTE]: Add KStream peek method

[Gwen Shapira]

+1 (binding)

On Wed, Feb 8, 2017 at 4:45 PM, Steven Schlansker
<sschlans...@opentable.com> wrote:
> Hi everyone,
>
> Thank you for constructive feedback on KIP-121, KStream.peek(ForeachAction<K, 
> V>) ;
> it seems like it is time to call a vote which I hope will pass easily :)
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-121%3A+Add+KStream+peek+method
>
> I believe the PR attached is already in good shape to consider merging:
>
> https://github.com/apache/kafka/pull/2493
>
> Thanks!
> Steven
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Gwen Shapira]

Just to clarify, we'll need to allow specifying topic and partition. I
don't think we want this on ALL partitions at once.

On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira <g...@confluent.io> wrote:
> That's what I'd like to see. For example, suppose a Connect task fails
> because it can't deserialize an event from a partition. Stop
> connector, move offset forward, start connector. Boom!
>
>
> On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io> wrote:
>> I am not sure about --reset-plus and --reset-minus
>>
>> Would this skip n messages forward/backward for each partitions?
>>
>>
>> -Matthias
>>
>> On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote:
>>> Great. I think I got the idea. What about this options:
>>>
>>> Scenarios:
>>>
>>> 1. Current status
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1´
>>>
>>> 2. To Datetime
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime
>>> 2017-01-01T00:00:00.000´
>>>
>>> 3. To Period
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´
>>>
>>> 4. To Earliest
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´
>>>
>>> 5. To Latest
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´
>>>
>>> 6. Minus 'n' offsets
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´
>>>
>>> 7. Plus 'n' offsets
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´
>>>
>>> 8. To specific offset
>>>
>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´
>>>
>>> Scopes:
>>>
>>> a. All topics used by Consumer Group
>>>
>>> Don't specify --topics
>>>
>>> b. Specific List of Topics
>>>
>>> Add list of values in --topics t1,t2,tn
>>>
>>> c. One Topic, all Partitions
>>>
>>> Add one topic and no partitions values: --topic t1
>>>
>>> d. One Topic, List of Partitions
>>>
>>> Add one topic and partitions values: --topic t1 --partitions 0,1,2
>>>
>>> About Reset Plan (JSON file):
>>>
>>> I think is still valid to have the option to persist reset configuration as
>>> a file, but I agree to give the option to run the tool without going down
>>> to the JSON file.
>>>
>>> Execution options:
>>>
>>> 1. Without execution argument (No args):
>>>
>>> Print out results (reset plan)
>>>
>>> 2. With --execute argument:
>>>
>>> Run reset process
>>>
>>> 3. With --output argument:
>>>
>>> Save result in a JSON format.
>>>
>>> 4. Only with --execute option and --reset-file (path to JSON)
>>>
>>> Reset based on file
>>>
>>> 4. Only with --verify option and --reset-file (path to JSON)
>>>
>>> Verify file values with current offsets
>>>
>>> I think we can remove --generate-and-execute because is a bit clumsy.
>>>
>>> With this options we will be able to execute with manual JSON configuration.
>>>
>>>
>>> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>)
>>> escribió:
>>>
>>>> Yes - using a tool like this to skip a set of consumer groups over a
>>>> corrupt/bad message is definitely appealing.
>>>>
>>>> B
>>>>
>>>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote:
>>>>
>>>>> I like the --reset-to-earliest and --reset-to-latest. In general,
>>>>> since the JSON route is the most challenging for users, we want to
>>>>> provide a lot of ways to do useful things without going there.
>>>>>
>>>>> Two things that can help:
>>>>>
>>>>> 1. A lot of times, users want to skip few messages that cause issues
>>>>> and continue. maybe just specifying the topic, partition and delta
>>>>> will be better than having to find the offset and write a JSON and
>>>>> validate the JSON etc.
>>>>>
>>>>> 2. Thinking if there are other common use-cases that we can make easy
>>>>> rather than just one generic but not 

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Gwen Shapira]

That's what I'd like to see. For example, suppose a Connect task fails
because it can't deserialize an event from a partition. Stop
connector, move offset forward, start connector. Boom!


On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> I am not sure about --reset-plus and --reset-minus
>
> Would this skip n messages forward/backward for each partitions?
>
>
> -Matthias
>
> On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote:
>> Great. I think I got the idea. What about this options:
>>
>> Scenarios:
>>
>> 1. Current status
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1´
>>
>> 2. To Datetime
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime
>> 2017-01-01T00:00:00.000´
>>
>> 3. To Period
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´
>>
>> 4. To Earliest
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´
>>
>> 5. To Latest
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´
>>
>> 6. Minus 'n' offsets
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´
>>
>> 7. Plus 'n' offsets
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´
>>
>> 8. To specific offset
>>
>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´
>>
>> Scopes:
>>
>> a. All topics used by Consumer Group
>>
>> Don't specify --topics
>>
>> b. Specific List of Topics
>>
>> Add list of values in --topics t1,t2,tn
>>
>> c. One Topic, all Partitions
>>
>> Add one topic and no partitions values: --topic t1
>>
>> d. One Topic, List of Partitions
>>
>> Add one topic and partitions values: --topic t1 --partitions 0,1,2
>>
>> About Reset Plan (JSON file):
>>
>> I think is still valid to have the option to persist reset configuration as
>> a file, but I agree to give the option to run the tool without going down
>> to the JSON file.
>>
>> Execution options:
>>
>> 1. Without execution argument (No args):
>>
>> Print out results (reset plan)
>>
>> 2. With --execute argument:
>>
>> Run reset process
>>
>> 3. With --output argument:
>>
>> Save result in a JSON format.
>>
>> 4. Only with --execute option and --reset-file (path to JSON)
>>
>> Reset based on file
>>
>> 4. Only with --verify option and --reset-file (path to JSON)
>>
>> Verify file values with current offsets
>>
>> I think we can remove --generate-and-execute because is a bit clumsy.
>>
>> With this options we will be able to execute with manual JSON configuration.
>>
>>
>> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>)
>> escribió:
>>
>>> Yes - using a tool like this to skip a set of consumer groups over a
>>> corrupt/bad message is definitely appealing.
>>>
>>> B
>>>
>>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote:
>>>
>>>> I like the --reset-to-earliest and --reset-to-latest. In general,
>>>> since the JSON route is the most challenging for users, we want to
>>>> provide a lot of ways to do useful things without going there.
>>>>
>>>> Two things that can help:
>>>>
>>>> 1. A lot of times, users want to skip few messages that cause issues
>>>> and continue. maybe just specifying the topic, partition and delta
>>>> will be better than having to find the offset and write a JSON and
>>>> validate the JSON etc.
>>>>
>>>> 2. Thinking if there are other common use-cases that we can make easy
>>>> rather than just one generic but not very usable method.
>>>>
>>>> Gwen
>>>>
>>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya
>>>> <quilcate.jo...@gmail.com> wrote:
>>>>> Thanks for the feedback!
>>>>>
>>>>> @Onur, @Gwen:
>>>>>
>>>>> Agree. Actually at the first draft I considered to have it inside
>>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone
>>>> tool
>>>>> to describe it clearly and focus it on reset functionality.
>>>>>
>>>>> But now that you mentioned, it does make sense to have it

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Gwen Shapira]

I like the --reset-to-earliest and --reset-to-latest. In general,
since the JSON route is the most challenging for users, we want to
provide a lot of ways to do useful things without going there.

Two things that can help:

1. A lot of times, users want to skip few messages that cause issues
and continue. maybe just specifying the topic, partition and delta
will be better than having to find the offset and write a JSON and
validate the JSON etc.

2. Thinking if there are other common use-cases that we can make easy
rather than just one generic but not very usable method.

Gwen

On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya
<quilcate.jo...@gmail.com> wrote:
> Thanks for the feedback!
>
> @Onur, @Gwen:
>
> Agree. Actually at the first draft I considered to have it inside
> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone tool
> to describe it clearly and focus it on reset functionality.
>
> But now that you mentioned, it does make sense to have it in
> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce it?
>
> Maybe something like this:
>
> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1 --topics t1
> --reset-from 2017-01-01T00:00:00.000 --output plan.json´
>
> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file
> plan.json´
>
> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file
> plan.json´
>
> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group cg1
> --topics t1 --reset-from 2017-01-01T00:00:00.000´
>
> @Gwen:
>
>> It looks exactly like the replica assignment tool
>
> It was influenced by ;-) I use the generate-verify-execute process here to
> make sure user will be aware of the result of this operation. At the
> beginning we considered only add a couple of options to Consumer Group
> Command:
>
> --rewind-to-timestamp and --rewind-to-period
>
> @Onur:
>
>> You can actually get away with overriding while members of the group are live
> with method 2 by using group information from DescribeGroupsRequest.
>
> This means that we need to have Consumer Group stopped before executing and
> start a new consumer internally to do this? Therefore, we won't be able to
> consider executing reset when ConsumerGroup is active? (trying to relate it
> with @Dong 5th question)
>
> @Dong:
>
>> Should we allow user to use wildcard to reset offset of all groups for a
> given topic as well?
>
> I haven't thought about this scenario. Could be interesting. Following the
> recommendation to add it into Consumer Group Command, in this case Group
> argument will be optional if there are only 1 topic. I think for multiple
> topic won't be that useful.
>
>> Should we allow user to specify timestamp per topic partition in the json
> file as well?
>
> Don't think this could be a valid from the tool, but if Reset Plan is
> generated, and user want to set the offset for a specific partition to
> other offset (eventually based on another timestamp), and execute it, it
> will be up to her/him.
>
>> Should the script take some credential file to make sure that this
> operation is authenticated given the potential impact of this operation?
>
> Haven't tried to secure brokers yet, but the tool should support
> authorization if it's enabled in the broker.
>
>> Should we provide constant to reset committed offset to earliest/latest
> offset of a partition, e.g. -1 indicates earliest offset and -2 indicates
> latest offset.
>
> I will go for something like ´--reset-to-earliest´ and ´--reset-to-latest´
>
>> Should we allow dynamic change of the comitted offset when consumer are
> running, such that consumer will seek to the newly committed offset and
> start consuming from there?
>
> Not sure about this. I will recommend to keep it simple and ask user to
> stop consumers first. But I would considered it if the trade-offs are
> clear.
>
> @Matthias
>
> Added :). And thanks a lot for your help to define this KIP!
>
>
>
> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>)
> escribió:
>
>> As long as the CLI is a bit consistent? Like, not just adding 3
>> arguments and a JSON parser to the existing tool, right?
>>
>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman
>> <onurkaraman.apa...@gmail.com> wrote:
>> > I think it makes sense to just add the feature to
>> kafka-consumer-groups.sh
>> >
>> > On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> wrote:
>> >
>> >> Thanks for the KIP. I'm super happy about adding the capability.
>> >>
>> >> I hate the interface, though. 

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Gwen Shapira]

As long as the CLI is a bit consistent? Like, not just adding 3
arguments and a JSON parser to the existing tool, right?

On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman
<onurkaraman.apa...@gmail.com> wrote:
> I think it makes sense to just add the feature to kafka-consumer-groups.sh
>
> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> wrote:
>
>> Thanks for the KIP. I'm super happy about adding the capability.
>>
>> I hate the interface, though. It looks exactly like the replica
>> assignment tool. A tool everyone loves so much that there are multiple
>> projects, open and closed, that try to fix it.
>>
>> Can we swap it with something that looks a bit more like the consumer
>> group tool? or the kafka streams reset tool? Consistency is helpful in
>> such cases. I spent some time learning existing tools and learning yet
>> another one is a deterrent.
>>
>> Gwen
>>
>>
>>
>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya
>> <quilcate.jo...@gmail.com> wrote:
>> > Hi all,
>> >
>> > I would like to propose a KIP to Add a tool to Reset Consumer Group
>> Offsets.
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets
>> >
>> > Please, take a look at the proposal and share your feedback.
>> >
>> > Thanks,
>> > Jorge.
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Gwen Shapira]

Thanks for the KIP. I'm super happy about adding the capability.

I hate the interface, though. It looks exactly like the replica
assignment tool. A tool everyone loves so much that there are multiple
projects, open and closed, that try to fix it.

Can we swap it with something that looks a bit more like the consumer
group tool? or the kafka streams reset tool? Consistency is helpful in
such cases. I spent some time learning existing tools and learning yet
another one is a deterrent.

Gwen



On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya
<quilcate.jo...@gmail.com> wrote:
> Hi all,
>
> I would like to propose a KIP to Add a tool to Reset Consumer Group Offsets.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets
>
> Please, take a look at the proposal and share your feedback.
>
> Thanks,
> Jorge.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Kafka Connect - Unknown magic byte

[Gwen Shapira]

ot;);
> props.put("value.serializer",
> "io.confluent.kafka.serializers.KafkaAvroSerializer");
> props.put("schema.registry.url", "http://localhost:8081;);
>
> String topic = "test";
>
> Producer<String, test> producer = new KafkaProducer<>(props);
>
> while (true) {
> test command = CommandGenerator.getNextTest();
> System.out.println("Generated event " + command.toString());
>
> ProducerRecord<String, test> record = new ProducerRecord<>(topic,
> UUID.randomUUID().toString(), command);
> producer.send(record);
> Thread.sleep(500);
> }
> }
>
> *Consumer*
> streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG,
> "create-order");
> streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9092");
> streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG,
> "localhost:2181");
> streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG,
> "http://localhost:8081;);
>
> final KStream<String, test> tests = builder.stream(Serdes.String(),
> testSpecificAvroSerde, "test");
>
> tests.map((id, command) -> {
> System.out.println("test id=" + id + " command=" + command);
> command.setId(9);
>
> return new KeyValue<>(UUID.randomUUID().toString(), command);
> })
> .through(Serdes.String(), testSpecificAvroSerde, "test2");
>
>
> *test.avsc*
> {
> "type": "record",
> "namespace": "com.foodpanda.command.avro",
> "name": "test",
> "fields": [{
> "name": "id",
> "type": "int"
> }, {
> "name": "product",
> "type": "string"
> }, {
> "name": "quantity",
> "type": "int"
> }, {
> "name": "price",
> "type": "float"
> }]
> }
>
> --
>
> Nick DeCoursin
> Software Engineer
> foodpanda
>
> Tel | +1 920 450 5434
>
> Mail | n.decour...@foodpanda.com
>
> Skype | nick.foodpanda
>
> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> USt-ID-Nr | DE 283789080
> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>
> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KIP-121 [Discuss]: Add KStream peek method

[Gwen Shapira]

Far better! Thank you!

On Tue, Feb 7, 2017 at 10:19 AM, Steven Schlansker
<sschlans...@opentable.com> wrote:
> Thanks for the feedback.  I improved the javadoc a bit, do you like it better?
>
> /**
>  * Perform an action on each record of {@code KStream}.
>  * This is a stateless record-by-record operation (cf. {@link 
> #process(ProcessorSupplier, String...)}).
>  *
>  * Peek is a non-terminal operation that triggers a side effect (such as 
> logging or statistics collection)
>  * and returns an unchanged stream.
>  *
>  * Note that since this operation is stateless, it may execute multiple 
> times for a single record in failure cases.
>  *
>  * @param action an action to perform on each record
>  * @see #process(ProcessorSupplier, String...)
>  */
> KStream<K, V> peek(final ForeachAction action);
>
> Updated in-place on the PR.
>
>> On Feb 7, 2017, at 2:19 AM, Michael Noll <mich...@confluent.io> wrote:
>>
>> Many thanks for the KIP and the PR, Steven!
>>
>> My opinion, too, is that we should consider including this.
>>
>> One thing that I would like to see clarified is the difference between the
>> proposed peek() and existing functions map() and foreach(), for instance.
>> My understanding (see also the Java 8 links below) is that:
>>
>> - Like `map`, `peek` will return a KStream.  This also means that, unlike
>> `foreach`, `peek` is not a terminal operation.
>> - The main purpose of `peek` is, similar to `foreach`, the *side effects*
>> (such as the metrics counter example in the KIP) -- and, on a related note,
>> also to express your *intent* to achieve such side effects in the first
>> place (which is similar to when to use `foreach` rather than `map`); and
>> typically you should not (must not?) modify the underlying stream itself
>> (unlike `map`, which is supposed to do exactly that).
>>
>> For reference, here are the descriptions of peek, map, foreach in Java 8.
>> I could have also included links to StackOverflow questions where people
>> were confused about when (not) to use peek. ;-)
>>
>> https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#peek-java.util.function.Consumer-
>> https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#map-java.util.function.Function-
>> https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#forEach-java.util.function.Consumer-
>>
>> Best wishes,
>> Michael
>>
>>
>>
>>
>>
>>
>> On Tue, Feb 7, 2017 at 10:37 AM, Damian Guy <damian@gmail.com> wrote:
>>
>>> Hi Steven,
>>> Thanks for the KIP. I think this is a worthy addition to the API.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Tue, 7 Feb 2017 at 09:30 Eno Thereska <eno.there...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I like the proposal, thank you. I have found it frustrating myself not to
>>>> be able to understand simple things, like how many records have been
>>>> currently processed. The peek method would allow those kinds of
>>> diagnostics
>>>> and debugging.
>>>>
>>>> Gwen, it is possible to do this with the existing functionality like map,
>>>> but you'd have to fake the map method. Also, it is not great using map
>>> for
>>>> things it was not intended for. Having an explicit peek makes it clearer
>>> in
>>>> my opinion.
>>>>
>>>> Thanks
>>>> Eno
>>>>
>>>>> On 7 Feb 2017, at 03:20, Gwen Shapira <g...@confluent.io> wrote:
>>>>>
>>>>> I've read the wiki and am unclear about the proposal. Can you provide
>>>>> something like a Javadoc for peek()? What would this method do?
>>>>>
>>>>> Also, forgive me if I'm missing an important point here, but can't I
>>>>> put the println statement in a map()?
>>>>>
>>>>> On Mon, Feb 6, 2017 at 5:48 PM, Matthias J. Sax <matth...@confluent.io
>>>>
>>>> wrote:
>>>>>> Steven,
>>>>>>
>>>>>> Thanks for your KIP. I move this discussion to dev mailing list --
>>> KIPs
>>>>>> need to be discussed there (and can be cc'ed to user list).
>>>>>>
>>>>>> Can you also add the KIP to the table "KIPs under discussion":
>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/con

Re: Format Kafka raw message to avro json format programatically + KafkaUtils

[Gwen Shapira]

You say the data is Avro, but your Spark example specifies the key and
value types as String and uses String decoders.

If the Spark example works fine, you can just run the console consumer
without the formatter and read the strings...

On Mon, Feb 6, 2017 at 7:41 PM, kush batra <kush.ba...@gmail.com> wrote:
> Hi Team,
>
> I am trying to implement below kafka-console-consumer command(works well &
> output the intended json data) functionality in the form of program using
> spark streaming.
>
> kafka-console-consumer.sh --zookeeper host..com:2181,host..com:2181
> --topic mytopic --formatter CustomAvroMessageFormatter --property
> "formatter-schema-file= schema.txt" > /var/tmp/myfile.json&
>
> I am able to read message from above topic using KafkaUtils spark streaming
> programatically as below using spark scala code which works well:
>
> object ConsumeTest {
>
> def main(args: Array[String]) {
>   val sc = new SparkContext("local[*]", "ConsumeKafkaMsg")
>   sc.setLogLevel("ERROR")
>   val ssc = new StreamingContext(sc, Seconds(1))
>
>   //To read from server
>   val kafkaParams = Map("metadata.broker.list" -> "brokername:9092")
>   val topics = List("mytopic").toSet
>
>   val lines = KafkaUtils.createDirectStream[
>String, String, StringDecoder, StringDecoder](ssc, kafkaParams,
> topics).map(_._2)
>
>   lines.print()
>
>   ssc.start()
>   ssc.awaitTermination()
>   }
>
> }
>
> However above program read message in raw format something similar to below:
>
> ��Cߣ�ߕ'윺~�_,��M˶/��Ѯ!眾�Vcusomtername client
> 2X3XX-sasadsad-4673-212c-dsdsadsad
> value
> ,"question"logName
> successstԇ���V
>
> against above command use custom kafka message formatter to convert raw
> format to json format using avro schema. I am unable to find out how to use
> command equivalent to message formatter in my above program which is
> important to achieve.
>
> Below is the probable avro schema(schema.txt) for reference(actually v
> complex what is available to process):
>
> {
>   "type" : "record",
>   "namespace" : "mynamespace",
>   "name" : "myname",
>   "fields" : [{
> "name":"field1",
> "type":{
>   "type":"record",
>   "name":"Eventfield1",
>   "fields":[{.}]
> }]
>   ]
> }
>
> Please help to implement the same.
>
> Regards,
> Kush



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KIP-121 [Discuss]: Add KStream peek method

[Gwen Shapira]

I've read the wiki and am unclear about the proposal. Can you provide
something like a Javadoc for peek()? What would this method do?

Also, forgive me if I'm missing an important point here, but can't I
put the println statement in a map()?

On Mon, Feb 6, 2017 at 5:48 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> Steven,
>
> Thanks for your KIP. I move this discussion to dev mailing list -- KIPs
> need to be discussed there (and can be cc'ed to user list).
>
> Can you also add the KIP to the table "KIPs under discussion":
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion
>
>
> Thanks.
>
>
> -Matthias
>
> On 2/6/17 3:35 PM, Steven Schlansker wrote:
>> Hello users@kafka,
>>
>> I would like to propose a small KIP on the Streams framework
>> that simply adds a KStream#peek implementation.
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-121%3A+Add+KStream+peek+method
>> https://issues.apache.org/jira/browse/KAFKA-4720
>> https://github.com/apache/kafka/pull/2493
>>
>> Please consider my contribution and hopefully you all like it and agree that 
>> it should be merged into 0.10.3 :)
>> If not, be gentle, this is my first KIP!
>>
>> Happy Monday,
>> Steven
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

[Gwen Shapira]

Sounds good :)

On Mon, Feb 6, 2017 at 5:40 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> Gwen,
>
> thanks for your feedback.
>
> I completely agree that KStreamBuilder#merge() is miss placed and should
> belong to KStream. However, I wanted to keep this KIP focus on one thing.
>
> As mentioned in a previous reply, we plan to have at least one more KIP
> to clean up DSL -- this future KIP should include exact this change.
>
>
> -Matthias
>
>
> On 2/6/17 4:26 PM, Gwen Shapira wrote:
>> I like the cleanup a lot :)
>>
>> The cleaner lines between PAPI and DSL are very helpful to beginners
>> who try to make sense of a long list of methods.
>>
>> I noticed that the "merge" method is still part of StreamBuilder. I
>> thought it belongs inside KStream. Merge seems a lot like the SQL
>> "union" operator, so I expect it to be a method of the same object as
>> "join". At least, it isn't immediately clear to me why "join" and
>> "merge" belong in two different levels of the hierarchy. They both
>> transform two (or more) streams into one.
>>
>> Gwen
>>
>> On Fri, Feb 3, 2017 at 3:33 PM, Matthias J. Sax <matth...@confluent.io> 
>> wrote:
>>> Hi All,
>>>
>>> I did prepare a KIP to do some cleanup some of Kafka's Streaming API.
>>>
>>> Please have a look here:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-120%3A+Cleanup+Kafka+Streams+builder+API
>>>
>>> Looking forward to your feedback!
>>>
>>>
>>> -Matthias
>>>
>>
>>
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

[Gwen Shapira]

I like the cleanup a lot :)

The cleaner lines between PAPI and DSL are very helpful to beginners
who try to make sense of a long list of methods.

I noticed that the "merge" method is still part of StreamBuilder. I
thought it belongs inside KStream. Merge seems a lot like the SQL
"union" operator, so I expect it to be a method of the same object as
"join". At least, it isn't immediately clear to me why "join" and
"merge" belong in two different levels of the hierarchy. They both
transform two (or more) streams into one.

Gwen

On Fri, Feb 3, 2017 at 3:33 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> Hi All,
>
> I did prepare a KIP to do some cleanup some of Kafka's Streaming API.
>
> Please have a look here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-120%3A+Cleanup+Kafka+Streams+builder+API
>
> Looking forward to your feedback!
>
>
> -Matthias
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: "End of Batch" event

[Gwen Shapira]

gt;> from these sources every 24 hours. The results will be KTable K.
>>>>
>>>> For day 1:
>>>> S1=[A, B, C]   =>  the result K = [A,B,C]
>>>>
>>>> S2=[D,E,F] =>   K will be [A,B,C,D,E,F]
>>>>
>>>> For day 2:
>>>>
>>>> S1=[A,B] because C is missing I have to remove it from K;  K= [A,B,D,E,F]
>>>> On the other hand, I will process A and B again in case of updates.
>>>>
>>>> In other words, I know how to process existent and new items, I'm not
>>> sure
>>>> how to remove items missing from the latest CSV file.
>>>>
>>>> If I can use Interactive Queries from inside the SourceTask to get a
>>>> snapshot of what currently in K for a specific source S, then I can send
>>>> delete message for the missing items by subtracting latest items in the
>>> CSV
>>>> from the items of that source in K.
>>>>
>>>> Thanks,
>>>>
>>>> On Tue, Jan 31, 2017 at 1:54 PM, Matthias J. Sax <matth...@confluent.io>
>>>> wrote:
>>>>
>>>>> I am not sure if I understand the complete scenario yet.
>>>>>
>>>>>> I need to delete all items from that source that
>>>>>> doesn't exist in the latest CSV file.
>>>>>
>>>>> Cannot follow here. I thought your CSV files provide the data you want
>>>>> to process. But it seems you also have a second source?
>>>>>
>>>>> How does your Streams app compute the items you want to delete? If you
>>>>> have this items in a KTable, you can access them from outside your
>>>>> application using Interactive Queries.
>>>>>
>>>>> Thus, you can monitor the app progress by observing committed offsets,
>>>>> and if finished, you query your KTable to extract the items you want to
>>>>> delete and do the cleanup.
>>>>>
>>>>> Does this make sense?
>>>>>
>>>>> For Interactive Queries see the docs and blog post:
>>>>>
>>>>> http://docs.confluent.io/current/streams/developer-
>>>>> guide.html#interactive-queries
>>>>>
>>>>> https://www.confluent.io/blog/unifying-stream-processing-
>>>>> and-interactive-queries-in-apache-kafka/
>>>>>
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 1/30/17 9:10 PM, Eric Dain wrote:
>>>>>> Thanks Matthias for your reply.
>>>>>>
>>>>>> I'm not trying to stop the application. I'm importing inventory from
>>> CSV
>>>>>> files coming from 3rd party sources. The CSVs are snapshots for each
>>>>>> source's inventory. I need to delete all items from that source that
>>>>>> doesn't exist in the latest CSV file.
>>>>>>
>>>>>> I was thinking of using "End of Batch" message to initiate that
>>> process.
>>>>> I
>>>>>> might need to do the clean-up as part of the Connect code instead, or
>>>>> there
>>>>>> is a better way of doing that?
>>>>>>
>>>>>> Thanks,
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 29, 2017 at 4:37 PM, Matthias J. Sax <
>>> matth...@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> currently, a Kafka Streams application is designed to "run forever"
>>> and
>>>>>>> there is no notion of "End of Batch" -- we have plans to add this
>>>>>>> though... (cf.
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 95%3A+Incremental+Batch+Processing+for+Kafka+Streams)
>>>>>>>
>>>>>>> Thus, right now you need to stop your application manually. You would
>>>>>>> need to observe the application's committed offsets (and lag) using
>>>>>>> bin/kafka-consumer-groups.sh (the application ID is user as group ID)
>>> to
>>>>>>> monitor the app's progress to see when it is done.
>>>>>>>
>>>>>>> Cf.
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/
>>>>>>> Kafka+Streams+Data+%28Re%29Processing+Scenarios
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 1/28/17 1:07 PM, Eric Dain wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm pretty new to Kafka Streams. I am using Kafka Streams to ingest
>>>>> large
>>>>>>>> csv file. I need to run some clean-up code after all records in the
>>>>> file
>>>>>>>> are processed. Is there a way to send "End of Batch" event that is
>>>>>>>> guaranteed to be processed after all records? If not is there
>>>>> alternative
>>>>>>>> solution?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: KAFKA Support

[Gwen Shapira]

Dear Lincu,

You have reached the Apache Kafka mailing list. We are an open source
community of volunteers, helping each other use Kafka to the best of
our abilities.

There are several companies that provide paid support for Kafka -
including, but not limited to Confluent, Cloudera, and Hortonworks.
Feel free to reach out to those companies for details on their support
model and pricing.

Cheers,

Gwen

On Mon, Jan 30, 2017 at 7:56 PM, Lincu Abraham <abrah...@vmware.com> wrote:
> Hi Team,
>
> We are planning to use KAFKA for our applications, can you please provide me 
> the list of support model with cost per year?
>
> Thanks
> Lincu



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Kafka docs for current trunk

[Gwen Shapira]

+1

On Tue, Jan 31, 2017 at 5:57 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> Hi,
>
> I want to collect feedback about the idea to publish docs for current
> trunk version of Apache Kafka.
>
> Currently, docs are only published for official release. Other projects
> also have docs for current SNAPSHOT version. So the question rises, if
> this would be helpful for Kafka community, too.
>
> The idea would be, to update SNAPSHOT docs (web page and JavaDocs) on a
> daily basis based on trunk (of course, fully automated).
>
>
> Looking forward to your feedback.
>
>
> -Matthias
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Fwd: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Gwen Shapira]

gt;> name with the user's provided names. Then from a user's point-view,
>>>>>>>>> if
>>>>>>>>>
>>>>>>>> they
>>>>>>>>
>>>>>>>>> ever want to query a KTable, they have to call materialize() with a
>>>>>>>>>
>>>>>>>> given
>>>>>>
>>>>>>> state store name. This approach has one awkwardness though, that
>>>>>>>>> serdes
>>>>>>>>>
>>>>>>>> and
>>>>>>>>
>>>>>>>>> state store names param are not separated and could be overlapped
>>>>>>>>> (see
>>>>>>>>> detailed comment #2 below).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2. This step does not need to be included in this KIP, but just as a
>>>>>>>>> reference / future work: as we have discussed before, we may enforce
>>>>>>>>> materialize KTable.join resulted KTables as well in the future. If
>>>>>>>>> we
>>>>>>>>>
>>>>>>>> do
>>>>>>
>>>>>>> that, then:
>>>>>>>>>
>>>>>>>>> a) KXX.agg resulted KTables are always materialized;
>>>>>>>>> b) KTable.agg requires the aggregating KTable to always be
>>>>>>>>> materialized
>>>>>>>>> (otherwise we would not know the old value);
>>>>>>>>> c) KTable.join resulted KTables are always materialized, and so are
>>>>>>>>> the
>>>>>>>>> joining KTables to always be materialized.
>>>>>>>>> d) KTable.filter/mapValues resulted KTables materialization depend
>>>>>>>>> on
>>>>>>>>>
>>>>>>>> its
>>>>>>
>>>>>>> parent's materialization;
>>>>>>>>>
>>>>>>>>> By recursive induction all KTables are actually always materialized,
>>>>>>>>>
>>>>>>>> and
>>>>>>
>>>>>>> then the effect of the "materialize()" is just for specifying the
>>>>>>>>> state
>>>>>>>>> store names. In this scenario, we do not need to send Change in
>>>>>>>>> repartition topics within joins any more, but only for repartitions
>>>>>>>>>
>>>>>>>> topics
>>>>>>>>
>>>>>>>>> within aggregations. Instead, we can just send a "tombstone" without
>>>>>>>>>
>>>>>>>> the
>>>>>>
>>>>>>> old value and we do not need to calculate joins twice (one more time
>>>>>>>>>
>>>>>>>> when
>>>>>>
>>>>>>> old value is received).
>>>>>>>>>
>>>>>>>>> 3. I'm wondering if it is worth-while to add a "KStream#toTable()"
>>>>>>>>>
>>>>>>>> function
>>>>>>>>
>>>>>>>>> which is interpreted as a dummy-aggregation where the new value
>>>>>>>>> always
>>>>>>>>> replaces the old value. I have seen a couple of use cases of this,
>>>>>>>>> for
>>>>>>>>> example, users want to read a changelog topic, apply some filters,
>>>>>>>>> and
>>>>>>>>>
>>>>>>>> then
>>>>>>>>
>>>>>>>>> materialize it into a KTable with state stores without creating
>>>>>>>>>
>>>>>>>> duplicated
>>>>>>>>
>>>>>>>>> changelog topics. With materialize() and toTable I'd imagine users
>>>>>>>>> can
>>>>>>>>> specify sth. like:
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> KStream stream = builder.stream("topic1").filter(..);
>>>>>>>>> KTable table = stream.toTable(..);
>>>>>>>>> table.materialize("state1");
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> And the library in this case could set store "state1" 's changelog
>>>>>>>>>
>>>>>>>> topic
>>>>>>
>>>>>>> to
>>>>>>>>
>>>>>>>>> be "topic1", and applying the filter on the fly while (re-)storing
>>>>>>>>> its
>>>>>>>>> state by reading from this topic, instead of creating a second
>>>>>>>>>
>>>>>>>> changelog
>>>>>>
>>>>>>> topic like "appID-state1-changelog" which is a semi-duplicate of
>>>>>>>>>
>>>>>>>> "topic1".
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Detailed:
>>>>>>>>>
>>>>>>>>> 1. I'm +1 with Michael regarding "#toStream"; actually I was
>>>>>>>>> thinking
>>>>>>>>>
>>>>>>>> about
>>>>>>>>
>>>>>>>>> renaming to "#toChangeLog" but after thinking a bit more I think
>>>>>>>>>
>>>>>>>> #toStream
>>>>>>>>
>>>>>>>>> is still better, and we can just mention in the javaDoc that it is
>>>>>>>>> transforming its underlying changelog stream to a normal stream.
>>>>>>>>> 2. As Damian mentioned, there are a few scenarios where the serdes
>>>>>>>>> are
>>>>>>>>> already specified in a previous operation whereas it is not known
>>>>>>>>>
>>>>>>>> before
>>>>>>
>>>>>>> calling materialize, for example:
>>>>>>>>> stream.groupByKey.agg(serde).materialize(serde) v.s.
>>>>>>>>>
>>>>>>>> table.mapValues(/*no
>>>>>>
>>>>>>> serde specified*/).materialize(serde). We need to specify what are
>>>>>>>>> the
>>>>>>>>> handling logic here.
>>>>>>>>> 3. We can remove "KTable#to" call as well, and enforce users to
>>>>>>>>> call "
>>>>>>>>> KTable.toStream.to" to be more clear.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Guozhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jan 18, 2017 at 3:22 AM, Eno Thereska <
>>>>>>>>> eno.there...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I think changing it to `toKStream` would make it absolutely clear
>>>>>>>>>> what
>>>>>>>>>>
>>>>>>>>> we
>>>>>>>>
>>>>>>>>> are converting it to.
>>>>>>>>>>
>>>>>>>>>> I'd say we should probably change the KStreamBuilder methods (but
>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>> in
>>>>>>
>>>>>>> this KIP).
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Eno
>>>>>>>>>>
>>>>>>>>>> On 17 Jan 2017, at 13:59, Michael Noll <mich...@confluent.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Rename toStream() to toKStream() for consistency.
>>>>>>>>>>>>
>>>>>>>>>>> Not sure whether that is really required. We also use
>>>>>>>>>>> `KStreamBuilder#stream()` and `KStreamBuilder#table()`, for
>>>>>>>>>>> example,
>>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>
>>>>>>>>> don't care about the "K" prefix.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jan 17, 2017 at 10:55 AM, Eno Thereska <
>>>>>>>>>>>
>>>>>>>>>> eno.there...@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks Damian, answers inline:
>>>>>>>>>>>>
>>>>>>>>>>>> On 16 Jan 2017, at 17:17, Damian Guy <damian@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP. Some comments:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. I'd probably rename materialized to materialize.
>>>>>>>>>>>>>
>>>>>>>>>>>> Ok.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. I don't think the addition of the new Log compaction mechanism
>>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>>>>
>>>>>>> necessary for this KIP, i.e, the KIP is useful without it. Maybe
>>>>>>>>>>>>>
>>>>>>>>>>>> that
>>>>>>>>
>>>>>>>>> should be a different KIP?
>>>>>>>>>>>>>
>>>>>>>>>>>> Agreed, already removed. Will do a separate KIP for that.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 3. What will happen when you call materialize on KTable that is
>>>>>>>>>>>>>
>>>>>>>>>>>> already
>>>>>>>>>>
>>>>>>>>>>> materialized? Will it create another StateStore (providing the
>>>>>>>>>>>>>
>>>>>>>>>>>> name
>>>>>>
>>>>>>> is
>>>>>>>>
>>>>>>>>> different), throw an Exception?
>>>>>>>>>>>>>
>>>>>>>>>>>> Currently an exception is thrown, but see below.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Have you considered overloading the existing KTable operations
>>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>
>>>>>>> add
>>>>>>>>>>>>
>>>>>>>>>>>>> a state store name? So if a state store name is provided, then
>>>>>>>>>>>>>
>>>>>>>>>>>> materialize
>>>>>>>>>>>>
>>>>>>>>>>>>> a state store? This would be my preferred approach as i don't
>>>>>>>>>>>>>
>>>>>>>>>>>> think
>>>>>>
>>>>>>> materialize is always a valid operation.
>>>>>>>>>>>>>
>>>>>>>>>>>> Ok I can see your point. This will increase the KIP size since
>>>>>>>>>>>> I'll
>>>>>>>>>>>>
>>>>>>>>>>> need
>>>>>>>>
>>>>>>>>> to enumerate all overloaded methods, but it's not a problem.
>>>>>>>>>>>>
>>>>>>>>>>>> 5. The materialize method will need ta value Serde as some
>>>>>>>>>>>>>
>>>>>>>>>>>> operations,
>>>>>>>>
>>>>>>>>> i.e., mapValues, join etc can change the value types
>>>>>>>>>>>>> 6. https://issues.apache.org/jira/browse/KAFKA-4609 - might
>>>>>>>>>>>>> mean
>>>>>>>>>>>>>
>>>>>>>>>>>> that
>>>>>>>>
>>>>>>>>> we
>>>>>>>>>>>>
>>>>>>>>>>>>> always need to materialize the StateStore for KTable-KTable
>>>>>>>>>>>>> joins.
>>>>>>>>>>>>>
>>>>>>>>>>>> If
>>>>>>>>
>>>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>>> is the case, then the KTable Join operators will also need Serde
>>>>>>>>>>>>> information.
>>>>>>>>>>>>>
>>>>>>>>>>>> I'll update the KIP with the serdes.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Eno
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Damian
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, 16 Jan 2017 at 16:44 Eno Thereska <
>>>>>>>>>>>>> eno.there...@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We created "KIP-114: KTable materialization and improved
>>>>>>>>>>>>>>
>>>>>>>>>>>>> semantics"
>>>>>>
>>>>>>> to
>>>>>>>>
>>>>>>>>> solidify the KTable semantics in Kafka Streams:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics
>>>>>>>>>>>>
>>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 114:+KTable+materialization+and+improved+semantics
>>>>>>>>>>>>
>>>>>>>>>>>>> Your feedback is appreciated.
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: kafka streams consumer partition assignment is uneven

[Gwen Shapira]

btw. in case you didn't find out yet (I just discovered this...), you
can get the entire topology by starting the stream, waiting a bit and
then printing "KafkaStreams.toString()" to console.

I found it useful and cool :)


On Tue, Jan 17, 2017 at 3:19 PM, Matthias J. Sax <matth...@confluent.io> wrote:
> Sorry for answering late.
>
> The mapping from partitions to threads also depend on the structure of
> your topology. As you mention that you have a quite complex one, I
> assume that this is the reason for the uneven distribution. I you want
> to dig deeper, it would be helpful to know the structure of your topology.
>
>
> -Matthias
>
> On 1/9/17 12:05 PM, Ara Ebrahimi wrote:
>> I meant I have 7 topics and each has 12 partitions. Considering that I have 
>> 4 streaming threads per node, I was expecting to see each thread process 1 
>> partition from each topics and 7 partitions total per streaming thread. But 
>> that’s not the case. Or perhaps you are saying the number of streaming 
>> threads should follow the total number of partitions across all 7 topics?!
>>
>> Ara.
>>
>>> On Jan 9, 2017, at 11:48 AM, Michael Noll <mich...@confluent.io> wrote:
>>>
>>> What does the processing topology of your Kafka Streams application look
>>> like, and what's the exact topic and partition configuration?  You say you
>>> have 12 partitions in your cluster, presumably across 7 topics -- that
>>> means that most topics have just a single partition.  Depending on your
>>> topology (e.g. if you have defined that single-partition topics A, B, C
>>> must be joined), Kafka Streams is forced to let one of your three Streams
>>> nodes process "more" topics/partitions than the other two nodes.
>>>
>>> -Michael
>>>
>>>
>>>
>>> On Mon, Jan 9, 2017 at 6:52 PM, Ara Ebrahimi <ara.ebrah...@argyledata.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have 3 kafka brokers, each with 4 disks. I have 12 partitions. I have 3
>>>> kafka streams nodes. Each is configured to have 4 streaming threads. My
>>>> topology is quite complex and I have 7 topics and lots of joins and states.
>>>>
>>>> What I have noticed is that each of the 3 kafka streams nodes gets
>>>> configured to process variables number of partitions of a topic. One node
>>>> is assigned to process 2 partitions of topic a and another one gets
>>>> assigned 5. Hence I end up with nonuniform throughput across these nodes.
>>>> One node ends up processing more data than the other.
>>>>
>>>> What’s going on? How can I make sure partitions assignment to kafka
>>>> streams nodes is uniform?
>>>>
>>>> On a similar topic, is there a way to make sure partition assignment to
>>>> disks across kafka brokers is also uniform? Even if I use a round-robin one
>>>> to pin partitions to broker, but there doesn’t seem to be a way to
>>>> uniformly pin partitions to disks. Or maybe I’m missing something here? I
>>>> end up with 2 partitions of topic a on disk 1 and 3 partitions of topic a
>>>> on disk 2. It’s a bit variable. Not totally random, but it’s not uniformly
>>>> distributed either.
>>>>
>>>> Ara.
>>>>
>>>>
>>>>
>>>> 
>>>>
>>>> This message is for the designated recipient only and may contain
>>>> privileged, proprietary, or otherwise confidential information. If you have
>>>> received it in error, please notify the sender immediately and delete the
>>>> original. Any other use of the e-mail by you is prohibited. Thank you in
>>>> advance for your cooperation.
>>>>
>>>> 
>>>>
>>>
>>>
>>>
>>> 
>>>
>>> This message is for the designated recipient only and may contain 
>>> privileged, proprietary, or otherwise confidential information. If you have 
>>> received it in error, please notify the sender immediately and delete the 
>>> original. Any other use of the e-mail by you is prohibited. Thank you in 
>>> advance for your cooperation.
>>>
>>> 
>>
>>
>>
>>
>> 
>>
>> This message is for the designated recipient only and may contain 
>> privileged, proprietary, or otherwise confidential information. If you have 
>> received it in error, please notify the sender immediately and delete the 
>> original. Any other use of the e-mail by you is prohibited. Thank you in 
>> advance for your cooperation.
>>
>> 
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[ANNOUNCE] New committer: Grant Henke

[Gwen Shapira]

The PMC for Apache Kafka has invited Grant Henke to join as a
committer and we are pleased to announce that he has accepted!

Grant contributed 88 patches, 90 code reviews, countless great
comments on discussions, a much-needed cleanup to our protocol and the
on-going and critical work on the Admin protocol. Throughout this, he
displayed great technical judgment, high-quality work and willingness
to contribute where needed to make Apache Kafka awesome.

Thank you for your contributions, Grant :)

-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Interpreting value of JMX metric for partition count

[Gwen Shapira]

The metric is "partition count per broker". You didn't mention how
many brokers and how many replicas you have for each topic, but if you
have 1 replica and 2 brokers, then this looks reasonable. Probably 35
partitions on one broker and 33 on the other or something similar. I'd
recommend getting at least 3 brokers and having at least 3 replicas on
each topic though - for availability.

On Tue, Jan 10, 2017 at 7:25 PM, Abhishek Agrawal
<abhishek.agrawal.1...@gmail.com> wrote:
> Hello Kafka Users,
>
>My kafka cluster has two topics: one with 50 partitions, another with 18
> partitions.
>
> The JMX bean *kafka.server:name=PartitionCount,type=ReplicaManager*, gives
> value as 35 when I try to probe using JMXTerm
>
> $>get -b kafka.server:name=PartitionCount,type=ReplicaManager Value
> #mbean = kafka.server:name=PartitionCount,type=ReplicaManager:
> Value = 35;
>
>
> Can someone help me understand if this value is supposed to be 'average
> partition count per topic'  or 'total partition count for all topics'?
>
> I want to have separate JMX metric for partition count of each topic. Can
> someone point me to configuration examples where this has been achieved?
>
> Regards,
> Abhishek



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: kafka-connect log4j

[Gwen Shapira]

Is your goal to simply log connect to file rather than to the console?
In this case your configuration is almost right. Just change the first
line in connect-log4j.properties to:

log4j.rootLogger=INFO, stdout, connectAppender

and then add the lines you have in your email.

Or you can get rid of stdout appender completely if you prefer.

You may find the log4j primer useful:
https://logging.apache.org/log4j/1.2/manual.html

On Tue, Jan 10, 2017 at 7:42 AM, Berryman, Eric <berry...@frib.msu.edu> wrote:
> Hello!
>
> Is there a log4j.appender.connectAppender?
>
> I noticed there is a log4j.appender.kafkaAppender.
> I was hoping to setup the connect-log4j.properties like kafka's.
>
> log4j.appender.connectAppender=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.connectAppender.DatePattern='.'-MM-dd-HH
> log4j.appender.connectAppender.File=${kafka.logs.dir}/connect.log
> log4j.appender.connectAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.connectAppender.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
>
> Thank you!
> Eric



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Json to JDBC using Kafka JDBC connector Sink

[Gwen Shapira]

Ewen: I think he was looking for exactly what you were guessing he
doesn't: "My goal is to pipe that json document in a postgres table
that has two columns: id and json."

Postgres has some nice built-in functions that make this actually
useful and not as nuts as it may appear.

As Ewen mentioned, our current JDBC sink doesn't do it. But it sounds
like a fairly trivial modification (especially if you use
auto-increment, so you don't even need to worry about IDs). You'll
want to use a String converter to make sure we are not parsing the
JSON.







On Tue, Jan 10, 2017 at 12:40 AM, Ewen Cheslack-Postava
<e...@confluent.io> wrote:
> Anything with a table structure is probably not going to handle schemaless
> data (i.e. JSON) very well without some extra help -- tables usually expect
> schemas and JSON doesn't have a schema. As it stands today, the JDBC sink
> connector will probably not handle your use case.
>
> To send schemaless data into a schema-based system, you'd probably need to
> impose/extract a schema. An upcoming feature called Single Message
> Transforms
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect
> could
> potentially help do this (in a generic way that doesn't depend on the
> Connector being used).
>
> The only alternative would be to update the JDBC sink to handle JSON data
> directly. Some databases might handle this if the entire record were
> converted to a JSON-type field (i.e. a single-column table), but I'm
> guessing you are looking for output that's a bit more structured than that.
>
> -Ewen
>
> On Mon, Jan 9, 2017 at 4:14 PM, Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
>
>> Hi,
>>
>> I’m wondering if the following is feasible…
>> I have a json document with pretty much 0 schema. The only thing I know for
>> sure is that it’s a json document.
>> My goal is to pipe that json document in a postgres table that has two
>> columns: id and json. The id column is basically topic+partition+offset (to
>> guarantee idempotence on upserts), and the json column is basically the
>> json document
>>
>> Is that feasible using the out of the box JDBC connector? I didn’t see any
>> support for “json type” fields
>>
>> Thanks,
>> Stephane
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Kafka SinkConnector for K-V store and Keys

[Gwen Shapira]

This may be a bit use-case dependent, but I think simply using the key
from the Kafka record as the KV key is a good start.
Another option is to use topic-partition-offset as the key. This has
the benefit of removing duplicates, but it also means that keys are no
longer meaningful for applications querying the KV store.

The nice thing is that now that we are adding transformations of
Connect, you can build one solution and users who don't like it can
use transformations to modify the key.

Gwen

On Tue, Jan 10, 2017 at 8:02 PM, Nilkanth Patel
<nilkanth.hpa...@gmail.com> wrote:
> Hello,
>
> I am writing a kafka sink connector for my product that is distributed
> table (Underneath a distribute K-V store where cluster of nodes are holding
> a different partitions/buckets of a table. This is based on Hash
> partitioned on Keys).
>
> When i write a SinkTask, I get the SinkRecord that contains both key and
> value.
>
> Can some one suggest/outline the general guidelines for keys to be used
> with K-V store from the SinkRecord.
>
> What should be the key for external K-V store to be used to store a records
> from kafka topics to external K-V store.
>
> Thanks,
> Nilkanth.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] Vote for KIP-101 - Leader Epochs

[Gwen Shapira]

+1 - thanks for tackling those old and painful bugs!

On Wed, Jan 4, 2017 at 9:24 AM, Ben Stopford <b...@confluent.io> wrote:
> Hi All
>
> We’re having some problems with this thread being subsumed by the [Discuss] 
> thread. Hopefully this one will appear distinct. If you see more than one, 
> please use this one.
>
> KIP-101 should now be ready for a vote. As a reminder the KIP proposes a 
> change to the replication protocol to remove the potential for replicas to 
> diverge.
>
> The KIP can be found here:  
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation
>  
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation>
>
> Please let us know your vote.
>
> B
>
>
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.1.1 RC1

[Gwen Shapira]

Thanks for everyone who voted. Having community vote of confidence in
bugfix releases is hugely important, especially as we are trying to
support more in-flight versions :)

On Wed, Dec 21, 2016 at 8:49 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> Hi Guozhang,
>
> Thanks for driving this release as it includes a few important fixes. I
> belatedly tested Scala 2.12 binaries using the quickstart,  +1
> (non-binding) from me.
>
> Ismael
>
> On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second, and hopefully the last candidate for the release of
>> Apache Kafka 0.10.1.1 before the break. This is a bug fix release and it
>> includes fixes and improvements from 30 JIRAs. See the release notes for
>> more details:
>>
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Tuesday, 20 December, 8pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> NOTE the artifacts include the ones built from Scala 2.12.1 and Java8,
>> which are treated a pre-alpha artifacts for the Scala community to try and
>> test it out:
>>
>> https://repository.apache.org/content/groups/staging/org/apa
>> che/kafka/kafka_2.12/0.10.1.1/
>>
>> We will formally add the scala 2.12 support in future minor releases.
>>
>>
>> * Javadoc:
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> c3638376708ee6c02dfe4e57747acae0126fa6e7
>>
>>
>> Thanks,
>> Guozhang
>>
>> --
>> -- Guozhang
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Kafka Summit 2017

[Gwen Shapira]

Hi Kafka Fans,

Just in case you didn't hear / read:

Last year was the first Kafka Summit and it was quite successful. So
we are doing two this year: May in NYC and Aug in SF.

You can read more details here: https://kafka-summit.org/ and you can
use the code "kafkcom17" for a $50 community discount. Early bird
registration for NYC is ending soon :)

I also encourage you to share your Kafka experience with the wider
community - got a cool use-case? Kafka tips and tricks? Amazing
streams application? The best PHP client?  Call for paper is open and
the conference committee is waiting for your abstracts :)

-- 
Gwen Shapira

Re: [VOTE] 0.10.1.1 RC1

[Gwen Shapira]

+1 (binding)

Validated signatures
Ran tests
Built from source distro
Tested binaries using the quickstart guide

Gwen

On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang <wangg...@gmail.com> wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second, and hopefully the last candidate for the release of
> Apache Kafka 0.10.1.1 before the break. This is a bug fix release and it
> includes fixes and improvements from 30 JIRAs. See the release notes for
> more details:
>
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, 20 December, 8pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> NOTE the artifacts include the ones built from Scala 2.12.1 and Java8,
> which are treated a pre-alpha artifacts for the Scala community to try and
> test it out:
>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/0.10.1.1/
>
> We will formally add the scala 2.12 support in future minor releases.
>
>
> * Javadoc:
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=c3638376708ee6c02dfe4e57747acae0126fa6e7
>
>
> Thanks,
> Guozhang
>
> --
> -- Guozhang



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Website Update, Part 2

[Gwen Shapira]

Hi,

Since we are breaking down the docs, we can no longer use ctrl-f to find
where to find specific things we are looking for... maybe it is time to add
a site search bar? I think google has something we can embed.

On Tue, Dec 13, 2016 at 6:12 PM, Guozhang Wang <wangg...@gmail.com> wrote:

> Folks,
>
> We are continuing to improve our website, and one of it is to break the
> single gigantic "documentation" page:
>
> https://kafka.apache.org/documentation/
>
> into sub-spaces and sub-pages for better visibility. As the first step of
> this effort, we will be gradually extract each section of this page into a
> separate page and then grow each one of them in their own sub-space.
>
> As of now, we have extract Streams section out of documentation as
>
> https://kafka.apache.org/documentation/streams
>
> while all the existing hashtags are preserved and re-directed via JS (many
> thanks to Derrick!) so that we do not loose any SEO. At the same time I
> have updated the "website doc contributions" wiki a bit with guidance on
> locally displaying and debugging doc changes with this refactoring:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+
> Documentation+Changes
>
>
> We are trying to do the same for Connect, Ops, Configs, APIs etc in the
> near future. Any comments, improvements, and contributions are welcome and
> encouraged.
>
>
> --
> -- Guozhang
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: Best approach to frequently restarting consumer process

[Gwen Shapira]

Can you clarify what you mean by "restart"? If you call
consumer.close() and consumer.subscribe() you will definitely trigger
a rebalance.

It doesn't matter if its "same consumer knocking", we already
rebalance when you call consumer.close().

Since we want both consumer.close() and consumer.subscribe() to cause
rebalance immediately (and not wait for heartbeat), I don't think
we'll be changing their behavior.

Depending on why consumers need to restart, I'm wondering if you can
restart other threads in your application but keep the consumer up and
running to avoid the rebalances.

On Tue, Dec 6, 2016 at 7:18 AM, Harald Kirsch <harald.kir...@raytion.com> wrote:
> We have consumer processes which need to restart frequently, say, every 5
> minutes. We have 10 of them so we are facing two restarts every minute on
> average.
>
> 1) It seems that nearly every time a consumer restarts  the group is
> rebalanced. Even if the restart takes less than the heartbeat interval.
>
> 2) My guess is that the group manager just cannot know that the same
> consumer is knocking at the door again.
>
> Are my suspicions (1) and (2) correct? Is there a chance to fix this such
> that a restart within the heartbeat interval does not lead to a re-balance?
> Would a well defined client.id help?
>
> Regards
> Harald
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Some general questions...

[Gwen Shapira]

Yeah, that's a good point - Kafka on Windows has few quirks because
most core Kafka developers are not windows experts and the big
deployments are almost all on Linux.

We discovered that most our .NET users actually run Kafka on Linux.
Turns out that installing few VMs with Linux and running Kafka is
fairly easy, but a programming language is not something you can
easily change.
Fortunately, thanks to librdkafka we can implement a good .NET client
without worrying about windows internals :)

On Tue, Dec 6, 2016 at 7:07 AM, Harald Kirsch <harald.kir...@raytion.com> wrote:
> This sounds like you might want to run the Kafka broker on Windows. Have a
> look at https://issues.apache.org/jira/browse/KAFKA-1194 for possible issues
> with regard to log cleaning.
>
> Regards,
> Harald.
>
>
>
> On 06.12.2016 00:50, Doyle, Keith wrote:
>>
>>
>>
>> We’re beginning to make use of Kafka, and it is encouraging.  But there
>> are a couple of questions I’ve had a hard time finding answers for.
>>
>>
>>
>> We’re using the rdkafka-dotnet client on the consumer side and it’s
>> straightforward as far as it goes.  However, documentation seems to be
>> scant—the Wiki points to a FAQ which has, like, two questions neither of
>> which are the questions we have.   And I can’t find a mailing list,
>> forum, blog, or other community where questions can be asked.  I found
>> some indication in the Git repository that there may be some API docs,
>> but it’s not at all clear exactly where those are.
>>
>>
>>
>> So I’m posting that question here because I can’t find anywhere else
>> that might be even remotely relevant to post it—where can I find out
>> more info about rdkafka and particularly rdkafka-dotnet, and some way to
>> ask questions that aren’t answered in the documentation?
>>
>>
>>
>> And second, my current question about rdkafka-dotnet, is the example
>> consumers both seem to read an entire message into memory.   We don’t
>> want to presume any particular message size, and may not want to cache
>> the entire message in memory while processing it.   Is there an
>> interface where we can consume messages via a stream, so that we can
>> read chunks of a message and process them based on some kind of batch
>> size that will allow us better control over memory usage?
>>
>>
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Some general questions...

[Gwen Shapira]

Confluent already supports a C client (the famous librdkafka). We are
indeed going to support a C# client, based on rdkafka-dotnet - we are
currently busy modifying the API a bit to fit our taste better :)



On Mon, Dec 5, 2016 at 6:34 PM, Tauzell, Dave
<dave.tauz...@surescripts.com> wrote:
> I don't know if any API to stream a message.  I don't suggest putting lots of 
> large messages onto Kafka.
>
> As far as documentation I hear that confluent is going to support a C and C# 
> client so you could try asking questions on the confluent mailing list.
>
> Dave
>
> On Dec 5, 2016, at 17:51, Doyle, Keith 
> <keith.do...@greenwayhealth.com<mailto:keith.do...@greenwayhealth.com>> wrote:
>
>
> We're beginning to make use of Kafka, and it is encouraging.  But there are a 
> couple of questions I've had a hard time finding answers for.
>
> We're using the rdkafka-dotnet client on the consumer side and it's 
> straightforward as far as it goes.  However, documentation seems to be 
> scant-the Wiki points to a FAQ which has, like, two questions neither of 
> which are the questions we have.   And I can't find a mailing list, forum, 
> blog, or other community where questions can be asked.  I found some 
> indication in the Git repository that there may be some API docs, but it's 
> not at all clear exactly where those are.
>
> So I'm posting that question here because I can't find anywhere else that 
> might be even remotely relevant to post it-where can I find out more info 
> about rdkafka and particularly rdkafka-dotnet, and some way to ask questions 
> that aren't answered in the documentation?
>
> And second, my current question about rdkafka-dotnet, is the example 
> consumers both seem to read an entire message into memory.   We don't want to 
> presume any particular message size, and may not want to cache the entire 
> message in memory while processing it.   Is there an interface where we can 
> consume messages via a stream, so that we can read chunks of a message and 
> process them based on some kind of batch size that will allow us better 
> control over memory usage?
>
>
> Thanks,
>
>
> --
>
> [Greenway_Health_PNG_large_NO_tag]
>
> Keith Doyle  |  Senior Software Engineer
> Greenway Health  |  4301 W. Boy Scout Blvd., Suite 800, Tampa, FL 33607
> (702) 256-9911 office  |  GreenwayHealth.com<http://www.greenwayhealth.com/>
> [cid:image003.png@01D04086.868CBCB0]<https://www.linkedin.com/company/greenway-health>[cid:image004.png@01D04086.868CBCB0]<https://www.facebook.com/GreenwayHealthLLC>[cid:image005.png@01D04086.868CBCB0]<https://twitter.com/greenway>
>
> NOTICE: This e-mail message and all attachments transmitted with it may 
> contain legally privileged and confidential information intended solely for 
> the use of the addressee. If the reader of this message is not the intended 
> recipient, you are hereby notified that any reading, dissemination, 
> distribution, copying, or other use of this message or its attachments is 
> strictly prohibited. If you have received this message in error, please 
> notify the sender immediately by electronic mail and delete this message and 
> all copies and backups thereof. Thank you. Greenway Health.
> This e-mail and any files transmitted with it are confidential, may contain 
> sensitive information, and are intended solely for the use of the individual 
> or entity to whom they are addressed. If you have received this e-mail in 
> error, please notify the sender by reply e-mail immediately and destroy all 
> copies of the e-mail and any attachments.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Disadvantages of Upgrading Kafka server without upgrading client libraries?

[Gwen Shapira]

Most people upgrade clients to enjoy new client features, fix bugs or
improve performance. If none of these apply, no need to upgrade.

Since you are upgrading to 0.10.1.0, read the upgrade docs closely -
there are specific server settings regarding the message format that
you need to configure a certain way if the clients are not upgraded.

Gwen

On Tue, Nov 29, 2016 at 6:30 AM, Tim Visher <tim.vis...@gmail.com> wrote:
> Hi Everyone,
>
> I have an install of Kafka 0.8.2.1 which I'm upgrading to 0.10.1.0. I see
> that Kafka 0.10.1.0 should be backwards compatible with client libraries
> written for older versions but that newer client libraries are only
> compatible with their version and up.
>
> My question is what disadvantages would there be to never upgrading the
> clients? I'm mainly asking because it would be advantageous to save some
> time here with a little technical debt if the costs weren't too high. If
> there are major issues then I can take on the client upgrade as well.
>
> Thanks in advance!
>
> --
>
> In Christ,
>
> Timmy V.
>
> http://blog.twonegatives.com/
> http://five.sentenc.es/ -- Spend less time on mail



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Kafka Clients Survey

[Gwen Shapira]

Hey Kafka Community,

I'm trying to take a pulse on the current state of the Kafka clients ecosystem.
Which languages are most popular in our community? What does the
community value in clients?

You can help me out by filling in the survey:
https://goo.gl/forms/cZg1CJyf1PuqivTg2

I will lock the survey and publish results by Jan 15.

Gwen

Re: Kafka - Roadmap - 2017

[Gwen Shapira]

Our community does not have a roadmap as such. But there are few
initiatives that are currently being worked on and are likely to be
included in 2017 releases:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

If members of the community will have additional improvements (or will
lose interest in some of the existing proposals) the roadmap for 2017
will change.

Per the new time-based-release policy, we anticipate upcoming releases
to happen on February 2017, June 2017 and October 2017 (if there will
be community interest and volunteer release managers).

Gwen



On Fri, Nov 25, 2016 at 11:51 AM, Costa Tsirbas <costa.tsir...@gmail.com> wrote:
> Hello,
>
> I was wondering if anyone from kafka (users/dev or commits) can send or
> direct me to the kafka software roadmap for 2017 and onwards.
>
> Let me know if I need to subscribe or if this is available somwhere on your
> website.
>
> Greatly appreciated!
>
> Costa Tsirbas
> 514.443.1439



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Oversized Message 40k

[Gwen Shapira]

This has been our experience as well. I think the largest we've seen
in production is 50MB.

If you have performance numbers you can share for the large messages,
I think we'll all appreciate :)

On Tue, Nov 22, 2016 at 1:04 PM, Tauzell, Dave
<dave.tauz...@surescripts.com> wrote:
> I ran tests with a mix of messages, some as large as 20MB.   These large 
> messages do slow down processing, but it still works.
>
> -Dave
>
> -Original Message-
> From: h...@confluent.io [mailto:h...@confluent.io]
> Sent: Tuesday, November 22, 2016 1:41 PM
> To: users@kafka.apache.org
> Subject: Re: Oversized Message 40k
>
> The default config handles messages up to 1MB so you should be fine.
>
> -hans
>
>> On Nov 22, 2016, at 4:00 AM, Felipe Santos <felip...@gmail.com> wrote:
>>
>> I read on documentation that kafka is not optimized for big messages,
>> what is considered a big message?
>>
>> For us the messages will be on average from 20k ~ 40k? Is this a real
>> problem?
>>
>> Thanks
>> --
>> Felipe Santos
> This e-mail and any files transmitted with it are confidential, may contain 
> sensitive information, and are intended solely for the use of the individual 
> or entity to whom they are addressed. If you have received this e-mail in 
> error, please notify the sender by reply e-mail immediately and destroy all 
> copies of the e-mail and any attachments.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: New Kafka user - Need some pointers to get started

[Gwen Shapira]

Pro tip: You can't use new (0.10) clients with old (0.7) brokers. They
are incompatible. You will need to easier upgrade your brokers or
downgrade your clients.

On Thu, Nov 17, 2016 at 7:38 AM, Raghav <raghavas...@gmail.com> wrote:
> Hello All
>
> I am a new Kafka users, and I am trying to create a publisher that produces
> a bunch of messages from a store and pushes it to broker.
>
> I am looking for best practices in writing a Kafka Producer. I looked at
> sample examples, and got confused because different versions of Kafka have
> used different API endpoints. I am on 0.10. Any best practices around which
> client version to use if the broker is still old (0.7) ?
>
> Can you point me to the some pointers which lists best practices about
> setting producer configs ? I am sure this is a very common questions, and
> should be available as a sticky note, but I could not find it.
>
> Thanks.
>
> --
> Raghav



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: --group flag for console consumer

[Gwen Shapira]

Thank you, Vahid!

On Wed, Nov 16, 2016 at 1:53 PM, Vahid S Hashemian
<vahidhashem...@us.ibm.com> wrote:
> I'll open a JIRA.
>
> Andrew, let me know if you want to take over the implementation.
> Otherwise, I'd be happy to work on it.
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Gwen Shapira <g...@confluent.io>
> To: Users <users@kafka.apache.org>
> Date:   11/16/2016 01:23 PM
> Subject:Re: --group flag for console consumer
>
>
>
> Makes sense to me. Do you want to contribute a pull request?
>
> On Wed, Nov 16, 2016 at 11:33 AM, Andrew Pennebaker
> <andrew.penneba...@gmail.com> wrote:
>> Could the kafka-console-consumer shell script please get a --group
> 
>> flag?
>>
>> Loading configs from properties files is helpful, but a direct --group
> flag
>> would be a simpler user interface for this common use case.
>>
>>
>> --
>> Cheers,
>> Andrew
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
>
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: --group flag for console consumer

[Gwen Shapira]

Makes sense to me. Do you want to contribute a pull request?

On Wed, Nov 16, 2016 at 11:33 AM, Andrew Pennebaker
<andrew.penneba...@gmail.com> wrote:
> Could the kafka-console-consumer shell script please get a --group 
> flag?
>
> Loading configs from properties files is helpful, but a direct --group flag
> would be a simpler user interface for this common use case.
>
>
> --
> Cheers,
> Andrew



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Kafka upgrade from 0.8.0 to 0.10.0.0

[Gwen Shapira]

What version of ZK are you using?

If you are using ZK 3.3, I'd recommend upgrading it before you upgrade Kafka.
If you are on ZK 3.4.5 or 3.4.6, you are good.

On Tue, Nov 15, 2016 at 8:33 AM, Amit Tank <amittankopensou...@gmail.com> wrote:
> Hi Johnny,
>
> I haven't tried it yet so I am not 100% sure which one should be upgraded
> in what order but the way I think is - you can upgrade brokers when you are
> not upgrading ZK too.  so if that holds true then the sequence do not
> matter.
>  Again I can be completely wrong here.May be some one else from the group
> can explain this in more detail.
>
> Thank you,
> Amit
>
> On Tuesday, November 15, 2016, ZHU Hua B <hua.b@alcatel-lucent.com>
> wrote:
>
>> Hi All,
>>
>>
>> Anyone knows if zookeeper upgrade should be executed before broker
>> upgrade? Thanks!
>>
>>
>>
>>
>>
>>
>> Best Regards
>>
>> Johnny
>>
>>
>> -Original Message-
>> From: ZHU Hua B
>> Sent: 2016年11月14日 15:07
>> To: users@kafka.apache.org <javascript:;>
>> Subject: RE: Kafka upgrade from 0.8.0 to 0.10.0.0
>>
>> Hi,
>>
>>
>> Thanks for your answer!
>>
>> Another question, if zookeeper upgrade should be executed before broker
>> upgrade? Thanks!
>>
>>
>>
>>
>>
>>
>> Best Regards
>>
>> Johnny
>>
>>
>> -Original Message-
>> From: Amit Tank [mailto:amittankopensou...@gmail.com <javascript:;>]
>> Sent: 2016年11月12日 1:49
>> To: users@kafka.apache.org <javascript:;>
>> Subject: Re: Kafka upgrade from 0.8.0 to 0.10.0.0
>>
>> Hi,
>>
>> I am not an expert but from what read and understood from HDFS
>> documentation, If you want to upgrade zookeeper, you can not avoid downtime.
>>
>> Thank you,
>> Amit
>>
>> On Thursday, November 10, 2016, ZHU Hua B <hua.b@alcatel-lucent.com
>> <javascript:;>>
>> wrote:
>>
>> > Hi,
>> >
>> >
>> > For a rolling upgrade, Kafka suggest upgrade the brokers one at a time
>> > (shut down the broker, update the code, and restart it) to avoid
>> > downtime during the upgrade.
>> > Usually, there is one zookeeper point to some brokers in a Kafka
>> > cluster, if the zookeeper should be upgraded also? If so, how to avoid
>> > downtime during zookeeper upgrade? Thanks!
>> >
>> >
>> >
>> >
>> >
>> >
>> > Best Regards
>> >
>> > Johnny
>> >
>> >
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Leader became -1 and no ISR for all topics/patitions

[Gwen Shapira]

Mind sharing how you got to this situation?

Did you restart brokers? did the replicas drop off the ISR one by one
or all together? Do you have unclear leader election enabled? What are
the errors you are seeing in the logs?

Once none of the partitions are available and lacking any additional
information, restarting the controller is probably the best choice,
followed by a complete restart of everything... but your cluster
really shouldn't get there.


On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung
<karthi.ssisams...@gmail.com> wrote:
> Hi,
>
> I am facing similar issue where couple of my partitions have a Leader as -1
> and No ISR. I tried to use kafka-reassign-partitions.sh and
> kafka-preferred-replica-election.sh tool and this did not help as the ISR
> was empty. Other users who faced this issue suggesting broker restart.
>
> Does any one has other solutions of gracefully handling this instead of
> restarting the brokers?



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: consumer client pause/resume/rebalance

[Gwen Shapira]

Yeah, we should mention that in the javadoc. Want to send a PR?

I like the "worse is better" philosophy - although we occasionally
choose complex implementation for simpler APIs (especially when it
comes to admin configurations).

On Tue, Nov 8, 2016 at 2:34 AM, Paul Mackles <pmack...@adobe.com> wrote:
> Hi Gwen - Makes sense. The way you explain it actually reminds me a little of 
> the "worse is better" philosophy: https://www.jwz.org/doc/worse-is-better.html
>
>
> Perhaps a mention in the javadoc for pause() and/or ConsumerRebalanceListener 
> would be sufficient.
>
> ____
> From: Gwen Shapira <g...@confluent.io>
> Sent: Monday, November 07, 2016 3:34:39 PM
> To: Users
> Subject: Re: consumer client pause/resume/rebalance
>
> I think the current behavior is fairly reasonable. Following a
> rebalance the entire state of the consumer changes - you may get an
> entirely new set of partitions. A common use-case for pause is to
> allow a consumer to keep polling and avoid getting new events while it
> is retrying to process existing events - well, following a rebalance,
> it is possible that another consumer owns the partition, is already
> re-processing these events and the entire state needs to be reset.
>
> I usually recommend developers to treat rebalance as a restart (since
> you are getting a whole new set of partitions) and just follow
> whatever process you'd follow to set up after a restart. Since pauses
> don't survive restarts, I wouldn't expect them to survive a rebalance
> either.
>
> I hope this helps explain the behavior?
>
> On Mon, Nov 7, 2016 at 9:53 AM, Paul Mackles <pmack...@adobe.com> wrote:
>> Using the  v0.9.0.1 consumer API, I recently learned that paused partitions 
>> can unexpectedly become become unpaused during a rebalance. I also found an 
>> old thread from the mailing list which corroborates this behavior:
>>
>>
>> http://grokbase.com/t/kafka/users/161wgzckze/new-consumer-pause-reset-behaviour
>>
>>
>> <http://grokbase.com/t/kafka/users/161wgzckze/new-consumer-pause-reset-behaviour>While
>>  I can maintain the partition state myself, it seems like it would be a lot 
>> easier if this were either handled internally by the consumer API (i.e. 
>> pause the partitions that were previously paused before resuming) and/or 
>> make the partition state available to the RebalanceListener.
>>
>>
>> I did not find any existing tickets in JIRA related to this so I am 
>> wondering if this is a valid bug/enhancement or if someone found a decent 
>> workaround. All of the consumer API examples that I have found do not appear 
>> to handle this scenario.
>>
>>
>> Here is the code snippet from he client I have been working on:
>>
>>
>> consumer.pause(consumer.assignment().toArray(EMPTYTPARRAY));
>>
>> while (!isWritable()) {
>>   // WARNING: if there is a rebalance, this call may return some records!!!
>>   consumer.poll(0);
>>   Uninterruptibles.sleepUninterruptibly(pauseWait, TimeUnit.MILLISECONDS);
>> }
>>
>> consumer.resume(consumer.assignment().toArray(EMPTYTPARRAY));
>>
>>
>> Thanks,
>>
>> Paul
>>
>>
>>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: consumer client pause/resume/rebalance

[Gwen Shapira]

I think the current behavior is fairly reasonable. Following a
rebalance the entire state of the consumer changes - you may get an
entirely new set of partitions. A common use-case for pause is to
allow a consumer to keep polling and avoid getting new events while it
is retrying to process existing events - well, following a rebalance,
it is possible that another consumer owns the partition, is already
re-processing these events and the entire state needs to be reset.

I usually recommend developers to treat rebalance as a restart (since
you are getting a whole new set of partitions) and just follow
whatever process you'd follow to set up after a restart. Since pauses
don't survive restarts, I wouldn't expect them to survive a rebalance
either.

I hope this helps explain the behavior?

On Mon, Nov 7, 2016 at 9:53 AM, Paul Mackles <pmack...@adobe.com> wrote:
> Using the  v0.9.0.1 consumer API, I recently learned that paused partitions 
> can unexpectedly become become unpaused during a rebalance. I also found an 
> old thread from the mailing list which corroborates this behavior:
>
>
> http://grokbase.com/t/kafka/users/161wgzckze/new-consumer-pause-reset-behaviour
>
>
> <http://grokbase.com/t/kafka/users/161wgzckze/new-consumer-pause-reset-behaviour>While
>  I can maintain the partition state myself, it seems like it would be a lot 
> easier if this were either handled internally by the consumer API (i.e. pause 
> the partitions that were previously paused before resuming) and/or make the 
> partition state available to the RebalanceListener.
>
>
> I did not find any existing tickets in JIRA related to this so I am wondering 
> if this is a valid bug/enhancement or if someone found a decent workaround. 
> All of the consumer API examples that I have found do not appear to handle 
> this scenario.
>
>
> Here is the code snippet from he client I have been working on:
>
>
> consumer.pause(consumer.assignment().toArray(EMPTYTPARRAY));
>
> while (!isWritable()) {
>   // WARNING: if there is a rebalance, this call may return some records!!!
>   consumer.poll(0);
>   Uninterruptibles.sleepUninterruptibly(pauseWait, TimeUnit.MILLISECONDS);
> }
>
> consumer.resume(consumer.assignment().toArray(EMPTYTPARRAY));
>
>
> Thanks,
>
> Paul
>
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: HDFS Connector Compression?

[Gwen Shapira]

Documentation patches are welcome :)

On Mon, Nov 7, 2016 at 9:06 AM, Henry Kim <henry@resonate.com> wrote:
> Nevermind, it seems snappy is supported by default for ParquetFormat. It was 
> not obvious with the fixed extension.
>
>
> private final static String EXTENSION = ".parquet";
>
> final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
> CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;
>
> final ParquetWriter writer =
> new AvroParquetWriter<>(path, avroSchema, compressionCodecName, 
> blockSize, pageSize);
>
>
>
> 
> From: Henry Kim
> Sent: Wednesday, November 2, 2016 2:46:27 PM
> To: users@kafka.apache.org
> Subject: HDFS Connector Compression?
>
>
> Is it possible to add compression to the HDFS Connector out of the box? Or 
> does it require code change?
>
>
> Thanks
>
> Henry Kim



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Added to Wiki please

[Gwen Shapira]

Hi Kenny,

First, thank you for letting the community know about your valuable service.
Second, the wiki you pointed to is for companies using Kafka, not for
vendors selling Kafka. We are trying to avoid commercializing the
community Wiki.  Notice that Heroku, Cloudera, Hortonworks, and
Confluent are all missing from the wiki.

If you have customers who are interested in being listed, we will
gladly include them - since they are using Kafka through your service.

Hope this clarifies the use of the Powered-By page.

Gwen

On Wed, Nov 2, 2016 at 12:11 PM, Kenny Gorman <ke...@eventador.io> wrote:
> Per the wiki, I am emailing the list for this. Can you please add us to 
> https://cwiki.apache.org/confluence/display/KAFKA/Powered+By?
>
> Eventador.io (https://www.eventador.io/) is a whole stack Kafka as-a-service 
> company. We enable developers to quickly create and painlessly manage 
> real-time data pipelines on Apache Kafka.
>
> Thx!!
> Kenny Gorman
> Founder
> www.eventador.io



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Character "/" and "\" not allowed in topic name

[Gwen Shapira]

We use topic names to generate directories on the file system, "/" is
not a valid character in directories (or if it is, it still makes
things like "ls" more complex), so we decided to simply do without it.



On Fri, Nov 4, 2016 at 8:46 AM, Paolo Patierno <ppatie...@live.com> wrote:
> Hi all,
>
>
> why the "/" character isn't allowed in a topic name ? If I try to use it I 
> receive an error on creation.
>
>
> If I try to use the "\" character ... no error on creation but this character 
> is omitted from the name. Why ?
>
>
> I.e. create "this\topic" ... it creates "thistopic"
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

[Gwen Shapira]

Congratulations! well deserved :)

On Mon, Oct 31, 2016 at 10:35 AM, Joel Koshy <jjkosh...@gmail.com> wrote:

> The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a
> committer and we are pleased to announce that he has accepted!
>
> Becket has made significant contributions to Kafka over the last two years.
> He has been deeply involved in a broad range of KIP discussions and has
> contributed several major features to the project. He recently completed
> the implementation of a series of improvements (KIP-31, KIP-32, KIP-33) to
> Kafka’s message format that address a number of long-standing issues such
> as avoiding server-side re-compression, better accuracy for time-based log
> retention, log roll and time-based indexing of messages.
>
> Congratulations Becket! Thank you for your many contributions. We are
> excited to have you on board as a committer and look forward to your
> continued participation!
>
> Joel
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: [VOTE] Add REST Server to Apache Kafka

[Gwen Shapira]

Oops. Sorry, didn't notice the 72h voting period has passed. You can
disregard.

Gwen

On Sat, Oct 29, 2016 at 4:29 PM, Gwen Shapira <g...@confluent.io> wrote:

> -1
>
> Kafka's development model is a good fit for critical path and
> well-established APIs. It doesn't work as well for add-ons that need to
> rapidly evolve. Merging communities with different development pace and
> models rarely ends well - I think the REST Proxy will benefit from being a
> separate project.
>
> On Tue, Oct 25, 2016 at 11:16 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
>> Hi All,
>>We are proposing to have a REST Server as part of  Apache Kafka
>> to provide producer/consumer/admin APIs. We Strongly believe having
>> REST server functionality with Apache Kafka will help a lot of users.
>> Here is the KIP that Mani Kumar wrote
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-80:+
>> Kafka+Rest+Server.
>> There is a discussion thread in dev list that had differing opinions on
>> whether to include REST server in Apache Kafka or not. You can read more
>> about that in this thread
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mb
>> ox/%3CCAMVt_AyMqeuDM39ZnSXGKtPDdE46sowmqhsXoP-+JMBCUV74Dw@
>> mail.gmail.com%3E
>>
>>   This is a VOTE thread to check interest in the community for
>> adding REST Server implementation in Apache Kafka.
>>
>> Thanks,
>> Harsha
>>
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: [VOTE] Add REST Server to Apache Kafka

[Gwen Shapira]

-1

Kafka's development model is a good fit for critical path and
well-established APIs. It doesn't work as well for add-ons that need to
rapidly evolve. Merging communities with different development pace and
models rarely ends well - I think the REST Proxy will benefit from being a
separate project.

On Tue, Oct 25, 2016 at 11:16 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Hi All,
>We are proposing to have a REST Server as part of  Apache Kafka
> to provide producer/consumer/admin APIs. We Strongly believe having
> REST server functionality with Apache Kafka will help a lot of users.
> Here is the KIP that Mani Kumar wrote
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 80:+Kafka+Rest+Server.
> There is a discussion thread in dev list that had differing opinions on
> whether to include REST server in Apache Kafka or not. You can read more
> about that in this thread
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mbox/%3CCAMVt_
> aymqeudm39znsxgktpdde46sowmqhsxop-+jmbcuv7...@mail.gmail.com%3E
>
>   This is a VOTE thread to check interest in the community for
> adding REST Server implementation in Apache Kafka.
>
> Thanks,
> Harsha
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: difficulty to delete a topic because of its syntax

[Gwen Shapira]

Just note that in general doing what Todd advice is pretty risky.
We've seen controllers get into all kinds of weird situations when
topics were deleted from ZK directly (including getting stuck in an
infinite loop, deleting unrelated topics and all kinds of strangeness)
- we have no tests for those scenarios so behavior can get really
unexpected.

On Thu, Oct 6, 2016 at 2:12 AM, Hamza HACHANI <hamza.hach...@supcom.tn> wrote:
> Thanks Todd,
>
>
> I've resolved it by suing what you told me.
>
> Thanks very much. But i think that there is a problem with kafka by letting 
> the saving names of topic and logs where there is a space as i showes in the 
> images.
>
> Have a good day to you all.
>
>
> Hamza
>
> 
> De : Hamza HACHANI <hamza.hach...@supcom.tn>
> Envoyé : mercredi 5 octobre 2016 19:23:00
> À : users@kafka.apache.org
> Objet : RE: difficulty to delete a topic because of its syntax
>
>
> Hi,
>
> Attached the files showing what i'm talking about.
>
>
> Hamza
>
> 
> De : Todd S <t...@borked.ca>
> Envoyé : mercredi 5 octobre 2016 07:25:48
> À : users@kafka.apache.org
> Objet : Re: difficulty to delete a topic because of its syntax
>
> You *could* go in to zookeeper and nuke the topic, then delete the files on
> disk
>
> Slightly more risky but it should work
>
> On Wednesday, 5 October 2016, Manikumar <manikumar.re...@gmail.com> wrote:
>
>> Kafka doesn't support white spaces in topic names.  Only support '.', '_'
>> and '-' these are allowed.
>> Not sure how you got white space in topic name.
>>
>> On Wed, Oct 5, 2016 at 8:19 PM, Hamza HACHANI <hamza.hach...@supcom.tn
>> <javascript:;>>
>> wrote:
>>
>> > Well ackwardly when i list the topics i find it but when i do delete it
>> it
>> > says that this topic does not exist.
>> >
>> > 
>> > De : Ben Davison <ben.davi...@7digital.com <javascript:;>>
>> > Envoyé : mercredi 5 octobre 2016 02:37:14
>> > À : users@kafka.apache.org <javascript:;>
>> > Objet : Re: difficulty to delete a topic because of its syntax
>> >
>> > Try putting "" or '' around the string when running the command.
>> >
>> > On Wed, Oct 5, 2016 at 3:29 PM, Hamza HACHANI <hamza.hach...@supcom.tn
>> <javascript:;>>
>> > wrote:
>> >
>> > > It's between "the" and "metric"
>> > >
>> > > 
>> > > De : Ali Akhtar <ali.rac...@gmail.com <javascript:;>>
>> > > Envoyé : mercredi 5 octobre 2016 02:16:33
>> > > À : users@kafka.apache.org <javascript:;>
>> > > Objet : Re: difficulty to delete a topic because of its syntax
>> > >
>> > > I don't see a space in that topic name
>> > >
>> > > On Wed, Oct 5, 2016 at 6:42 PM, Hamza HACHANI <hamza.hach...@supcom.tn
>> <javascript:;>>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I created a topic called device-connection-invert-key-value-the
>> > > > metric-changelog.
>> > > >
>> > > > I insit that there is a space in it.
>> > > >
>> > > >
>> > > >
>> > > > Now that i want to delete it because my  cluster can no longer work
>> > > > correctly i can't do it as it  only reads the first part of it : (
>> > > > device-connection-invert-key-value-the) which obviously it doesn't
>> > find.
>> > > >
>> > > > Does some body have a wolution to delete it ?
>> > > >
>> > > > Thanks in advance.
>> > > >
>> > > >
>> > > > Hamza
>> > > >
>> > > >
>> > >
>> >
>> > --
>> >
>> >
>> > This email, including attachments, is private and confidential. If you
>> have
>> > received this email in error please notify the sender and delete it from
>> > your system. Emails are not secure and may contain viruses. No liability
>> > can be accepted for viruses that might be transferred by this email or
>> any
>> > attachment. Any unauthorised copying of this message or unauthorised
>> > distribution and publication of the information contained herein are
>> > prohibited.
>> >
>> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
>> > Registered in England and Wales. Registered No. 04843573.
>> >
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Tuning for high RAM and 10GBe

[Gwen Shapira]

Out of curiosity - what is "Golang's Kafka interface"? Are you
referring to Sarama client?

On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cdste...@gmail.com> wrote:
> Hello,
>
> The last thread available regarding 10GBe is about 2 years old, with no
> obvious recommendations on tuning.
>
> Is there a more complex tuning guide than the example production config
> available on Kafka's main site? Anything other than the list of possible
> configs?
>
> I currently have access to a rather substantial academic cluster to test
> on, including multiple machines with the following hardware:
>
> 10GBe NICs
> 250GB RAM each
> SSDs on each
> (also, optional access to single NVMe)
>
> Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
> producer pushing to logs on the localhost, using no replication and reading
> from/logging to SSD. If it helps, I can post my configs. I've tried
> fiddling with a bunch of broker configs as well as producer configs,
> raising the memory limits, max message size, io threads etc.
>
> Since the last post from 2014 indicates that there is no public
> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
> this hardware if we can get it tuned up properly.
>
> What kind of broker/producer/consumer settings would you recommend?
>
> Thanks!
> - chris



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog