Re: [VOTE] 2.0.0 RC1

2018-07-03 Thread Brett Rann 
+1 tentative
rolling upgrade of tiny shared staging multitenacy (200+ consumer groups)
cluster from 1.1 to 2.0.0-rc1. cluster looks healthy. Will monitor.

On Tue, Jul 3, 2018 at 8:18 AM Harsha  wrote:

> +1.
>
> 1) Ran unit tests
> 2) 3 node cluster , tested basic operations.
>
> Thanks,
> Harsha
>
> On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" <
> vahidhashem...@us.ibm.com> wrote:
>
> >
> >
> >
> > +1 (non-binding)
> >
> > Built from source and ran quickstart successfully on Ubuntu (with Java
> 8).
> >
> >
> > Minor: It seems this doc update PR is not included in the RC:
> > https://github.com/apache/kafka/pull/5280
> 
> > Guozhang seems to have wanted to cherry-pick it to 2.0.
> >
> > Thanks Rajini!
> > --Vahid
> >
> >
> >
> >
> > From: Rajini Sivaram < rajinisiva...@gmail.com >
> > To: dev < d...@kafka.apache.org >, Users < users@kafka.apache.org >,
> > kafka-clients < kafka-clie...@googlegroups.com >
> > Date: 06/29/2018 11:36 AM
> > Subject: [VOTE] 2.0.0 RC1
> >
> >
> >
> > Hello Kafka users, developers and client-developers,
> >
> >
> > This is the second candidate for release of Apache Kafka 2.0.0.
> >
> >
> > This is a major version release of Apache Kafka. It includes 40 new KIPs
> > and
> >
> > several critical bug fixes. Please see the 2.0.0 release plan for more
> > details:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
> 
> >
> >
> >
> > A few notable highlights:
> >
> > - Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
> > (KIP-277)
> > - SASL/OAUTHBEARER implementation (KIP-255)
> > - Improved quota communication and customization of quotas (KIP-219,
> > KIP-257)
> > - Efficient memory usage for down conversion (KIP-283)
> > - Fix log divergence between leader and follower during fast leader
> > failover (KIP-279)
> > - Drop support for Java 7 and remove deprecated code including old
> > scala
> > clients
> > - Connect REST extension plugin, support for externalizing secrets and
> > improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > - Scala API for Kafka Streams and other Streams API improvements
> > (KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> >
> > Release notes for the 2.0.0 release:
> >
> > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/RELEASE_NOTES.html
> 
> >
> >
> >
> >
> > *** Please download, test and vote by Tuesday, July 3rd, 4pm PT
> >
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> >
> > http://kafka.apache.org/KEYS
> 
> >
> >
> >
> > * Release artifacts to be voted upon (source and binary):
> >
> > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/
> 
> >
> >
> >
> > * Maven artifacts to be voted upon:
> >
> > https://repository.apache.org/content/groups/staging/
> 
> >
> >
> >
> > * Javadoc:
> >
> > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/javadoc/
> 
> >
> >
> >
> > * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> >
> > https://github.com/apache/kafka/tree/2.0.0-rc1
> 
> >
> >
> >
> > * Documentation:
> >
> > http://kafka.apache.org/20/documentation.html
> 
> >
> >
> >
> > * Protocol:
> >
> > http://kafka.apache.org/20/protocol.html
> 
> >
> >
> >
> > * Successful Jenkins builds for the 2.0 branch:
> >
> > Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.0-jdk8/66/
> 
> >
> >
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/2.0/15/
> 
> >
> >
> >
> >
> > Please test and verify the release artifacts and submit a vote for this
> RC
> >
> > or report any issues so that we can fix them and roll out a new RC ASAP!
> >
> > Although this release vote requires PMC votes to pass, testing, votes,
> and
> >
> > bug
> > reports are valuable and appreciated from everyone.
> >
> >
> > Thanks,
> >
> >
> > Rajini
> >
> >
> >
> >
> >
> >
> >
> >
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Re: Kafka Streams release

2018-07-03 Thread Guozhang Wang 
Hello Ayushi,

We are cutting a new RC for 2.0.0 now, and but for now we do not have a new
tentative date for it, hopefully it would be less than a week.


Guozhang

On Mon, Jul 2, 2018 at 9:57 PM, Ayushi Sharma  wrote:

> When is the new Streams(2.0.0) release?
> Tentative date was 27th June. Please revert.
>
> Thanks
> Ayushi
>



-- 
-- Guozhang

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

2018-07-03 Thread Yishun Guan 
Nice! Thanks~

On Tue, Jul 3, 2018, 12:16 PM Ismael Juma  wrote:

> Thanks Matthias!
>
> On Tue, 3 Jul 2018, 11:31 Matthias J. Sax,  wrote:
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA512
> >
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 0.11.0.3.
> >
> >
> > This is a bug fix release and it includes fixes and improvements from
> > 27 JIRAs, including a few critical bugs.
> >
> >
> > All of the changes in this release can be found in the release notes:
> >
> >
> > https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.
> > html
> > <
> https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.html
> >
> >
> >
> >
> > You can download the source release from:
> >
> >
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.
> > 3-src.tgz
> > <
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.3-src.tgz
> >
> >
> >
> > and binary releases from:
> >
> >
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.11-0.
> > 11.0.3.tgz
> > (Scala 2.11)
> >
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.12-0.
> > 11.0.3.tgz
> > (Scala 2.12)
> >
> >
> > -
> 
> > - ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream records
> > to one or more Kafka topics.
> >
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming
> > the input streams to output streams.
> >
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.three key capabilities:
> >
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> >
> > Apache Kafka is in use at large and small companies worldwide,
> > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > Zalando, among others.
> >
> >
> >
> > A big thank you for the following 26 contributors to this release!
> >
> >
> > Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> > Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> > Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> > Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> > Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> > Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
> >
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > http://kafka.apache.org/
> >
> >
> > Thank you!
> >
> >
> > Regards,
> >  -Matthias
> > -BEGIN PGP SIGNATURE-
> > Comment: GPGTools - https://gpgtools.org
> >
> > iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> > g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> > eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> > ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> > UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> > /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> > DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> > SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> > 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> > iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> > x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> > +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> > =4xXe
> > -END PGP SIGNATURE-
> >
>

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

2018-07-03 Thread Jason Gustafson 
Awesome. Thanks Matthias!

On Tue, Jul 3, 2018 at 12:44 PM, Yishun Guan  wrote:

> Nice! Thanks~
>
> On Tue, Jul 3, 2018, 12:16 PM Ismael Juma  wrote:
>
> > Thanks Matthias!
> >
> > On Tue, 3 Jul 2018, 11:31 Matthias J. Sax,  wrote:
> >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA512
> > >
> > > The Apache Kafka community is pleased to announce the release for
> > > Apache Kafka 0.11.0.3.
> > >
> > >
> > > This is a bug fix release and it includes fixes and improvements from
> > > 27 JIRAs, including a few critical bugs.
> > >
> > >
> > > All of the changes in this release can be found in the release notes:
> > >
> > >
> > > https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/
> RELEASE_NOTES.
> > > html
> > > <
> > https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/
> RELEASE_NOTES.html
> > >
> > >
> > >
> > >
> > > You can download the source release from:
> > >
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka-0.11.0.
> > > 3-src.tgz
> > > <
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka-0.11.0.3-src.tgz
> > >
> > >
> > >
> > > and binary releases from:
> > >
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka_2.11-0.
> > > 11.0.3.tgz
> > > (Scala 2.11)
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka_2.12-0.
> > > 11.0.3.tgz
> > > (Scala 2.12)
> > >
> > >
> > > -
> > 
> > > - ---
> > >
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream records
> > > to one or more Kafka topics.
> > >
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming
> > > the input streams to output streams.
> > >
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.three key capabilities:
> > >
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > >
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> > > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > > Zalando, among others.
> > >
> > >
> > >
> > > A big thank you for the following 26 contributors to this release!
> > >
> > >
> > > Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> > > Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> > > Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> > > Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> > > Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> > > Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
> > >
> > >
> > > We welcome your help and feedback. For more information on how to
> > > report problems, and to get involved, visit the project website at
> > > http://kafka.apache.org/
> > >
> > >
> > > Thank you!
> > >
> > >
> > > Regards,
> > >  -Matthias
> > > -BEGIN PGP SIGNATURE-
> > > Comment: GPGTools - https://gpgtools.org
> > >
> > > iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> > > g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> > > eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> > > ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> > > UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> > > /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> > > DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> > > SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> > > 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> > > iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> > > x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> > > +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> > > =4xXe
> > > -END PGP SIGNATURE-
> > >
> >
>

[ANNOUNCE] Apache Kafka 0.10.2.2 Released

2018-07-03 Thread Matthias J. Sax 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

The Apache Kafka community is pleased to announce the release for
Apache Kafka 0.10.2.2.


This is a bug fix release and it includes fixes and improvements from
29 JIRAs, including a few critical bugs.


All of the changes in this release can be found in the release notes:


https://dist.apache.org/repos/dist/release/kafka/0.10.2.2/RELEASE_NOTES.
html



You can download the source release from:


https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.2/kafka-0.10.2.
2-src.tgz


and binary releases from:


https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.2/kafka_2.11-0.
10.2.2.tgz
(Scala 2.11)

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.2/kafka_2.12-0.
10.2.2.tgz
(Scala 2.12)


- 
- ---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records
to one or more Kafka topics.


** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.


** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming
the input streams to output streams.


** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.three key capabilities:



With these APIs, Kafka can be used for two broad classes of application:


** Building real-time streaming data pipelines that reliably get data
between systems or applications.


** Building real-time streaming applications that transform or react
to the streams of data.



Apache Kafka is in use at large and small companies worldwide,
including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.



A big thank you for the following 30 contributors to this release!


Ewen Cheslack-Postava, Matthias J. Sax, Randall Hauch, Eno Thereska,
Damian Guy, Rajini Sivaram, Colin P. Mccabe, Kelvin Rutt, Kyle
Winkelman, Max Zheng, Guozhang Wang, Xavier Léauté, Konstantine
Karantasis, Paolo Patierno, Robert Yokota, Tommy Becker, Arjun Satish,
Xi Hu, Armin Braun, Edoardo Comar, Gunnar Morling, Gwen Shapira,
Hooman Broujerdi, Ismael Juma, Jaikiran Pai, Jarek Rudzinski, Jason
Gustafson, Jun Rao, Manikumar Reddy, Maytee Chinavanichkit


We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://kafka.apache.org/


Thank you!


Regards,
 -Matthias


-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls70woACgkQDccxaWtL
g1+Xzw//Rb7K691p0R2qPOixZfllEuO926C9dIjiq9XA+dZrabgC4tMgAtE07Pf4
i6ZUeIqVLH3IDYIKji92K+JUIWpu6fdmCc999bJUOJG+zABMbO0uRYm7/4LwfMPR
kfjxRhxu31ewvafs3crE4Kfkekw4FLFIwHiaz3i/mKC1Ty6V4oiJcwHP4PZizE2r
rTNbt0ZHzviiBH3klOoDh+ZZFwbDZn7EHUXm8o9fiiC52o/7TIqVWwmNzZJlNGRc
bxC3boGXAXjgBwm7iqxBgkPku/kTTWpxj6jkHbS2NQfCZE5V7INQC2HlnynPHc7j
m2F2plSvKOm4gi54q6SSiXkjcXA2dBJDe3y/jNpckXSQ31sNXsTi6vbRMkMPj8dJ
j0SKhFoSCDpWejgLkUMg6hZgepgz7G1uYHA9K8SfCyCooqxsEY4I3dClNOySORly
4brdjZWpclhCn+zpekqBFZ9Sn3ipG4MOvH64chPEvYnysHkRH26FqXNPOK185V0Z
Czl0dL0aEoJWZ3LxLTSoFkncKgqrcE00q4VknK3zGW65tlQ1DqTXtK3Ta1q8vX98
PCCR4Tjhu0RcBAV2L4o43itKzIaLCp9lElA1341oQUB+tiPRA0GvWGg36EomehzF
1qdbjBug91CLyefZVVeEfTiqmNAYNyR1Zmx99rryx+Fp+5Ek9YI=
=yjnJ
-END PGP SIGNATURE-

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

2018-07-03 Thread Ismael Juma 
Thanks Matthias!

On Tue, 3 Jul 2018, 11:31 Matthias J. Sax,  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 0.11.0.3.
>
>
> This is a bug fix release and it includes fixes and improvements from
> 27 JIRAs, including a few critical bugs.
>
>
> All of the changes in this release can be found in the release notes:
>
>
> https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.
> html
> 
>
>
>
> You can download the source release from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.
> 3-src.tgz
> 
>
>
> and binary releases from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.11-0.
> 11.0.3.tgz
> (Scala 2.11)
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.12-0.
> 11.0.3.tgz
> (Scala 2.12)
>
>
> - 
> - ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records
> to one or more Kafka topics.
>
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.three key capabilities:
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
>
>
> A big thank you for the following 26 contributors to this release!
>
>
> Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
>
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
>
> Thank you!
>
>
> Regards,
>  -Matthias
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> =4xXe
> -END PGP SIGNATURE-
>

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

2018-07-03 Thread Guozhang Wang 
Thanks Matthias for driving the release!

On Tue, Jul 3, 2018 at 11:31 AM, Matthias J. Sax  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 0.11.0.3.
>
>
> This is a bug fix release and it includes fixes and improvements from
> 27 JIRAs, including a few critical bugs.
>
>
> All of the changes in this release can be found in the release notes:
>
>
> https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.
> html
>
>
>
> You can download the source release from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.
> 3-src.tgz
>
>
> and binary releases from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.11-0.
> 11.0.3.tgz
> (Scala 2.11)
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.12-0.
> 11.0.3.tgz
> (Scala 2.12)
>
>
> - 
> - ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records
> to one or more Kafka topics.
>
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.three key capabilities:
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
>
>
> A big thank you for the following 26 contributors to this release!
>
>
> Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
>
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
>
> Thank you!
>
>
> Regards,
>  -Matthias
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> =4xXe
> -END PGP SIGNATURE-
>



-- 
-- Guozhang

[ANNOUNCE] Apache Kafka 0.11.0.3 Released

2018-07-03 Thread Matthias J. Sax 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

The Apache Kafka community is pleased to announce the release for
Apache Kafka 0.11.0.3.


This is a bug fix release and it includes fixes and improvements from
27 JIRAs, including a few critical bugs.


All of the changes in this release can be found in the release notes:


https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.
html



You can download the source release from:


https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.
3-src.tgz


and binary releases from:


https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.11-0.
11.0.3.tgz
(Scala 2.11)

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.12-0.
11.0.3.tgz
(Scala 2.12)


- 
- ---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records
to one or more Kafka topics.


** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.


** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming
the input streams to output streams.


** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.three key capabilities:



With these APIs, Kafka can be used for two broad classes of application:


** Building real-time streaming data pipelines that reliably get data
between systems or applications.


** Building real-time streaming applications that transform or react
to the streams of data.



Apache Kafka is in use at large and small companies worldwide,
including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.



A big thank you for the following 26 contributors to this release!


Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma


We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://kafka.apache.org/


Thank you!


Regards,
 -Matthias
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
/AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
+m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
=4xXe
-END PGP SIGNATURE-

Re: Possible bug? Duplicates when searching kafka stream state store with caching

2018-07-03 Thread Christian Henry 
Nope, we're setting retainDuplicates to false.

On Tue, Jul 3, 2018 at 6:55 AM, Damian Guy  wrote:

> Hi,
>
> When you create your window store do you have `retainDuplicates` set to
> `true`? i.e., assuming you use `Stores.persistentWindowStore(...)` is the
> last param `true`?
>
> Thanks,
> Damian
>
> On Mon, 2 Jul 2018 at 17:29 Christian Henry 
> wrote:
>
> > We're using the latest Kafka (1.1.0). I'd like to note that when we
> > encounter duplicates, the window is the same as well.
> >
> > My original code was a bit simplifier -- we also insert into the store if
> > iterator.hasNext() as well, before returning null. We're using a window
> > store because we have a punctuator that runs every few minutes to count
> > GUIDs with similar metadata, and reports that in a healthcheck. Since our
> > healthcheck window is less than the retention period of the store
> > (retention period might be 1 hour, healthcheck window is ~5 min), the
> > window store seemed like a good way to efficiently query all of the most
> > recent data. Note that since the healthcheck punctuator needs to
> aggregate
> > on all the recent values, it has to do a *fetchAll(start, end) *which is
> > how these duplicates are affecting us.
> >
> > On Fri, Jun 29, 2018 at 7:32 PM, Guozhang Wang 
> wrote:
> >
> > > Hello Christian,
> > >
> > > Since you are calling fetch(key, start, end) I'm assuming that
> > > duplicateStore
> > > is a WindowedStore. With a windowed store, it is possible that a single
> > key
> > > can fall into multiple windows, and hence be returned from the
> > > WindowStoreIterator,
> > > note its type is , V>
> > >
> > > So I'd first want to know
> > >
> > > 1) which Kafka version are you using.
> > > 2) why you'd need a window store, and if yes, could you consider using
> > the
> > > single point fetch (added in KAFKA-6560) other than the range query
> > (which
> > > is more expensive as well).
> > >
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Fri, Jun 29, 2018 at 11:38 AM, Christian Henry <
> > > christian.henr...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'll first describe a simplified view of relevant parts of our setup
> > > (which
> > > > should be enough to repro), describe the behavior we're seeing, and
> > then
> > > > note some information I've come across after digging in a bit.
> > > >
> > > > We have a kafka stream application, and one of our transform steps
> > keeps
> > > a
> > > > state store to filter out messages with a previously seen GUID. That
> > is,
> > > > our transform looks like:
> > > >
> > > > public KeyValue transform(byte[] key, String guid) {
> > > > try (WindowStoreIterator iterator =
> > > > duplicateStore.fetch(correlationId, start, now)) {
> > > > if (iterator.hasNext()) {
> > > > return null;
> > > > } else {
> > > > duplicateStore.put(correlationId, some metadata);
> > > > return new KeyValue<>(key, message);
> > > > }
> > > > }}
> > > >
> > > > where the duplicateStore is a persistent windowed store with caching
> > > > enabled.
> > > >
> > > > I was debugging some tests and found that sometimes when calling
> > > > *all()* or *fetchAll()
> > > > *on the duplicate store and stepping through the iterator, it would
> > > return
> > > > the same guid more than once, even if it was only inserted into the
> > store
> > > > once. More specifically, if I had the following guids sent to the
> > stream:
> > > > [1, 2, ... 9] (for 9 values total), sometimes it would
> > return
> > > > 10 values, with one (or more) of the values being returned twice by
> the
> > > > iterator. However, this would not show up with a *fetch(guid)* on
> that
> > > > specific guid. For instance, if 1 was being returned twice by
> > > > *fetchAll()*, calling *duplicateStore.fetch("1", start, end)*
> will
> > > > still return an iterator with size of 1.
> > > >
> > > > I dug into this a bit more by setting a breakpoint in
> > > > *SegmentedCacheFunction#compareSegmentedKeys(cacheKey,
> > > > storeKey)* and watching the two input values as I looped through the
> > > > iterator using "*while(iterator.hasNext()) { print(iterator.next())
> > }*".
> > > In
> > > > one test, the duplicate value was 6, and saw the following
> behavior
> > > > (trimming off the segment values from the byte input):
> > > > -- compareSegmentedKeys(cacheKey = 6, storeKey = 2)
> > > > -- next() returns 6
> > > > and
> > > > -- compareSegmentedKeys(cacheKey = 7, storeKey = 6)
> > > > -- next() returns 6
> > > > Besides those, the input values are the same and the output is as
> > > expected.
> > > > Additionally, a coworker noted that the number of duplicates always
> > > matches
> > > > the number of times *Long.compare(cacheSegmentId, storeSegmentId)
> > > *returns
> > > > a non-zero value, indicating that duplicates are likely arising due
> to
> > > the
> > > > segment comparison.
> > > >
> > >
> > >
> > >
> > >

Re: [kafka-clients] [VOTE] 1.0.2 RC1

2018-07-03 Thread Rajini Sivaram 
Hi Matthias,

+1 (binding)

Thank you for running the release.

Ran quick start with binary, tests with source, checked javadocs.

Regards,

Rajini

On Mon, Jul 2, 2018 at 9:34 PM, Harsha  wrote:

> +1.
>
> 1) Ran unit tests
> 2) 3 node cluster , tested basic operations.
>
> Thanks,
> Harsha
>
> On Mon, Jul 2nd, 2018 at 11:57 AM, Jun Rao  wrote:
>
> >
> >
> >
> > Hi, Matthias,
> >
> > Thanks for the running the release. Verified quickstart on scala 2.12
> > binary. +1
> >
> > Jun
> >
> > On Fri, Jun 29, 2018 at 10:02 PM, Matthias J. Sax <
> matth...@confluent.io >
> >
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 1.0.2.
> > >
> > > This is a bug fix release addressing 27 tickets:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
> > >
> > > Release notes for the 1.0.2 release:
> > > http://home.apache.org/~mjsax/kafka-1.0.2-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by end of next week (7/6/18).
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~mjsax/kafka-1.0.2-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~mjsax/kafka-1.0.2-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
> > > https://github.com/apache/kafka/releases/tag/1.0.2-rc1
> > >
> > > * Documentation:
> > > http://kafka.apache.org/10/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/10/protocol.html
> > >
> > > * Successful Jenkins builds for the 1.0 branch:
> > > Unit/integration tests: https://builds.apache.org/job/
> kafka-1.0-jdk7/214/
> >
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/1.0/225/
> > >
> > > /**
> > >
> > > Thanks,
> > > -Matthias
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups
> > > "kafka-clients" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > an
> > > email to kafka-clients+ unsubscr...@googlegroups.com.
> > > To post to this group, send email to kafka-clie...@googlegroups.com.
> > > Visit this group at https://groups.google.com/group/kafka-clients.
> > > To view this discussion on the web visit https://groups.google.com/d/
> > > msgid/kafka-clients/ca183ad4-9285-e423-3850-261f9dfec044%40
> confluent.io.
> >
> > > For more options, visit https://groups.google.com/d/optout.
> > >
> >
> >
> >
> >
>

Offset reprocess loop

2018-07-03 Thread adrien ruffie 
Hello all,


we have in our infrastructure, 3 brokers.

Sometimes we order a "reprocess all" certain flow, but we are facing a problem 
...

After relaunch reprocessing at the beginning offset, and arrived at a number 
offset, it loops several times by returning to an previous offset.


for example, we start to offset 0 and arrived to 4987 offset, the processing 
return to 3127 (it's a example) ...

We have a topic with only one partition and a replication factor of 3


Do you already see/have this kind of problem ?


Best regards,


Adrien

Re: RecordTooLargeException

2018-07-03 Thread jerryrichardson 

I cannot increase the max size of messages stored in Kafka.  How do I limit 
them to avoid the RecordTooLargeException?

2. Jul 2018 18:29 by jiangtao@zuora.com 
.INVALID:


> You can consider increase `max.request.size` ​a little big (the default
> value is `1048576`.), after checking Kafka client source code, they count
> [`key size` + `value size` + `header size` + others] together, so it's
> possible the calculated size is a little bigger than the default value.
>
> please check with > https://kafka.apache.org/documentation/#configuration 
> > .
>
> On Mon, Jul 2, 2018 at 5:08 PM, <> jerryrichard...@tutanota.com 
> > > wrote:
>
>> Hi all,
>>
>> I get this error even when my records are smaller than the 112 byte
>> limit:
>>
>> org.apache.kafka.common.errors.RecordTooLargeException: The request
>> included a message larger than the max message size the server will accept.
>>
>> How do I ensure that my producer doesn't send records that are too large?
>>
>> Thanks in advance for any suggestions and help.
>>

How to connect kafka with Industrial Sensor Devices

2018-07-03 Thread 
Hi

I wanted to bridge the communication  between industrial sensor devices which  
gives the data in raw format  like heat, temperature,  humidity, light, velocity
Does also sensor device which emit data  in binary or digital form  how the 
connection can be established to Kafka
IF the sensor devices are wifi connected what should be the preferred bandwidth 
it should have
>From TCP IP port will kafka connect and received the data from sensor device

Could you please through some light on the above

Thanks,
Sudheer.S
Johnson & Johnson
Project Manager- MVP Project
Bangalore, India
Cell: +91-9845102723

Kafka Streams release

2018-07-03 Thread Ayushi Sharma 
When is the new Streams(2.0.0) release?
Tentative date was 27th June. Please revert.

Thanks
Ayushi

How to get best concurrency performance while using KafkaProducer?

2018-07-03 Thread Rohan Kulkarni 
KafkaProducer javadoc

 says that using single instance will give faster performance than using
multiple instances "*The producer is thread safe and sharing a single
producer instance across threads will generally be faster than having
multiple instances.*" - what is the explanation for this?

On the contrary while going through KafkaProducer.send() code flow

 found out that there is a sychronized block on TopicPartition object, so
we should be able achieve better performance if we are using multiple
producer instances...?

Please advise on the recommended approach to achieve best performance with
KafkaProducer.

Re: Possible bug? Duplicates when searching kafka stream state store with caching

2018-07-03 Thread Damian Guy 
Hi,

When you create your window store do you have `retainDuplicates` set to
`true`? i.e., assuming you use `Stores.persistentWindowStore(...)` is the
last param `true`?

Thanks,
Damian

On Mon, 2 Jul 2018 at 17:29 Christian Henry 
wrote:

> We're using the latest Kafka (1.1.0). I'd like to note that when we
> encounter duplicates, the window is the same as well.
>
> My original code was a bit simplifier -- we also insert into the store if
> iterator.hasNext() as well, before returning null. We're using a window
> store because we have a punctuator that runs every few minutes to count
> GUIDs with similar metadata, and reports that in a healthcheck. Since our
> healthcheck window is less than the retention period of the store
> (retention period might be 1 hour, healthcheck window is ~5 min), the
> window store seemed like a good way to efficiently query all of the most
> recent data. Note that since the healthcheck punctuator needs to aggregate
> on all the recent values, it has to do a *fetchAll(start, end) *which is
> how these duplicates are affecting us.
>
> On Fri, Jun 29, 2018 at 7:32 PM, Guozhang Wang  wrote:
>
> > Hello Christian,
> >
> > Since you are calling fetch(key, start, end) I'm assuming that
> > duplicateStore
> > is a WindowedStore. With a windowed store, it is possible that a single
> key
> > can fall into multiple windows, and hence be returned from the
> > WindowStoreIterator,
> > note its type is , V>
> >
> > So I'd first want to know
> >
> > 1) which Kafka version are you using.
> > 2) why you'd need a window store, and if yes, could you consider using
> the
> > single point fetch (added in KAFKA-6560) other than the range query
> (which
> > is more expensive as well).
> >
> >
> >
> > Guozhang
> >
> >
> > On Fri, Jun 29, 2018 at 11:38 AM, Christian Henry <
> > christian.henr...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'll first describe a simplified view of relevant parts of our setup
> > (which
> > > should be enough to repro), describe the behavior we're seeing, and
> then
> > > note some information I've come across after digging in a bit.
> > >
> > > We have a kafka stream application, and one of our transform steps
> keeps
> > a
> > > state store to filter out messages with a previously seen GUID. That
> is,
> > > our transform looks like:
> > >
> > > public KeyValue transform(byte[] key, String guid) {
> > > try (WindowStoreIterator iterator =
> > > duplicateStore.fetch(correlationId, start, now)) {
> > > if (iterator.hasNext()) {
> > > return null;
> > > } else {
> > > duplicateStore.put(correlationId, some metadata);
> > > return new KeyValue<>(key, message);
> > > }
> > > }}
> > >
> > > where the duplicateStore is a persistent windowed store with caching
> > > enabled.
> > >
> > > I was debugging some tests and found that sometimes when calling
> > > *all()* or *fetchAll()
> > > *on the duplicate store and stepping through the iterator, it would
> > return
> > > the same guid more than once, even if it was only inserted into the
> store
> > > once. More specifically, if I had the following guids sent to the
> stream:
> > > [1, 2, ... 9] (for 9 values total), sometimes it would
> return
> > > 10 values, with one (or more) of the values being returned twice by the
> > > iterator. However, this would not show up with a *fetch(guid)* on that
> > > specific guid. For instance, if 1 was being returned twice by
> > > *fetchAll()*, calling *duplicateStore.fetch("1", start, end)* will
> > > still return an iterator with size of 1.
> > >
> > > I dug into this a bit more by setting a breakpoint in
> > > *SegmentedCacheFunction#compareSegmentedKeys(cacheKey,
> > > storeKey)* and watching the two input values as I looped through the
> > > iterator using "*while(iterator.hasNext()) { print(iterator.next())
> }*".
> > In
> > > one test, the duplicate value was 6, and saw the following behavior
> > > (trimming off the segment values from the byte input):
> > > -- compareSegmentedKeys(cacheKey = 6, storeKey = 2)
> > > -- next() returns 6
> > > and
> > > -- compareSegmentedKeys(cacheKey = 7, storeKey = 6)
> > > -- next() returns 6
> > > Besides those, the input values are the same and the output is as
> > expected.
> > > Additionally, a coworker noted that the number of duplicates always
> > matches
> > > the number of times *Long.compare(cacheSegmentId, storeSegmentId)
> > *returns
> > > a non-zero value, indicating that duplicates are likely arising due to
> > the
> > > segment comparison.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Benchmark tests of producer

2018-07-03 Thread darekAsz 
Hi,
I make a benchmark test of producent to check, how fast producer send data
to kafka broker.
There are my results: RESULTS 
Can someone explain me why results look like they look?
I send 100 mln messages 64B size. Why early logs are much faster? I noticed
that hardly always the 3-4 logs at beggining have bigger speed. I want to
maximize my speed so maybe understanding this  this can help me.

Maybe you have idea, how i can speed up this? I increased partitions
number, now I test with Ack = 0. What I can do any more?

Best regards