[VOTE] 2.2.1 RC1

[Vahid Hashemian]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 2.2.1.

Compared to RC0, this release candidate also fixes the following issues:

   - [KAFKA-6789] - Add retry logic in AdminClient requests
   - [KAFKA-8348] - Document of kafkaStreams improvement
   - [KAFKA-7633] - Kafka Connect requires permission to create internal
   topics even if they exist
   - [KAFKA-8240] - Source.equals() can fail with NPE
   - [KAFKA-8335] - Log cleaner skips Transactional mark and batch record,
   causing unlimited growth of __consumer_offsets
   - [KAFKA-8352] - Connect System Tests are failing with 404

Release notes for the 2.2.1 release:
https://home.apache.org/~vahid/kafka-2.2.1-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, May 16, 9:00 pm PT.

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~vahid/kafka-2.2.1-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~vahid/kafka-2.2.1-rc1/javadoc/

* Tag to be voted upon (off 2.2 branch) is the 2.2.1 tag:
https://github.com/apache/kafka/releases/tag/2.2.1-rc1

* Documentation:
https://kafka.apache.org/22/documentation.html

* Protocol:
https://kafka.apache.org/22/protocol.html

* Successful Jenkins builds for the 2.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/115/

Thanks!
--Vahid

Re: Kafka Connect - HDFS or FileStream

[Hans Jespersen]

Can you just use kafka-console-consumer and just redirect the output into a
file?

-hans


On Mon, May 13, 2019 at 1:55 PM Vinay Jain  wrote:

> Hi
>
> The data needs to be transferred to some other system in other network, and
> due to some security reasons, the other systems cannot be exposed . So the
> available mechanism is file based integration. Is there a production ready
> Kafka connect adapter which can create files in local directory.
>
> Regards
> Vinay Jain
>
> On Mon, May 13, 2019 at 3:41 PM Robin Moffatt  wrote:
>
> > Can you explain more about why you're writing a file with the data?
> > Presumably, this is for another application to consume; could it not take
> > the data from Kafka directly, whether with a native client or over the
> REST
> > proxy?
> > Oftentimes local files are unnecessary 'duck tape' for integration that
> can
> > be done in a better way.
> >
> >
> > --
> >
> > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff
> >
> >
> > On Mon, 13 May 2019 at 01:35, Vinay Jain  wrote:
> >
> > > Hi
> > >
> > > I would like to consume a Topic and save the AVRO messages in local
> > > directory in the AVRO file formats. As per Kafka Connect File Stream
> > > documentation, it is not for production use.
> > >
> > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am
> not
> > > sure if it can also write to the Local directory if we pass in the
> > variable
> > > hdfs.url the URL for local file system instead of HDFS path.
> > >
> > > Will this work or are there any other ready made options which can be
> > used
> > > for the same.
> > >
> > > Regards
> > >
> > > Vinay
> > >
> >
>

Re: Kafka Connect - HDFS or FileStream

[Vinay Jain]

Hi

The data needs to be transferred to some other system in other network, and
due to some security reasons, the other systems cannot be exposed . So the
available mechanism is file based integration. Is there a production ready
Kafka connect adapter which can create files in local directory.

Regards
Vinay Jain

On Mon, May 13, 2019 at 3:41 PM Robin Moffatt  wrote:

> Can you explain more about why you're writing a file with the data?
> Presumably, this is for another application to consume; could it not take
> the data from Kafka directly, whether with a native client or over the REST
> proxy?
> Oftentimes local files are unnecessary 'duck tape' for integration that can
> be done in a better way.
>
>
> --
>
> Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Mon, 13 May 2019 at 01:35, Vinay Jain  wrote:
>
> > Hi
> >
> > I would like to consume a Topic and save the AVRO messages in local
> > directory in the AVRO file formats. As per Kafka Connect File Stream
> > documentation, it is not for production use.
> >
> > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am not
> > sure if it can also write to the Local directory if we pass in the
> variable
> > hdfs.url the URL for local file system instead of HDFS path.
> >
> > Will this work or are there any other ready made options which can be
> used
> > for the same.
> >
> > Regards
> >
> > Vinay
> >
>

Re: Please give me permission to create KIP,Thanks!

[Bill Bejeck]

Thanks for your interest in contributing!

You should be all set now.

Thanks,
Bill

On Mon, May 13, 2019 at 12:34 PM hu xiaohua 
wrote:

>
>
> 
> I want create my KIP ,but I havn't a permission,please give me permission
> to create KIP,My Wiki ID is Flowermin,thanks.
>

Please give me permission to create KIP,Thanks!

[hu xiaohua]

I want create my KIP ,but I havn't a permission,please give me permission to 
create KIP,My Wiki ID is Flowermin,thanks.

Re: Customers getting duplicate emails

[John Roesler]

Hi Ashok,

In general, what Ryanne said is correct. For example, if you send an
email, and the send times out (but the email actually did get sent),
your app cannot distinguish this from a failure in which the send
times out before the email makes it out. Therefore, your only option
would be to retry, which may result in duplicate messages. This
obviously has nothing to do with Kafka, it's just a fact of life with
distributed systems.

However, there's some possibility that you can use Kafka's guarantees
to reduce your exposure to duplicate emails... But it's hard to say
without more information about the structure of your application. Feel
free to provide more info, and we'll see if we can give you some
advice.

Thanks,
-John

On Fri, May 10, 2019 at 10:11 AM Ryanne Dolan  wrote:
>
> Kafka only supports exactly-once and idempotency within the context of
> streams apps where records are consumed and produced within the same
> cluster. As soon as you touch the outside world in a non-idempotent way,
> e.g. by sending an email, these guarantees fall away. It is essentially
> impossible to guarantee that an intended action occurs no more than once
> while also guaranteeing that no intended action is skipped. This is a
> problem with distributed systems in general.
>
> You might use something like a cache to prevent most spurious emails -- if
> the cache already has record of the email being sent, don't resend it --
> but this will not prevent all duplicates.
>
> Ryanne
>
> On Fri, May 10, 2019 at 7:26 AM ASHOK MACHERLA  wrote:
>
> > *Dear Team*
> >
> >
> >
> > In our project, for SMS/Email purpose we are using Kafka cluster and
> > Real-time Notification which is our custom application.
> >
> >
> >
> > We are sending to messages from *Kafka to Real-time Notification, and
> > then SMTP Gateway* servers.
> >
> >
> >
> > Our problem is ,sometimes customers are getting same email for multiple
> > times.
> >
> > During this time consumer is goes to Rebalancing mode.
> >
> >
> >
> > How to overcome this,
> >
> > Right now, we have 10 partitions for Kafka topic and 10 consumers.
> >
> >
> >
> > Can you please suggest us to fix this one.
> >
> >
> >
> > If you required any information/logs, I’ll share to you .
> >
> >
> >
> > Please help us, Thanks
> >
> >
> >
> >
> >
> > Sent from Mail  for
> > Windows 10
> >
> >
> >

RE: Kafka upgrade process details

[Chylek, Artur]

Hi,
Thanks for clarification. The KIP you posted is a great piece of information 😊

Regards,
Artur

-Original Message-
From: Kamal Chandraprakash [mailto:kamal.chandraprak...@gmail.com] 
Sent: Friday, May 10, 2019 3:50 PM
To: users@kafka.apache.org
Subject: Re: Kafka upgrade process details

Sabre Email Notification: This email is from an EXTERNAL source. Please use 
caution when clicking on links or opening attachments from an unknown or 
suspicious sender.To report a suspected phishing email, browse to: 
help.sabre.com > Risk & Security > Report a Security Issue

__
Hi,

In Kafka v2.1.0, the OffsetCommit Request/Response schema version is changed to 
v4 for the *__consumer_offsets* topic.
If you upgrade Kafka to v2.1.0 & higher and changed the inter.broker.protocol 
version to 2.1, then you cannot revert back to older versions as it doesn't 
know how to read the *__consumer_offsets *topic with the latest offset message 
schema.

Indeed, if you want to revert back to the older versions, you have to delete 
the* __consumer_offsets* topic. Note: Once that topic is deleted, all your 
consumer groups will start to read from the latest/earliest available message 
as per your `auto.offset.reset` configuration. Please correct me if I am wrong.

https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D211-253A-2BRevise-2BExpiration-2BSemantics-2Bof-2BConsumer-2BGroup-2BOffsets&d=DwIFaQ&c=FXJfUb1oWgygD0uNz-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=FoNa_w6hwhTP2oeFsNP8mygpNhrxoIPqzjssJmXtpsY&s=9nYYkBcasdjeivQfxJyoBXW005zgTFUa8BRAf598l2o&e=

Note: The `__consumer_offsets` topic is read by the broker after each restart 
to load the last commited offset of the consumer groups.
So, once you change the `inter.broker.protocol.version` to 2.1 the OffsetCommit 
schema version changes to v4.

On Thu, May 9, 2019 at 4:21 PM Chylek, Artur  wrote:

> Thanks for prompt response.
> I am not sure I understand correctly, but I am still confused why 
> switching inter.broker.protocol.version in the last step would make 
> the process irreversible.
> If we agree that log conversion to a new format is applied when a new 
> value of log.message.format.version or broker's default one is applied 
> then according to documentation switching back to old version of 
> broker will make no harm. So either a broker is able to read a newer 
> versions of log and dismiss parts he doesn’t understand or the 
> conversion itself occurs only at certain combination of 
> inter.broker.protocol.version and log.message.format.version. Or maybe 
> I am completely wrong with my assumptions.
>
> Regards,
> Artur
>
> -Original Message-
> From: M. Manna [mailto:manme...@gmail.com]
> Sent: Thursday, May 09, 2019 12:19 PM
> To: Kafka Users 
> Subject: Re: Kafka upgrade process details
>
> Sabre Email Notification: This email is from an EXTERNAL source. 
> Please use caution when clicking on links or opening attachments from 
> an unknown or suspicious sender.To report a suspected phishing email, browse 
> to:
> help.sabre.com > Risk & Security > Report a Security Issue
>
> __
> Artur,
>
>
>
> The upgrade process is such that
>
>
>
> 1) You ensure that there is a hard-check on protocol version if not 
> exists
>
> already. As you have already mentioned above, in #3 - it's to ensure 
> that
>
> min verson for msg formats are being adhered to before upgrade.
>
> 2) broker protocol version is to ensure that when you do rolling 
> upgrade
>
> there is a minimum compatibility between inter-broker comms (if that 
> makes
>
> sense).
>
>
>
> API versions are there for a specific reason (to ensure compatbility 
> is
>
> maintained first before the protocol/msg formats are fixed). Also, it
>
> ensures that a regular upgrade (e.g. from bugfix->major release) 
> happens
>
> seamlessly via rolling restart. If you want to get techincal about it, 
> you
>
> can look at kafka Table load and config load in github codebase.
>
>
>
> Once you've initiated a rolling restart by setting
>
> inter.broker.protocl.version and log.messge.format.version there is no 
> way
>
> back - but i am happy to be proven wrong as I have only done rolling
>
> restart successfully (never needed to go back :) )
>
>
>
>
>
>
>
> On Thu, 9 May 2019 at 09:54, Chylek, Artur  wrote:
>
>
>
> > Hi,
>
> > I read the documentation about upgrading Kafka(
>
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_2
> 1_documentation.html-23upgrade-5F2-5F1-5F0&d=DwIBaQ&c=FXJfUb1oWgygD0uN
> z-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=3g5HRSTfPHijn_j
> hhzYIB88jScofj68jB1AmrcMPBC0&s=qiYRoeea_IxCjRGx1Vi7ylwk2vuCiYGGoPhhGYm
> iwos&e=)
> but I have
>
> > questions that I believe the documentation doesn't cover. I am 
> > planning
> to
>
> > upgrade Kafka from 2.0.0 to 2.1.0 and would li