Re: [ANNOUNCE] Apache Kafka 2.2.0

[Guozhang Wang]

Thanks Matthias for the release!

On Tue, Mar 26, 2019 at 9:31 PM Dongjin Lee  wrote:

> Congratulations and thanks for your great work, Matthias!!
>
> Best,
> Dongjin
>
> On Wed, Mar 27, 2019 at 4:49 AM Stephane Maarek  >
> wrote:
>
> > Congratulations on this amazing release! Lots of cool new features :)
> >
> > I've also released a YouTube video that will hopefully help the community
> > get up to speed: https://www.youtube.com/watch?v=kaWbp1Cnfo4=5s
> >
> > Happy watching!
> >
> > On Tue, Mar 26, 2019 at 7:02 PM Matthias J. Sax 
> wrote:
> >
> > > The Apache Kafka community is pleased to announce the release for
> Apache
> > > Kafka 2.2.0
> > >
> > >  - Added SSL support for custom principal name
> > >  - Allow SASL connections to periodically re-authenticate
> > >  - Command line tool bin/kafka-topics.sh adds AdminClient support
> > >  - Improved consumer group management
> > >- default group.id is `null` instead of empty string
> > >  - API improvement
> > >- Producer: introduce close(Duration)
> > >- AdminClient: introduce close(Duration)
> > >- Kafka Streams: new flatTransform() operator in Streams DSL
> > >- KafkaStreams (and other classed) now implement AutoClosable to
> > > support try-with-resource
> > >- New Serdes and default method implementations
> > >  - Kafka Streams exposed internal client.id via ThreadMetadata
> > >  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
> > > output `NaN` as default value
> > >
> > > All of the changes in this release can be found in the release notes:
> > > https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html
> > >
> > >
> > > You can download the source and binary release (Scala 2.11 and 2.12)
> > > from: https://kafka.apache.org/downloads#2.2.0
> > >
> > >
> > >
> >
> ---
> > >
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream records
> to
> > > one or more Kafka topics.
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming
> the
> > > input streams to output streams.
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> including
> > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> Rabobank,
> > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > >
> > > A big thank you for the following 98 contributors to this release!
> > >
> > > Alex Diachenko, Andras Katona, Andrew Schofield, Anna Povzner, Arjun
> > > Satish, Attila Sasvari, Benedict Jin, Bert Roos, Bibin Sebastian, Bill
> > > Bejeck, Bob Barrett, Boyang Chen, Bridger Howell, cadonna, Chia-Ping
> > > Tsai, Chris Egerton, Colin Hicks, Colin P. Mccabe, Colin Patrick
> McCabe,
> > > cwildman, Cyrus Vafadari, David Arthur, Dhruvil Shah, Dong Lin, Edoardo
> > > Comar, Flavien Raynaud, forficate, Gardner Vickers, Guozhang Wang, Gwen
> > > (Chen) Shapira, hackerwin7, hejiefang, huxi, Ismael Juma, Jacek
> > > Laskowski, Jakub Scholz, Jarek Rudzinski, Jason Gustafson, Jingguo Yao,
> > > John Eismeier, John Roesler, Jonathan Santilli, jonathanskrzypek, Jun
> > > Rao, Kamal Chandraprakash, Kan Li, Konstantine Karantasis, lambdaliu,
> > > Lars Francke, layfe, Lee Dongjin, linyli001, lu.ke...@berkeley.edu,
> > > Lucas Bradstreet, Magesh Nandakumar, Manikumar Reddy, Manikumar Reddy
> O,
> > > Manohar Vanam, Mark Cho, Mathieu Chataigner, Matthias J. Sax, Matthias
> > > Wessendorf, matus-cuper, Max Zheng, Mayuresh Gharat, Mickael Maison,
> > > mingaliu, Nikolay, occho, Pasquale Vazzana, Radai Rosenblatt, Rajini
> > > Sivaram, Randall Hauch, Renato Mefi, Richard Yu, Robert Yokota, Ron
> > > Dagostino, ryannatesmith, Samuel Hawker, Satish Duggana, Sayat,
> seayoun,
> > > Shawn Nguyen, slim, Srinivas Reddy, Stanislav Kozlovski, Stig Rohde
> > > Døssing, Suman, Tom Bentley, u214578, Vahid Hashemian, Viktor Somogyi,
> > > Viktor Somogyi-Vass, Xi Yang, Xiongqi Wu, ying-zheng, Yishun Guan,
> > > 

Re: [ANNOUNCE] Apache Kafka 2.2.0

[Dongjin Lee]

Congratulations and thanks for your great work, Matthias!!

Best,
Dongjin

On Wed, Mar 27, 2019 at 4:49 AM Stephane Maarek 
wrote:

> Congratulations on this amazing release! Lots of cool new features :)
>
> I've also released a YouTube video that will hopefully help the community
> get up to speed: https://www.youtube.com/watch?v=kaWbp1Cnfo4=5s
>
> Happy watching!
>
> On Tue, Mar 26, 2019 at 7:02 PM Matthias J. Sax  wrote:
>
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 2.2.0
> >
> >  - Added SSL support for custom principal name
> >  - Allow SASL connections to periodically re-authenticate
> >  - Command line tool bin/kafka-topics.sh adds AdminClient support
> >  - Improved consumer group management
> >- default group.id is `null` instead of empty string
> >  - API improvement
> >- Producer: introduce close(Duration)
> >- AdminClient: introduce close(Duration)
> >- Kafka Streams: new flatTransform() operator in Streams DSL
> >- KafkaStreams (and other classed) now implement AutoClosable to
> > support try-with-resource
> >- New Serdes and default method implementations
> >  - Kafka Streams exposed internal client.id via ThreadMetadata
> >  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
> > output `NaN` as default value
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release (Scala 2.11 and 2.12)
> > from: https://kafka.apache.org/downloads#2.2.0
> >
> >
> >
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 98 contributors to this release!
> >
> > Alex Diachenko, Andras Katona, Andrew Schofield, Anna Povzner, Arjun
> > Satish, Attila Sasvari, Benedict Jin, Bert Roos, Bibin Sebastian, Bill
> > Bejeck, Bob Barrett, Boyang Chen, Bridger Howell, cadonna, Chia-Ping
> > Tsai, Chris Egerton, Colin Hicks, Colin P. Mccabe, Colin Patrick McCabe,
> > cwildman, Cyrus Vafadari, David Arthur, Dhruvil Shah, Dong Lin, Edoardo
> > Comar, Flavien Raynaud, forficate, Gardner Vickers, Guozhang Wang, Gwen
> > (Chen) Shapira, hackerwin7, hejiefang, huxi, Ismael Juma, Jacek
> > Laskowski, Jakub Scholz, Jarek Rudzinski, Jason Gustafson, Jingguo Yao,
> > John Eismeier, John Roesler, Jonathan Santilli, jonathanskrzypek, Jun
> > Rao, Kamal Chandraprakash, Kan Li, Konstantine Karantasis, lambdaliu,
> > Lars Francke, layfe, Lee Dongjin, linyli001, lu.ke...@berkeley.edu,
> > Lucas Bradstreet, Magesh Nandakumar, Manikumar Reddy, Manikumar Reddy O,
> > Manohar Vanam, Mark Cho, Mathieu Chataigner, Matthias J. Sax, Matthias
> > Wessendorf, matus-cuper, Max Zheng, Mayuresh Gharat, Mickael Maison,
> > mingaliu, Nikolay, occho, Pasquale Vazzana, Radai Rosenblatt, Rajini
> > Sivaram, Randall Hauch, Renato Mefi, Richard Yu, Robert Yokota, Ron
> > Dagostino, ryannatesmith, Samuel Hawker, Satish Duggana, Sayat, seayoun,
> > Shawn Nguyen, slim, Srinivas Reddy, Stanislav Kozlovski, Stig Rohde
> > Døssing, Suman, Tom Bentley, u214578, Vahid Hashemian, Viktor Somogyi,
> > Viktor Somogyi-Vass, Xi Yang, Xiongqi Wu, ying-zheng, Yishun Guan,
> > Zhanxiang (Patrick) Huang
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > https://kafka.apache.org/
> >
> > Thank you!
> >
> >
> > Regards,
> >
> > Matthias
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical 

Build failed in Jenkins: kafka-trunk-jdk8 #3497

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-3522: Add RocksDBTimestampedSegmentedBytesStore (#6186)

--
[...truncated 4.71 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Jun Rao]

Hi, Jason,

Thanks for the KIP. Just a couple of more comments.

200. I am wondering if we really need the replica.selection.policy config
in the consumer. A slight variant is that we (1) let the consumer always
fetch from the PreferredReplica and (2) provide a default implementation of
ReplicaSelector that always returns the leader replica in select() for
backward compatibility. Then, we can get rid of replica.selection.policy in
the consumer. The benefits are that (1) fewer configs, (2) affinity
optimization can potentially be turned on with just a broker side change
(assuming affinity can be determined w/o client rack.id).

201. I am wondering if PreferredReplica in the protocol should be named
PreferredReadReplica since it's intended for reads?

Jun

On Mon, Mar 25, 2019 at 9:07 AM Jason Gustafson  wrote:

> Hi All, discussion on the KIP seems to have died down, so I'd like to go
> ahead and start a vote. Here is a link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> .
>
> +1 from me (duh)
>
> -Jason
>

Re: [DISCUSS] KIP-433: Provide client API version to authorizer

[Ying Zheng]

I have rewritten the KIP. The new proposal is adding a new configuration
min.api.version in Kafka broker.

Please review the new KIP. Thank you!

On Fri, Mar 1, 2019 at 11:06 AM Colin McCabe  wrote:

> On Wed, Feb 27, 2019, at 15:53, Harsha wrote:
> > HI Colin,
> > Overlooked the IDEMPOTENT_WRITE ACL. This along with
> > client.min.version should solve the cases proposed in the KIP.
> > Can we turn this KIP into adding min.client.version config to broker
> > and it could be part of the dynamic config .
>
> +1, sounds like a good idea.
>
> Colin
>
>
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Feb 27, 2019, at 12:17 PM, Colin McCabe wrote:
> > > On Tue, Feb 26, 2019, at 16:33, Harsha wrote:
> > > > Hi Colin,
> > > >
> > > > "> I think Ismael and Gwen here bring up a good point.  The version
> of the
> > > > > request is a technical detail that isn't really related to
> > > > > authorization.  There are a lot of other technical details like
> this
> > > > > like the size of the request, the protocol it came in on, etc.
> None of
> > > > > them are passed to the authorizer-- they all have configuration
> knobs
> > > > > to control how we handle them.  If we add this technical detail,
> > > > > logically we'll have to start adding all the others, and the
> authorizer
> > > > > API will get really bloated.  It's better to keep it focused on
> > > > > authorization, I think."
> > > >
> > > > probably my previous email is not clear but I am agreeing with
> Gwen's point.
> > > > I am not in favor of extending authorizer to support this.
> > > >
> > > >
> > > > "> Another thing to consider is that if we add a new broker
> configuration
> > > > > that lets us set a minimum client version which is allowed, that
> could
> > > > > be useful to other users as well.  On the other hand, most users
> are
> > > > > not likely to write a custom authorizer to try to take advantage
> of
> > > > > version information being passed to the authorizer.  So, I think
> using> a configuration is clearly the better way to go here.  Perhaps it
> can
> > > > > be a KIP-226 dynamic configuration to make this easier to deploy?"
> > > >
> > > > Although minimum client version might help to a certain extent there
> > > > are other cases where we want users to not start using transactions
> for
> > > > example. My proposal in the previous thread was to introduce another
> > > > module/interface, let's say
> > > > "SupportedAPIs" which will take in dynamic configuration to check
> which
> > > > APIs are allowed.
> > > > It can throw UnsupportedException just like we are throwing
> > > > Authorization Exception.
> > >
> > > Hi Harsha,
> > >
> > > We can already prevent people from using transactions using ACLs,
> > > right?  That's what the IDEMPOTENT_WRITE ACL was added for.
> > >
> > > In general, I think users should be able to think of ACLs in terms of
> > > "what can I do" rather than "how is it implemented."  For example,
> > > maybe some day we will replace FetchRequest with GetStuffRequest.  But
> > > users who have READ permission on a topic shouldn't have to change
> > > anything.  So I think the Authorizer interface should not be aware of
> > > individual RPC types or message versions.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > n Tue, Feb 26, 2019, at 10:04 AM, Colin McCabe wrote:
> > > > > Hi Harsha,
> > > > >
> > > > > I think Ismael and Gwen here bring up a good point.  The version
> of the
> > > > > request is a technical detail that isn't really related to
> > > > > authorization.  There are a lot of other technical details like
> this
> > > > > like the size of the request, the protocol it came in on, etc.
> None of
> > > > > them are passed to the authorizer-- they all have configuration
> knobs
> > > > > to control how we handle them.  If we add this technical detail,
> > > > > logically we'll have to start adding all the others, and the
> authorizer
> > > > > API will get really bloated.  It's better to keep it focused on
> > > > > authorization, I think.
> > > > >
> > > > > Another thing to consider is that if we add a new broker
> configuration
> > > > > that lets us set a minimum client version which is allowed, that
> could
> > > > > be useful to other users as well.  On the other hand, most users
> are
> > > > > not likely to write a custom authorizer to try to take advantage
> of
> > > > > version information being passed to the authorizer.  So, I think
> using
> > > > > a configuration is clearly the better way to go here.  Perhaps it
> can
> > > > > be a KIP-226 dynamic configuration to make this easier to deploy?
> > > > >
> > > > > cheers,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Mon, Feb 25, 2019, at 15:43, Harsha wrote:
> > > > > > Hi Ying,
> > > > > > I think the question is can we add a module in the core
> which
> > > > > > can take up the dynamic config and does a block certain APIs.
> This
> > > > > > module will 

[jira] [Created] (KAFKA-8160) To add ACL with SSL authentication

[suseendramani (JIRA)]

suseendramani created KAFKA-8160:


 Summary: To add ACL with SSL authentication
 Key: KAFKA-8160
 URL: https://issues.apache.org/jira/browse/KAFKA-8160
 Project: Kafka
  Issue Type: New Feature
  Components: consumer, producer 
Affects Versions: 1.1.0
Reporter: suseendramani


We want to setup the SSL based authentication along with ACL in place.  Is that 
doable and can it be added as a feature ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[George Li]

 Hi Ismael,

Thanks,  I understand your points. I will add the RPC mechanism for 
reassignments to KIP-236.  I can think of a few Requests/Responses 
corresponding to the old Scala AdminClient using ZK:  

SubmitReassignments (--execute)
StatusReassignments (--verify)
ChangeOrRemoveReassignmentsThrottle (--verify)
GenerateReassignments (--generate)
CancelPendingReassignments (new in KIP-236)

To clarify the "structure" change of KIP-236, the ZK public interface remains 
the same:  {topic, partition, new_replicas},  the user client generate/submit 
the reassignment plan the same way as before.  the new "original_replicas" in 
the ZK node is added by admin client before writing to the ZK node.  A bit 
similar to the "log_dirs".  User's direct modification of ZK 
/admin/reassign_partitions is strongly discouraged. 

The original_replicas info is essential for cancellation/rollback of the 
reassignments still pending. 

Thanks,
George

On Monday, March 25, 2019, 4:43:30 PM PDT, Ismael Juma  
wrote:  
 
 Hi George,

The goal is not to prevent people from updating ZK directly. The goal is to
offer a solution where people don't have to. If people then decide to avoid
the recommended path, they can deal with the consequences. However, if we
add another structure in ZK and no RPC mechanism, then there is no
recommended path apart from updating ZK (implicitly making it an API for
users).

Ismael

On Mon, Mar 25, 2019 at 3:57 PM George Li 
wrote:

>  Thanks Ismael.  One question, even switch to submitting reassignments via
> RPC instead of Zookeeper.  The reassignment data will still persist in
> ZooKeeper node /admin/reassign_partitions (e.g. when Controller failover it
> can resume reassignments)?  If yes, how this can keep someone from
> modifying ZK (/admin/reassign_partitions) directly ?
>
>
> Thanks,
> George
>
>    On Saturday, March 23, 2019, 1:07:11 PM PDT, Ismael Juma <
> isma...@gmail.com> wrote:
>
>  Thanks for the KIP, making reassignment more flexible is definitely
> welcome. As others have mentioned, I think we need to do it via the Kafka
> protocol and not via ZK. The latter introduces an implicit API that other
> tools will depend on causing migration challenges. This has already
> happened with the existing ZK based interface and we should avoid
> introducing more tech debt here.
>
> Ismael
>
> On Sat, Mar 23, 2019, 12:09 PM Colin McCabe  wrote:
>
> > On Thu, Mar 21, 2019, at 20:51, George Li wrote:
> > >  Hi Colin,
> > >
> > > I agree with your proposal of having administrative APIs through RPC
> > > instead of ZooKeeper. But seems like it will incur significant changes
> > > to both submitting reassignments and this KIP's cancelling pending
> > > reassignments.
> > >
> > > To make this KIP simple and moving along, I will be happy to do another
> > > follow-up KIP to change all reassignment related operations via RP
> >
> > Thanks, George.  I think doing it as a two-step process is fine, but I
> > suspect it would be much easier and quicker to do the RPC conversion
> first,
> > and the interruptible part later.  The reason is because a lot of the
> > things that people have brought up as concerns with this KIP are really
> > issues with the API (how will people interact with ZK, how does access
> > control work, what does the format look like in ZK) that will just go
> away
> > once we have an RPC.
> >
> > > Just curious,  KIP-4 includes Topics/ACL related operations. In
> > > addition to Reassignments,  any other operations should be done via
> > > RPC?
> >
> > I think all of the administrative shell scripts have been converted
> except
> > kafka-configs.sh.  I believe there is a KIP for that conversion.
> > Reassigning partitions is probably the biggest KIP-4 gap we have right
> now.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >    On Wednesday, March 20, 2019, 5:28:59 PM PDT, Colin McCabe
> > >  wrote:
> > >
> > >  Hi George,
> > >
> > > One big problem here is that administrative APIs should be done through
> > > RPCs, not through ZooKeeper.  KIP-4 (Command line and centralized
> > > administrative operations) describes the rationale for this.  We want
> > > public and stable APIs that don't depend on the internal representation
> > > of stuff in ZK, which will change over time.  Tools shouldn't have to
> > > integrate with ZK or understand the internal data structures of Kafka
> > > to make administrative changes.  ZK doesn't have a good security,
> > > access control, or compatibility story.
> > >
> > > We should create an official reassignment RPC for Kafka.  This will
> > > solve many of the problems discussed in this thread, I think.  For
> > > example, with an RPC, users won't be able to make changes unless they
> > > have ALTER on KafkaCluster.  That avoids the problem of random users
> > > making changes without the administrator knowing.  Also, if multiple
> > > users are making changes, there is no risk that they will overwrite
> > > each other's 

[jira] [Created] (KAFKA-8159) Multi-key range queries with negative keyFrom results in unexpected behavior

[Sophie Blee-Goldman (JIRA)]

Sophie Blee-Goldman created KAFKA-8159:
--

 Summary: Multi-key range queries with negative keyFrom results in 
unexpected behavior
 Key: KAFKA-8159
 URL: https://issues.apache.org/jira/browse/KAFKA-8159
 Project: Kafka
  Issue Type: Bug
Reporter: Sophie Blee-Goldman


If a user creates a queryable state store using one of the signed built-in 
serdes (eg Integer) for the key, there is nothing preventing records with 
negative keys from being inserted and/or fetched individually. However if the 
user tries to query the store for a range of keys starting with a negative 
number, unexpected behavior results that is store-specific.

 

For RocksDB stores with caching disabled, Streams will silently miss and 
negative keys and return those from the range [0, keyTo]. 

 

For in-memory stores and ANY store with caching enabled, Streams will throw an 
unchecked exception and crash.

 

This situation should be handled more gracefully, or users should be informed 
of this limitation and the result should at least be consist across types of 
store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-2.2-jdk8 #74

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Add 2.2.0 upgrade instructions (#6501)

--
[...truncated 2.73 MB...]
kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithEmptyIsr PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
PASSED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > testUpdatingOfflinePartitionsCount 
STARTED

kafka.controller.PartitionStateMachineTest > testUpdatingOfflinePartitionsCount 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testUpdatingOfflinePartitionsCountDuringTopicDeletion STARTED

kafka.controller.PartitionStateMachineTest > 
testUpdatingOfflinePartitionsCountDuringTopicDeletion PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown PASSED

kafka.controller.PartitionStateMachineTest > 

[jira] [Created] (KAFKA-8158) Add EntityType for Kafka RPC fields

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-8158:
--

 Summary: Add EntityType for Kafka RPC fields
 Key: KAFKA-8158
 URL: https://issues.apache.org/jira/browse/KAFKA-8158
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe


Add an EntityType for Kafka RPC fields so that we know what type they should be.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-0.11.0-jdk7 #414

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: fix flaky QueryableStateIntegrationTest (#6458)

--
[...truncated 1.55 MB...]

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses FAILED
java.lang.AssertionError: Condition not met within timeout 12. Did not 
receive all 202 records from topic output-concurrent-4
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:274)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:286)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1022)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:451)

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache PASSED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldRestoreStateFromSourceTopic STARTED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldRestoreStateFromSourceTopic PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKTableKTable STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKTableKTable PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > testLeftKTableKTable 
STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > testLeftKTableKTable 
PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testLeftKStreamKStream STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testLeftKStreamKStream PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testLeftKStreamKTable STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testLeftKStreamKTable PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKTableKTable STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKTableKTable PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKStream STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKStream PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKStreamKStream STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKStreamKStream PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKTable STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKTable PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 

Re: [ANNOUNCE] Apache Kafka 2.2.0

[Stephane Maarek]

Congratulations on this amazing release! Lots of cool new features :)

I've also released a YouTube video that will hopefully help the community
get up to speed: https://www.youtube.com/watch?v=kaWbp1Cnfo4=5s

Happy watching!

On Tue, Mar 26, 2019 at 7:02 PM Matthias J. Sax  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.2.0
>
>  - Added SSL support for custom principal name
>  - Allow SASL connections to periodically re-authenticate
>  - Command line tool bin/kafka-topics.sh adds AdminClient support
>  - Improved consumer group management
>- default group.id is `null` instead of empty string
>  - API improvement
>- Producer: introduce close(Duration)
>- AdminClient: introduce close(Duration)
>- Kafka Streams: new flatTransform() operator in Streams DSL
>- KafkaStreams (and other classed) now implement AutoClosable to
> support try-with-resource
>- New Serdes and default method implementations
>  - Kafka Streams exposed internal client.id via ThreadMetadata
>  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
> output `NaN` as default value
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.11 and 2.12)
> from: https://kafka.apache.org/downloads#2.2.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 98 contributors to this release!
>
> Alex Diachenko, Andras Katona, Andrew Schofield, Anna Povzner, Arjun
> Satish, Attila Sasvari, Benedict Jin, Bert Roos, Bibin Sebastian, Bill
> Bejeck, Bob Barrett, Boyang Chen, Bridger Howell, cadonna, Chia-Ping
> Tsai, Chris Egerton, Colin Hicks, Colin P. Mccabe, Colin Patrick McCabe,
> cwildman, Cyrus Vafadari, David Arthur, Dhruvil Shah, Dong Lin, Edoardo
> Comar, Flavien Raynaud, forficate, Gardner Vickers, Guozhang Wang, Gwen
> (Chen) Shapira, hackerwin7, hejiefang, huxi, Ismael Juma, Jacek
> Laskowski, Jakub Scholz, Jarek Rudzinski, Jason Gustafson, Jingguo Yao,
> John Eismeier, John Roesler, Jonathan Santilli, jonathanskrzypek, Jun
> Rao, Kamal Chandraprakash, Kan Li, Konstantine Karantasis, lambdaliu,
> Lars Francke, layfe, Lee Dongjin, linyli001, lu.ke...@berkeley.edu,
> Lucas Bradstreet, Magesh Nandakumar, Manikumar Reddy, Manikumar Reddy O,
> Manohar Vanam, Mark Cho, Mathieu Chataigner, Matthias J. Sax, Matthias
> Wessendorf, matus-cuper, Max Zheng, Mayuresh Gharat, Mickael Maison,
> mingaliu, Nikolay, occho, Pasquale Vazzana, Radai Rosenblatt, Rajini
> Sivaram, Randall Hauch, Renato Mefi, Richard Yu, Robert Yokota, Ron
> Dagostino, ryannatesmith, Samuel Hawker, Satish Duggana, Sayat, seayoun,
> Shawn Nguyen, slim, Srinivas Reddy, Stanislav Kozlovski, Stig Rohde
> Døssing, Suman, Tom Bentley, u214578, Vahid Hashemian, Viktor Somogyi,
> Viktor Somogyi-Vass, Xi Yang, Xiongqi Wu, ying-zheng, Yishun Guan,
> Zhanxiang (Patrick) Huang
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Matthias
>

Build failed in Jenkins: kafka-trunk-jdk8 #3496

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Add 2.2.0 upgrade instructions (#6501)

--
[...truncated 2.34 MB...]
org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectSchemaless STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectSchemaless PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnectNull 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectBadSchema PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectNull PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testToConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > testFromConnect 
PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue STARTED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFromConnectInvalidValue PASSED

org.apache.kafka.connect.converters.LongConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.LongConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.LongConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.LongConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.LongConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.LongConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.LongConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.LongConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.LongConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.LongConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.LongConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.LongConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.LongConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.LongConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 

[jira] [Reopened] (KAFKA-1149) Please delete old releases from mirroring system

[Sebb (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebb reopened KAFKA-1149:
-

There are now 9 releases on the mirrors.

Please respect the 3rd party mirrors and remove all non-current releases.

If necessary, the download page can continue to point to old releases on the 
archive server.

> Please delete old releases from mirroring system
> 
>
> Key: KAFKA-1149
> URL: https://issues.apache.org/jira/browse/KAFKA-1149
> Project: Kafka
>  Issue Type: Bug
> Environment: http://www.apache.org/dist/kafka/old_releases/
>Reporter: Sebb
>Priority: Major
>
> To reduce the load on the ASF mirrors, projects are required to delete old 
> releases [1]
> Please can you remove all non-current releases?
> Thanks!
> [Note that older releases are always available from the ASF archive server]
> [1] http://www.apache.org/dev/release.html#when-to-archive



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[ANNOUNCE] Apache Kafka 2.2.0

[Matthias J. Sax]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.2.0

 - Added SSL support for custom principal name
 - Allow SASL connections to periodically re-authenticate
 - Command line tool bin/kafka-topics.sh adds AdminClient support
 - Improved consumer group management
   - default group.id is `null` instead of empty string
 - API improvement
   - Producer: introduce close(Duration)
   - AdminClient: introduce close(Duration)
   - Kafka Streams: new flatTransform() operator in Streams DSL
   - KafkaStreams (and other classed) now implement AutoClosable to
support try-with-resource
   - New Serdes and default method implementations
 - Kafka Streams exposed internal client.id via ThreadMetadata
 - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
output `NaN` as default value

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.11 and 2.12)
from: https://kafka.apache.org/downloads#2.2.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 98 contributors to this release!

Alex Diachenko, Andras Katona, Andrew Schofield, Anna Povzner, Arjun
Satish, Attila Sasvari, Benedict Jin, Bert Roos, Bibin Sebastian, Bill
Bejeck, Bob Barrett, Boyang Chen, Bridger Howell, cadonna, Chia-Ping
Tsai, Chris Egerton, Colin Hicks, Colin P. Mccabe, Colin Patrick McCabe,
cwildman, Cyrus Vafadari, David Arthur, Dhruvil Shah, Dong Lin, Edoardo
Comar, Flavien Raynaud, forficate, Gardner Vickers, Guozhang Wang, Gwen
(Chen) Shapira, hackerwin7, hejiefang, huxi, Ismael Juma, Jacek
Laskowski, Jakub Scholz, Jarek Rudzinski, Jason Gustafson, Jingguo Yao,
John Eismeier, John Roesler, Jonathan Santilli, jonathanskrzypek, Jun
Rao, Kamal Chandraprakash, Kan Li, Konstantine Karantasis, lambdaliu,
Lars Francke, layfe, Lee Dongjin, linyli001, lu.ke...@berkeley.edu,
Lucas Bradstreet, Magesh Nandakumar, Manikumar Reddy, Manikumar Reddy O,
Manohar Vanam, Mark Cho, Mathieu Chataigner, Matthias J. Sax, Matthias
Wessendorf, matus-cuper, Max Zheng, Mayuresh Gharat, Mickael Maison,
mingaliu, Nikolay, occho, Pasquale Vazzana, Radai Rosenblatt, Rajini
Sivaram, Randall Hauch, Renato Mefi, Richard Yu, Robert Yokota, Ron
Dagostino, ryannatesmith, Samuel Hawker, Satish Duggana, Sayat, seayoun,
Shawn Nguyen, slim, Srinivas Reddy, Stanislav Kozlovski, Stig Rohde
Døssing, Suman, Tom Bentley, u214578, Vahid Hashemian, Viktor Somogyi,
Viktor Somogyi-Vass, Xi Yang, Xiongqi Wu, ying-zheng, Yishun Guan,
Zhanxiang (Patrick) Huang

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Matthias

Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams

[Boyang Chen]

Sounds good Guozhang! I will kick another discussion thread for KIP-444.

Boyang


From: Guozhang Wang 
Sent: Thursday, March 21, 2019 9:17 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams

Hello Boyang,

I've made another thorough pass over this KIP and I'd like to spilt it into
two parts: the first part, covered in KIP-429 would be touching on Consumer
Coordinator only to have incremental rebalance protocol in place. The
second part (for now I've reserved KIP number 444 for it) would contain all
the changes on StreamsPartitionAssginor to allow warming up new members.

I think the first part, a.k.a. the current updated KIP-429 is ready for
review and discussions again. Would love to hear people's feedbacks and
ideas.

Guozhang



On Mon, Mar 4, 2019 at 10:09 AM Boyang Chen  wrote:

> Thanks Guozhang for the great questions. Answers are inlined:
>
> 1. I'm still not sure if it's worthwhile to add a new type of "learner
> task" in addition to "standby task": if the only difference is that for the
> latter, we would consider workload balance while for the former we would
> not, I think we can just adjust the logic of StickyTaskAssignor a bit to
> break that difference. Adding a new type of task would be adding a lot of
> code complexity, so if we can still piggy-back the logic on a standby-task
> I would prefer to do so.
> In the proposal we stated that we are not adding a new type of task
> implementation. The
> learner task shall share the same implementation with normal standby task,
> only that we
> shall tag the standby task with learner and prioritize the learner tasks
> replay effort.
> 2. One thing that's still not clear from the KIP wiki itself is which layer
> would the logic be implemented at. Although for most KIPs we would not
> require internal implementation details but only public facing API updates,
> for a KIP like this I think it still requires to flesh out details on the
> implementation design. More specifically: today Streams embed a full
> fledged Consumer client, which hard-code a ConsumerCoordinator inside,
> Streams then injects a StreamsPartitionAssignor to its pluggable
> PartitionAssignor interface and inside the StreamsPartitionAssignor we also
> have a TaskAssignor interface whose default implementation is
> StickyPartitionAssignor. Streams partition assignor logic today sites in
> the latter two classes. Hence the hierarchy today is:
>
> KafkaConsumer -> ConsumerCoordinator -> StreamsPartitionAssignor ->
> StickyTaskAssignor.
>
> We need to think about where the proposed implementation would take place
> at, and personally I think it is not the best option to inject all of them
> into the StreamsPartitionAssignor / StickyTaskAssignor since the logic of
> "triggering another rebalance" etc would require some coordinator logic
> which is hard to mimic at PartitionAssignor level. On the other hand, since
> we are embedding a KafkaConsumer client as a whole we cannot just replace
> ConsumerCoordinator with a specialized StreamsCoordinator like Connect does
> in KIP-415. So I'd like to maybe split the current proposal in both
> consumer layer and streams-assignor layer like we did in KIP-98/KIP-129.
> And then the key thing to consider is how to cut off the boundary so that
> the modifications we push to ConsumerCoordinator would be beneficial
> universally for any consumers, while keep the Streams-specific logic at the
> assignor level.
> Yes, that's also my ideal plan. The details for the implementation are
> depicted
> in this doc<
> https://docs.google.com/document/d/1me2a5wvxAZT1QE6HkwyDl7C2TiBQlKN3Dpw_I1ro91U/edit#heading=h.qix74qdmekae>,
> and I have explained the reasoning on why we want to push a
> global change of replacing ConsumerCoordinator with StreamCoordinator. The
> motivation
> is that KIP space is usually used for public & algorithm level change, not
> for internal
> implementation details.
>
> 3. Depending on which design direction we choose, our migration plan would
> also be quite different. For example, if we stay with ConsumerCoordinator
> whose protocol type is "consumer" still, and we can manage to make all
> changes agnostic to brokers as well as to old versioned consumers, then our
> migration plan could be much easier.
> Yes, the upgrade plan was designed to take the new StreamCoordinator
> approach
> which means we shall define a new protocol type. For existing application
> we could only
> maintain the same `consumer` protocol type is because current broker only
> allows
> change of protocol type when the consumer group is empty. It is of course
> user-unfriendly to force
> a wipe-out for the entire application, and I don't think maintaining old
> protocol type would greatly
> impact ongoing services using new stream coordinator. WDYT?
>
> 4. I think one major issue related to this KIP is that today, in the
> StickyPartitionAssignor, we always try to honor stickiness over workload
> 

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Stephane Maarek]

It's going to change quite a few things for learners, but this is an
awesome idea!
+1 (non-binding)

On Tue, Mar 26, 2019 at 3:35 PM Viktor Somogyi-Vass 
wrote:

> +1 (non-binding) very good proposal.
>
> On Mon, Mar 25, 2019 at 7:19 PM David Arthur  wrote:
>
> > +1
> >
> > Thanks, Jason!
> >
> > On Mon, Mar 25, 2019 at 1:23 PM Eno Thereska 
> > wrote:
> >
> > > +1 (non-binding)
> > > Thanks for updating the KIP and addressing my previous comments.
> > >
> > > Eno
> > >
> > > On Mon, Mar 25, 2019 at 4:35 PM Ryanne Dolan 
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Great stuff, thanks.
> > > >
> > > > Ryanne
> > > >
> > > > On Mon, Mar 25, 2019, 11:08 AM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Hi All, discussion on the KIP seems to have died down, so I'd like
> to
> > > go
> > > > > ahead and start a vote. Here is a link to the KIP:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > > .
> > > > >
> > > > > +1 from me (duh)
> > > > >
> > > > > -Jason
> > > > >
> > > >
> > >
> >
> >
> > --
> > David Arthur
> >
>

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

[Viktor Somogyi-Vass]

+1 (non-binding) very good proposal.

On Mon, Mar 25, 2019 at 7:19 PM David Arthur  wrote:

> +1
>
> Thanks, Jason!
>
> On Mon, Mar 25, 2019 at 1:23 PM Eno Thereska 
> wrote:
>
> > +1 (non-binding)
> > Thanks for updating the KIP and addressing my previous comments.
> >
> > Eno
> >
> > On Mon, Mar 25, 2019 at 4:35 PM Ryanne Dolan 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Great stuff, thanks.
> > >
> > > Ryanne
> > >
> > > On Mon, Mar 25, 2019, 11:08 AM Jason Gustafson 
> > wrote:
> > >
> > > > Hi All, discussion on the KIP seems to have died down, so I'd like to
> > go
> > > > ahead and start a vote. Here is a link to the KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > .
> > > >
> > > > +1 from me (duh)
> > > >
> > > > -Jason
> > > >
> > >
> >
>
>
> --
> David Arthur
>

[jira] [Created] (KAFKA-8157) Missing "key.serializer" exception when setting "segment index bytes"

[Cristian D (JIRA)]

Cristian D created KAFKA-8157:
-

 Summary: Missing "key.serializer" exception when setting "segment 
index bytes"
 Key: KAFKA-8157
 URL: https://issues.apache.org/jira/browse/KAFKA-8157
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.2.0
 Environment: ubuntu 18.10, localhost and Aiven too
Reporter: Cristian D


As a `kafka-streams` user,

When I set the "segment index bytes" property

Then I would like to have internal topics with the specified allocated disk 
space

 

At the moment, when setting the "topic.segment.index.bytes" property, the 
application is exiting with following exception: 
{code:java}
Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
Missing required configuration "key.serializer" which has no default value.
{code}
Tested with `kafka-streams` v2.0.0 and v2.2.0.

 

Stack trace:
{code:java}
Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
Missing required configuration "key.serializer" which has no default value.
 at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:474)
 at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:464)
 at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
 at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75)
 at 
org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:392)
 at 
org.apache.kafka.streams.StreamsConfig.getMainConsumerConfigs(StreamsConfig.java:1014)
 at 
org.apache.kafka.streams.processor.internals.StreamThread.create(StreamThread.java:666)
 at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:718)
 at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:634)
 at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:544)
 at app.Main.main(Main.java:36)

{code}
A demo application simulating the exception:

https://github.com/razorcd/java-snippets-and-demo-projects/tree/master/kafkastreamsdemo

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)