date:20200128

Re: [DISCUSS] KIP-567: Kafka Cluster Audit

2020-01-28 Thread Владимир Беруненко

Hi Nikolai!

>Can you, please, make it more specific?
Why does a business want to have this information?
>What are the use-cases for it?
>Who will be analyzing these events and how?
>Why it’s not convenient to implement it with third-party tools?

This is required by the guys from information security to detect potential
threats of violation the rules for providing access to the data layer.
Analysis in the audit system is usually based on identifying
uncharacteristic integrations. The key feature of the audit, is that
cluster administrators do not have access to modification of audit events.
Third-party tools are great for data analysis, but its not good idea to use
it for audit events collection.

>It’s not clear for me where and when AuditEvents will be sent?
Who will be the receiver of events?

In my opinion, sending an audit event should be initiated when the broker
receives a request that matches the audit parameters. Each organization has
its own receiver system, so a common interface is required that the
organization’s development team can implement to integrate with their audit
system.

Best wishes, Vladimir

Hello, Igor.
>
> Thanks for the KIP.
> I have a couple of comments for it:
>
> > Motivation
> > It is highly demanded in most businesses to have the ability of
> obtaining audit information in case someone changes cluster configuration
> (like creation/deletion/modify/description of any topic or ACLs).
>
> Can you, please, make it more specific?
> Why does a business want to have this information?
> What are the use-cases for it?
> Who will be analyzing these events and how?
> Why it’s not convenient to implement it with third-party tools?
>
> It’s not clear for me where and when AuditEvents will be sent?
> Who will be the receiver of events?
>
> > void audit(AuditEvent event);
>
> 1. `audit` name sounds too general for me. How about `onEvent`?
> 2. Should we introduce a special marker interface for audit events?
> `AuditableRequest`, for example?
>
> > public interface AuditEvent {
> >  String guid();
>
> Where this `guid` comes from?
> Will it be the same on each node that receives an auditable event?
> Do we have `guid` for any extensions of `AbstractRequest`?
> If this field is `guid` why do we format this as a String on the API level?
>
> > One or more of AuditExtension's implementations can be configured via
> the configuration audit.extension.classes
>
> Can you, please, add a full list of proposed new configuration properties
> and examples for each to clarify your intentions?
>
>
> > 24 янв. 2020 г., в 18:12, Alexander Dunayevsky 
> написал(а):
> >
> > Hello Igor,
> >
> > Thanks for your KIP 
> > It would be great to adopt this functionality and getting the best of
> > tracking cluster activity.
> >
> > +1 vote from me
> >
> > Cheers,
> > Alex Dunayevsky
> >
> >
> > On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, 
> wrote:
> >
> >> Motivation:
> >>
> >>
> >> *It is highly demanded in most businesses to have the ability of
> obtaining
> >> audit information in case someone changes cluster configuration (like
> >> creation/deletion/modify/description of any topic or ACLs).We may add
> this
> >> ability. Since audit requirements are so broad, it's impractical to
> support
> >> all of them.Hence we have to provide ability for users to plug resources
> >> helping to achieve required capabilities.*
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
> >>
> >>
> >> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов :
> >>
> >>> Hello there.
> >>> Please review this KIP.
> >>> Thanks.
> >>>
> >>
>
>

[jira] [Created] (KAFKA-9480) Value for Task-level Metric process-rate is Constant Zero

2020-01-28 Thread Bruno Cadonna (Jira)

Bruno Cadonna created KAFKA-9480:


 Summary: Value for Task-level Metric process-rate is Constant Zero 
 Key: KAFKA-9480
 URL: https://issues.apache.org/jira/browse/KAFKA-9480
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.4.0
Reporter: Bruno Cadonna


The value for task-level metric process-rate is constant zero. The value should 
reflect the number of calls to {{process()}}  on source processors which 
clearly cannot be constant zero. 
This behavior applies to built-in metrics version {{latest}}. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-01-28 Thread Jason Gustafson (Jira)

Jason Gustafson created KAFKA-9479:
--

 Summary: Describe consumer group --all-groups shows header for 
each entry
 Key: KAFKA-9479
 URL: https://issues.apache.org/jira/browse/KAFKA-9479
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
print output like the following:

{code}
GROUP  COORDINATOR (ID)  
ASSIGNMENT-STRATEGY  STATE   #MEMBERS
group1 localhost:9092 (3) rangeStable  1

   

GROUP  COORDINATOR (ID)  
ASSIGNMENT-STRATEGY  STATE   #MEMBERS
group2  localhost:9092 (3) rangeStable  1   

 
{code}

It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1111

2020-01-28 Thread Apache Jenkins Server

See 


Changes:

[rajinisivaram] KAFKA-9460: Enable only TLSv1.2 by default and disable other TLS


--
[...truncated 2.85 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp

Build failed in Jenkins: kafka-trunk-jdk8 #4187

2020-01-28 Thread Apache Jenkins Server

See 


Changes:

[rajinisivaram] KAFKA-9460: Enable only TLSv1.2 by default and disable other TLS


--
[...truncated 2.73 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
>

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

2020-01-28 Thread Nikolay Izhikov

KIP adopted by 
https://github.com/apache/kafka/commit/172409c44b8551e2315bd93044a8a95ccda4699f

> 27 янв. 2020 г., в 13:10, Nikolay Izhikov  написал(а):
> 
> Thanks everyone!
> 
> After 3+ business days since this thread started, I'm concluding the vote
> on KIP-553.
> 
> The KIP has passed with:
> 
> 4 binding votes from Mickael Maison, Manikumar, Rajini Sivaram, M. Manna.
> 2 non-binding vote from Ted Yu, Ron Dagostino.
> 
> Thank you all for voting!
> 
>> 22 янв. 2020 г., в 14:43, M. Manna  написал(а):
>> 
>> +1 (binding). A simple, and yet powerful enforcement of TLS version.
>> 
>> Thanks for this KIP :)
>> 
>> On Tue, 21 Jan 2020 at 20:39, Mickael Maison 
>> wrote:
>> 
>>> +1 (binding)
>>> Thanks
>>> 
>>> On Tue, Jan 21, 2020 at 7:58 PM Ron Dagostino  wrote:
 
 +1 (non-binding)
 
 Ron
 
 On Tue, Jan 21, 2020 at 11:29 AM Manikumar 
>>> wrote:
> 
> +1 (binding).
> 
> Thanks for the KIP.
> 
> 
> On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:
> 
>> +1
>> 
>> On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram <
>>> rajinisiva...@gmail.com>
>> wrote:
>> 
>>> +1 (binding)
>>> 
>>> Thanks for the KIP!
>>> 
>>> Regards,
>>> 
>>> Rajini
>>> 
>>> 
>>> On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков <
>>> nizhi...@apache.org>
>>> wrote:
>>> 
 Hello.
 
 I would like to start vote for KIP-553: Disable all SSL protocols
>> except
 TLSV1.2 by default.
 
 KIP -
 
>>> 
>> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
 Discussion thread -
 
>>> 
>> 
>>> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
>>> 
>> 
>>> 
>

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

2020-01-28 Thread Ted Yu

+1

On Tue, Jan 28, 2020 at 10:52 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP, Brian!
>
> Regards,
>
> Rajini
>
> On Thu, Jan 23, 2020 at 7:34 PM Jason Gustafson 
> wrote:
>
> > Sounds good. +1 from me.
> >
> > On Thu, Jan 23, 2020 at 9:00 AM Brian Byrne  wrote:
> >
> > > Thanks Jason,
> > >
> > > I'm in favor of the latter: metadata.max.idle.ms. I agree that
> > describing
> > > it as a "period" is inaccurate. With metadata.max.idle.ms, it also
> > aligns
> > > with metadata.max.age.ms for determining refresh period (which is an
> > > actual
> > > period).
> > >
> > > I've updated the docs.
> > >
> > > Thanks,
> > > Brian
> > >
> > > On Wed, Jan 22, 2020 at 6:19 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Thanks for the proposal. Looks good overall. I wanted to suggest a
> > > possible
> > > > name change. I was considering something like `
> > > idle.metadata.expiration.ms
> > > > `
> > > > or maybe `metadata.max.idle.ms`. Thoughts?
> > > >
> > > > -Jason
> > > >
> > > >
> > > > On Tue, Jan 21, 2020 at 11:38 AM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Got it.
> > > > >
> > > > > I was proposing that we do the "delayed async batch" but I think
> your
> > > > > argument for complexity and pushing it out of the scope is
> > convincing,
> > > so
> > > > > instead I propose we do the synchronous mini batching still but
> > > obviously
> > > > > it is already there :)  I'm +1 on the current proposal scope.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Jan 21, 2020 at 10:16 AM Brian Byrne 
> > > > wrote:
> > > > >
> > > > > > Hi Guozhang,
> > > > > >
> > > > > > Ah, sorry, I misunderstood. Actually, this is solved for us
> today.
> > > How
> > > > > the
> > > > > > producer works is that it maintains at most one inflight metadata
> > > fetch
> > > > > > request at any time, where each request is tagged with the
> current
> > > > > > (monotonically increasing) request version. This version is
> bumped
> > > > > whenever
> > > > > > a new topic is encountered, and metadata fetching will continue
> to
> > > > > process
> > > > > > while the latest metadata response's version is below the current
> > > > > version.
> > > > > >
> > > > > > So if a metadata request is in flight, and a number of threads
> > > produce
> > > > to
> > > > > > new topics, they'll be added to the working set but the next
> > metadata
> > > > > > request won't take place until the outstanding one returns. So
> > their
> > > > > > updates will be batched together. As you suggest, we can have a
> > > simple
> > > > > list
> > > > > > that tracks unknown topics to isolate new vs. old topics.
> > > > > >
> > > > > > Thanks,
> > > > > > Brian
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 21, 2020 at 10:04 AM Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Brian,
> > > > > > >
> > > > > > > I think I buy the complexity and extra end-to-end-latency
> > argument
> > > :)
> > > > > I'm
> > > > > > > fine with delaying the asynchronous tech fetching to future
> works
> > > and
> > > > > > keep
> > > > > > > the current KIP's scope as-is for now. Under that case can we
> > > > consider
> > > > > > just
> > > > > > > a minor implementation detail (since it is not affecting public
> > > APIs
> > > > we
> > > > > > > probably do not even need to list it, but just thinking loud
> > here):
> > > > > > >
> > > > > > > In your proposal when we request for a topic of unknown
> metadata,
> > > we
> > > > > are
> > > > > > > going to directly set the topic name as that singleton in the
> > > > request.
> > > > > > I'm
> > > > > > > wondering for the scenario that KAFKA-8904 described, if the
> > > > > > producer#send
> > > > > > > for thousands of new topics are triggered sequentially by a
> > single
> > > > > thread
> > > > > > > or concurrent threads? If it's the latter, and we expect in
> such
> > > > > > scenarios
> > > > > > > we may have multiple topics being requests within a very short
> > > time,
> > > > > then
> > > > > > > we can probably do sth. like this internally in a synchronized
> > > > manner:
> > > > > > >
> > > > > > > 1) put the topic name into a list, as "unknown topics", then
> > > > > > > 2) exhaust the list, and put all topics from that list to the
> > > > request;
> > > > > if
> > > > > > > the list is empty, it means it has been emptied by another
> thread
> > > so
> > > > we
> > > > > > > skip sending a new request and just wait for the returned
> > metadata
> > > > > > refresh.
> > > > > > >
> > > > > > > In most cases the list would just be a singleton with the one
> > that
> > > > > thread
> > > > > > > has just enqueued, but under extreme scenarios it can help
> > > batching a
> > > > > few
> > > > > > > topic names probably (of course, I'm thinking about very
> extreme
> > > > cases
> > > > > > > here, assuming that's was what we've seen in 8904). Since these
> > two
> > > > > steps
> > > > > > > are very light-weighted, doing that in

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-28 Thread Jason Gustafson

+1

Nice to see this API finally resolved.

-Jason

On Tue, Jan 28, 2020 at 4:30 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP, Brian!
>
> Regards,
>
> Rajini
>
> On Mon, Jan 27, 2020 at 10:35 PM Colin McCabe  wrote:
>
> > Thanks, Brian.
> >
> > +1 (binding)
> >
> > best,
> > Colin
> >
> > On Sun, Jan 26, 2020, at 07:56, Brian Byrne wrote:
> > > Hello all,
> > >
> > > I'd like to start a vote on KIP-456: Add quota-specific APIs to the
> Admin
> > > Client, redux
> > >
> > > The KIP is here:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
> > >
> > > The discussion thread is here:
> > >
> >
> https://lists.apache.org/thread.html/802de7a2167677bf647a729df74e828b6acd4185f2eb0a25ddb939f6%40%3Cdev.kafka.apache.org%3E
> > >
> > > Thanks,
> > > Brian
> > >
> >
>

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

2020-01-28 Thread Rajini Sivaram

+1 (binding)

Thanks for the KIP, Brian!

Regards,

Rajini

On Thu, Jan 23, 2020 at 7:34 PM Jason Gustafson  wrote:

> Sounds good. +1 from me.
>
> On Thu, Jan 23, 2020 at 9:00 AM Brian Byrne  wrote:
>
> > Thanks Jason,
> >
> > I'm in favor of the latter: metadata.max.idle.ms. I agree that
> describing
> > it as a "period" is inaccurate. With metadata.max.idle.ms, it also
> aligns
> > with metadata.max.age.ms for determining refresh period (which is an
> > actual
> > period).
> >
> > I've updated the docs.
> >
> > Thanks,
> > Brian
> >
> > On Wed, Jan 22, 2020 at 6:19 PM Jason Gustafson 
> > wrote:
> >
> > > Thanks for the proposal. Looks good overall. I wanted to suggest a
> > possible
> > > name change. I was considering something like `
> > idle.metadata.expiration.ms
> > > `
> > > or maybe `metadata.max.idle.ms`. Thoughts?
> > >
> > > -Jason
> > >
> > >
> > > On Tue, Jan 21, 2020 at 11:38 AM Guozhang Wang 
> > wrote:
> > >
> > > > Got it.
> > > >
> > > > I was proposing that we do the "delayed async batch" but I think your
> > > > argument for complexity and pushing it out of the scope is
> convincing,
> > so
> > > > instead I propose we do the synchronous mini batching still but
> > obviously
> > > > it is already there :)  I'm +1 on the current proposal scope.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Jan 21, 2020 at 10:16 AM Brian Byrne 
> > > wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > Ah, sorry, I misunderstood. Actually, this is solved for us today.
> > How
> > > > the
> > > > > producer works is that it maintains at most one inflight metadata
> > fetch
> > > > > request at any time, where each request is tagged with the current
> > > > > (monotonically increasing) request version. This version is bumped
> > > > whenever
> > > > > a new topic is encountered, and metadata fetching will continue to
> > > > process
> > > > > while the latest metadata response's version is below the current
> > > > version.
> > > > >
> > > > > So if a metadata request is in flight, and a number of threads
> > produce
> > > to
> > > > > new topics, they'll be added to the working set but the next
> metadata
> > > > > request won't take place until the outstanding one returns. So
> their
> > > > > updates will be batched together. As you suggest, we can have a
> > simple
> > > > list
> > > > > that tracks unknown topics to isolate new vs. old topics.
> > > > >
> > > > > Thanks,
> > > > > Brian
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jan 21, 2020 at 10:04 AM Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Hi Brian,
> > > > > >
> > > > > > I think I buy the complexity and extra end-to-end-latency
> argument
> > :)
> > > > I'm
> > > > > > fine with delaying the asynchronous tech fetching to future works
> > and
> > > > > keep
> > > > > > the current KIP's scope as-is for now. Under that case can we
> > > consider
> > > > > just
> > > > > > a minor implementation detail (since it is not affecting public
> > APIs
> > > we
> > > > > > probably do not even need to list it, but just thinking loud
> here):
> > > > > >
> > > > > > In your proposal when we request for a topic of unknown metadata,
> > we
> > > > are
> > > > > > going to directly set the topic name as that singleton in the
> > > request.
> > > > > I'm
> > > > > > wondering for the scenario that KAFKA-8904 described, if the
> > > > > producer#send
> > > > > > for thousands of new topics are triggered sequentially by a
> single
> > > > thread
> > > > > > or concurrent threads? If it's the latter, and we expect in such
> > > > > scenarios
> > > > > > we may have multiple topics being requests within a very short
> > time,
> > > > then
> > > > > > we can probably do sth. like this internally in a synchronized
> > > manner:
> > > > > >
> > > > > > 1) put the topic name into a list, as "unknown topics", then
> > > > > > 2) exhaust the list, and put all topics from that list to the
> > > request;
> > > > if
> > > > > > the list is empty, it means it has been emptied by another thread
> > so
> > > we
> > > > > > skip sending a new request and just wait for the returned
> metadata
> > > > > refresh.
> > > > > >
> > > > > > In most cases the list would just be a singleton with the one
> that
> > > > thread
> > > > > > has just enqueued, but under extreme scenarios it can help
> > batching a
> > > > few
> > > > > > topic names probably (of course, I'm thinking about very extreme
> > > cases
> > > > > > here, assuming that's was what we've seen in 8904). Since these
> two
> > > > steps
> > > > > > are very light-weighted, doing that in a synchronized block would
> > not
> > > > > hurt
> > > > > > the concurrency too much.
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 21, 2020 at 9:39 AM Brian Byrne  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Guozhang,
> > > > > > >
> > > > > > > Your understanding of the rationale is accurate, and what you
> > > suggest
> > > > > is
> > > > > > > completely

Build failed in Jenkins: kafka-trunk-jdk8 #4186

2020-01-28 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Update lz4, jetty and other minor dependency bumps (#8008)


--
[...truncated 2.84 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

Re: [DISCUSS] KIP-390: Allow fine-grained configuration for compression (Rebooted)

2020-01-28 Thread Dongjin Lee

Hi Guozhang,

Sorry for the late reply. Let me have a detailed look on how each codec
uses compression buffer.

About compression levels, gzip and lz4 also supports this feature.

Thanks,
Dongjin

On Tue, Jan 21, 2020 at 4:31 AM Guozhang Wang  wrote:

> Hello Dongjin,
>
> I'm wondering if you have looked into the different implementor's buffer
> usage? So far as I read from the code:
>
> 1. LZ4 used a shared 64KB for decompression, and when reading it used
> ByteBuffer copy from the decompression buffer.
> 2. Snappy used shared uncompressed buffer, but when reading it uses
> SnappyNative.arrayCopy
> JNI which could be slow.
> 3. GZIP used shared 8KB (inflator), and another shared 16KB for reading it
> out with System.arraycopy.
> 4. ZSTD: dynamically allocate 128KB (i.e. not shared), and also use read
> internal for skip, so skip does not benefit that much.
>
> It seems to me that for different types the buffer used quite differently.
> Also, aside from ZSTD, are there any other types that have levels?
>
>
> Guozhang
>
>
> On Mon, Jun 24, 2019 at 4:30 PM Dongjin Lee  wrote:
>
> > Hello. Here is the new discussion thread for KIP-390: Allow fine-grained
> > configuration for compression.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Allow+fine-grained+configuration+for+compression
> >
> > Here is some history: Initially, the draft implementation was done with
> > traditional configuration scheme (Type A). Then, Becket Qin (
> > becket@gmail.com) and Mickael Maison (mickael.mai...@gmail.com)
> > proposed that the map style configuration
> > like listener.security.protocol.map
> > or max.connections.per.ip.overrides (Type B) would be better. From then
> on,
> > the discussion got struck.
> >
> > So last weekend, I re-implemented the feature against the latest trunk,
> for
> > all public interface alternatives (i.e., Type A & B.), and updated the
> KIP
> > document. You can find the details in this PR:
> > https://github.com/apache/kafka/pull/5927
> >
> > Please have a look when you are free. All kinds of feedbacks are
> welcomed!
> >
> > Regards,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> > *github:  github.com/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>
>
> --
> -- Guozhang
>
-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[jira] [Resolved] (KAFKA-7854) Behavior change in controller picking up partition reassignment tasks since 1.1.0

2020-01-28 Thread Ismael Juma (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-7854.

Resolution: Won't Fix

I'm marking this as "Won't fix" since KIP-455 introduced a Kafka protocol API 
that provides the desired functionality. Setting reassignment via the znode is 
deprecated and will be removed in a future release.

> Behavior change in controller picking up partition reassignment tasks since 
> 1.1.0
> -
>
> Key: KAFKA-7854
> URL: https://issues.apache.org/jira/browse/KAFKA-7854
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller
>Reporter: Zhanxiang (Patrick) Huang
>Priority: Major
>
> After [https://github.com/apache/kafka/pull/4143,] the controller does not 
> subscribe to data change on /admin/reassign_partitions any more (in order to 
> avoid unnecessarily loading the reassignment data again after controller 
> updating the znode) as opposed to the previous kafka versions. However, there 
> are systems built around kafka relying on the previous behavior to 
> incrementally update the list of partition reassignment since kafka does not 
> natively support that.
>  
> For example, [cruise control|https://github.com/linkedin/cruise-control] can 
> rely on the previous behavior (controller listening to data changes) to 
> maintain the reassignment concurrency by dynamically updating the data in the 
> reassignment znode instead of waiting for the current batch to finish and 
> doing reassignment batch by batch, which can significantly reduce the 
> rebalance time in production clusters. Although directly updating the znode 
> can somehow be viewed as an anti-pattern in the long term, this is necessary 
> since kafka does not natively support incrementally submit more reassignment 
> tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise 
> control no longer works because the controller behavior has changed. This 
> reveals the following problems:
>  * These behavior changes may be viewed as internal changes so compatibility 
> is not guaranteed but I think by convention people do view this as public 
> interfaces and rely on the compatibility. In this case, I think we should 
> clearly document the data contract for the partition reassignment task to 
> avoid misusage and making controller changes that break the defined data 
> contract. There may be other cases (e.g. topic deletion) whose data contracts 
> need to be clearly defined and we should keep it in mind when making 
> controller changes.
>  * Kafka does not natively support incrementally submit more reassignment 
> tasks. If we do want to support that nicely, we should consider change how we 
> store the reassignment data to store the data in child nodes and let the 
> controller listen on child node changes, similar to what we do for 
> /admin/delete_topics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

2020-01-28 Thread Dongjin Lee

Hi Smiklos,

Thanks for your interest in this issue. I am the author of this PR and now
rebasing the code to the latest trunk.

I have some questions:

   1. Could you share how you conducted the benchmark? I want to run the
   full validation with all cases.
   2. As you can see in the PR, it proposes three configuration
   alternatives. Which one do you prefer?

Thanks,

Dongjin

+1. Sorry for the late reply. I was working on another issue.

On Wed, Jan 15, 2020 at 2:01 AM smiklos  wrote:

> Hi,
>
> Is there any update on this? I've done performance test with Avro data
> and Snappy compression.
>
> Setting the buffer from 32kb to 128kb brings a rough 10% decrease in
> storage which is a big deal.
>
> I could offer working on this as well.
>
> Best regards,
>
> Miklos
>
>
> --
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[jira] [Created] (KAFKA-9478) Controller may stop react on partition reassignment command in ZooKeeper

2020-01-28 Thread Ivan Yurchenko (Jira)

Ivan Yurchenko created KAFKA-9478:
-

 Summary: Controller may stop react on partition reassignment 
command in ZooKeeper
 Key: KAFKA-9478
 URL: https://issues.apache.org/jira/browse/KAFKA-9478
 Project: Kafka
  Issue Type: Bug
  Components: controller, core
Affects Versions: 2.4.0, 2.4.1
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


Seemingly after 
[bdf2446ccce592f3c000290f11de88520327aa19|https://github.com/apache/kafka/commit/bdf2446ccce592f3c000290f11de88520327aa19],
 the controller may stop watching {{/admin/reassign_partitions}} node in 
ZooKeeper and consequently accept partition reassignment commands via ZooKeeper.

I'm not 100% sure that bdf2446ccce592f3c000290f11de88520327aa19 causes this, 
but it doesn't reproduce on 
[3fe6b5e951db8f7184a4098f8ad8a1afb2b2c1a0|https://github.com/apache/kafka/commit/3fe6b5e951db8f7184a4098f8ad8a1afb2b2c1a0]
 - the one right before it.

Also, reproduces on the trunk HEAD 
[a87decb9e4df5bfa092c26ae4346f65c426f1321|https://github.com/apache/kafka/commit/a87decb9e4df5bfa092c26ae4346f65c426f1321].
h1. How to reproduce

1. Run ZooKeeper and two Kafka brokers.

2. Create a topic with 100 partitions and place them on Broker 0:
{code:bash}
distro/bin/kafka-topics.sh --bootstrap-server localhost:9092,localhost:9093 
--create \
--topic foo \
--replica-assignment $(for i in {0..99}; do echo -n "0,"; done | sed 
's/.$$//')
{code}
3. Add some data:
{code:bash}
seq 1 100 | bin/kafka-console-producer.sh --broker-list 
localhost:9092,localhost:9093 --topic foo
{code}
4. Create the partition reassignment node {{/admin/reassign_partitions}} in Zoo 
and shortly after that update the data in the node (even the same value will 
do). I made a simple Python script for this:
{code:python}
import time
import json
from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

reassign = {
"version": 1,
"partitions":[]
}
for p in range(100):
reassign["partitions"].append({"topic": "foo", "partition": p, 
"replicas": [1]})

zk.create("/admin/reassign_partitions", json.dumps(reassign).encode())

time.sleep(0.05)

zk.set("/admin/reassign_partitions", json.dumps(reassign).encode())
{code}
4. Observe that the controller doesn't react on further updates to 
{{/admin/reassign_partitions}} and doesn't delete the node.

Also, it can be confirmed with
{code:bash}
echo wchc | nc 127.0.0.1 2181
{code}
that there is no watch on the node in ZooKeeper (for this, you should run 
ZooKeeper with {{4lw.commands.whitelist=*}}).

Since it's about timing, it might not work on first attempt, so you might need 
to do 4 a couple of times. However, the reproducibility rate is pretty high.

The data in the topic and the big amount of partitions are not needed per se, 
only to make the timing more favourable.

Controller re-election will solve the issue, but a new controller can be put in 
this state the same way.
h1. Proposed solution

TBD, suggestions are welcome.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1110

2020-01-28 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Update lz4, jetty and other minor dependency bumps (#8008)


--
[...truncated 2.85 MB...]

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

2020-01-28 Thread Rajini Sivaram

+1 (binding)

Thanks for the KIP, Brian!

Regards,

Rajini

On Mon, Jan 27, 2020 at 10:35 PM Colin McCabe  wrote:

> Thanks, Brian.
>
> +1 (binding)
>
> best,
> Colin
>
> On Sun, Jan 26, 2020, at 07:56, Brian Byrne wrote:
> > Hello all,
> >
> > I'd like to start a vote on KIP-456: Add quota-specific APIs to the Admin
> > Client, redux
> >
> > The KIP is here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
> >
> > The discussion thread is here:
> >
> https://lists.apache.org/thread.html/802de7a2167677bf647a729df74e828b6acd4185f2eb0a25ddb939f6%40%3Cdev.kafka.apache.org%3E
> >
> > Thanks,
> > Brian
> >
>

[VOTE] KIP-561: Regex Expressions Support for ConsumerGroupCommand

2020-01-28 Thread Alexander Dunayevsky

Hello guys,

Your votes will be much appreciated to bring this KIP to life and make
Kafka administration easier for all of us 

https://cwiki.apache.org/confluence/display/KAFKA/KIP-561%3A+Regex+Support+for+ConsumerGroupCommand

Discussion threads can be found here:
*Main Discussion: [DISCUSS] KIP-561: Regex Expressions Support for
ConsumerGroupCommand
*
*Previous Discussion 1*: Re: Multiple Consumer Group Management

*Previous Discussion 2*: Re: ConsumerGroupCommand tool improvement?


Thanks for your feedback,
Alex Dunayevsky

Re: [DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

2020-01-28 Thread Alexander Dunayevsky

Hello Nikolay,

I appreciate your feedback and thanks for your advice. I did receive
positive feedback from the community.
I'm starting the vote session right away then.

Best Regards,
Alex Dunayevsky

On Tue, 28 Jan 2020, 12:49 Nikolay Izhikov,  wrote:

> Hello, Alexander.
>
> As I can see from the previous discussion - you got positive feedback from
> the community.
> If you resolved all the comments and suggestions I think you should
> consider starting voting for this KIP.
>
> > 28 янв. 2020 г., в 10:56, Alexander Dunayevsky 
> написал(а):
> >
> > Any additional feedback on this?
> >
> > Best Regards,
> > Alex Dunayevsky
> >
> >
> > On Thu, 23 Jan 2020, 11:39 Alexander Dunayevsky, <
> a.a.dunayev...@gmail.com>
> > wrote:
> >
> >> Hello guys,
> >>
> >> Let's discuss KIP-561 Regex Support for ConsumerGroupCommand:
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-561%3A+Regex+Support+for+ConsumerGroupCommand
> >>
> >> Functionality already implemented and waiting to be reviewed.
> >>
> >> Best Regards,
> >> Alex Dunayevsky
> >>
> >>
> >> On Thu, 16 Jan 2020, 14:25 Alex D,  wrote:
> >>
> >>> Hello, guys,
> >>>
> >>> Please review Regex Expressions Support for ConsumerGroupCommand
> >>> improvement proposal
> >>>
> >>>   - *Previous Discussion 1*: Re: Multiple Consumer Group Management
> >>>   
> >>>   - *Previous Discussion 2*: Re: ConsumerGroupCommand tool improvement?
> >>>   
> >>>
> >>> *JIRA*: KAFKA-7817 Multiple Consumer Group Management with Regex
> >>> 
> >>>
> >>> *PR*: #6700 
> >>>
> >>
>
>

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

2020-01-28 Thread Viktor Somogyi-Vass

Hi Rajini,

I rebased my older PR and double checked it. It'll work with a new resource
type without adding new fields the ACL admin client APIs. As I mentioned
though, it'll be good to increment their version though to allow more
graceful handling of the protocol compatibilities as an older broker won't
know about the User resource type and probably will fail with a
serialization error whereas if they match the protocol the client could
detect it's an older broker and wouldn't allow the request. I'll append
this to the KIP.
Please let me know if we're good to continue with this.

Best,
Viktor

On Mon, Jan 20, 2020 at 5:45 PM Viktor Somogyi-Vass 
wrote:

> Hi Rajini,
>
> 1) I think we can to keep the conventions in the tool. As an addition we
> wouldn't have to retain certain characters (for creating the list).
> 2) Yes, so based on 1) and this --users changes to --user-principal (and
> accepts one single user principal).
> 3) Looking at it again probably we'll want to increase the version of the
> ACL protocols as new resource and operation types are getting added and
> currently sending such requests to old brokers would result in
> serialization errors. So it would be nicer to handle them on the API
> handshake. Besides this I don't see if we need to do anything else as these
> operations should be able to handle these changes on the code level. I'll
> make sure to test this ACL scenario and report back about it (although I
> need a few days as the code I have is very old and contains a lot of
> conflicts with the current trunk). Please let me know if I'm missing
> something here.
>
> Thanks,
> Viktor
>
> On Fri, Jan 17, 2020 at 5:23 PM Rajini Sivaram 
> wrote:
>
>> Hi Viktor,
>>
>> Thanks for the KIP. A few questions:
>>
>> 1) kafka-acls.sh has options like* --topic* that specifies a single topic.
>> Is there a reason why we want to have *--users* instead of *--user *with a
>> single user?
>> 2) We use user principal rather than just the name everywhere else. Can we
>> do the same here, or do we not want to treat this as a principal?
>> 3) If we update AclCommand, don't we also need equivalent AdminClient
>> changes to configure this ACL? I believe we are deprecating ZK-based ACL
>> updates, so we need to add this to AdminClient?
>>
>> Regards,
>>
>> Rajini
>>
>> On Fri, Jan 17, 2020 at 3:15 PM Viktor Somogyi-Vass <
>> viktorsomo...@gmail.com>
>> wrote:
>>
>> > Hi Jun & Richard,
>> >
>> > Jun, thanks for your feedback and vote.
>> >
>> > 100. Thanks, I'll correct that.
>> >
>> > 101. (@Richard) in this case the principal names will be something like
>> > "CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown"
>> unless
>> > principal mapping or builder is defined (refer to [1]). I think Jun was
>> > referring to this case which is correct, semicolon seems to be a better
>> fit
>> > in this case.
>> >
>> > Viktor
>> >
>> > https://docs.confluent.io/current/kafka/authorization.html
>> >
>> > On Thu, Jan 16, 2020 at 11:45 PM Richard Yu > >
>> > wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > Can the SSL username really include the comma?
>> > >
>> > > From what I could tell, when I searched it up, I couldn't find
>> anything
>> > > that indicated comma can be a delimiter.
>> > > A related doc below:
>> > > https://knowledge.digicert.com/solution/SO12401.html
>> > >
>> > > Cheers,
>> > > Richard
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Jan 16, 2020 at 1:37 PM Jun Rao  wrote:
>> > >
>> > > > Hi, Viktor,
>> > > >
>> > > > Thanks for the KIP. +1 from me. Just a couple of minor comments
>> below.
>> > > >
>> > > > 100. CreateDelegationTokenResponse/DescribeDelegationTokenResponse.
>> It
>> > > > seems that "validVersions" should be "0-2".
>> > > >
>> > > > 101. The option --users "owner1,owner2" in AclCommand. Since SSL
>> user
>> > > name
>> > > > can include comma, perhaps we could use semicolon as the separator.
>> > > >
>> > > > Jun
>> > > >
>> > > > On Wed, Jan 15, 2020 at 2:11 AM Viktor Somogyi-Vass <
>> > > > viktorsomo...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hey folks, bumping this again as KIP freeze is nearing and I hope
>> to
>> > > get
>> > > > > this into the next release.
>> > > > > We need only one binding vote.
>> > > > >
>> > > > > Thanks,
>> > > > > Viktor
>> > > > >
>> > > > > On Thu, Jan 9, 2020 at 1:56 PM Viktor Somogyi-Vass <
>> > > > > viktorsomo...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Bumping this in the hope of a vote or additional feedback.
>> > > > > >
>> > > > > > Viktor
>> > > > > >
>> > > > > > On Tue, Dec 3, 2019 at 1:07 PM Viktor Somogyi-Vass <
>> > > > > > viktorsomo...@gmail.com> wrote:
>> > > > > >
>> > > > > >> Hi Folks,
>> > > > > >>
>> > > > > >> I'd like to bump this once more in the hope of a binding vote
>> or
>> > any
>> > > > > >> additional feedback.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Viktor
>> > > > > >>
>> > > > > >> On Fri, Oct 25, 2019 at 2:24 PM Viktor Somogyi-Vass <
>> > > > > >> viktorsomo...@gmail.com> wrote:
>>

Re: [DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

2020-01-28 Thread Nikolay Izhikov

Hello, Alexander.

As I can see from the previous discussion - you got positive feedback from the 
community.
If you resolved all the comments and suggestions I think you should consider 
starting voting for this KIP.

> 28 янв. 2020 г., в 10:56, Alexander Dunayevsky  
> написал(а):
> 
> Any additional feedback on this?
> 
> Best Regards,
> Alex Dunayevsky
> 
> 
> On Thu, 23 Jan 2020, 11:39 Alexander Dunayevsky, 
> wrote:
> 
>> Hello guys,
>> 
>> Let's discuss KIP-561 Regex Support for ConsumerGroupCommand:
>> 
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-561%3A+Regex+Support+for+ConsumerGroupCommand
>> 
>> Functionality already implemented and waiting to be reviewed.
>> 
>> Best Regards,
>> Alex Dunayevsky
>> 
>> 
>> On Thu, 16 Jan 2020, 14:25 Alex D,  wrote:
>> 
>>> Hello, guys,
>>> 
>>> Please review Regex Expressions Support for ConsumerGroupCommand
>>> improvement proposal
>>> 
>>>   - *Previous Discussion 1*: Re: Multiple Consumer Group Management
>>>   
>>>   - *Previous Discussion 2*: Re: ConsumerGroupCommand tool improvement?
>>>   
>>> 
>>> *JIRA*: KAFKA-7817 Multiple Consumer Group Management with Regex
>>> 
>>> 
>>> *PR*: #6700 
>>> 
>>

Re: [DISCUSS] KIP-567: Kafka Cluster Audit

[jira] [Created] (KAFKA-9480) Value for Task-level Metric process-rate is Constant Zero

[jira] [Created] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

Build failed in Jenkins: kafka-trunk-jdk11 #1111

Build failed in Jenkins: kafka-trunk-jdk8 #4187

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

Build failed in Jenkins: kafka-trunk-jdk8 #4186

Re: [DISCUSS] KIP-390: Allow fine-grained configuration for compression (Rebooted)

[jira] [Resolved] (KAFKA-7854) Behavior change in controller picking up partition reassignment tasks since 1.1.0

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[jira] [Created] (KAFKA-9478) Controller may stop react on partition reassignment command in ZooKeeper

Build failed in Jenkins: kafka-trunk-jdk11 #1110

Re: [VOTE] KIP-546: Add quota-specific APIs to the Admin Client, redux

[VOTE] KIP-561: Regex Expressions Support for ConsumerGroupCommand

Re: [DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Re: [DISCUSS] KIP-561: Regex Expressions Support for ConsumerGroupCommand

20 matches

Site Navigation

Mail list logo

Footer information