Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Eno Thereska]

Ok thanks, if you guys are seeing this at LinkedIn then the motivation
makes more sense.

Eno

On Tue, Aug 21, 2018 at 5:39 PM, Becket Qin  wrote:

> Hi Eno,
>
> Thanks for the comments. This KIP is not really about improving the
> performance in general. It is about ensuring the cluster state can still be
> updated quickly even if the brokers are under heavy load.
>
> We have seen quite often that it took dozens of seconds for a broker to
> process the requests sent by the controller when the cluster is under heavy
> load. This leads to the issues Lucas mentioned in the motivation part.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> > On Aug 20, 2018, at 11:33 PM, Eno Thereska 
> wrote:
> >
> > Hi folks,
> >
> > I looked at the previous numbers that Lucas provided (thanks!) but it's
> > still not clear to me whether the performance benefits justify the added
> > complexity. I'm looking for some intuition here (a graph would be great
> but
> > not required): for a small/medium/large cluster, what are the expected
> > percentage of control requests today that will benefit from the change?
> > It's a bit hard to go through this level of detail without knowing the
> > expected end-to-end benefit. The best folks to answer this might be ones
> > running such clusters, and ideally should pitch in with some data.
> >
> > Thanks
> > Eno
> >
> > On Mon, Aug 20, 2018 at 7:29 AM, Becket Qin 
> wrote:
> >
> >> Hi Lucas,
> >>
> >> In KIP-103, we introduced a convention to define and look up the
> listeners.
> >> So it would be good if the later KIPs can follow the same convention.
> >>
> >> From what I understand, the advertised.listeners is actually designed
> for
> >> our purpose, i.e. providing a list of listeners that can be used in
> >> different cases. In KIP-103 it was used to separate internal traffic
> from
> >> the external traffic. It is not just for the user traffic or data
> >> only. So adding
> >> a controller listener is not repurposing the config. Also, ZK structure
> is
> >> only visible to brokers, the clients will still only see the listeners
> they
> >> are seeing today.
> >>
> >> For this KIP, we are essentially trying to separate the controller
> traffic
> >> from the inter-broker data traffic. So adding a new
> >> listener.name.for.controller config seems reasonable. The behavior would
> >> be:
> >> 1. If the listener.name.for.controller is set, the broker-controller
> >> communication will go through that listener.
> >> 2. Otherwise, the controller traffic falls back to use
> >> inter.broker.listener.name or inter.broker.security.protocol, which is
> the
> >> current behavior.
> >>
> >> Regarding updating the security protocol with one line change v.s
> two-lines
> >> change, I am a little confused, can you elaborate?
> >>
> >> Regarding the possibility of hurry and misreading. It is the system
> admin's
> >> responsibility to configure the right listener to ensure that different
> >> kinds of traffic are using the correct endpoints. So I think it is
> better
> >> that we always follow the same of convention instead of doing it in
> >> different ways.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >>
> >>
> >> On Fri, Aug 17, 2018 at 4:34 AM, Lucas Wang 
> wrote:
> >>
> >>> Thanks for the review, Becket.
> >>>
> >>> (1) After comparing the two approaches, I still feel the current
> writeup
> >> is
> >>> a little better.
> >>> a. The current writeup asks for an explicit endpoint while reusing the
> >>> existing "inter.broker.listener.name" with the exactly same semantic,
> >>> and your proposed change asks for a new listener name for controller
> >> while
> >>> reusing the existing "advertised.listeners" config with a slight
> semantic
> >>> change since a new controller endpoint needs to be added to it.
> >>> Hence conceptually the current writeup requires one config change
> instead
> >>> of two.
> >>> Also with one listener name, e.g. INTERNAL, for inter broker traffic,
> >>> instead of two, e.g. "INTERNAL" and "CONTROLLER",
> >>> if an operator decides to switch from PLAINTEXT to SSL for internal
> >>> traffic, chances are that she wants to upgrade
> >>> both controller connections and data connections, she only needs to
> >> update
> >>> one line in
> >>> the "listener.security.protocol.map" config, and avoids possible
> >> mistakes.
> >>>
> >>>
> >>> b. When this KIP is picked up by an operator who is in a hurry without
> >>> reading the docs, if she sees a
> >>> new listener name for controller is required, and chances are there is
> >>> already a list of listeners,
> >>> it's possible for her to simply choose an existing listener name,
> without
> >>> explicitly creating
> >>> the new CONTROLLER listener and endpoints. If this is done, Kafka will
> be
> >>> run with the existing
> >>> behavior, defeating the purpose of this KIP.
> >>> In comparison, if she sees a separate endpoint is being asked, I feel
> >> it's
> >>> unlikely for her to
> >>> copy and paste an existing endpoint.
> 

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Dongjin Lee]

Jason,

> I think we would only use this error code when we /know/ that zstd was in
use and the client doesn't support it? This is true if either 1) the
message needs down-conversion and we encounter a zstd compressed message,
or 2) if the topic is explicitly configured to use zstd.

Yes, it is right. And you know, the case 1 includes 1.a) old clients'
request v0, v1 records or 1.b) implicit zstd, the compression type of
"producer" with Zstd compressed data.

> However, if the compression type is set to "producer," then the fetched
data may or may not be compressed with zstd. In this case, we return the
data to the client and expect it to fail parsing. Is that correct?

Exactly.

Following your message, I reviewed the implementation of
`KafkaApis#handleFetchRequest,` which handles the fetch request. And found
that the information we can use is like the following:

1. Client's fetch request version. (`versionId` variable)
2. Log's compression type. (`logConfig` variable)

We can't detect the actual compression type of the data, unless we inspect
the `RecordBatch` included in the `Records` instance (i.e.,
`unconvertedRecords` variable.) Since it requires some performance issue,
it is not our option - in short, we can't be sure if given chunks of data
are compressed with zstd or not.

So, conclusion: we can return an error in the case of 1.a and 2 easily,
with the information above. In the case 1.b (implicit zstd), we can just
return the data by do nothing special and expect it to fail parsing.

Thanks,
Dongjin

On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma  wrote:

> Jason, that's an interesting point regarding the Java client. Do we know
> what clients in other languages do in these cases?
>
> Ismael
>
> On Tue, 21 Aug 2018, 17:30 Jason Gustafson,  wrote:
>
> > Hi Dongjin,
> >
> > One of the complications is that old versions of the API will not expect
> a
> > new error code. However, since we expect this to be a fatal error anyway
> > for old clients, it may still be more useful to return the correct error
> > code. For example, the Kafka clients use the following code to convert
> the
> > error code:
> >
> > public static Errors forCode(short code) {
> > Errors error = codeToError.get(code);
> > if (error != null) {
> > return error;
> > } else {
> > log.warn("Unexpected error code: {}.", code);
> > return UNKNOWN_SERVER_ERROR;
> > }
> > }
> >
> > If we return an unsupported error code, it will be converted to an
> UNKNOWN
> > error, but at least we will get the message in the log with the correct
> > code. That seems preferable to returning a misleading error code. So I
> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
> > older versions.
> >
> > Also, one question just to check my understanding. I think we would only
> > use this error code when we /know/ that zstd was in use and the client
> > doesn't support it? This is true if either 1) the message needs
> > down-conversion and we encounter a zstd compressed message, or 2) if the
> > topic is explicitly configured to use zstd. However, if the compression
> > type is set to "producer," then the fetched data may or may not be
> > compressed with zstd. In this case, we return the data to the client and
> > expect it to fail parsing. Is that correct?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee  wrote:
> >
> > > Ismael, Jason and all,
> > >
> > > I rewrote the backward compatibility strategy & its alternatives like
> > > following, based on Ismael & Jason's comments. Since it is not updated
> to
> > > the wiki yet, don't hesitate to give me a message if you have any
> opinion
> > > on it.
> > >
> > > ```
> > > *Backward Compatibility*
> > >
> > > We need to establish some backward-compatibility strategy for the case
> an
> > > old client subscribes a topic using ZStandard implicitly (i.e.,
> > > 'compression.type' configuration of given topic is 'producer' and the
> > > producer compressed the records with ZStandard). We have the following
> > > options for this situation:
> > >
> > > *A. Support ZStandard to the old clients which can understand v0, v1
> > > messages only.*
> > >
> > > This strategy necessarily requires the down-conversion of v2 message
> > > compressed with Zstandard into v0 or v1 messages, which means a
> > > considerable performance degradation. So we rejected this strategy.
> > >
> > > *B. Bump the API version and support only v2-available clients*
> > >
> > > With this approach, we can message the old clients that they are old
> and
> > > should be upgraded. However, there are still several options for the
> > Error
> > > code.
> > >
> > > *B.1. INVALID_REQUEST (42)*
> > >
> > > This option gives the client so little information; the user can be
> > > confused about why the client worked correctly in the past suddenly
> > > encounters a problem. So we rejected this strategy.
> > >
> > > *B.2

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Dongjin Lee]

Jason and Ismael,

It seems like the only thing we need to regard if we define a new error
code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the implementation of
the other language clients, right? At least, this strategy causes any
problem for Java client. Do I understand correctly?

Thanks,
Dongjin

On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee  wrote:

> Jason,
>
> > I think we would only use this error code when we /know/ that zstd was
> in use and the client doesn't support it? This is true if either 1) the
> message needs down-conversion and we encounter a zstd compressed message,
> or 2) if the topic is explicitly configured to use zstd.
>
> Yes, it is right. And you know, the case 1 includes 1.a) old clients'
> request v0, v1 records or 1.b) implicit zstd, the compression type of
> "producer" with Zstd compressed data.
>
> > However, if the compression type is set to "producer," then the fetched
> data may or may not be compressed with zstd. In this case, we return the
> data to the client and expect it to fail parsing. Is that correct?
>
> Exactly.
>
> Following your message, I reviewed the implementation of
> `KafkaApis#handleFetchRequest,` which handles the fetch request. And found
> that the information we can use is like the following:
>
> 1. Client's fetch request version. (`versionId` variable)
> 2. Log's compression type. (`logConfig` variable)
>
> We can't detect the actual compression type of the data, unless we inspect
> the `RecordBatch` included in the `Records` instance (i.e.,
> `unconvertedRecords` variable.) Since it requires some performance issue,
> it is not our option - in short, we can't be sure if given chunks of data
> are compressed with zstd or not.
>
> So, conclusion: we can return an error in the case of 1.a and 2 easily,
> with the information above. In the case 1.b (implicit zstd), we can just
> return the data by do nothing special and expect it to fail parsing.
>
> Thanks,
> Dongjin
>
> On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma  wrote:
>
>> Jason, that's an interesting point regarding the Java client. Do we know
>> what clients in other languages do in these cases?
>>
>> Ismael
>>
>> On Tue, 21 Aug 2018, 17:30 Jason Gustafson,  wrote:
>>
>> > Hi Dongjin,
>> >
>> > One of the complications is that old versions of the API will not
>> expect a
>> > new error code. However, since we expect this to be a fatal error anyway
>> > for old clients, it may still be more useful to return the correct error
>> > code. For example, the Kafka clients use the following code to convert
>> the
>> > error code:
>> >
>> > public static Errors forCode(short code) {
>> > Errors error = codeToError.get(code);
>> > if (error != null) {
>> > return error;
>> > } else {
>> > log.warn("Unexpected error code: {}.", code);
>> > return UNKNOWN_SERVER_ERROR;
>> > }
>> > }
>> >
>> > If we return an unsupported error code, it will be converted to an
>> UNKNOWN
>> > error, but at least we will get the message in the log with the correct
>> > code. That seems preferable to returning a misleading error code. So I
>> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
>> > older versions.
>> >
>> > Also, one question just to check my understanding. I think we would only
>> > use this error code when we /know/ that zstd was in use and the client
>> > doesn't support it? This is true if either 1) the message needs
>> > down-conversion and we encounter a zstd compressed message, or 2) if the
>> > topic is explicitly configured to use zstd. However, if the compression
>> > type is set to "producer," then the fetched data may or may not be
>> > compressed with zstd. In this case, we return the data to the client and
>> > expect it to fail parsing. Is that correct?
>> >
>> > Thanks,
>> > Jason
>> >
>> >
>> >
>> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee 
>> wrote:
>> >
>> > > Ismael, Jason and all,
>> > >
>> > > I rewrote the backward compatibility strategy & its alternatives like
>> > > following, based on Ismael & Jason's comments. Since it is not
>> updated to
>> > > the wiki yet, don't hesitate to give me a message if you have any
>> opinion
>> > > on it.
>> > >
>> > > ```
>> > > *Backward Compatibility*
>> > >
>> > > We need to establish some backward-compatibility strategy for the
>> case an
>> > > old client subscribes a topic using ZStandard implicitly (i.e.,
>> > > 'compression.type' configuration of given topic is 'producer' and the
>> > > producer compressed the records with ZStandard). We have the following
>> > > options for this situation:
>> > >
>> > > *A. Support ZStandard to the old clients which can understand v0, v1
>> > > messages only.*
>> > >
>> > > This strategy necessarily requires the down-conversion of v2 message
>> > > compressed with Zstandard into v0 or v1 messages, which means a
>> > > considerable performance degradation. So we rejected this strategy.
>> > >
>> > > *B. Bump the

[jira] [Resolved] (KAFKA-7279) partitionsFor implicitly creates topic for the existent topic

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7279.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-7320/KIP-361

> partitionsFor implicitly creates topic for the existent topic
> -
>
> Key: KAFKA-7279
> URL: https://issues.apache.org/jira/browse/KAFKA-7279
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 2.0.0
>Reporter: huxihx
>Priority: Major
>
> With `auto.create.topics.enable` set to true, the non-existent topic got 
> created when invoking `Consumer#partitionsFor`. Is it deliberately as 
> designed?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Viktor Somogyi-Vass]

Hi All,

I'd like to start a vote on this KIP (
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242)
which aims to refactor ExtendedSerializer/Serializer and
ExtendedDeserializer/Deserializer.

To summarize what's the motivation:

When headers were introduced by KIP-82 the ExtendedSerializer and
ExtendedDeserializer classes were created in order to keep interface
compatibility but still add `T deserialize(String topic, Headers headers,
byte[] data);` and `byte[] serialize(String topic, Headers headers, T
data);` methods that consume the headers for serialization/deserialization.
The reason for doing so was that Kafka at that time needed be compatbile
with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423) we'll
try consolidate the way we're using these in a backward compatible fashion:
deprecating the Extended* classes and moving the aforementioned methods up
in the class hierarchy.

I'd be happy to get votes or additional feedback on this.

Viktor

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Kamal Chandraprakash]

+1

Thanks for the KIP!

On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass 
wrote:

> Hi All,
>
> I'd like to start a vote on this KIP (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242)
> which aims to refactor ExtendedSerializer/Serializer and
> ExtendedDeserializer/Deserializer.
>
> To summarize what's the motivation:
>
> When headers were introduced by KIP-82 the ExtendedSerializer and
> ExtendedDeserializer classes were created in order to keep interface
> compatibility but still add `T deserialize(String topic, Headers headers,
> byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> data);` methods that consume the headers for serialization/deserialization.
> The reason for doing so was that Kafka at that time needed be compatbile
> with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423) we'll
> try consolidate the way we're using these in a backward compatible fashion:
> deprecating the Extended* classes and moving the aforementioned methods up
> in the class hierarchy.
>
> I'd be happy to get votes or additional feedback on this.
>
> Viktor
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ted Yu]

+1
 Original message From: Kamal Chandraprakash 
 Date: 8/22/18  3:19 AM  (GMT-08:00) To: 
dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate 
ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer 
+1

Thanks for the KIP!

On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass 
wrote:

> Hi All,
>
> I'd like to start a vote on this KIP (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242)
> which aims to refactor ExtendedSerializer/Serializer and
> ExtendedDeserializer/Deserializer.
>
> To summarize what's the motivation:
>
> When headers were introduced by KIP-82 the ExtendedSerializer and
> ExtendedDeserializer classes were created in order to keep interface
> compatibility but still add `T deserialize(String topic, Headers headers,
> byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> data);` methods that consume the headers for serialization/deserialization.
> The reason for doing so was that Kafka at that time needed be compatbile
> with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423) we'll
> try consolidate the way we're using these in a backward compatible fashion:
> deprecating the Extended* classes and moving the aforementioned methods up
> in the class hierarchy.
>
> I'd be happy to get votes or additional feedback on this.
>
> Viktor
>

Re: [VOTE] KIP-346 - Improve LogCleaner behavior on error

[Stanislav Kozlovski]

Hi everybody,

@Jason - I've updated the section. Thanks for the reminder

I'm glad to say that the vote *has passed* with 3 binding votes (Jason,
Gwen, Harsha) and 6 non-binding votes (Dhruvil, Colin, Mickael, Manikumar,
Ray, Ted, Thomas).

The PR is ready for review at https://github.com/apache/kafka/pull/5439

On Tue, Aug 21, 2018 at 4:55 PM Jason Gustafson  wrote:

> +1 Thanks for the KIP! I'd suggest mentioning the configurations that were
> previously proposed in the rejected alternatives section. We may reconsider
> them in the future.
>
> On Mon, Aug 13, 2018 at 9:48 AM, Dhruvil Shah 
> wrote:
>
> > Thanks for the KIP, Stanislav! +1 (non-binding)
> >
> > - Dhruvil
> >
> > On Mon, Aug 13, 2018 at 9:39 AM Colin McCabe  wrote:
> >
> > > +1 (non-binding)
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Aug 7, 2018, at 04:19, Stanislav Kozlovski wrote:
> > > > Hey everybody,
> > > > I'm starting a vote on KIP-346
> > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 346+-+Improve+LogCleaner+behavior+on+error
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > >
> >
>


-- 
Best,
Stanislav

Build failed in Jenkins: kafka-trunk-jdk8 #2916

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Split at first occurrence of '=' in kafka.py props parsing

--
[...truncated 2.48 MB...]

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveSensor STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveSensor PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveNullSensor STARTED

org.apache.kafka.streams.processor.internals.StreamsMet

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

[Viktor Somogyi-Vass]

Hi Manikumar,

Implementation-wise is it just a filter over the returned ACL listing or do
you plan to add new methods to the Authorizer as well?

Thanks,
Viktor

On Fri, Aug 17, 2018 at 9:18 PM Priyank Shah  wrote:

> +1(non-binding)
>
> Thanks.
> Priyank
>
> On 8/16/18, 6:01 AM, "Manikumar"  wrote:
>
> Hi all,
>
> I have created a minor KIP to add support to list ACLs per principal
> using
> AclCommand (kafka-acls.sh)
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-357%3A++Add+support+to+list+ACLs+per+principal
>
> Please take a look.
>
> Thanks,
>
>
>

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Satish Duggana]

Congrats Dong Lin!

On Wed, Aug 22, 2018 at 10:08 AM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:

> Congratulations, Dong!
>
> On Wed, Aug 22, 2018 at 6:20 AM Dhruvil Shah  wrote:
>
> > Congratulations, Dong!
> >
> > On Tue, Aug 21, 2018 at 4:38 PM Jason Gustafson 
> > wrote:
> >
> > > Congrats!
> > >
> > > On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang 
> wrote:
> > >
> > > > Congrats Dong!
> > > >
> > > > -Ray
> > > >
> > > >
> > > > On 8/21/18 9:33 AM, Becket Qin wrote:
> > > >
> > > >> Congrats, Dong!
> > > >>
> > > >> On Aug 21, 2018, at 11:03 PM, Eno Thereska 
> > > >>> wrote:
> > > >>>
> > > >>> Congrats Dong!
> > > >>>
> > > >>> Eno
> > > >>>
> > > >>> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu 
> wrote:
> > > >>>
> > > >>> Congratulation Dong!
> > > 
> > >  On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
> > >  viktorsomo...@gmail.com>
> > >  wrote:
> > > 
> > >  Congrats Dong! :)
> > > >
> > > > On Tue, Aug 21, 2018 at 10:09 AM James Cheng <
> wushuja...@gmail.com
> > >
> > > >
> > >  wrote:
> > > 
> > > > Congrats Dong!
> > > >>
> > > >> -James
> > > >>
> > > >> On Aug 20, 2018, at 3:54 AM, Ismael Juma 
> > wrote:
> > > >>>
> > > >>> Hi everyone,
> > > >>>
> > > >>> Dong Lin became a committer in March 2018. Since then, he has
> > > >>>
> > > >> remained
> > > 
> > > > active in the community and contributed a number of patches,
> > reviewed
> > > >>> several pull requests and participated in numerous KIP
> > > discussions. I
> > > >>>
> > > >> am
> > > >
> > > >> happy to announce that Dong is now a member of the
> > > >>> Apache Kafka PMC.
> > > >>>
> > > >>> Congratulation Dong! Looking forward to your future
> > contributions.
> > > >>>
> > > >>> Ismael, on behalf of the Apache Kafka PMC
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Satish Duggana]

+1

On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:

> +1
>  Original message From: Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
> +1
>
> Thanks for the KIP!
>
> On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on this KIP (
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=87298242)
> > which aims to refactor ExtendedSerializer/Serializer and
> > ExtendedDeserializer/Deserializer.
> >
> > To summarize what's the motivation:
> >
> > When headers were introduced by KIP-82 the ExtendedSerializer and
> > ExtendedDeserializer classes were created in order to keep interface
> > compatibility but still add `T deserialize(String topic, Headers headers,
> > byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> > data);` methods that consume the headers for
> serialization/deserialization.
> > The reason for doing so was that Kafka at that time needed be compatbile
> > with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423)
> we'll
> > try consolidate the way we're using these in a backward compatible
> fashion:
> > deprecating the Extended* classes and moving the aforementioned methods
> up
> > in the class hierarchy.
> >
> > I'd be happy to get votes or additional feedback on this.
> >
> > Viktor
> >
>

Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

[Adam Bellemare]

Hi All

I am looking for performance metrics related to Kafka Streams and KSQL. I
have been scouring various blogs, including the confluent one, looking for
any current performance metrics or benchmarks, official or otherwise, on
both Kafka Streams and KSQL for Kafka 2.x +. Unfortunately, almost
everything I am finding is 0.x.

In this particular blog post on KSQL, there is the following quotation:

> For example, our soak testing cluster has racked up over 1,000 hours and
runs KSQL workloads 24×7. The performance tests we conduct allow us to
understand performance characteristics of stateless and stateful KSQL
queries. We currently run over 42 different tests that collect more than
700 metrics.

I assume that there is also some information related to Kafka Streams in
similar tests. Does anyone know where I can find these results? Or does
anyone have any blog posts or other materials that look at the performance
of either one of these for Kafka 2.x ?

For context, I am asking this question to get a better understanding of
current Kafka Streams / KSQL performance, such that contributors can
understand the prioritization of performance-related improvements vs.
feature-related improvements.

Thanks
Adam

Re: Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

[Adam Bellemare]

Blog post in question:
https://www.confluent.io/blog/ksql-february-release-streaming-sql-for-apache-kafka/

On Wed, Aug 22, 2018 at 10:01 AM, Adam Bellemare 
wrote:

> Hi All
>
> I am looking for performance metrics related to Kafka Streams and KSQL. I
> have been scouring various blogs, including the confluent one, looking for
> any current performance metrics or benchmarks, official or otherwise, on
> both Kafka Streams and KSQL for Kafka 2.x +. Unfortunately, almost
> everything I am finding is 0.x.
>
> In this particular blog post on KSQL, there is the following quotation:
>
> > For example, our soak testing cluster has racked up over 1,000 hours
> and runs KSQL workloads 24×7. The performance tests we conduct allow us to
> understand performance characteristics of stateless and stateful KSQL
> queries. We currently run over 42 different tests that collect more than
> 700 metrics.
>
> I assume that there is also some information related to Kafka Streams in
> similar tests. Does anyone know where I can find these results? Or does
> anyone have any blog posts or other materials that look at the performance
> of either one of these for Kafka 2.x ?
>
> For context, I am asking this question to get a better understanding of
> current Kafka Streams / KSQL performance, such that contributors can
> understand the prioritization of performance-related improvements vs.
> feature-related improvements.
>
> Thanks
> Adam
>

Re: [DISCUSS] KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metrics

[Kevin Lu]

Bumping this as it has been a week without any responses.

Regards,
Kevin

On Wed, Aug 15, 2018 at 11:40 PM Kevin Lu  wrote:

> Hi friends! :)
>
> I believe we currently have a gap in KafkaConsumer metrics for errors
> since the KafkaConsumer is complex and are many places where things can go
> wrong. Currently, these failures are logged and certain ones can be
> inferred from the existing metrics (ex. heartbeat-rate).
>
> This KIP seeks to improve monitoring and alerting for the consumer by
> providing metrics for the Fetcher class.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-356%3A+Add+KafkaConsumer+fetch-error-rate+and+fetch-error-total+metrics
>
> There are also a few other places in the Fetcher where errors may happen
> (parsing completed fetches, offset requests, etc) but it may be appropriate
> to monitor them in separate metrics.
>
> Any thoughts?
>
> Thanks!
>
> Regards,
> Kevin
>
>
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Jason Gustafson]

Hey Viktor,

This is a nice cleanup. Just a couple quick questions:

1. Rather than returning null for the default `deserialize(topic, data)`,
would it be better to throw UnsupportedOperationException? I assume that
internally we'll always invoke the api which takes headers. Similarly for
`serialize(topic, data)`.
2. Would it make sense to have default no-op implementations for
`configure` and `close`?

Thanks,
Jason

On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana 
wrote:

> +1
>
> On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:
>
> > +1
> >  Original message From: Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00) To:
> > dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> > ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
> > +1
> >
> > Thanks for the KIP!
> >
> > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on this KIP (
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=87298242)
> > > which aims to refactor ExtendedSerializer/Serializer and
> > > ExtendedDeserializer/Deserializer.
> > >
> > > To summarize what's the motivation:
> > >
> > > When headers were introduced by KIP-82 the ExtendedSerializer and
> > > ExtendedDeserializer classes were created in order to keep interface
> > > compatibility but still add `T deserialize(String topic, Headers
> headers,
> > > byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> > > data);` methods that consume the headers for
> > serialization/deserialization.
> > > The reason for doing so was that Kafka at that time needed be
> compatbile
> > > with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423)
> > we'll
> > > try consolidate the way we're using these in a backward compatible
> > fashion:
> > > deprecating the Extended* classes and moving the aforementioned methods
> > up
> > > in the class hierarchy.
> > >
> > > I'd be happy to get votes or additional feedback on this.
> > >
> > > Viktor
> > >
> >
>

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

[Manikumar]

Hi Viktor,
We already have a method in Authorizer interface to get acls for a given
principal.
We will use this method to fetch acls and filter the results for requested
Resources.
Authorizer {
   def getAcls(principal: KafkaPrincipal): Map[Resource, Set[Acl]]
}
Currently AdminClient API doesn't have a API to fetch acls for a given
principal.
So while using AclCommand with AdminClient API (KIP-332), we just filter
the results returned
from describeAcls API. We can add new AdminClient API/new
DescribeAclsRequest if required in future.

Updated the KIP. Thanks for the review.

Thanks,

On Wed, Aug 22, 2018 at 5:53 PM Viktor Somogyi-Vass 
wrote:

> Hi Manikumar,
>
> Implementation-wise is it just a filter over the returned ACL listing or do
> you plan to add new methods to the Authorizer as well?
>
> Thanks,
> Viktor
>
> On Fri, Aug 17, 2018 at 9:18 PM Priyank Shah 
> wrote:
>
> > +1(non-binding)
> >
> > Thanks.
> > Priyank
> >
> > On 8/16/18, 6:01 AM, "Manikumar"  wrote:
> >
> > Hi all,
> >
> > I have created a minor KIP to add support to list ACLs per principal
> > using
> > AclCommand (kafka-acls.sh)
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-357%3A++Add+support+to+list+ACLs+per+principal
> >
> > Please take a look.
> >
> > Thanks,
> >
> >
> >
>

[DISCUSS] KIP-362: Dynamic Session Window Support

[Lei Chen]

Hi All,

I created a KIP to add dynamic gap session window support to Kafka Streams
DSL.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-362%3A+Support+dynamic+gap+session+window

Please take a look,

Thanks,
Lei

[jira] [Resolved] (KAFKA-6343) OOM as the result of creation of 5k topics

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-6343.

   Resolution: Fixed
 Assignee: Alex Dunayevsky
Fix Version/s: 2.1.0

Merged the PR to trunk.

> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.1, 0.11.0.2, 1.0.0
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>Assignee: Alex Dunayevsky
>Priority: Major
> Fix For: 2.1.0
>
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays. 
> Wait until brokers will finish loading them. This will actually never happen, 
> since all brokers will go down one by one after approx 10-15 minutes or more, 
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
>  
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
>   val zkConnect = "grid978:2185"
>   var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
>   for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
>   topic = s"${topic.toString}",
>   partitions= 10,
>   replicationFactor = 2,
>   zkUtils   = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
>   }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to 
> start again. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7325) Support dynamic gap session window

[Lei Chen (JIRA)]

Lei Chen created KAFKA-7325:
---

 Summary: Support dynamic gap session window
 Key: KAFKA-7325
 URL: https://issues.apache.org/jira/browse/KAFKA-7325
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Lei Chen


Currently, Kafka Streams DSL only supports fixed-gap session window. However, 
in some circumstances, the gap is more dynamic and can vary depending on other 
factors: the statistical aggregation result, liquidity of the records, etc. In 
such cases, allowing the user to define a dynamic-gap session is important. 
[KIP-362|https://cwiki.apache.org/confluence/display/KAFKA/KIP-362%3A+Support+dynamic+gap+session+window]
 is created to address this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (KAFKA-7325) Support dynamic gap session window

[Daniel Roy]

unsuscribe

On Wed, Aug 22, 2018 at 7:22 PM, Lei Chen (JIRA)  wrote:

> Lei Chen created KAFKA-7325:
> ---
>
>  Summary: Support dynamic gap session window
>  Key: KAFKA-7325
>  URL: https://issues.apache.org/jira/browse/KAFKA-7325
>  Project: Kafka
>   Issue Type: Improvement
>   Components: streams
> Reporter: Lei Chen
>
>
> Currently, Kafka Streams DSL only supports fixed-gap session window.
> However, in some circumstances, the gap is more dynamic and can vary
> depending on other factors: the statistical aggregation result, liquidity
> of the records, etc. In such cases, allowing the user to define a
> dynamic-gap session is important. [KIP-362|https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-362%3A+Support+dynamic+gap+session+window] is
> created to address this.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>



-- 



you belong, find where with AI

+44 747 247 8328
dan...@osero.io

[DISCUSS] KIP-363: Allow performance tools to print final results to output file

[Attila Sasvári]

Hi all,

I have created a minor KIP to allow consumer and producer performance tools
to print final results to output file in CSV format.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file

Please take a look and share your thoughts!

Thanks,
Attila

[jira] [Created] (KAFKA-7326) Let KStream.print() to flush on each printed line

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-7326:


 Summary: Let KStream.print() to flush on each printed line
 Key: KAFKA-7326
 URL: https://issues.apache.org/jira/browse/KAFKA-7326
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


Today, {{KStream.print()}} is implemented as a special "foreach" function as 
below:

{code}
@Override
public void apply(final K key, final V value) {
final String data = String.format("[%s]: %s", label, mapper.apply(key, 
value));
printWriter.println(data);
}
{code}

Note that since {{this.printWriter = new PrintWriter(new 
OutputStreamWriter(outputStream, StandardCharsets.UTF_8));}}, without flushing 
the writer we do not guarantee that printed lines are written to the underlying 
`outputStream` in time.

Since {{KStream.print()}} is mainly for debugging / testing / demoing purposes, 
not for performance, I think it is okay to enforce auto flushing.

This would include:

1. set {{autoFlush}} in the constructor of printWriter.
2. document in java-docs of {{KStream.print}} that this is for debug / testing 
purposes only, and it will try to flush on each record print, and hence should 
not be used for production usage if performance requirement is key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Jason Gustafson]

Hey Dongjin,

Yeah that's right. For what it's worth, librdkafka also appears to handle
unexpected error codes. I expect that most client implementations would
either pass through the raw type or convert to an enum using something like
what the java client does. Since we're expecting the client to fail anyway,
I'm probably in favor of using the UNSUPPORTED_COMPRESSION_TYPE error code.

-Jason

On Wed, Aug 22, 2018 at 1:46 AM, Dongjin Lee  wrote:

> Jason and Ismael,
>
> It seems like the only thing we need to regard if we define a new error
> code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the implementation of
> the other language clients, right? At least, this strategy causes any
> problem for Java client. Do I understand correctly?
>
> Thanks,
> Dongjin
>
> On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee  wrote:
>
> > Jason,
> >
> > > I think we would only use this error code when we /know/ that zstd was
> > in use and the client doesn't support it? This is true if either 1) the
> > message needs down-conversion and we encounter a zstd compressed message,
> > or 2) if the topic is explicitly configured to use zstd.
> >
> > Yes, it is right. And you know, the case 1 includes 1.a) old clients'
> > request v0, v1 records or 1.b) implicit zstd, the compression type of
> > "producer" with Zstd compressed data.
> >
> > > However, if the compression type is set to "producer," then the fetched
> > data may or may not be compressed with zstd. In this case, we return the
> > data to the client and expect it to fail parsing. Is that correct?
> >
> > Exactly.
> >
> > Following your message, I reviewed the implementation of
> > `KafkaApis#handleFetchRequest,` which handles the fetch request. And
> found
> > that the information we can use is like the following:
> >
> > 1. Client's fetch request version. (`versionId` variable)
> > 2. Log's compression type. (`logConfig` variable)
> >
> > We can't detect the actual compression type of the data, unless we
> inspect
> > the `RecordBatch` included in the `Records` instance (i.e.,
> > `unconvertedRecords` variable.) Since it requires some performance issue,
> > it is not our option - in short, we can't be sure if given chunks of data
> > are compressed with zstd or not.
> >
> > So, conclusion: we can return an error in the case of 1.a and 2 easily,
> > with the information above. In the case 1.b (implicit zstd), we can just
> > return the data by do nothing special and expect it to fail parsing.
> >
> > Thanks,
> > Dongjin
> >
> > On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma  wrote:
> >
> >> Jason, that's an interesting point regarding the Java client. Do we know
> >> what clients in other languages do in these cases?
> >>
> >> Ismael
> >>
> >> On Tue, 21 Aug 2018, 17:30 Jason Gustafson,  wrote:
> >>
> >> > Hi Dongjin,
> >> >
> >> > One of the complications is that old versions of the API will not
> >> expect a
> >> > new error code. However, since we expect this to be a fatal error
> anyway
> >> > for old clients, it may still be more useful to return the correct
> error
> >> > code. For example, the Kafka clients use the following code to convert
> >> the
> >> > error code:
> >> >
> >> > public static Errors forCode(short code) {
> >> > Errors error = codeToError.get(code);
> >> > if (error != null) {
> >> > return error;
> >> > } else {
> >> > log.warn("Unexpected error code: {}.", code);
> >> > return UNKNOWN_SERVER_ERROR;
> >> > }
> >> > }
> >> >
> >> > If we return an unsupported error code, it will be converted to an
> >> UNKNOWN
> >> > error, but at least we will get the message in the log with the
> correct
> >> > code. That seems preferable to returning a misleading error code. So I
> >> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even
> for
> >> > older versions.
> >> >
> >> > Also, one question just to check my understanding. I think we would
> only
> >> > use this error code when we /know/ that zstd was in use and the client
> >> > doesn't support it? This is true if either 1) the message needs
> >> > down-conversion and we encounter a zstd compressed message, or 2) if
> the
> >> > topic is explicitly configured to use zstd. However, if the
> compression
> >> > type is set to "producer," then the fetched data may or may not be
> >> > compressed with zstd. In this case, we return the data to the client
> and
> >> > expect it to fail parsing. Is that correct?
> >> >
> >> > Thanks,
> >> > Jason
> >> >
> >> >
> >> >
> >> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee 
> >> wrote:
> >> >
> >> > > Ismael, Jason and all,
> >> > >
> >> > > I rewrote the backward compatibility strategy & its alternatives
> like
> >> > > following, based on Ismael & Jason's comments. Since it is not
> >> updated to
> >> > > the wiki yet, don't hesitate to give me a message if you have any
> >> opinion
> >> > > on it.
> >> > >
> >> > > ```
> >> > > *Backward Compatibility*
> >> > >
> >> > > 

Re: Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

[Guozhang Wang]

Hello Adam,

Thanks for your interests in working on Kafka Streams / KSQL potential
performance improvements (I thought the non-key joining will take most of
your time :P )

Currently there is no published performance numbers for latest versions of
Streams AFAIK. Personally I ran the Streams SimpleBenchmark (
https://github.com/apache/kafka/blob/trunk/tests/
kafkatest/benchmarks/streams/streams_simple_benchmark_test.py) and profile
it if necessary trying to figure out the performance bottlenecks. If you
are interested you can follow similar approaches, there are also some JIRAs
open for potential performance improvements as well:

https://issues.apache.org/jira/issues/?jql=project%20%
3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%
20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%
20AND%20component%20%3D%20%22streams%22%20%20AND%
20labels%20%3D%20performance%20%20


Guozhang

On Wed, Aug 22, 2018 at 7:02 AM, Adam Bellemare 
wrote:

> Blog post in question:
> https://www.confluent.io/blog/ksql-february-release-streamin
> g-sql-for-apache-kafka/
>
> On Wed, Aug 22, 2018 at 10:01 AM, Adam Bellemare  >
> wrote:
>
> > Hi All
> >
> > I am looking for performance metrics related to Kafka Streams and KSQL. I
> > have been scouring various blogs, including the confluent one, looking
> for
> > any current performance metrics or benchmarks, official or otherwise, on
> > both Kafka Streams and KSQL for Kafka 2.x +. Unfortunately, almost
> > everything I am finding is 0.x.
> >
> > In this particular blog post on KSQL, there is the following quotation:
> >
> > > For example, our soak testing cluster has racked up over 1,000 hours
> > and runs KSQL workloads 24×7. The performance tests we conduct allow us
> to
> > understand performance characteristics of stateless and stateful KSQL
> > queries. We currently run over 42 different tests that collect more than
> > 700 metrics.
> >
> > I assume that there is also some information related to Kafka Streams in
> > similar tests. Does anyone know where I can find these results? Or does
> > anyone have any blog posts or other materials that look at the
> performance
> > of either one of these for Kafka 2.x ?
> >
> > For context, I am asking this question to get a better understanding of
> > current Kafka Streams / KSQL performance, such that contributors can
> > understand the prioritization of performance-related improvements vs.
> > feature-related improvements.
> >
> > Thanks
> > Adam
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Dongjin Lee]

Jason,

Great. +1 for UNSUPPORTED_COMPRESSION_TYPE.

Best,
Dongjin

On Thu, Aug 23, 2018 at 8:19 AM Jason Gustafson  wrote:

> Hey Dongjin,
>
> Yeah that's right. For what it's worth, librdkafka also appears to handle
> unexpected error codes. I expect that most client implementations would
> either pass through the raw type or convert to an enum using something like
> what the java client does. Since we're expecting the client to fail anyway,
> I'm probably in favor of using the UNSUPPORTED_COMPRESSION_TYPE error code.
>
> -Jason
>
> On Wed, Aug 22, 2018 at 1:46 AM, Dongjin Lee  wrote:
>
> > Jason and Ismael,
> >
> > It seems like the only thing we need to regard if we define a new error
> > code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the implementation of
> > the other language clients, right? At least, this strategy causes any
> > problem for Java client. Do I understand correctly?
> >
> > Thanks,
> > Dongjin
> >
> > On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee  wrote:
> >
> > > Jason,
> > >
> > > > I think we would only use this error code when we /know/ that zstd
> was
> > > in use and the client doesn't support it? This is true if either 1) the
> > > message needs down-conversion and we encounter a zstd compressed
> message,
> > > or 2) if the topic is explicitly configured to use zstd.
> > >
> > > Yes, it is right. And you know, the case 1 includes 1.a) old clients'
> > > request v0, v1 records or 1.b) implicit zstd, the compression type of
> > > "producer" with Zstd compressed data.
> > >
> > > > However, if the compression type is set to "producer," then the
> fetched
> > > data may or may not be compressed with zstd. In this case, we return
> the
> > > data to the client and expect it to fail parsing. Is that correct?
> > >
> > > Exactly.
> > >
> > > Following your message, I reviewed the implementation of
> > > `KafkaApis#handleFetchRequest,` which handles the fetch request. And
> > found
> > > that the information we can use is like the following:
> > >
> > > 1. Client's fetch request version. (`versionId` variable)
> > > 2. Log's compression type. (`logConfig` variable)
> > >
> > > We can't detect the actual compression type of the data, unless we
> > inspect
> > > the `RecordBatch` included in the `Records` instance (i.e.,
> > > `unconvertedRecords` variable.) Since it requires some performance
> issue,
> > > it is not our option - in short, we can't be sure if given chunks of
> data
> > > are compressed with zstd or not.
> > >
> > > So, conclusion: we can return an error in the case of 1.a and 2 easily,
> > > with the information above. In the case 1.b (implicit zstd), we can
> just
> > > return the data by do nothing special and expect it to fail parsing.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma 
> wrote:
> > >
> > >> Jason, that's an interesting point regarding the Java client. Do we
> know
> > >> what clients in other languages do in these cases?
> > >>
> > >> Ismael
> > >>
> > >> On Tue, 21 Aug 2018, 17:30 Jason Gustafson, 
> wrote:
> > >>
> > >> > Hi Dongjin,
> > >> >
> > >> > One of the complications is that old versions of the API will not
> > >> expect a
> > >> > new error code. However, since we expect this to be a fatal error
> > anyway
> > >> > for old clients, it may still be more useful to return the correct
> > error
> > >> > code. For example, the Kafka clients use the following code to
> convert
> > >> the
> > >> > error code:
> > >> >
> > >> > public static Errors forCode(short code) {
> > >> > Errors error = codeToError.get(code);
> > >> > if (error != null) {
> > >> > return error;
> > >> > } else {
> > >> > log.warn("Unexpected error code: {}.", code);
> > >> > return UNKNOWN_SERVER_ERROR;
> > >> > }
> > >> > }
> > >> >
> > >> > If we return an unsupported error code, it will be converted to an
> > >> UNKNOWN
> > >> > error, but at least we will get the message in the log with the
> > correct
> > >> > code. That seems preferable to returning a misleading error code.
> So I
> > >> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even
> > for
> > >> > older versions.
> > >> >
> > >> > Also, one question just to check my understanding. I think we would
> > only
> > >> > use this error code when we /know/ that zstd was in use and the
> client
> > >> > doesn't support it? This is true if either 1) the message needs
> > >> > down-conversion and we encounter a zstd compressed message, or 2) if
> > the
> > >> > topic is explicitly configured to use zstd. However, if the
> > compression
> > >> > type is set to "producer," then the fetched data may or may not be
> > >> > compressed with zstd. In this case, we return the data to the client
> > and
> > >> > expect it to fail parsing. Is that correct?
> > >> >
> > >> > Thanks,
> > >> > Jason
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee 
> > >> wrote:
> 

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

[Matthias J. Sax]

Thanks a lot for the KIP.

From my understanding, the idea of the KIP is to improve the public API
at DSL level. However, not all public methods listed are part of DSL
level API, but part of runtime API. Those methods are called during
processing and are on the hot code path. I am not sure, if we want to
update those methods. We should carefully think about this, and consider
to keep Long/long type to keep runtime overhead small. Note, that the
methods I mention are not required to specify a program using the DSL
and thus is questionable if the DSL API would be improved if we change
the methods.

It's unfortunate, that some part of the public API stretch the DSL
builder part as well as the runtime part...

This affects the following methods (please double check if I missed any):

 - Windows#windowsFor()
 - Window#start()
 - Window#end()
 - JoinWindows#windowFor()
 - SessionWindows#inactivitiyGap()
 - TimeWindows#windowFor()
 - UnlimitedWindows#windowFor()
 - ProcessorContext#schedule()
 - ReadOnlyWindowStore#fetch() (2x) and #fetchAll()
 - SessionStore#findSessions() (2x)

maybe
 - TimeWindowedDeserializer#getWindowSize() (it's unused atm, but I
could imagine that it might be use on the hot code path in the furture)

So methods have "dual" use and might be called externally and internally:

 - Window#start()
 - Window#end()
 - ReadOnlyWindowStore#fetch() (2x) and #fetchAll()
 - SessionStore#findSessions() (2x)

Thus, it might make sense to keep old and just add new ones? End users
can use the "nicer" new ones, while we can still use the existing ones
internally? Not sure if it would be possible to keep the old ones
without exposing them as public API?

Let me know what you think about this.


-Matthias



On 8/21/18 11:41 PM, Nikolay Izhikov wrote:
> Dear, commiters.
> 
> Please, pay attention to this KIP and share your opinion.
> 
> В Вт, 21/08/2018 в 11:14 -0500, John Roesler пишет:
>> I'll solicit more reviews. Let's get at least one committer to chime in
>> before we start a vote (since we need their approval anyway).
>> -John
>>
>> On Mon, Aug 20, 2018 at 12:39 PM Nikolay Izhikov 
>> wrote:
>>
>>> Hello, Ted.
>>>
>>> Thanks for the comment.
>>>
>>> I've edit KIP and change proposal to `windowSize`.
>>>
>>> Guys, any other comments?
>>>
>>>
>>> В Вс, 19/08/2018 в 14:57 -0700, Ted Yu пишет:
 bq. // or just Duration windowSize();

 +1 to the above choice.
 The duration is obvious from the return type. For getter methods, we
>>>
>>> don't
 use get as prefix (as least for new code).

 Cheers

 On Sun, Aug 19, 2018 at 8:03 AM Nikolay Izhikov 
>>>
>>> wrote:

> Hello, John.
>
> Thank you very much for your feedback!
> I've addressed all your comments.
> Please, see my answers and let my know is anything in KIP [1] needs to
>>>
>>> be
> improved.
>
>> The correct choice is actually "Instant", not> "LocalDateTime"
>
> I've changed the methods proposed in KIP [1] to use Instant.
>
>> I noticed some recent APIs are> missing (see KIP-328)
>> those APIs were just added and have never been released... you can
>>>
>>> just
>
> replace them.
>
> I've added new methods to KIP [1].
> Not released methods marked for remove.
>
>> any existing method that's already deprecated, don't bother
>
> transitioning it to Duration.
>
> Fixed.
>
>> IllegalArgumentException... we should plan to mention this in the
>
> javadoc for those methods.
>
> Got it.
>
>> In Stores, windowSize and segmentInterval should also be durations.
>
> Fixed.
>
>> StreamsMetrics, recordLatency ... this one is better left alone.
>
> OK. I removed this method from KIP [1].
>
> Two more questions question about implementation:
>
> 1. We have serveral methods without parameters.
> In java we can't have two methods with parameters with the same name.
> It wouldn't compile.
> So we have to rename new methods. Please, see suggested names and share
> your thoughts:
>
> Windows {
> long size() -> Duration windowSize();
> }
>
> Window {
> long start() -> Instant startTime();
> long end() -> Instant endTime();
> }
>
> SessionWindows {
> long inactivityGap() -> Duration inactivityGapDuration();
> }
>
> TimeWindowedDeserializer {
> Long getWindowSize() -> Duration getWindowSizeDuration(); // or
>>>
>>> just
> Duration windowSize();
> }
>
> SessionBytesStoreSupplier {
> long retentionPeriod() -> Duration retentionPeriodDuration();
> }
>
> WindowBytesStoreSupplier {
> long windowSize() -> Duration windowSizeDuration();
> long retentionPeriod() -> Duration retentionPeriodDuration();
> }
>
> 2. Do we want to use Duration and Instant inside API implementations?
>
> IGNITE-7277: "Durations potential

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Dhruvil Shah]

Hi Matthias,

Do you mean something like "suppress.auto.create.topic"? I am leaning a bit
towards "allow.auto.create.topics" but I don't have a strong preference
either. Let's wait to hear if anyone else has an opinion on this.

Thanks,
Dhruvil

On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax 
wrote:

> Thanks for the KIP Dhruvil!
>
> I agree with Jason's comment. An alternative might be to use "suppress"
> what would revert the logic of "allow". Not sure which one is more
> intuitive and I am fine with both (no personal preference). Just wanted
> to mention it as an alternative.
>
> Don't have any further comments/question so far.
>
>
> -Matthias
>
>
>
> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> > Hey Dhruvil,
> >
> > I would suggest using the verb "allow" rather than "enable. The consumer
> > cannot enable auto topic creation because it is configured on the broker.
> > All it can do is prevent it from happening if it is enabled.
> >
> > -Jason
> >
> > On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
> wrote:
> >
> >> Hi,
> >>
> >> I would like to start discussion on KIP-361 that proposes we add a
> consumer
> >> configuration to disable auto topic creation.
> >>
> >> Link to the KIP:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> >> Configuration+to+Disable+Auto+Topic+Creation
> >>
> >> Suggestions and feedback are welcome!
> >>
> >> Thanks,
> >> Dhruvil
> >>
> >
>
>

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ted Yu]

We may also consider :

"suppress.auto.topic.creation"

or

"allow.auto.topic.creation"

w.r.t. suppress or allow, I don't have strong opinion either. It's just a
matter of choosing the proper default value.

Cheers

On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah  wrote:

> Hi Matthias,
>
> Do you mean something like "suppress.auto.create.topic"? I am leaning a bit
> towards "allow.auto.create.topics" but I don't have a strong preference
> either. Let's wait to hear if anyone else has an opinion on this.
>
> Thanks,
> Dhruvil
>
> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax 
> wrote:
>
> > Thanks for the KIP Dhruvil!
> >
> > I agree with Jason's comment. An alternative might be to use "suppress"
> > what would revert the logic of "allow". Not sure which one is more
> > intuitive and I am fine with both (no personal preference). Just wanted
> > to mention it as an alternative.
> >
> > Don't have any further comments/question so far.
> >
> >
> > -Matthias
> >
> >
> >
> > On 8/21/18 4:42 PM, Jason Gustafson wrote:
> > > Hey Dhruvil,
> > >
> > > I would suggest using the verb "allow" rather than "enable. The
> consumer
> > > cannot enable auto topic creation because it is configured on the
> broker.
> > > All it can do is prevent it from happening if it is enabled.
> > >
> > > -Jason
> > >
> > > On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> I would like to start discussion on KIP-361 that proposes we add a
> > consumer
> > >> configuration to disable auto topic creation.
> > >>
> > >> Link to the KIP:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> > >> Configuration+to+Disable+Auto+Topic+Creation
> > >>
> > >> Suggestions and feedback are welcome!
> > >>
> > >> Thanks,
> > >> Dhruvil
> > >>
> > >
> >
> >
>

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Brandon Kirchner]

“allow=false” seems a bit more intuitive to me than “suppress=false”

Brandon

> On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
> 
> We may also consider :
> 
> "suppress.auto.topic.creation"
> 
> or
> 
> "allow.auto.topic.creation"
> 
> w.r.t. suppress or allow, I don't have strong opinion either. It's just a
> matter of choosing the proper default value.
> 
> Cheers
> 
>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah  wrote:
>> 
>> Hi Matthias,
>> 
>> Do you mean something like "suppress.auto.create.topic"? I am leaning a bit
>> towards "allow.auto.create.topics" but I don't have a strong preference
>> either. Let's wait to hear if anyone else has an opinion on this.
>> 
>> Thanks,
>> Dhruvil
>> 
>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax 
>> wrote:
>> 
>>> Thanks for the KIP Dhruvil!
>>> 
>>> I agree with Jason's comment. An alternative might be to use "suppress"
>>> what would revert the logic of "allow". Not sure which one is more
>>> intuitive and I am fine with both (no personal preference). Just wanted
>>> to mention it as an alternative.
>>> 
>>> Don't have any further comments/question so far.
>>> 
>>> 
>>> -Matthias
>>> 
>>> 
>>> 
 On 8/21/18 4:42 PM, Jason Gustafson wrote:
 Hey Dhruvil,
 
 I would suggest using the verb "allow" rather than "enable. The
>> consumer
 cannot enable auto topic creation because it is configured on the
>> broker.
 All it can do is prevent it from happening if it is enabled.
 
 -Jason
 
 On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
>>> wrote:
 
> Hi,
> 
> I would like to start discussion on KIP-361 that proposes we add a
>>> consumer
> configuration to disable auto topic creation.
> 
> Link to the KIP:
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> Configuration+to+Disable+Auto+Topic+Creation
> 
> Suggestions and feedback are welcome!
> 
> Thanks,
> Dhruvil
> 
 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Dhruvil Shah]

Hi Ismael,

Thanks for the comments. Replies below.

1. We could throw an InvalidConfigurationException at run-time either when
building the MetadataRequest or when an ApiVersions response is received.
Because this is a configuration problem, I think users would likely see the
exception on the first call to poll. Does this sound reasonable?

2. I think we could log a warning when KafkaConsumer is instantiated with
the default configuration. Is there a plan to deprecate the server
configuration? Is so, we could employ the same strategy in terms of when to
switch the default value and then eventually remove the config altogether.

- Dhruvil

On Tue, Aug 21, 2018 at 7:36 PM Ismael Juma  wrote:

> Thanks for the KIP. A few questions/comments:
>
> 1. It seems hard to reason about if we just disregard the config for older
> brokers. Maybe we should throw an error if the brokers don't support it and
> let users explicitly change the config if they want to.
>
> 2. We probably want to switch the default and eventually remove this config
> in a future version. What's the path to making that happen? One option
> would be to warn if people rely on the default as a first step (or warn
> every time it's used).
>
> Ismael
>
> On 21 Aug 2018 3:56 pm, "Dhruvil Shah"  wrote:
>
> Hi,
>
> I would like to start discussion on KIP-361 that proposes we add a consumer
> configuration to disable auto topic creation.
>
> Link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>
> Suggestions and feedback are welcome!
>
> Thanks,
> Dhruvil
>

[jira] [Created] (KAFKA-7327) kafak主节点cpu 内存持续飙高，不回收，最后服务挂掉问题？

[ruiliang (JIRA)]

ruiliang created KAFKA-7327:
---

 Summary: kafak主节点cpu 内存持续飙高，不回收，最后服务挂掉问题？
 Key: KAFKA-7327
 URL: https://issues.apache.org/jira/browse/KAFKA-7327
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 1.1.0
 Environment: linux centos7 
Reporter: ruiliang


Xmlipcregsvc-> 172.18.58.184:60686 (CLOSE_WAIT) has many such ports to close 
waiting, which is the application connection side.Why wait?Memory nor recycling 
three services are 2 nuclear 4 gb of memory, this before is kafka3G, found that 
memory, the heap memory, and then I will limit kfaka up to 2 g, but the master 
node to run after a period of time, and submitted to the heap memory and heap 
memory leak, I free -m looked at it and really have 100 MB of memory, I don't 
know where memory use, kafka made up 80% of the process of memory, CPU by more 
than 100%, what reason is this?The configuration parameters have been checked 
with the official website. The default is not acceptable.

XmlIpcRegSvc->172.18.58.184:60686 (CLOSE_WAIT) 
有很多这个样的端口关闭等待，这是应用连接端。为什么一直等待呢？内存也没有回收 我3台服务是 2核 4G 
内存，这之前给的是kafka3G,发现内存没了，报堆外内存溢出，然后我就限制kfaka最大为2G，但主节点跑一段时间后，又报堆内存溢出和堆外内存溢出，我free
 -m看了一下，内存确实还有100MB了，不知内存用在那里，kafka 这个进程暂用完了 80%的内存，cpu 
100%多了，这是什么原因呢?配置参数和官网核对了一下，全用默认的也不行，

` 1772 liandong 20 0 6398984 2.146g 16112 S 101.3 58.0 93:59.72 
/usr/local/jdk1.8/bin/java -Xmx2G -Xms1G -server -XX:+UseG1GC 
-XX:+HeapDumpOnOutOfMemoryError -XX:MaxGCPauseMillis=20 
-XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent 
-Djava.awt.headless=true -XX:MaxDirectMemorySize=512m 
-Xloggc:/data/kafka/bin/../logs/kafkaSer+...`

kafka server.log log
`[2018-08-23 07:56:11,788] INFO [GroupCoordinator 0]: Stabilized group 
consumer.web.log generation 268 (__consumer_offsets-24) 
(kafka.coordinator.group.GroupCoordinator)
[2018-08-23 07:56:12,054] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2018-08-23 07:56:13,846] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2018-08-23 07:56:15,673] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:694)
 at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
 at sun.nio.ch.IOUtil.read(IOUtil.java:195)
 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
 at 
org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:104)
 at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:145)
 at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
 at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
 at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
 at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:557)
 at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:495)
 at org.apache.kafka.common.network.Selector.poll(Selector.java:424)
 at kafka.network.Processor.poll(SocketServer.scala:628)
 at kafka.network.Processor.run(SocketServer.scala:545)
 at java.lang.Thread.run(Thread.java:748)
[2018-08-23 07:56:16,379] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space`


172.18.58.184:speedtrace (CLOSE_WAIT) 172.18.58.184 是 kafka client connect
lsof -i | grep java 
`java 1772 liandong 83u IPv4 7990697 0t0 TCP *:36145 (LISTEN)
java 1772 liandong 84u IPv4 7990698 0t0 TCP *:9099 (LISTEN)
java 1772 liandong 85u IPv4 7990701 0t0 TCP *:40745 (LISTEN)
java 1772 liandong 100u IPv4 7990709 0t0 TCP 
prod_data_kafka_2:44688->prod_data_zk:eforward (ESTABLISHED)
java 1772 liandong 193u IPv4 7989816 0t0 TCP prod_data_kafka_2:XmlIpcRegSvc 
(LISTEN)
java 1772 liandong 224u IPv4 8019955 0t0 TCP 
prod_data_kafka_2:9099->172.18.58.184:47430 (ESTABLISHED)
java 1772 liandong 228u IPv4 8018733 0t0 TCP 
prod_data_kafka_2:XmlIpcRegSvc->172.18.58.184:33032 (CLOSE_WAIT)
java 1772 liandong 229u IPv4 7990859 0t0 TCP 
prod_data_kafka_2:XmlIpcRegSvc->172.18.58.184:51334 (ESTABLISHED)
java 1772 liandong 230u IPv4 8022506 0t0 TCP 
prod_data_kafka_2:36145->172.18.58.184:46112 (ESTABLISHED)
java 1772 liandong 235u IPv4 7989829 0t0 TCP 
prod_data_kafka_2:32976->prod_data_kafka_1:XmlIpcRegSvc (ESTABLISHED)
java 1772 liandong 236u IPv4 804 0t0 TCP 
prod_data_kafka_2:36145->172.18.58.184:46024 (ESTABLISHED)
java 1772 liandong 243u IPv4 7998548 0t0 TCP 
prod_data_kafka_2:XmlIpcRegSvc->prod_data_kafka_3:39816 (ESTABLISHED)
java 1772 liandong 247u IPv4 7998555 0t0 TCP 
prod_data_kafka_2:33206->prod_data_kafka_3:XmlIpcRegSvc (ESTABLIS

Jenkins build is back to normal : kafka-trunk-jdk8 #2917

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7328) Java version for Kafka

[Vasudevan Seshadri (JIRA)]

Vasudevan Seshadri created KAFKA-7328:
-

 Summary: Java version for Kafka
 Key: KAFKA-7328
 URL: https://issues.apache.org/jira/browse/KAFKA-7328
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.0
Reporter: Vasudevan Seshadri


Kafka-zookeeper runs very well on Java 1.6, while all other components like 
kafka-server or kafka-consumer or kafka-topic doesn't run on Java 1.6.

Error was: unsuported minor.major.version 52

I downloaded the latest version of Kafka from - 
[https://www.apache.org/dyn/closer.cgi?path=/kafka/2.0.0/kafka_2.11-2.0.0.tgz]

Suggestion: I think following has to be implemented

1. Quick Start Page (https://kafka.apache.org/quickstart) : Clearly mention the 
pre-requisites as Java 1.8+ 

2. Also, add checks in starter scripts to check Java version. If it's not a 
compatible version of Java, don't even start anything and call out clearly. 

While kafka-zookeeper was running without any issues and others had issues, it 
took a while to identify what the actual problem and had to search the 
documentation section 6.4 to understand Java 1.8 is required.

 

Thanks,

Vasu



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

1 seems reasonable to me, would be good to hear other opinions.

With regards to 2, yes, we discussed deprecating and eventually removing
the server config. But we'd like to make it easier to get auto create
behaviour on the producer (and maybe the consumer although that's less
clear) before we do that. Since the broker has to support older clients,
it's probably going to be a long time before we can remove it there though.

Ismael

On Wed, Aug 22, 2018 at 7:21 PM Dhruvil Shah  wrote:

> Hi Ismael,
>
> Thanks for the comments. Replies below.
>
> 1. We could throw an InvalidConfigurationException at run-time either when
> building the MetadataRequest or when an ApiVersions response is received.
> Because this is a configuration problem, I think users would likely see the
> exception on the first call to poll. Does this sound reasonable?
>
> 2. I think we could log a warning when KafkaConsumer is instantiated with
> the default configuration. Is there a plan to deprecate the server
> configuration? Is so, we could employ the same strategy in terms of when to
> switch the default value and then eventually remove the config altogether.
>
> - Dhruvil
>
> On Tue, Aug 21, 2018 at 7:36 PM Ismael Juma  wrote:
>
> > Thanks for the KIP. A few questions/comments:
> >
> > 1. It seems hard to reason about if we just disregard the config for
> older
> > brokers. Maybe we should throw an error if the brokers don't support it
> and
> > let users explicitly change the config if they want to.
> >
> > 2. We probably want to switch the default and eventually remove this
> config
> > in a future version. What's the path to making that happen? One option
> > would be to warn if people rely on the default as a first step (or warn
> > every time it's used).
> >
> > Ismael
> >
> > On 21 Aug 2018 3:56 pm, "Dhruvil Shah"  wrote:
> >
> > Hi,
> >
> > I would like to start discussion on KIP-361 that proposes we add a
> consumer
> > configuration to disable auto topic creation.
> >
> > Link to the KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >
> > Suggestions and feedback are welcome!
> >
> > Thanks,
> > Dhruvil
> >
>

[jira] [Created] (KAFKA-7329) Continuous warning message of LEADER_NOT_AVAILABLE

[Vasudevan Seshadri (JIRA)]

Vasudevan Seshadri created KAFKA-7329:
-

 Summary: Continuous warning message of LEADER_NOT_AVAILABLE
 Key: KAFKA-7329
 URL: https://issues.apache.org/jira/browse/KAFKA-7329
 Project: Kafka
  Issue Type: Bug
  Components: consumer, producer 
Affects Versions: 2.0.0
 Environment: macOS - High Sierra; Java 1.8
Reporter: Vasudevan Seshadri


I am running kafka version kafka_2.11-2.0.0. I have followed the instruction 
mentioned in quick start and was able to run zookeeper and server (broker with 
id=0) without any issues. Note: I have NOT changed any config file entries. 
Everything is same as downloaded by zip file

I also have created two topics as "test" and "topic_test"

Issue: Whenever I run producer or consumer and try to publish or consume on any 
of the above topics, following error is thrown continuously/non-stop: 

[2018-08-22 22:36:34,380] WARN [Producer clientId=console-producer] Error while 
fetching metadata with correlation id 1 : \{topic_test=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)

[2018-08-22 22:36:34,474] WARN [Producer clientId=console-producer] Error while 
fetching metadata with correlation id 2 : \{topic_test=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)

[2018-08-22 22:36:34,579] WARN [Producer clientId=console-producer] Error while 
fetching metadata with correlation id 3 : \{topic_test=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)

[2018-08-22 22:36:34,685] WARN [Producer clientId=console-producer] Error while 
fetching metadata with correlation id 4 : \{topic_test=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)

Am I missing any settings?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Matthias J. Sax]

@Dhruvil:

Yes, I had "suppress.auto.topic.creation" or "allow.auto.topic.creation"
in mind.


@Brandon:

I think it would be "allow=false" or "suppress=true" to enable this
feature. Ie, default values would be "allow=true" or "suppress=false".
Not sure if this changes any preferences for you.



-Matthias



On 8/22/18 8:29 PM, Ismael Juma wrote:
> 1 seems reasonable to me, would be good to hear other opinions.
> 
> With regards to 2, yes, we discussed deprecating and eventually removing
> the server config. But we'd like to make it easier to get auto create
> behaviour on the producer (and maybe the consumer although that's less
> clear) before we do that. Since the broker has to support older clients,
> it's probably going to be a long time before we can remove it there though.
> 
> Ismael
> 
> On Wed, Aug 22, 2018 at 7:21 PM Dhruvil Shah  wrote:
> 
>> Hi Ismael,
>>
>> Thanks for the comments. Replies below.
>>
>> 1. We could throw an InvalidConfigurationException at run-time either when
>> building the MetadataRequest or when an ApiVersions response is received.
>> Because this is a configuration problem, I think users would likely see the
>> exception on the first call to poll. Does this sound reasonable?
>>
>> 2. I think we could log a warning when KafkaConsumer is instantiated with
>> the default configuration. Is there a plan to deprecate the server
>> configuration? Is so, we could employ the same strategy in terms of when to
>> switch the default value and then eventually remove the config altogether.
>>
>> - Dhruvil
>>
>> On Tue, Aug 21, 2018 at 7:36 PM Ismael Juma  wrote:
>>
>>> Thanks for the KIP. A few questions/comments:
>>>
>>> 1. It seems hard to reason about if we just disregard the config for
>> older
>>> brokers. Maybe we should throw an error if the brokers don't support it
>> and
>>> let users explicitly change the config if they want to.
>>>
>>> 2. We probably want to switch the default and eventually remove this
>> config
>>> in a future version. What's the path to making that happen? One option
>>> would be to warn if people rely on the default as a first step (or warn
>>> every time it's used).
>>>
>>> Ismael
>>>
>>> On 21 Aug 2018 3:56 pm, "Dhruvil Shah"  wrote:
>>>
>>> Hi,
>>>
>>> I would like to start discussion on KIP-361 that proposes we add a
>> consumer
>>> configuration to disable auto topic creation.
>>>
>>> Link to the KIP:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>>>
>>> Suggestions and feedback are welcome!
>>>
>>> Thanks,
>>> Dhruvil
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Dhruvil Shah]

To be clear, we will allow auto topic creation only when server config
auto.create.topics.enable=true and consumer config
allow.auto.create.topics=true; when either is false, we would not create
the topic if it does not exist.

"suppress" is the opposite of "allow", so
setting suppress.auto.create.topics=false would mean that we do _not_ allow
auto topic creation; when set to true, the server configuration will
determine whether we allow automatic creation or not.

I think "allow" is easier to understand but I am open to suggestions.

- Dhruvil

On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner 
wrote:

> “allow=false” seems a bit more intuitive to me than “suppress=false”
>
> Brandon
>
> > On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
> >
> > We may also consider :
> >
> > "suppress.auto.topic.creation"
> >
> > or
> >
> > "allow.auto.topic.creation"
> >
> > w.r.t. suppress or allow, I don't have strong opinion either. It's just a
> > matter of choosing the proper default value.
> >
> > Cheers
> >
> >> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah 
> wrote:
> >>
> >> Hi Matthias,
> >>
> >> Do you mean something like "suppress.auto.create.topic"? I am leaning a
> bit
> >> towards "allow.auto.create.topics" but I don't have a strong preference
> >> either. Let's wait to hear if anyone else has an opinion on this.
> >>
> >> Thanks,
> >> Dhruvil
> >>
> >> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax 
> >> wrote:
> >>
> >>> Thanks for the KIP Dhruvil!
> >>>
> >>> I agree with Jason's comment. An alternative might be to use "suppress"
> >>> what would revert the logic of "allow". Not sure which one is more
> >>> intuitive and I am fine with both (no personal preference). Just wanted
> >>> to mention it as an alternative.
> >>>
> >>> Don't have any further comments/question so far.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>>
>  On 8/21/18 4:42 PM, Jason Gustafson wrote:
>  Hey Dhruvil,
> 
>  I would suggest using the verb "allow" rather than "enable. The
> >> consumer
>  cannot enable auto topic creation because it is configured on the
> >> broker.
>  All it can do is prevent it from happening if it is enabled.
> 
>  -Jason
> 
>  On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
> >>> wrote:
> 
> > Hi,
> >
> > I would like to start discussion on KIP-361 that proposes we add a
> >>> consumer
> > configuration to disable auto topic creation.
> >
> > Link to the KIP:
> >
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> > Configuration+to+Disable+Auto+Topic+Creation
> >
> > Suggestions and feedback are welcome!
> >
> > Thanks,
> > Dhruvil
> >
> 
> >>>
> >>>
> >>
>

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Dhruvil Shah]

*"suppress" is the opposite of "allow", so
setting suppress.auto.create.topics=false would mean that we do _not_ allow
auto topic creation; when set to true, the server configuration will
determine whether we allow automatic creation or not.*

Sorry, I meant suppress.auto.create.topics=true above to disallow auto
topic creation.


On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah  wrote:

> To be clear, we will allow auto topic creation only when server config
> auto.create.topics.enable=true and consumer config
> allow.auto.create.topics=true; when either is false, we would not create
> the topic if it does not exist.
>
> "suppress" is the opposite of "allow", so
> setting suppress.auto.create.topics=false would mean that we do _not_ allow
> auto topic creation; when set to true, the server configuration will
> determine whether we allow automatic creation or not.
>
> I think "allow" is easier to understand but I am open to suggestions.
>
> - Dhruvil
>
> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> brandon.kirch...@gmail.com> wrote:
>
>> “allow=false” seems a bit more intuitive to me than “suppress=false”
>>
>> Brandon
>>
>> > On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
>> >
>> > We may also consider :
>> >
>> > "suppress.auto.topic.creation"
>> >
>> > or
>> >
>> > "allow.auto.topic.creation"
>> >
>> > w.r.t. suppress or allow, I don't have strong opinion either. It's just
>> a
>> > matter of choosing the proper default value.
>> >
>> > Cheers
>> >
>> >> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah 
>> wrote:
>> >>
>> >> Hi Matthias,
>> >>
>> >> Do you mean something like "suppress.auto.create.topic"? I am leaning
>> a bit
>> >> towards "allow.auto.create.topics" but I don't have a strong preference
>> >> either. Let's wait to hear if anyone else has an opinion on this.
>> >>
>> >> Thanks,
>> >> Dhruvil
>> >>
>> >> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax > >
>> >> wrote:
>> >>
>> >>> Thanks for the KIP Dhruvil!
>> >>>
>> >>> I agree with Jason's comment. An alternative might be to use
>> "suppress"
>> >>> what would revert the logic of "allow". Not sure which one is more
>> >>> intuitive and I am fine with both (no personal preference). Just
>> wanted
>> >>> to mention it as an alternative.
>> >>>
>> >>> Don't have any further comments/question so far.
>> >>>
>> >>>
>> >>> -Matthias
>> >>>
>> >>>
>> >>>
>>  On 8/21/18 4:42 PM, Jason Gustafson wrote:
>>  Hey Dhruvil,
>> 
>>  I would suggest using the verb "allow" rather than "enable. The
>> >> consumer
>>  cannot enable auto topic creation because it is configured on the
>> >> broker.
>>  All it can do is prevent it from happening if it is enabled.
>> 
>>  -Jason
>> 
>>  On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
>> >>> wrote:
>> 
>> > Hi,
>> >
>> > I would like to start discussion on KIP-361 that proposes we add a
>> >>> consumer
>> > configuration to disable auto topic creation.
>> >
>> > Link to the KIP:
>> >
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
>> > Configuration+to+Disable+Auto+Topic+Creation
>> >
>> > Suggestions and feedback are welcome!
>> >
>> > Thanks,
>> > Dhruvil
>> >
>> 
>> >>>
>> >>>
>> >>
>>
>

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Matthias J. Sax]

Thanks for the summary!

We might want to add a diagram/table to the docs when we add this
feature (with whatever config name we choose) to explain how broker
config `auto.create.topics.enable` and the consumer config work together.

I think both options are equally easy to understand. "allow" means
follow the broker config, while "suppress" implies ignore the broker
config and don't auto-create.


-Matthias


On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> *"suppress" is the opposite of "allow", so
> setting suppress.auto.create.topics=false would mean that we do _not_ allow
> auto topic creation; when set to true, the server configuration will
> determine whether we allow automatic creation or not.*
> 
> Sorry, I meant suppress.auto.create.topics=true above to disallow auto
> topic creation.
> 
> 
> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah  wrote:
> 
>> To be clear, we will allow auto topic creation only when server config
>> auto.create.topics.enable=true and consumer config
>> allow.auto.create.topics=true; when either is false, we would not create
>> the topic if it does not exist.
>>
>> "suppress" is the opposite of "allow", so
>> setting suppress.auto.create.topics=false would mean that we do _not_ allow
>> auto topic creation; when set to true, the server configuration will
>> determine whether we allow automatic creation or not.
>>
>> I think "allow" is easier to understand but I am open to suggestions.
>>
>> - Dhruvil
>>
>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
>> brandon.kirch...@gmail.com> wrote:
>>
>>> “allow=false” seems a bit more intuitive to me than “suppress=false”
>>>
>>> Brandon
>>>
 On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:

 We may also consider :

 "suppress.auto.topic.creation"

 or

 "allow.auto.topic.creation"

 w.r.t. suppress or allow, I don't have strong opinion either. It's just
>>> a
 matter of choosing the proper default value.

 Cheers

> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah 
>>> wrote:
>
> Hi Matthias,
>
> Do you mean something like "suppress.auto.create.topic"? I am leaning
>>> a bit
> towards "allow.auto.create.topics" but I don't have a strong preference
> either. Let's wait to hear if anyone else has an opinion on this.
>
> Thanks,
> Dhruvil
>
> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax >>>
> wrote:
>
>> Thanks for the KIP Dhruvil!
>>
>> I agree with Jason's comment. An alternative might be to use
>>> "suppress"
>> what would revert the logic of "allow". Not sure which one is more
>> intuitive and I am fine with both (no personal preference). Just
>>> wanted
>> to mention it as an alternative.
>>
>> Don't have any further comments/question so far.
>>
>>
>> -Matthias
>>
>>
>>
>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
>>> Hey Dhruvil,
>>>
>>> I would suggest using the verb "allow" rather than "enable. The
> consumer
>>> cannot enable auto topic creation because it is configured on the
> broker.
>>> All it can do is prevent it from happening if it is enabled.
>>>
>>> -Jason
>>>
>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
>> wrote:
>>>
 Hi,

 I would like to start discussion on KIP-361 that proposes we add a
>> consumer
 configuration to disable auto topic creation.

 Link to the KIP:

>>
>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
 Configuration+to+Disable+Auto+Topic+Creation

 Suggestions and feedback are welcome!

 Thanks,
 Dhruvil

>>>
>>
>>
>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-7330) Kakfa 0.10.2.1 producer close method issue

[vinay (JIRA)]

vinay created KAFKA-7330:


 Summary: Kakfa 0.10.2.1 producer close method issue
 Key: KAFKA-7330
 URL: https://issues.apache.org/jira/browse/KAFKA-7330
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.2.1
 Environment: linux
Reporter: vinay


We are creating new producer connections each time but in separate threads(pool 
of 40 and all threads will be used most of the time) and closing it when work 
in done. its a java process.

Not always but sometimes close connection takes more than 10 seconds and 
observerd that particular message was never published.

Could you please help.

 

{{Properties prop = new Properties(); }}

{{prop.put("bootstarp.servers",-);}}

{{prop.put("acks","all"); }}

{{//some ssl properties }}

{{--- }}

{{//ends KafkaProducer }}

{{connection = null; }}

{{try { }}

{{connection = new KafkaProducer(props); msg.setTopic(topic); }}

{{msg.setDate(new Date());}}

{{ connection.send(msg);}}

{{ } catch() {}}

{{ } finally { }}

{{connection.close();// sometimes it takes more than 10 secs and the above 
message was not send }}

{{}}}

Producer config :

max.in.flight.requests.per.connection = 5

acks= all

batch.size = 16384

linger.ms = 0

max.block.ms = 6

request.timeout.ms = 5000

retries = 0

---

 

Even  i see below warning most of the times,

Failed to send SSL Close message:

java.io.IOException: Unexpected status returned by SSLEngine.wrap, expected 
CLOSED, received OK. Will not send close message to peer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1921) Delete offsets for a group with kafka offset storage

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1921.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6275

> Delete offsets for a group with kafka offset storage
> 
>
> Key: KAFKA-1921
> URL: https://issues.apache.org/jira/browse/KAFKA-1921
> Project: Kafka
>  Issue Type: Improvement
>  Components: offset manager
>Reporter: Onur Karaman
>Priority: Major
>
> There is currently no way to delete offsets for a consumer group when using 
> kafka offset storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6314) Add a tool to delete kafka based consumer offsets for a given group

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6314.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6275

> Add a tool to delete kafka based consumer offsets for a given group
> ---
>
> Key: KAFKA-6314
> URL: https://issues.apache.org/jira/browse/KAFKA-6314
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer, core, tools
>Reporter: Tom Scott
>Priority: Minor
>
> Add a tool to delete kafka based consumer offsets for a given group similar 
> to the reset tool. It could look something like this:
> kafka-consumer-groups --bootstrap-server localhost:9092 --delete-offsets 
> --group somegroup
> The case for this is as follows:
> 1. Consumer group with id: group1 subscribes to topic1
> 2. The group is stopped 
> 3. The subscription changed to topic2 but the id is kept as group1
> Now the out output of kafka-consumer-groups --describe for the group will 
> show topic1 even though the group is not subscribed to that topic. This is 
> bad for monitoring as it will show lag on topic1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)