[jira] [Resolved] (KAFKA-6257) KafkaConsumer was hung when bootstrap servers was not existed

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-6257.
-
Resolution: Duplicate

Closing this as duplicate since no contradicting information was added.

> KafkaConsumer was hung when bootstrap servers was not existed
> -
>
> Key: KAFKA-6257
> URL: https://issues.apache.org/jira/browse/KAFKA-6257
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Brian Clark
>Priority: Minor
>
> Could anyone help me on this?
> We have an issue if we entered an non-existed host:port for bootstrap.servers 
> property on KafkaConsumer. The created KafkaConsumer was hung forever.
> the debug message:
> java.net.ConnectException: Connection timed out: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
> at 
> org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
> at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:359)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:432)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:223)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:200)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
> at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
> at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> [2017-08-28 09:20:56,400] DEBUG Node -1 disconnected. 
> (org.apache.kafka.clients.NetworkClient)
> [2017-08-28 09:20:56,400] WARN Connection to node -1 could not be 
> established. Broker may not be available. 
> (org.apache.kafka.clients.NetworkClient)
> [2017-08-28 09:20:56,400] DEBUG Give up sending metadata request since no 
> node is available (org.apache.kafka.clients.NetworkClient)
> [2017-08-28 09:20:56,450] DEBUG Initialize connection to node -1 for sending 
> metadata request (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-229: DeleteGroups API

[Dong Lin]

Hey Vahid,

Thanks for the KIP! This is certainly a useful one and users have been
asking about the ability to delete group from the Kafka offset topic in my
past experience.

It seems that the protocol of the new request/response should probably
include more fields fields. For example, it may be useful to include
throttle_time_ms field and a request level error code in
DeleteGroupsResponse. The request level error code can possibly show error
such as NOT_COORDINATOR_FOR_GROUP.

Also, user may want to use this feature programmatically. Do you think we
should add a corresponding API in AminClient to delete groups? If so, can
you specify the new AdminClient API in the KIP?

Thanks,
Dong


On Tue, Nov 28, 2017 at 4:03 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi everyone,
>
> I started KIP-229 and proposed a consumer group deletion API for
> Kafka-based group management to address KAFKA-6275:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 229%3A+DeleteGroups+API
> Your feedback and suggestions are welcome!
>
> Thanks.
> --Vahid
>
>
>

[GitHub] kafka pull request #4273: KAFKA-4827: Porting fix for KAFKA-4827 to v0.10 an...

[wicknicks]

GitHub user wicknicks opened a pull request:

https://github.com/apache/kafka/pull/4273

KAFKA-4827: Porting fix for KAFKA-4827 to v0.10 and v0.11



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wicknicks/kafka KAFKA-4827-0.10.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4273


commit c6e8dc1edb074a2b677de43031d8b59fec4a5e1e
Author: Arjun Satish 
Date:   2017-11-10T17:59:42Z

Correctly encode special chars while creating URI objects

Signed-off-by: Arjun Satish 

commit 02ddc84850acdc5dc55dc146e48744be2346a929
Author: Arjun Satish 
Date:   2017-11-10T23:48:13Z

Replace URLEncoder with URIBuilder

Also, update the tests to pass some additional characters in the connector
name along with adding a decode step using the URI class.

Signed-off-by: Arjun Satish 

commit a1cd574cf1987e59cb89fc33da12aa1ced8bf9f5
Author: Arjun Satish 
Date:   2017-11-28T22:33:16Z

Porting fix for KAFKA-4827 to v0.10 and v0.11

Signed-off-by: Arjun Satish 




---

答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

[Hu Xi]

Hi Rao Jun,


Already updated the patch per your suggestion and it seems there are no further 
feedbacks on this KIP.  Could we vote now?



发件人: Jun Rao 
发送时间: 2017年11月23日 6:21
收件人: dev@kafka.apache.org
主题: Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead 
metrics to KafkaConsumer

Hi, Hu,

There are two types of names. One is the sensor name, which has to be
unique. It's fine if the sensor name includes the topic/partition as the
prefix since the sensor name is only a string and is not exposed to jmx.
The second name is the metric name, which will be used in jmx. Currently,
the existing lag metric name uses topic/partition as the prefix. KIP-225
tries to change the metric name to use topic/partition as the tag. We can
just do the same thing for lead by using tags in the metric name.

Thanks,

Jun

On Mon, Nov 20, 2017 at 10:14 PM, Hu Xi  wrote:

> Hi Jun,
>
>
> Seems the prefix that is used to be the unique Sensor name cannot be
> removed, so should we keep the prefix?
>
>
> 
> 发件人: Jun Rao 
> 发送时间: 2017年11月21日 3:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Hu,
>
> For the new partition level metrics that you are adding, it seems it's
> better to just add the topic/partition tag instead of using them in the
> prefix. For the existing lag metrics, we can fix them in KIP-225.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi  wrote:
>
> > Jun,
> >
> >
> > Thanks for the comments. Do you think it'd better to add topic/partition
> > tags for those metrics as well as keep the prefix? If those prefixes
> should
> > really be removed, does this KIP need to do the same thing for `lag`
> ones?
> >
> > 
> > 发件人: Jun Rao 
> > 发送时间: 2017年11月18日 8:55
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition lead metrics to KafkaConsumer
> >
> > Hi, Charly,
> >
> > Thanks for the input. It makes sense.
> >
> > Hi, Hu,
> >
> > Perhaps we can keep the per partition records-lead-min and
> records-lead-avg
> > as you had before, but just add the topic and the partition as the tags
> > instead of prefix of the metric name.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Nov 15, 2017 at 4:58 AM, charly molter 
> > wrote:
> >
> > > Hi Jun, Jiangle,
> > >
> > > I'd just like to clarify that KIP-225 seems to be using per partition
> > > metric the same way as KIP-223 seems to be doing.
> > >
> > > I believe avg and max are still necessary because the MetricsReporter
> > > doesn't work in a "push" manner and the "Value" measurableStat will
> only
> > > keep the last recorded entry.
> > > Therefore a MetricsReporter usually polls to grab a current view with
> > Value
> > > this view is incomplete so it becomes not possible to compute the
> > > Max/Min/Avg.
> > > Max/Min/Avg uses SampledStats which work with a rolling window of
> samples
> > > and therefore periodic polling would work.
> > >
> > > This is why I believe it's necessary to keep Avg, Min and Max for these
> > > metrics as otherwise we wouldn't be able to recompute it in an external
> > > monitoring system.
> > >
> > > Am I wrong thinking this?
> > >
> > > Thanks,
> > > Charly
> > >
> > >
> > > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao  wrote:
> > >
> > > > Hi, Charly,
> > > >
> > > > Thanks for KIP-225. Your proposal looks reasonable.
> > > >
> > > > Hi, Jiangjie,
> > > >
> > > > Do you think the approach that KIP-225 proposes is better for
> exposing
> > > the
> > > > per partition metric? Also, do we really need the per partition
> > > > record-lag-avg
> > > > and record-lag-max? It seems that an external monitoring system can
> > > always
> > > > derive that from the per partition record-lag.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> > charly.mol...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jun, Hu,
> > > > >
> > > > > I have KIP-225 open for adding tags to records-lag:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=74686649
> > > > >
> > > > > I have a patch more or less ready so I could probably get the fix
> > > checked
> > > > > in (after the vote) and you could build on top of it. Otherwise we
> > > could
> > > > > merge both KIPs if you want but they do sound different to me.
> > > > >
> > > > > Thanks!
> > > > > Charly
> > > > >
> > > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi 
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > >
> > > > > > Let me double confirm with your comments:
> > > > > >
> > > > > > 1 remove partition-level records-lead-avg and records-lead-min
> > since
> > > > they
> > > > > > both can be deduced by external monitoring system.
> > > > > >
> > > > > > 2 Tag partition-level records-lead with topic&partition info
> > > 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Jason Gustafson]

Hi Rajini,

One quick question I was wondering about is whether this could be used to
update the inter-broker protocol version or the message format version?
Potentially then we'd only need one rolling restart to upgrade the cluster.
I glanced quickly through the uses of this config in the code and it seems
like it might be possible. Not sure if there are any complications you or
others can think of.

Thanks,
Jason

On Tue, Nov 28, 2017 at 2:48 PM, Rajini Sivaram 
wrote:

> Hi Colin,
>
> Thank you for reviewing the KIP.
>
> *kaka-configs.sh* will be converted to use *AdminClient* under KAFKA-5722.
> This is targeted for the next release (1.1.0). Under this KIP, we will
> implement *AdminClient#alterConfigs* for the dynamic configs listed in the
> KIP. This will include validation of the configs and will return
> appropriate errors if configs are invalid. Integration tests will also be
> added using AdmnClient. Only the actual conversion of ConfigCommand to use
> AdminClient will be left to be done under KAFKA-5722.
>
> Once KAFKA-5722 is implemented,* kafka-confgs.sh* can be used to obtain the
> current configuration, which can be redirected to a text file to create a
> static *server.properties* file. This should help when downgrading - but it
> does require brokers to be running. We can also document how to obtain the
> properties using *zookeeper-shell.sh* while downgrading if brokers are
> down.
>
> If we rename properties, we should add the new property to ZK based on the
> value of the old property when the upgraded broker starts up. But we would
> probably leave the old property as is. The old property will be returned
> and used as a synonym only as long as the broker is on a version where it
> is still valid. But it can remain in ZK and be updated if downgrading - it
> will be up to the user to update the old property if downgrading or delete
> it if not needed. Renaming properties is likely to be confusing in any case
> even without dynamic configs, so hopefully it will be very rare.
>
>
> Rajini
>
> On Tue, Nov 28, 2017 at 7:47 PM, Colin McCabe  wrote:
>
> > Hi Rajini,
> >
> > This seems like a nice improvement!
> >
> > One thing that is a bit concerning is that, if bin/kafka-configs.sh is
> > used, there is no  way for the broker to give feedback or error
> > messages.  The broker can't say "sorry, I can't reconfigure that in that
> > way."  Or even "that configuration property is not reconfigurable in
> > this version of the software."  It seems like in the examples give,
> > users will simply set a configuration using bin/kafka-configs.sh, but
> > then they have to check the broker log files to see if it could actually
> > be applied.  And even if it couldn't be applied, then it still lingers
> > in ZooKeeper.
> >
> > This seems like it would lead to a lot of user confusion, since they get
> > no feedback when reconfiguring something.  For example, there will be a
> > lot of scenarios where someone finds a reconfiguration command on
> > Google, runs it, but then it doesn't do anything because the software
> > version is different.  And there's no error message or feedback about
> > this.  It just doesn't work.
> >
> > To prevent this, I think we should convert bin/kafka-configs.sh to use
> > AdminClient's AlterConfigsRequest.  This allows us to detect scenarios
> > where, because of a typo, different software version, or a value of the
> > wrong type (eg. string vs. int), the given configuration cannot be
> > applied.  We really should convert kafka-configs.sh to use AdminClient
> > anyway, for all the usual reasons-- people want to lock down ZooKeeper,
> > ACLs should be enforced, internal representations should be hidden, we
> > should support environments where ZK is not exposed, etc. etc.
> >
> > Another issue that I see here is, how does this interact with downgrade?
> >  Presumably, if the user downgrades to a version that doesn't support
> > KIP-226, all the dynamic configurations stored in ZK revert to whatever
> > value they have in the local config files.  Do we need to have a utility
> > that can reify the actual applied configuration into a local text file,
> > to make downgrades less painful?
> >
> > With regard to upgrades, what happens if we change the name of a
> > configuration key in the future?  For example, if we decide to rename
> > min.insync.replicas to min.in.sync.replicas, presumably we will
> > deprecate the old key name.  And then perhaps it will be removed in a
> > future release, such as Apache Kafka 2.0.  In this scenario, should the
> > Kafka upgrade process change the name of the configuration key in ZK
> > from min.insync.replicas to min.in.sync.replicas?  Obviously this will
> > make downgrades harder, if so.  But if it doesn't, then removing
> > deprecated configuration key synonyms might become very difficult.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Nov 22, 2017, at 12:52, Rajini Sivaram wrote:
> > > Hi Tom,
> > >
> > > No, I am not proposing this as a 

[jira] [Resolved] (KAFKA-6279) Connect metrics do not get cleaned up for a source connector that doesn't stop properly

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-6279.
--
Resolution: Duplicate

> Connect metrics do not get cleaned up for a source connector that doesn't 
> stop properly
> ---
>
> Key: KAFKA-6279
> URL: https://issues.apache.org/jira/browse/KAFKA-6279
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 1.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Critical
>
> Connect will call {{stop()}} on a source task to signal that it should "_stop 
> trying to poll for new data 
> and interrupt any outstanding poll() requests_" (see the 
> [JavaDoc|http://kafka.apache.org/10/javadoc/org/apache/kafka/connect/source/SourceTask.html#stop--]
>  for this method).
> Unfortunately, not all connectors properly adhere to this expectation. Since 
> the metrics for the source task are cleaned up only when the worker source 
> task's thread completes, a task whose {{poll()}} method blocks forever will 
> current prevent its thread from completing. So, we need to change how the 
> metrics are cleaned up to ensure this always happens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[DISCUSS] KIP-229: DeleteGroups API

[Vahid S Hashemian]

Hi everyone,

I started KIP-229 and proposed a consumer group deletion API for 
Kafka-based group management to address KAFKA-6275: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-229%3A+DeleteGroups+API
Your feedback and suggestions are welcome!

Thanks.
--Vahid

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Rajini Sivaram]

Hi Colin,

Thank you for reviewing the KIP.

*kaka-configs.sh* will be converted to use *AdminClient* under KAFKA-5722.
This is targeted for the next release (1.1.0). Under this KIP, we will
implement *AdminClient#alterConfigs* for the dynamic configs listed in the
KIP. This will include validation of the configs and will return
appropriate errors if configs are invalid. Integration tests will also be
added using AdmnClient. Only the actual conversion of ConfigCommand to use
AdminClient will be left to be done under KAFKA-5722.

Once KAFKA-5722 is implemented,* kafka-confgs.sh* can be used to obtain the
current configuration, which can be redirected to a text file to create a
static *server.properties* file. This should help when downgrading - but it
does require brokers to be running. We can also document how to obtain the
properties using *zookeeper-shell.sh* while downgrading if brokers are down.

If we rename properties, we should add the new property to ZK based on the
value of the old property when the upgraded broker starts up. But we would
probably leave the old property as is. The old property will be returned
and used as a synonym only as long as the broker is on a version where it
is still valid. But it can remain in ZK and be updated if downgrading - it
will be up to the user to update the old property if downgrading or delete
it if not needed. Renaming properties is likely to be confusing in any case
even without dynamic configs, so hopefully it will be very rare.


Rajini

On Tue, Nov 28, 2017 at 7:47 PM, Colin McCabe  wrote:

> Hi Rajini,
>
> This seems like a nice improvement!
>
> One thing that is a bit concerning is that, if bin/kafka-configs.sh is
> used, there is no  way for the broker to give feedback or error
> messages.  The broker can't say "sorry, I can't reconfigure that in that
> way."  Or even "that configuration property is not reconfigurable in
> this version of the software."  It seems like in the examples give,
> users will simply set a configuration using bin/kafka-configs.sh, but
> then they have to check the broker log files to see if it could actually
> be applied.  And even if it couldn't be applied, then it still lingers
> in ZooKeeper.
>
> This seems like it would lead to a lot of user confusion, since they get
> no feedback when reconfiguring something.  For example, there will be a
> lot of scenarios where someone finds a reconfiguration command on
> Google, runs it, but then it doesn't do anything because the software
> version is different.  And there's no error message or feedback about
> this.  It just doesn't work.
>
> To prevent this, I think we should convert bin/kafka-configs.sh to use
> AdminClient's AlterConfigsRequest.  This allows us to detect scenarios
> where, because of a typo, different software version, or a value of the
> wrong type (eg. string vs. int), the given configuration cannot be
> applied.  We really should convert kafka-configs.sh to use AdminClient
> anyway, for all the usual reasons-- people want to lock down ZooKeeper,
> ACLs should be enforced, internal representations should be hidden, we
> should support environments where ZK is not exposed, etc. etc.
>
> Another issue that I see here is, how does this interact with downgrade?
>  Presumably, if the user downgrades to a version that doesn't support
> KIP-226, all the dynamic configurations stored in ZK revert to whatever
> value they have in the local config files.  Do we need to have a utility
> that can reify the actual applied configuration into a local text file,
> to make downgrades less painful?
>
> With regard to upgrades, what happens if we change the name of a
> configuration key in the future?  For example, if we decide to rename
> min.insync.replicas to min.in.sync.replicas, presumably we will
> deprecate the old key name.  And then perhaps it will be removed in a
> future release, such as Apache Kafka 2.0.  In this scenario, should the
> Kafka upgrade process change the name of the configuration key in ZK
> from min.insync.replicas to min.in.sync.replicas?  Obviously this will
> make downgrades harder, if so.  But if it doesn't, then removing
> deprecated configuration key synonyms might become very difficult.
>
> best,
> Colin
>
>
> On Wed, Nov 22, 2017, at 12:52, Rajini Sivaram wrote:
> > Hi Tom,
> >
> > No, I am not proposing this as a way to configure replication quotas.
> > When
> > you describe broker configs using AdminClient, you will see all the
> > configs
> > persisted in /configs/brokers/ in ZooKeeper and this includes
> > leader.replication.throttled.rate, follower.replication.throttled.rate
> > etc. But
> > the broker configs that can be altered using AdminClient as a result of
> > this KIP are those explicitly stated in the KIP (does not include any of
> > the quota configs).
> >
> > Regards,
> >
> > Rajini
> >
> > On Wed, Nov 22, 2017 at 7:54 PM, Tom Bentley 
> > wrote:
> >
> > > Hi Rajini,
> > >
> > > Just to clarify, are you proposing this as a way to co

Build failed in Jenkins: kafka-trunk-jdk9 #226

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6255; Add ProduceBench to Trogdor

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 58877a0deacd6c13436af83ef5be1a6f75a3ac4a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 58877a0deacd6c13436af83ef5be1a6f75a3ac4a
Commit message: "KAFKA-6255; Add ProduceBench to Trogdor"
 > git rev-list 1a1d923f252e9b1576dad6f7285f237feb064f64 # timeout=10
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins6617992994547749440.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins6617992994547749440.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 4 hr 57 min old

ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
Not sending mail to unregistered user rajinisiva...@googlemail.com

[GitHub] kafka pull request #4272: KAFKA-4827: Porting fix for KAFKA-4827 to v0.10 an...

[wicknicks]

GitHub user wicknicks opened a pull request:

https://github.com/apache/kafka/pull/4272

KAFKA-4827: Porting fix for KAFKA-4827 to v0.10 and v0.11

Back porting the fix in this PR (https://github.com/apache/kafka/pull/4205) 
to v0.10 and v0.11.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wicknicks/kafka KAFKA-4827-0.10.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4272


commit 5df843e7945bb92009c0718a31bb6af42775b0c4
Author: Arjun Satish 
Date:   2017-11-10T17:59:42Z

Correctly encode special chars while creating URI objects

Signed-off-by: Arjun Satish 

commit d2979100bb433e992894b5df8e387ca84a66cf95
Author: Arjun Satish 
Date:   2017-11-10T23:48:13Z

Replace URLEncoder with URIBuilder

Also, update the tests to pass some additional characters in the connector
name along with adding a decode step using the URI class.

Signed-off-by: Arjun Satish 

commit 2251846a36107a37d89160097b1ea75575c7550b
Author: Arjun Satish 
Date:   2017-11-28T22:33:16Z

Porting fix for KAFKA-4827 to v0.10 and v0.11

Signed-off-by: Arjun Satish 




---

[GitHub] kafka pull request #4245: KAFKA-6255: Add ProduceBench to Trogdor

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4245


---

[jira] [Resolved] (KAFKA-6255) Add ProduceBench to Trogdor

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-6255.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

Issue resolved by pull request 4245
[https://github.com/apache/kafka/pull/4245]

> Add ProduceBench to Trogdor
> ---
>
> Key: KAFKA-6255
> URL: https://issues.apache.org/jira/browse/KAFKA-6255
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 1.1.0
>
>
> Add ProduceBench, a benchmark of producer latency, to Trogdor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #3006

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6255; Add ProduceBench to Trogdor

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 58877a0deacd6c13436af83ef5be1a6f75a3ac4a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 58877a0deacd6c13436af83ef5be1a6f75a3ac4a
Commit message: "KAFKA-6255; Add ProduceBench to Trogdor"
 > git rev-list 1a1d923f252e9b1576dad6f7285f237feb064f64 # timeout=10
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/jenkins6593241682362356385.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins6593241682362356385.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] Parsing 12 files in 

[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=58877a0deacd6c13436af83ef5be1a6f75a3ac4a, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #3005
Recording test results
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 hr 16 min old

ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
Not sending mail to unregistered user rajinisiva...@googlemail.com

Build failed in Jenkins: kafka-trunk-jdk8 #2245

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6255; Add ProduceBench to Trogdor

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 58877a0deacd6c13436af83ef5be1a6f75a3ac4a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 58877a0deacd6c13436af83ef5be1a6f75a3ac4a
Commit message: "KAFKA-6255; Add ProduceBench to Trogdor"
 > git rev-list 1a1d923f252e9b1576dad6f7285f237feb064f64 # timeout=10
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/jenkins1541542912226643958.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins1541542912226643958.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] Parsing 10 files in 

[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
[FINDBUGS] Successfully parsed file 

 with 0 unique warnings and 0 duplicates.
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=58877a0deacd6c13436af83ef5be1a6f75a3ac4a, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #2239
Recording test results
ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 hr 56 min old

ERROR: No tool found matching GRADLE_3_4_RC_2_HOME
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user rajinisiva...@googlemail.com

[jira] [Created] (KAFKA-6280) Allow for additional archive types to be loaded from plugin.path in Connect

[Konstantine Karantasis (JIRA)]

Konstantine Karantasis created KAFKA-6280:
-

 Summary: Allow for additional archive types to be loaded from 
plugin.path in Connect
 Key: KAFKA-6280
 URL: https://issues.apache.org/jira/browse/KAFKA-6280
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Konstantine Karantasis
Assignee: Konstantine Karantasis
Priority: Minor
 Fix For: 1.1.0


Additionally to uber {{.jar}} archives, seems it would be nice if one could 
load also `zip` archives of appropriately packaged Connect plugins. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4271: KAFKA-5526: Additional `--describe` views for Cons...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/4271

KAFKA-5526: Additional `--describe` views for ConsumerGroupCommand (KIP-175)

The `--describe` option of ConsumerGroupCommand is expanded to support:
* `--describe` or `--describe --offsets`: listing of current group offsets
* `--describe --members` or `--describe --members --verbose`: listing of 
group members
* `--describe --state`: group status

Example: With a single partition topic `test1` and a double partition topic 
`test2`, consumers `consumer1` and `consumer11` subscribed to `test`, consumers 
`consumer2` and `consumer22` and `consumer222` subscribed to `test2`, and all 
consumers belonging to group `test-group`, this is an output example of the new 
options above for `test-group`:

```
--describe, or --describe --offsets:

TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG 
CONSUMER-ID HOSTCLIENT-ID
test2   0  0   0   0   
consumer2-bad9496d-0889-47ab-98ff-af17d9460382  /127.0.0.1  consumer2
test2   1  0   0   0   
consumer22-c45e6ee2-0c7d-44a3-94a8-9627f63fb411 /127.0.0.1  consumer22
test1   0  0   0   0   
consumer1-d51b0345-3194-4305-80db-81a68fa6c5bf  /127.0.0.1  consumer1
```

```
--describe --members

CONSUMER-ID  HOSTCLIENT-ID  
 #PARTITIONS
consumer2-bad9496d-0889-47ab-98ff-af17d9460382   /127.0.0.1  consumer2  
 1
consumer222-ed2108cd-d368-41f1-8514-5b72aa835bcc /127.0.0.1  
consumer222 0
consumer11-dc8295d7-8f3f-4438-9b11-7270bab46760  /127.0.0.1  consumer11 
 0
consumer22-c45e6ee2-0c7d-44a3-94a8-9627f63fb411  /127.0.0.1  consumer22 
 1
consumer1-d51b0345-3194-4305-80db-81a68fa6c5bf   /127.0.0.1  consumer1  
 1
```

```
--describe --members --verbose

CONSUMER-ID  HOSTCLIENT-ID  
 #PARTITIONS ASSIGNMENT
consumer2-bad9496d-0889-47ab-98ff-af17d9460382   /127.0.0.1  consumer2  
 1   test2(0)
consumer222-ed2108cd-d368-41f1-8514-5b72aa835bcc /127.0.0.1  
consumer222 0   -
consumer11-dc8295d7-8f3f-4438-9b11-7270bab46760  /127.0.0.1  consumer11 
 0   -
consumer22-c45e6ee2-0c7d-44a3-94a8-9627f63fb411  /127.0.0.1  consumer22 
 1   test2(1)
consumer1-d51b0345-3194-4305-80db-81a68fa6c5bf   /127.0.0.1  consumer1  
 1   test1(0)
```

```
--describe --state

ASSIGNMENT-STRATEGY   STATE#MEMBERS
range Stable   5
```

Note that this PR also addresses the issue reported in 
[KAFKA-6158](https://issues.apache.org/jira/browse/KAFKA-6158) by dynamically 
setting the width of columns `TOPIC`, `CONSUMER-ID`, `HOST` and `CLIENT-ID`. 
This avoid truncation of column values when they go over the current fixed 
width of these columns.

The code has been restructured to better support testing of individual 
values and also the console output. Unit tests have been updated and extended 
to take advantage of this restructuring.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-5526

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4271.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4271


commit 1973db23d2f29191ec56b56a3040c1a2b0c00ef4
Author: Vahid Hashemian 
Date:   2017-11-28T20:08:37Z

KAFKA-5526: Additional `--describe` views for ConsumerGroupCommand (KIP-175)

The `--describe` option of ConsumerGroupCommand is expanded to support:
* `--describe` or `--describe --offsets`: listing of current group offsets
* `--describe --members` or `--describe --members --verbose`: listing of 
group members
* `--describe --state`: group status

Example: With a single partition topic `test1` and a double partition topic 
`test2`, consumers `consumer1` and `consumer11` subscribed to `test`, consumers 
`consumer2` and `consumer22` and `consumer222` subscribed to `test2`, and all 
consumers belonging to group `test-group`, this is an output example of the new 
options above for `test-group`:

```
--describe, or

Build failed in Jenkins: kafka-trunk-jdk8 #2244

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6274: Use topic plus dash as prefix of auto-generated state 
store

--
[...truncated 1.91 MB...]

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > 
testToStringIsLowerCase STARTED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > 
testToStringIsLowerCase PASSED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > testForValue 
STARTED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > testForValue 
PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigCast STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigCast PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigRegexRouter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigRegexRouter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigSetSchemaMetadata STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigSetSchemaMetadata PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampConverter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampConverter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigHoistField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigHoistField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigMaskField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigMaskField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigInsertField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigInsertField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigFlatten STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigFlatten PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigReplaceField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigReplaceField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampRouter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampRouter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigValueToKey STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigValueToKey PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigExtractField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigExtractField PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > unconfiguredTransform 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > unconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorFailedBasicValidation STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorFailed

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

Jan,

Thanks for your input, I can understand now that the oldValue is also
exposed in user customized `filter` function and hence want record context
we should expose is a problem. And I think it does brings a good point to
consider for KIP-159. The discussions maybe a bit confusing to reader
though, and hence I'd like to summarize the status quo and with a proposal:

In today's Streams DSL, when a KTable is created either from a source
topic, or from an stateful operator, we will materialize the KTable with a
backing state store; on the other hand, KTables created from a non-stateful
operator like filter, will not be backed by a state store by default unless
users indicate so (e.g. using the overloaded function with the queryable
name or store supplier).

For example:

KTable table1 = builder.table("topic");  // a
state store created for table1
KTable table2 = table1.filter(..);
// no state store created for table2
KTable table3 = table1.filter(.., "storeName");  // a state
store created for table3
KTable table4 = table1.groupBy(..).aggregate(..);// a state
store created for table4

Because of that, the filter() operator above on table1 will always be
exposed with oldValue and newValue; Damian's point is that, we may optimize
the first case such that table1 will only be materialized if users asked so
(e.g. using the overloaded function with a store supplier), and in which
case, we do not need to pass newValue / oldValue pairs (I think this is
what Jan suggests as well, i.e. do filtering before materializing, so that
we can have a smaller backed state store as well). But this optimization
does not eliminate the possibilities that we may still need to do filter if
users does specify "yes I do want to the source KTable itself to be
materialized, please". So the concern about how to expose the record
context in such cases still persists.


With that, regarding to KIP-159 itself, here are my thoughts:

1) if we restrict the scope of exposing record context only to source
KTables / KStreams I felt the KIP itself does not bring much value given
its required API change because only the SourceKStream can safely maintain
its records context, and for SourceKTable if it is materialized, then even
non-stateful operators like Join may still have a concern about exposing
the record context.

2) an alternative idea is we provide the semantics on how record context
would be inherited across the operators for KTable / KStream and expose it
in all operators (similarly in PAPI we would expose a much simpler
contract), and make it as a public contract that Streams library will
guarantee moving forward even we optimize our topology builder; it may not
align perfectly with the linear algebraic semantics but practically
applicable for most cases; if users semantics do not fit in the provided
contract, then they may need to get this themselves (embed such information
in the value payload, for example).

If people do not like the second idea, I'd suggest we hold on pursuing the
first direction since to me its beneficial scope is too limited compared to
its cost.


Guozhang



On Fri, Nov 24, 2017 at 1:39 AM, Jan Filipiak 
wrote:

> Cleary we show the oldValue to the user. We have to, because we filter
> after the store.
> https://github.com/axbaretto/kafka/blob/master/streams/src/m
> ain/java/org/apache/kafka/streams/kstream/internals/KTableFilter.java#L96
>
> I cannot help you following this. It is really obvious and I am running
> out of tools for explaining.
>
> Thanks for understanding my point to put filter before. Not only would it
> make the store smaller. It would make this feature reasonably possible and
> the framework easier. Interestingly it would also help to move IQ into more
> reasonable directions. And it might help understand that we do not need any
> intermediate representation of the topology,
>
> KIP-182 I have no clue what everyone has with their "bytestores" so
> broken. But putting another store after doesn't help when the store before
> is the problem.
>
>
>
>
> On 24.11.2017 05:08, Matthias J. Sax wrote:
>
>>  From a DSL point of view, users only see the new value on a
>> KTable#filter anyway. So why should it be an issue that we use
>>  pair under the hood?
>>
>> User sees newValue and gets corresponding RecordContext. I can't see any
>> issue here?
>>
>> I cannot follow here:
>>
>> Even when we have a statefull operation last. We move it to the very
 first processor (KtableSource)
 and therefore cant present a proper RecordContext.

>>>
>>
>> With regard to `builder.table().filter()`:
>>
>> I see you point that it would be good to be able to apply the filter()
>> first to reduce the stat store size of the table. But how is this
>> related to KIP-159?
>>
>> Btw: with KIP-182, I am wondering if this would not be possible, by
>> putting a custom dummy store into the table and materialize the filter
>> result afterwards? It's not a nice way

[GitHub] kafka pull request #3448: KAFKA-5526: Report group consumers while offsets a...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/3448


---

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Colin McCabe]

Hi Rajini,

This seems like a nice improvement!

One thing that is a bit concerning is that, if bin/kafka-configs.sh is
used, there is no  way for the broker to give feedback or error
messages.  The broker can't say "sorry, I can't reconfigure that in that
way."  Or even "that configuration property is not reconfigurable in
this version of the software."  It seems like in the examples give,
users will simply set a configuration using bin/kafka-configs.sh, but
then they have to check the broker log files to see if it could actually
be applied.  And even if it couldn't be applied, then it still lingers
in ZooKeeper.

This seems like it would lead to a lot of user confusion, since they get
no feedback when reconfiguring something.  For example, there will be a
lot of scenarios where someone finds a reconfiguration command on
Google, runs it, but then it doesn't do anything because the software
version is different.  And there's no error message or feedback about
this.  It just doesn't work.

To prevent this, I think we should convert bin/kafka-configs.sh to use
AdminClient's AlterConfigsRequest.  This allows us to detect scenarios
where, because of a typo, different software version, or a value of the
wrong type (eg. string vs. int), the given configuration cannot be
applied.  We really should convert kafka-configs.sh to use AdminClient
anyway, for all the usual reasons-- people want to lock down ZooKeeper,
ACLs should be enforced, internal representations should be hidden, we
should support environments where ZK is not exposed, etc. etc.

Another issue that I see here is, how does this interact with downgrade?
 Presumably, if the user downgrades to a version that doesn't support
KIP-226, all the dynamic configurations stored in ZK revert to whatever
value they have in the local config files.  Do we need to have a utility
that can reify the actual applied configuration into a local text file,
to make downgrades less painful?

With regard to upgrades, what happens if we change the name of a
configuration key in the future?  For example, if we decide to rename 
min.insync.replicas to min.in.sync.replicas, presumably we will
deprecate the old key name.  And then perhaps it will be removed in a
future release, such as Apache Kafka 2.0.  In this scenario, should the
Kafka upgrade process change the name of the configuration key in ZK
from min.insync.replicas to min.in.sync.replicas?  Obviously this will
make downgrades harder, if so.  But if it doesn't, then removing
deprecated configuration key synonyms might become very difficult.

best,
Colin


On Wed, Nov 22, 2017, at 12:52, Rajini Sivaram wrote:
> Hi Tom,
> 
> No, I am not proposing this as a way to configure replication quotas.
> When
> you describe broker configs using AdminClient, you will see all the
> configs
> persisted in /configs/brokers/ in ZooKeeper and this includes
> leader.replication.throttled.rate, follower.replication.throttled.rate
> etc. But
> the broker configs that can be altered using AdminClient as a result of
> this KIP are those explicitly stated in the KIP (does not include any of
> the quota configs).
> 
> Regards,
> 
> Rajini
> 
> On Wed, Nov 22, 2017 at 7:54 PM, Tom Bentley 
> wrote:
> 
> > Hi Rajini,
> >
> > Just to clarify, are you proposing this as a way to configure interbroker
> > throttling/quotas? I don't think you are, just wanted to check (since
> > KIP-179 proposes a different mechanism for setting them which supports
> > their automatic removal).
> >
> > Cheers,
> >
> > Tom
> >
> > On 22 November 2017 at 18:28, Rajini Sivaram 
> > wrote:
> >
> > > I have made an update to the KIP to optionally return all the config
> > > synonyms in *DescribeConfigsResponse*. The synonyms are returned in the
> > > order of precedence. AlterConfigsResponse will not be modified by the
> > KIP.
> > > Since many configs already have various overrides (e.g. topic configs
> > with
> > > broker overrides, security properties with listener name overrides) and
> > we
> > > will be adding more levels with dynamic configs, it will be useful to
> > > obtain the full list in order of precedence.
> > >
> > > On Tue, Nov 21, 2017 at 11:24 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > You can quote the config name, but it is not necessary for deleting a
> > > > config since the name doesn't contain any special characters that
> > > requires
> > > > quoting.
> > > >
> > > > On Mon, Nov 20, 2017 at 9:20 PM, Ted Yu  wrote:
> > > >
> > > >> Thanks for the quick response.
> > > >>
> > > >> It seems the config following --delete-config should be quoted.
> > > >>
> > > >> Cheers
> > > >>
> > > >> On Mon, Nov 20, 2017 at 12:02 PM, Rajini Sivaram <
> > > rajinisiva...@gmail.com
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > Ted,
> > > >> >
> > > >> > Have added an example for --delete-config.
> > > >> >
> > > >> > On Mon, Nov 20, 2017 at 7:42 PM, Ted Yu 
> > wrote:
> > > >> >
> > > >> > > bq. There is a --de

Re: [ANNOUNCE] New committer: Onur Karaman

[Jason Gustafson]

Sorry for being late to the party, but congratulations Onur!



On Wed, Nov 8, 2017 at 1:47 AM, Sandeep Nemuri  wrote:

> Congratulations Onur!!
>
> On Wed, Nov 8, 2017 at 9:19 AM, UMESH CHAUDHARY 
> wrote:
>
> > Congratulations Onur!
> >
> > On Tue, 7 Nov 2017 at 21:44 Jun Rao  wrote:
> >
> > > Affan,
> > >
> > > All known problems in the controller are described in the doc linked
> from
> > > https://issues.apache.org/jira/browse/KAFKA-5027.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed  wrote:
> > >
> > > > Congrats Onur,
> > > >
> > > > Can you also share the document where all known problems are listed;
> I
> > am
> > > > assuming these bugs are still valid for the current stable release.
> > > >
> > > > Affan
> > > >
> > > > - Affan
> > > >
> > > > On Mon, Nov 6, 2017 at 10:24 PM, Jun Rao  wrote:
> > > >
> > > > > Hi, everyone,
> > > > >
> > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> committer
> > > Onur
> > > > > Karaman.
> > > > >
> > > > > Onur's most significant work is the improvement of Kafka
> controller,
> > > > which
> > > > > is the brain of a Kafka cluster. Over time, we have accumulated
> > quite a
> > > > few
> > > > > correctness and performance issues in the controller. There have
> been
> > > > > attempts to fix controller issues in isolation, which would make
> the
> > > code
> > > > > base more complicated without a clear path of solving all problems.
> > > Onur
> > > > is
> > > > > the one who took a holistic approach, by first documenting all
> known
> > > > > issues, writing down a new design, coming up with a plan to deliver
> > the
> > > > > changes in phases and executing on it. At this point, Onur has
> > > completed
> > > > > the two most important phases: making the controller single
> threaded
> > > and
> > > > > changing the controller to use the async ZK api. The former fixed
> > > > multiple
> > > > > deadlocks and race conditions. The latter significantly improved
> the
> > > > > performance when there are many partitions. Experimental results
> show
> > > > that
> > > > > Onur's work reduced the controlled shutdown time by a factor of 100
> > > times
> > > > > and the controller failover time by a factor of 3 times.
> > > > >
> > > > > Congratulations, Onur!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *  Regards*
> *  Sandeep Nemuri*
>

[GitHub] kafka pull request #4268: KAFKA-6274: Use topic plus dash as prefix of auto-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4268


---

[jira] [Resolved] (KAFKA-6274) Improve KTable Source state store auto-generated names

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6274.
--
   Resolution: Fixed
Fix Version/s: 1.0.1
   1.1.0

Issue resolved by pull request 4268
[https://github.com/apache/kafka/pull/4268]

> Improve KTable Source state store auto-generated names
> --
>
> Key: KAFKA-6274
> URL: https://issues.apache.org/jira/browse/KAFKA-6274
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 1.1.0, 1.0.1
>
>
> When the source KTable is generated without the store name specified, the 
> auto-generated store name use {{topic}} as the store name prefix. This would 
> generate the store name as
> {code}
> Processor: KTABLE-SOURCE-31 (stores: 
> [windowed-node-countsSTATE-STORE-29])
>   --> none
>   <-- KSTREAM-SOURCE-30
> {code}
> We'd better improve the auto-generated store name as 
> {{[topic-name]-STATE-STORE-suffix}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk8 #2243

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6170; KIP-220 Part 2: Break dependency of Assignor on 
StreamThread

--
[...truncated 1.90 MB...]

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > 
testToStringIsLowerCase STARTED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > 
testToStringIsLowerCase PASSED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > testForValue 
STARTED

org.apache.kafka.connect.runtime.rest.entities.ConnectorTypeTest > testForValue 
PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigCast STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigCast PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigRegexRouter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigRegexRouter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigSetSchemaMetadata STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigSetSchemaMetadata PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampConverter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampConverter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigHoistField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigHoistField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigMaskField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigMaskField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigInsertField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigInsertField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigFlatten STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigFlatten PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigReplaceField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigReplaceField PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampRouter STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigTimestampRouter PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigValueToKey STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigValueToKey PASSED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigExtractField STARTED

org.apache.kafka.connect.runtime.TransformationConfigTest > 
testEmbeddedConfigExtractField PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > unconfiguredTransform 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > unconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSinkConnector STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSinkConnector PASSED

org.apach

[GitHub] kafka pull request #4270: [WIP] KAFKA-6150: Purge repartition topics

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/4270

[WIP] KAFKA-6150: Purge repartition topics

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
K6150-purge-repartition-topics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4270






---

[jira] [Created] (KAFKA-6279) Connect metrics do not get cleaned up for a source connector that doesn't stop properly

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6279:


 Summary: Connect metrics do not get cleaned up for a source 
connector that doesn't stop properly
 Key: KAFKA-6279
 URL: https://issues.apache.org/jira/browse/KAFKA-6279
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Randall Hauch
Assignee: Randall Hauch


Connect will call {{stop()}} on a source task to signal that it should "_stop 
trying to poll for new data 
and interrupt any outstanding poll() requests_" (see the 
[JavaDoc|http://kafka.apache.org/10/javadoc/org/apache/kafka/connect/source/SourceTask.html#stop--]
 for this method).

Unfortunately, not all connectors properly adhere to this expectation. Since 
the metrics for the source task are cleaned up only when the worker source 
task's thread completes, a task whose {{poll()}} method blocks forever will 
current prevent its thread from completing. So, we need to change how the 
metrics are cleaned up to ensure this always happens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4224: KAFKA-6170; KIP-220 Part 2: Break dependency of As...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4224


---

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

[Colin McCabe]

On Mon, Nov 27, 2017, at 22:30, Jan Filipiak wrote:
> Hi Colin, thank you  for this KIP, it can become a really useful thing.
> 
> I just scanned through the discussion so far and wanted to start a 
> thread to make as decision about keeping the
> cache with the Connection / Session or having some sort of UUID indN exed 
> global Map.
> 
> Sorry if that has been settled already and I missed it. In this case 
> could anyone point me to the discussion?

Hi Jan,

I don't think anyone has discussed the idea of tying the cache to an
individual TCP session yet.  I agree that since the cache is intended to
be used only by a single follower or client, it's an interesting thing
to think about.

I guess the obvious disadvantage is that whenever your TCP session
drops, you have to make a full fetch request rather than an incremental
one.  It's not clear to me how often this happens in practice -- it
probably depends a lot on the quality of the network.  From a code
perspective, it might also be a bit difficult to access data associated
with the Session from classes like KafkaApis (although we could refactor
it to make this easier).

It's also clear that even if we tie the cache to the session, we still
have to have limits on the number of caches we're willing to create. 
And probably we should reserve some cache slots for each follower, so
that clients don't take all of them.

> 
> Id rather see a protocol in which the client is hinting the broker that, 
> he is going to use the feature instead of a client
> realizing that the broker just offered the feature (regardless of 
> protocol version which should only indicate that the feature
> would be usable).

Hmm.  I'm not sure what you mean by "hinting."  I do think that the
server should have the option of not accepting incremental requests from
specific clients, in order to save memory space.

> This seems to work better with a per 
> connection/session attached Metadata than with a Map and could allow for
> easier client implementations.
> It would also make Client-side code easier as there wouldn't be any 
> Cache-miss error Messages to handle.

It is nice not to have to handle cache-miss responses, I agree. 
However, TCP sessions aren't exposed to most of our client-side code. 
For example, when the Producer creates a message and hands it off to the
NetworkClient, the NC will transparently re-connect and re-send a
message if the first send failed.  The higher-level code will not be
informed about whether the TCP session was re-established, whether an
existing TCP session was used, and so on.  So overall I would still lean
towards not coupling this to the TCP session...

best,
Colin

> 
>   Thank you again for the KIP. And again, if this was clarified already 
> please drop me a hint where I could read about it.
> 
> Best Jan
> 
> 
> 
> 
> 
> On 21.11.2017 22:02, Colin McCabe wrote:
> > Hi all,
> >
> > I created a KIP to improve the scalability and latency of FetchRequest:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
> >
> > Please take a look.
> >
> > cheers,
> > Colin
> 

[jira] [Resolved] (KAFKA-6249) Interactive query downtime when node goes down even with standby replicas

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6249.

Resolution: Duplicate

> Interactive query downtime when node goes down even with standby replicas
> -
>
> Key: KAFKA-6249
> URL: https://issues.apache.org/jira/browse/KAFKA-6249
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Charles Crain
>
> In a multi-node Kafka Streams application that uses interactive queries, the 
> queryable store will become unavailable (throw InvalidStateStoreException) 
> for up to several minutes when a node goes down.  This happens regardless of 
> how many nodes are in the application as well as how many standby replicas 
> are configured.
> My expectation is that if a standby replica is present, that the interactive 
> query would fail over to the live replica immediately causing negligible 
> downtime for interactive queries.  Instead, what appears to happen is that 
> the queryable store is down for however long it takes for the nodes to 
> completely rebalance (this takes a few minutes for a couple GB of total data 
> in the queryable store's backing topic).
> I am filing this as a bug, realizing that it may in fact be a feature 
> request.  However, until there is a way we can use interactive queries with 
> minimal (~zero) downtime on node failure, we are having to entertain other 
> strategies for serving queries (e.g. manually materializing the topic to an 
> external resilient store such as Cassandra) in order to meet our SLAs.
> If there is a way to minimize the downtime of interactive queries on node 
> failure that I am missing, I would like to know what it is.
> Our team is super-enthusiastic about Kafka Streams and we're keen to use it 
> for just about everything!  This is our only major roadblock.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Support for Gradle 2.X

[Jakub Scholz]

Hi,

According to the GitHub page, "Kafka requires Gradle 2.0 or higher.".
However, when I try to build Kafka using Gradle 2.7, it doesn't work
because of the com.github.johnrengelman.shadow plugin which in the version
used by Kafka requires Gradle 3.0+. There seems to be already a JIRA for
it: https://issues.apache.org/jira/browse/KAFKA-5895

I was wondering if supporting only Gradle 3.0 and higher is intentional and
the bug is in the README file on GitHub or if it is a bug in the Gradle
build and the intention is to continue support Gradle 2.0+.

Could someone help to clarify this?

Thanks & Regards
Jakub

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Matt Farmer]

Hi there, sorry for the delay in responding. Last week had a holiday and
several busy work days in it so I'm just now getting around to responding.

> We would only exclude
> exception Streams can handle itself (like ProducerFencedException) --
> thus, if the handler has code to react to this, it would not be bad, as
> this code is just never called.
[...]
> Thus, I am still in favor of not calling the ProductionExceptionHandler
> for fatal exception.

Acknowledged, so is ProducerFencedException the only kind of exception I
need to change my behavior on? Or are there other types I need to check? Is
there a comprehensive list somewhere?

> About the "always continue" case. Sounds good to me to remove it (not
> sure why we need it in test package?)

I include it in the test package because I have tests that assert that if
the record collector impl encounters an Exception and receives a CONTINUE
that it actually does CONTINUE.

> What is there reasoning for invoking the handler only for the first
> exception?

I didn't want to invoke the handler in places where the CONTINUE or FAIL
result would just be ignored. Presumably, after a FAIL has been returned,
subsequent exceptions are likely to be repeats or noise from my
understanding of the code paths. If this is incorrect we can revisit.

Once I get the answers to these questions I can make another pass on the
pull request!

Matt

On Mon, Nov 20, 2017 at 4:07 PM Matthias J. Sax 
wrote:

> Thanks for following up!
>
> One thought about an older reply from you:
>
>  I strongly disagree here. The purpose of this handler isn't *just* to
>  make a decision for streams. There may also be desirable side effects
> that
>  users wish to cause when production exceptions occur. There may be
>  side-effects that they wish to cause when AuthenticationExceptions
> occur,
>  as well. We should still give them the hooks to preform those side
> effects.
>
> And your follow up:
>
> >>- I think I would rather invoke it for all exceptions that could
> occur
> >>from attempting to produce - even those exceptions were returning
> CONTINUE
> >>may not be a good idea (e.g. Authorization exception). Until there
> is a
> >>different type for exceptions that are totally fatal (for example a
> >>FatalStreamsException or some sort), maintaining a list of
> exceptions that
> >>you can intercept with this handler and exceptions you cannot would
> be
> >>really error-prone and hard to keep correct.
>
> I understand what you are saying, however, consider that Streams needs
> to die for a fatal exception. Thus, if you call the handler for a fatal
> exception, we would  need to ignore the return value and fail -- this
> seems to be rather intuitive. Furthermore, users can register an
> uncaught-exception-handler and side effects for fatal errors can be
> triggered there.
>
> Btw: an AuthorizationException is fatal -- not sure what you mean by an
> "totally fatal" exception -- there is no superlative to fatal from my
> understanding.
>
> About maintaining a list of exceptions: I don't think this is too hard,
> and users also don't need to worry about it IMHO. We would only exclude
> exception Streams can handle itself (like ProducerFencedException) --
> thus, if the handler has code to react to this, it would not be bad, as
> this code is just never called. In case that there is an exception
> Streams could actually handle but we still call the handler (ie, bug),
> there should be no harm either -- the handler needs to return either
> CONTINUE or FAIL and we would obey. It could only happen, that Streams
> dies---as request by the user(!)---even if Streams could actually handle
> the error and move on. But this seems to be not a issue from my point of
> view.
>
> Thus, I am still in favor of not calling the ProductionExceptionHandler
> for fatal exception.
>
>
>
> About the "always continue" case. Sounds good to me to remove it (not
> sure why we need it in test package?) and to rename the "failing"
> handler to "Default" (even if "default" is less descriptive and I would
> still prefer "Fail" in the name).
>
>
> Last question:
>
> >>   - Continue to *only* invoke it on the first exception that we
> >>   encounter (before sendException is set)
>
>
> What is there reasoning for invoking the handler only for the first
> exception?
>
>
>
>
> -Matthias
>
> On 11/20/17 10:43 AM, Matt Farmer wrote:
> > Alright, here are some updates I'm planning to make after thinking on
> this
> > for awhile:
> >
> >- Given that the "always continue" handler isn't something I'd
> recommend
> >for production use as is, I'm going to move it into the test
> namespace and
> >remove it from mention in the public API.
> >- I'm going to rename the "AlwaysFailProductionExceptionHandler" to
> >"DefaultProductionExceptionHandler"
> >- Given that the API for the exception handler involves being invoked
> by
> >streams to make a decision about whe