[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306318#comment-15306318
 ] 

Greg Fodor commented on KAFKA-3769:
---

Actually, additionally it looks like the code path for fetching from RocksDB 
spends most of its time recording the latency metrics :(

> KStream job spending 60% of time writing metrics
> 
>
> Key: KAFKA-3769
> URL: https://issues.apache.org/jira/browse/KAFKA-3769
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
>Priority: Critical
>
> I've been profiling a complex streams job, and found two major hotspots when 
> writing metrics, which take up about 60% of the CPU time of the job. (!) A PR 
> is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306321#comment-15306321
 ] 

Greg Fodor commented on KAFKA-3769:
---

It seems it might be desirable to have a way to just flip off some or all of 
the metrics.

> KStream job spending 60% of time writing metrics
> 
>
> Key: KAFKA-3769
> URL: https://issues.apache.org/jira/browse/KAFKA-3769
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
>Priority: Critical
>
> I've been profiling a complex streams job, and found two major hotspots when 
> writing metrics, which take up about 60% of the CPU time of the job. (!) A PR 
> is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3770) KStream job should be able to specify linger.ms

[Greg Fodor (JIRA)]

Greg Fodor created KAFKA-3770:
-

 Summary: KStream job should be able to specify linger.ms
 Key: KAFKA-3770
 URL: https://issues.apache.org/jira/browse/KAFKA-3770
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Greg Fodor
Assignee: Guozhang Wang


The default linger.ms hardcoded into the StreamsConfig class of 100ms is 
problematic for jobs that have lots of tasks, since this latency can accrue. It 
seems useful to be able to override the linger.ms in the StreamsConfig. 
Attached is a PR which allows this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Add support for linger.ms override

[gfodor]

GitHub user gfodor opened a pull request:

https://github.com/apache/kafka/pull/1448

Add support for linger.ms override



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/AltspaceVR/kafka gfodor/kafka-3770

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1448


commit 7639b61c353b7c5d40bcf27a5e7cc0d114f26d93
Author: Greg Fodor 
Date:   2016-05-30T07:59:27Z

Add support for linger.ms override




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3770) KStream job should be able to specify linger.ms

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306371#comment-15306371
 ] 

Greg Fodor commented on KAFKA-3770:
---

https://github.com/apache/kafka/pull/1448

> KStream job should be able to specify linger.ms
> ---
>
> Key: KAFKA-3770
> URL: https://issues.apache.org/jira/browse/KAFKA-3770
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
>
> The default linger.ms hardcoded into the StreamsConfig class of 100ms is 
> problematic for jobs that have lots of tasks, since this latency can accrue. 
> It seems useful to be able to override the linger.ms in the StreamsConfig. 
> Attached is a PR which allows this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306381#comment-15306381
 ] 

Greg Fodor commented on KAFKA-3769:
---

Consider the PR a first pass, please advise on how we may want to deal with the 
fact that for KStream jobs with lots of tasks, etc, the overhead of writing the 
various process/poll/latency metrics is immense.

> KStream job spending 60% of time writing metrics
> 
>
> Key: KAFKA-3769
> URL: https://issues.apache.org/jira/browse/KAFKA-3769
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
>Priority: Critical
>
> I've been profiling a complex streams job, and found two major hotspots when 
> writing metrics, which take up about 60% of the CPU time of the job. (!) A PR 
> is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control

[Michael Noll (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306409#comment-15306409
 ] 

Michael Noll commented on KAFKA-3478:
-

Latest documentation link (covers flow control with timestamps for Apache Kafka 
0.10.0.0):
http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps

> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to 
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/2.1.0-alpha1/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer 
> control of the input streams, for example, with two input streams, one of 
> them always reading from offset 0 upon (re)-starting, and the other reading 
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that 
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer 
> control over these frameworks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Avoid trace logging computation in `che...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1449

MINOR: Avoid trace logging computation in `checkEnoughReplicasReachOffset`

`numAcks` is only used in the `trace` logging statement so it should be a 
`def` and not a `val`. Also took the chance to improve the code and 
documentation a little.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
minor-avoid-trace-logging-computation-in-partition

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1449


commit 8fdf820e68053b3fe1170a38a9e167afb96caa37
Author: Ismael Juma 
Date:   2016-05-30T09:37:40Z

Avoid computing `numAcks` in `checkEnoughReplicasReachOffset` unless trace 
logging is enabled

commit e1a357e94339dbc8d5533649964d708d5770ce74
Author: Ismael Juma 
Date:   2016-05-30T09:38:31Z

Improve code style in `checkEnoughReplicasReachOffset` and 
`DelayedProduce.tryComplete`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

unsubcribe

[Hafsa Asif]

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Joel Pfaff (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306654#comment-15306654
 ] 

Joel Pfaff commented on KAFKA-2729:
---

We have hit that as well on 0.9.0.1 today, same logs, and only a reboot of the 
faulty broker recovered the problem.

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: HOTFIX: updated JavaDoc example for 0.9 tech-p...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1450

HOTFIX: updated JavaDoc example for 0.9 tech-prev to 0.10



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka hotfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1450


commit 2bbb18ba61cf7d3638e2142f16ca4e931c6a691e
Author: Matthias J. Sax 
Date:   2016-05-30T15:58:59Z

HOTFIX: updated JavaDoc example for 0.9 tech-prev to 0.10




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-3771) Improving Kafka code

[Rekha Joshi (JIRA)]

Rekha Joshi created KAFKA-3771:
--

 Summary: Improving Kafka code
 Key: KAFKA-3771
 URL: https://issues.apache.org/jira/browse/KAFKA-3771
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.10.0.0
Reporter: Rekha Joshi
Assignee: Rekha Joshi


Improve Kafka core code :

Remove redundant val modifier for case class constructor
Use flatMap instead of map and flatten
Use isEmpty, NonEmpty, isDefined as appropriate
Use head, keys and keySet where appropriate
Use contains, diff and find where appropriate
toString has no parameters, no side effect hence without () use consistently
Remove unnecessary return and semi colons, parentheses



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3771; Improving Kafka core code

[rekhajoshm]

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/kafka/pull/1451

KAFKA-3771; Improving Kafka core code

- Used flatMap instead of map and flatten
- Use isEmpty, NonEmpty, isDefined as appropriate
- Used head, keys and keySet where appropriate
- Used contains, diff and find where appropriate
- Removed redundant val modifier for case class constructor
- toString has no parameters, no side effect hence without () consistent 
usage
- Removed unnecessary return , parentheses and semi colons.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/kafka KAFKA-3771

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1451


commit c9a66992b1095616f87c5748f210b973ebc7eb01
Author: Rekha Joshi 
Date:   2016-05-26T17:48:37Z

Merge pull request #2 from apache/trunk

Apache Kafka trunk pull

commit 8d7fb005cb132440e7768a5b74257d2598642e0f
Author: Rekha Joshi 
Date:   2016-05-30T21:37:43Z

Merge pull request #3 from apache/trunk

Apache Kafka trunk pull

commit 7b3273de9d7310695701bd2407992d6c69517f47
Author: Joshi 
Date:   2016-05-30T23:00:58Z

KAFKA-3771; Improving Kafka core code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3771) Improving Kafka code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306997#comment-15306997
 ] 

ASF GitHub Bot commented on KAFKA-3771:
---

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/kafka/pull/1451

KAFKA-3771; Improving Kafka core code

- Used flatMap instead of map and flatten
- Use isEmpty, NonEmpty, isDefined as appropriate
- Used head, keys and keySet where appropriate
- Used contains, diff and find where appropriate
- Removed redundant val modifier for case class constructor
- toString has no parameters, no side effect hence without () consistent 
usage
- Removed unnecessary return , parentheses and semi colons.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/kafka KAFKA-3771

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1451


commit c9a66992b1095616f87c5748f210b973ebc7eb01
Author: Rekha Joshi 
Date:   2016-05-26T17:48:37Z

Merge pull request #2 from apache/trunk

Apache Kafka trunk pull

commit 8d7fb005cb132440e7768a5b74257d2598642e0f
Author: Rekha Joshi 
Date:   2016-05-30T21:37:43Z

Merge pull request #3 from apache/trunk

Apache Kafka trunk pull

commit 7b3273de9d7310695701bd2407992d6c69517f47
Author: Joshi 
Date:   2016-05-30T23:00:58Z

KAFKA-3771; Improving Kafka core code




> Improving Kafka code
> 
>
> Key: KAFKA-3771
> URL: https://issues.apache.org/jira/browse/KAFKA-3771
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Rekha Joshi
>Assignee: Rekha Joshi
>
> Improve Kafka core code :
> Remove redundant val modifier for case class constructor
> Use flatMap instead of map and flatten
> Use isEmpty, NonEmpty, isDefined as appropriate
> Use head, keys and keySet where appropriate
> Use contains, diff and find where appropriate
> toString has no parameters, no side effect hence without () use consistently
> Remove unnecessary return and semi colons, parentheses



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Remove synchronized as the tasks are ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1441


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #663

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: Remove synchronized as the tasks are executed sequentially

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-us1 (Ubuntu ubuntu ubuntu-us golang-ppa) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 3fd9be49ac35adaca401f58552b3ffa68f8d4eaa 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3fd9be49ac35adaca401f58552b3ffa68f8d4eaa
 > git rev-list 404b696bea58aca17fbe528aed03cb3c94516c39 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5501146644400975570.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 19.337 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7931868777399717588.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 19.101 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66

[jira] [Updated] (KAFKA-3335) Kafka Connect hangs in shutdown hook

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3335:
-
Affects Version/s: 0.9.0.1

> Kafka Connect hangs in shutdown hook
> 
>
> Key: KAFKA-3335
> URL: https://issues.apache.org/jira/browse/KAFKA-3335
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Ben Kirwin
>
> The `Connect` class can run into issues during start, such as:
> {noformat}
> Exception in thread "main" org.apache.kafka.connect.errors.ConnectException: 
> Could not look up partition metadata for offset backing store topic in 
> allotted period. This could indicate a connectivity issue, unavailable topic 
> partitions, or if this is your first use of the topic it may have taken too 
> long to create.
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:130)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {noformat}
> This exception halts the startup process. It also triggers the shutdown 
> hook... which blocks waiting for the service to start up before calling stop. 
> This causes the process to hang forever.
> There's a few things that could be done here, but it would be nice to bound 
> the amount of time the process spends trying to exit gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3335) Kafka Connect hangs in shutdown hook

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307077#comment-15307077
 ] 

Ewen Cheslack-Postava commented on KAFKA-3335:
--

[~shikhar] The shutdown hook is added at the beginning to make sure we clean up 
even if something happens during startup -- any services that did get started 
up should be properly cleaned up.

I think a relevant piece of info that was missing is that it looks like this 
was against the 0.9 releases (or a version of trunk after 0.9 and before 0.10) 
and the code has since been cleaned up a bit. The startLatch wasn't previously 
in a finally block which explains why it was never triggered. Since that's 
fixed, it won't block the subsequent stop() call. I've validated by manually 
triggering an exception in both the 0.9.0.1 code and the trunk code and the 
issue is only reproduced in the old release.

> Kafka Connect hangs in shutdown hook
> 
>
> Key: KAFKA-3335
> URL: https://issues.apache.org/jira/browse/KAFKA-3335
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Ben Kirwin
>
> The `Connect` class can run into issues during start, such as:
> {noformat}
> Exception in thread "main" org.apache.kafka.connect.errors.ConnectException: 
> Could not look up partition metadata for offset backing store topic in 
> allotted period. This could indicate a connectivity issue, unavailable topic 
> partitions, or if this is your first use of the topic it may have taken too 
> long to create.
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:130)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {noformat}
> This exception halts the startup process. It also triggers the shutdown 
> hook... which blocks waiting for the service to start up before calling stop. 
> This causes the process to hang forever.
> There's a few things that could be done here, but it would be nice to bound 
> the amount of time the process spends trying to exit gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3335) Kafka Connect hangs in shutdown hook

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3335.
--
   Resolution: Fixed
Fix Version/s: 0.10.0.0

Fixed independently by commit 5092e7f8347 "MINOR: Connect hangs on startup 
failure"

> Kafka Connect hangs in shutdown hook
> 
>
> Key: KAFKA-3335
> URL: https://issues.apache.org/jira/browse/KAFKA-3335
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Ben Kirwin
> Fix For: 0.10.0.0
>
>
> The `Connect` class can run into issues during start, such as:
> {noformat}
> Exception in thread "main" org.apache.kafka.connect.errors.ConnectException: 
> Could not look up partition metadata for offset backing store topic in 
> allotted period. This could indicate a connectivity issue, unavailable topic 
> partitions, or if this is your first use of the topic it may have taken too 
> long to create.
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:130)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {noformat}
> This exception halts the startup process. It also triggers the shutdown 
> hook... which blocks waiting for the service to start up before calling stop. 
> This causes the process to hang forever.
> There's a few things that could be done here, but it would be nice to bound 
> the amount of time the process spends trying to exit gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1327

[Apache Jenkins Server]

See 

[GitHub] kafka pull request: Replace all pattern match on boolean value by ...

[satendrakumar06]

Github user satendrakumar06 closed the pull request at:

https://github.com/apache/kafka/pull/1444


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3768) Replace all pattern match on boolean value by if/elase block.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307207#comment-15307207
 ] 

ASF GitHub Bot commented on KAFKA-3768:
---

Github user satendrakumar06 closed the pull request at:

https://github.com/apache/kafka/pull/1444


> Replace all pattern match on boolean value by if/elase block.
> -
>
> Key: KAFKA-3768
> URL: https://issues.apache.org/jira/browse/KAFKA-3768
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Satendra Kumar
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Scala recommend  use if/else block instead of  pattern match on boolean 
> values.
> For example:
> {code:title=Comparasion.scala|borderStyle=solid}
> class Comparasion {
> def method1(flag: Boolean): String = {
>   flag match {
>  case true => "TRUE"
>  case false => "FALSE"
>}
> }
>   def method2(flag: Boolean): String = {
>   if(flag) {
>"TRUE"
>  }else {
>"FALSE"
>  }
>   }
> }
> {code}
> Byte code comparison between method1 and method2:
> scala>javap -cp Comparasion
> {code:title=Comparasion.class|borderStyle=solid}
> Compiled from ""
> public class Comparasion {
>   public java.lang.String method1(boolean);
> Code:
>0: iload_1
>1: istore_2
>2: iconst_1
>3: iload_2
>4: if_icmpne 13
>7: ldc   #9  // String TRUE
>9: astore_3
>   10: goto  21
>   13: iconst_0
>   14: iload_2
>   15: if_icmpne 23
>   18: ldc   #11 // String FALSE
>   20: astore_3
>   21: aload_3
>   22: areturn
>   23: new   #13 // class scala/MatchError
>   26: dup
>   27: iload_2
>   28: invokestatic  #19 // Method 
> scala/runtime/BoxesRunTime.boxToBoolean:(Z)Ljava/lang/Boolean;
>   31: invokespecial #23 // Method 
> scala/MatchError."":(Ljava/lang/Object;)V
>   34: athrow
>   public java.lang.String method2(boolean);
> Code:
>0: iload_1
>1: ifeq  9
>4: ldc   #9  // String TRUE
>6: goto  11
>9: ldc   #11 // String FALSE
>   11: areturn
>   public Comparasion();
> Code:
>0: aload_0
>1: invokespecial #33 // Method 
> java/lang/Object."":()V
>4: return
> }
> {code}
> method1 have 23 line of byte code and method2 have only 6 line  byte code. 
> Pattern match are more expensive comparison to if/else block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

kafka consumer group is rebalancing

[Sunny Gupta]

Hi,
I am using Kafka .9 and new Java consumer. I am polling inside a loop. I am
getting commitfailedexcption because of group rebalance, when code try to
execute consumer.commitSycn . Please note, I am adding session.timeout.ms
as 3 and heartbeat.interval.ms as 1 to consumer and polling happens
for sure with in 3. Can anyone help me out. Please let me know if any
information is needed.
I am using 3 node kafka cluster.
Thanks,
Sunny