Build failed in Jenkins: kafka-trunk-jdk8 #3003

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H32 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 3593, done.
remote: Counting objects:   0% (1/3593)   remote: Counting objects:   
1% (36/3593)   remote: Counting objects:   2% (72/3593)   
remote: Counting objects:   3% (108/3593)   remote: Counting objects:   
4% (144/3593)   remote: Counting objects:   5% (180/3593)   
remote: Counting objects:   6% (216/3593)   remote: Counting objects:   
7% (252/3593)   remote: Counting objects:   8% (288/3593)   
remote: Counting objects:   9% (324/3593)   remote: Counting objects:  
10% (360/3593)   remote: Counting objects:  11% (396/3593)   
remote: Counting objects:  12% (432/3593)   remote: Counting objects:  
13% (468/3593)   remote: Counting objects:  14% (504/3593)   
remote: Counting objects:  15% (539/3593)   remote: Counting objects:  
16% (575/3593)   remote: Counting objects:  17% (611/3593)   
remote: Counting objects:  18% (647/3593)   remote: Counting objects:  
19% (683/3593)   remote: Counting objects:  20% (719/3593)   
remote: Counting objects:  21% (755/3593)   remote: Counting objects:  
22% (791/3593)   remote: Counting objects:  23% (827/3593)   
remote: Counting objects:  24% (863/3593)   remote: Counting objects:  
25% (899/3593)   remote: Counting objects:  26% (935/3593)   
remote: Counting objects:  27% (971/3593)   remote: Counting objects:  
28% (1007/3593)   remote: Counting objects:  29% (1042/3593)   
remote: Counting objects:  30% (1078/3593)   remote: Counting objects:  
31% (1114/3593)   remote: Counting objects:  32% (1150/3593)   
remote: Counting objects:  33% (1186/3593)   remote: Counting objects:  
34% (1222/3593)   remote: Counting objects:  35% (1258/3593)   
remote: Counting objects:  36% (1294/3593)   remote: Counting objects:  
37% (1330/3593)   remote: Counting objects:  38% (1366/3593)   
remote: Counting objects:  39% (1402/3593)   remote: Counting objects:  
40% (1438/3593)   remote: Counting objects:  41% (1474/3593)   
remote: Counting objects:  42% (1510/3593)   remote: Counting objects:  
43% (1545/3593)   remote: Counting objects:  44% (1581/3593)   
remote: Counting objects:  45% (1617/3593)   remote: Counting objects:  
46% (1653/3593)   remote: Counting objects:  47% (1689/3593)   
remote: Counting objects:  48% (1725/3593)   remote: Counting objects:  
49% (1761/3593)   remote: Counting objects:  50% (1797/3593)   
remote: Counting objects:  51% (1833/3593)   remote: Counting objects:  
52% (1869/3593)   remote: Counting objects:  53% (1905/3593)   
remote: Counting objects:  54% (1941/3593)   remote: Counting objects:  
55% (1977/3593)   remote: Counting objects:  56% (2013/3593)   
remote: Counting objects:  57% (2049/3593)   

[jira] [Created] (KAFKA-7458) Avoid enforced processing during bootstrap phase

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-7458:


 Summary: Avoid enforced processing during bootstrap phase
 Key: KAFKA-7458
 URL: https://issues.apache.org/jira/browse/KAFKA-7458
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


In KAFKA-3514, we introduced a new config for allowing users to delay enforcing 
processing without all input topic partitions to have data. This config's 
default value is 0, which means that as long as the first fetch does not 
contains some records for all the partitions it will fall into enforced 
processing immediately, which is a high risk especially under bootstrap case.

We should consider leveraging on pause / resume to make sure that upon 
starting, some partition indeed does not have any data before we fall into 
enforced processing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk10 #530

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #3002

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2668, done.
remote: Counting objects:   0% (1/2668)   remote: Counting objects:   
1% (27/2668)   remote: Counting objects:   2% (54/2668)   
remote: Counting objects:   3% (81/2668)   remote: Counting objects:   
4% (107/2668)   remote: Counting objects:   5% (134/2668)   
remote: Counting objects:   6% (161/2668)   remote: Counting objects:   
7% (187/2668)   remote: Counting objects:   8% (214/2668)   
remote: Counting objects:   9% (241/2668)   remote: Counting objects:  
10% (267/2668)   remote: Counting objects:  11% (294/2668)   
remote: Counting objects:  12% (321/2668)   remote: Counting objects:  
13% (347/2668)   remote: Counting objects:  14% (374/2668)   
remote: Counting objects:  15% (401/2668)   remote: Counting objects:  
16% (427/2668)   remote: Counting objects:  17% (454/2668)   
remote: Counting objects:  18% (481/2668)   remote: Counting objects:  
19% (507/2668)   remote: Counting objects:  20% (534/2668)   
remote: Counting objects:  21% (561/2668)   remote: Counting objects:  
22% (587/2668)   remote: Counting objects:  23% (614/2668)   
remote: Counting objects:  24% (641/2668)   remote: Counting objects:  
25% (667/2668)   remote: Counting objects:  26% (694/2668)   
remote: Counting objects:  27% (721/2668)   remote: Counting objects:  
28% (748/2668)   remote: Counting objects:  29% (774/2668)   
remote: Counting objects:  30% (801/2668)   remote: Counting objects:  
31% (828/2668)   remote: Counting objects:  32% (854/2668)   
remote: Counting objects:  33% (881/2668)   remote: Counting objects:  
34% (908/2668)   remote: Counting objects:  35% (934/2668)   
remote: Counting objects:  36% (961/2668)   remote: Counting objects:  
37% (988/2668)   remote: Counting objects:  38% (1014/2668)   
remote: Counting objects:  39% (1041/2668)   remote: Counting objects:  
40% (1068/2668)   remote: Counting objects:  41% (1094/2668)   
remote: Counting objects:  42% (1121/2668)   remote: Counting objects:  
43% (1148/2668)   remote: Counting objects:  44% (1174/2668)   
remote: Counting objects:  45% (1201/2668)   remote: Counting objects:  
46% (1228/2668)   remote: Counting objects:  47% (1254/2668)   
remote: Counting objects:  48% (1281/2668)   remote: Counting objects:  
49% (1308/2668)   remote: Counting objects:  50% (1334/2668)   
remote: Counting objects:  51% (1361/2668)   remote: Counting objects:  
52% (1388/2668)   remote: Counting objects:  53% (1415/2668)   
remote: Counting objects:  54% (1441/2668)   remote: Counting objects:  
55% 

Re: [STREAMS] Qs on code of State Managers (AbstractStateManager, ProcessorStateManager and GlobalStateManagerImpl)

[Guozhang Wang]

Hello Jacek,

Your observation is valid, currently the code hierarchy of
AbstractStateManager, ProcessorStateManager and GlobalStateManagerImpl are
a bit intrusive to each other and would better be cleaned a bit.

Feel free to create a JIRA with your observed placed to clean up and submit
a PR.


Guozhang


On Tue, Sep 25, 2018 at 1:56 AM, Jacek Laskowski  wrote:

> Hi Devs,
>
> I'm reviewing state managers and noticed that ProcessorStateManager has
> registerGlobalStateStores method that is used exclusively when StreamTask
> is created [1] (that seems a bit weird given global state stores
> are GlobalStateManagerImpl's actually).
>
> That leads to another question about AbstractStateManager and its two
> internal registries stores and globalStores that seem to be managed
> by ProcessorStateManager and GlobalStateManagerImpl, respectively (except
> the above case when ProcessorStateManager deals with global state store).
>
> Two questions then:
>
> 1. Why does ProcessorStateManager do registerGlobalStateStores and register
> global state stores (since they are registered and managed elsewhere if
> there are global state stores in use)?
>
> 2. Why does AbstractStateManager have stores and globalStores since they
> should really be in ProcessorStateManager and GlobalStateManagerImpl,
> respectively?
>
> There must be something I don't understand yet and hence the questions.
> Please help. Thanks.
>
> [1]
> https://github.com/apache/kafka/blob/trunk/streams/src/
> main/java/org/apache/kafka/streams/processor/internals/
> StreamTask.java#L240
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> Mastering Spark SQL https://bit.ly/mastering-spark-sql
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
> Follow me at https://twitter.com/jaceklaskowski
>



-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk10 #529

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 379211134740268b570fc8edd59ae78df0dffee9
error: Could not read 57d7f11e38e41892191f6fe87faae8f23aa0362e
remote: Enumerating objects: 3640, done.
remote: Counting objects:   0% (1/3640)   remote: Counting objects:   
1% (37/3640)   remote: Counting objects:   2% (73/3640)   
remote: Counting objects:   3% (110/3640)   remote: Counting objects:   
4% (146/3640)   remote: Counting objects:   5% (182/3640)   
remote: Counting objects:   6% (219/3640)   remote: Counting objects:   
7% (255/3640)   remote: Counting objects:   8% (292/3640)   
remote: Counting objects:   9% (328/3640)   remote: Counting objects:  
10% (364/3640)   remote: Counting objects:  11% (401/3640)   
remote: Counting objects:  12% (437/3640)   remote: Counting objects:  
13% (474/3640)   remote: Counting objects:  14% (510/3640)   
remote: Counting objects:  15% (546/3640)   remote: Counting objects:  
16% (583/3640)   remote: Counting objects:  17% (619/3640)   
remote: Counting objects:  18% (656/3640)   remote: Counting objects:  
19% (692/3640)   remote: Counting objects:  20% (728/3640)   
remote: Counting objects:  21% (765/3640)   remote: Counting objects:  
22% (801/3640)   remote: Counting objects:  23% (838/3640)   
remote: Counting objects:  24% (874/3640)   remote: Counting objects:  
25% (910/3640)   remote: Counting objects:  26% (947/3640)   
remote: Counting objects:  27% (983/3640)   remote: Counting objects:  
28% (1020/3640)   remote: Counting objects:  29% (1056/3640)   
remote: Counting objects:  30% (1092/3640)   remote: Counting objects:  
31% (1129/3640)   remote: Counting objects:  32% (1165/3640)   
remote: Counting objects:  33% (1202/3640)   remote: Counting objects:  
34% (1238/3640)   remote: Counting objects:  35% (1274/3640)   
remote: Counting objects:  36% (1311/3640)   remote: Counting objects:  
37% (1347/3640)   remote: Counting objects:  38% (1384/3640)   
remote: Counting objects:  39% (1420/3640)   remote: Counting objects:  
40% (1456/3640)   remote: Counting objects:  41% (1493/3640)   
remote: Counting objects:  42% (1529/3640)   remote: Counting objects:  
43% (1566/3640)   remote: Counting objects:  44% (1602/3640)   
remote: Counting objects:  45% (1638/3640)   remote: Counting objects:  
46% (1675/3640)   remote: Counting objects:  47% (1711/3640)   
remote: Counting objects:  48% (1748/3640)   remote: Counting objects:  
49% (1784/3640)   remote: Counting objects:  50% (1820/3640)   
remote: Counting objects:  51% (1857/3640)   remote: Counting objects:  
52% (1893/3640)   remote: Counting objects:  53% (1930/3640)   
remote: Counting objects:  54% (1966/3640)   remote: Counting objects:  
55% 

[jira] [Resolved] (KAFKA-6620) Documentation about "exactly_once" doesn't mention "transaction.state.log.min.isr"

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6620.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> Documentation about "exactly_once" doesn't mention 
> "transaction.state.log.min.isr" 
> ---
>
> Key: KAFKA-6620
> URL: https://issues.apache.org/jira/browse/KAFKA-6620
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation, streams
>Reporter: Daniel Qian
>Assignee: Lee Dongjin
>Priority: Major
> Fix For: 2.1.0
>
>
> Documentation about "processing.guarantee" says:
> {quote}The processing guarantee that should be used. Possible values are 
> {{at_least_once}}(default) and {{exactly_once}}. Note that exactly-once 
> processing requires a cluster of at least three brokers by default what is 
> the recommended setting for production; *for development you can change this, 
> by adjusting broker setting* 
> `{color:#FF}*transaction.state.log.replication.factor*{color}`
> {quote}
> If one only set *transaction.state.log.replication.factor=1* but leave 
> *transaction.state.log.min.isr* with default value (which is 2) the Streams 
> Application will break.
> Hope you guys modify the doc, thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2999

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2674, done.
remote: Counting objects:   0% (1/2674)   remote: Counting objects:   
1% (27/2674)   remote: Counting objects:   2% (54/2674)   
remote: Counting objects:   3% (81/2674)   remote: Counting objects:   
4% (107/2674)   remote: Counting objects:   5% (134/2674)   
remote: Counting objects:   6% (161/2674)   remote: Counting objects:   
7% (188/2674)   remote: Counting objects:   8% (214/2674)   
remote: Counting objects:   9% (241/2674)   remote: Counting objects:  
10% (268/2674)   remote: Counting objects:  11% (295/2674)   
remote: Counting objects:  12% (321/2674)   remote: Counting objects:  
13% (348/2674)   remote: Counting objects:  14% (375/2674)   
remote: Counting objects:  15% (402/2674)   remote: Counting objects:  
16% (428/2674)   remote: Counting objects:  17% (455/2674)   
remote: Counting objects:  18% (482/2674)   remote: Counting objects:  
19% (509/2674)   remote: Counting objects:  20% (535/2674)   
remote: Counting objects:  21% (562/2674)   remote: Counting objects:  
22% (589/2674)   remote: Counting objects:  23% (616/2674)   
remote: Counting objects:  24% (642/2674)   remote: Counting objects:  
25% (669/2674)   remote: Counting objects:  26% (696/2674)   
remote: Counting objects:  27% (722/2674)   remote: Counting objects:  
28% (749/2674)   remote: Counting objects:  29% (776/2674)   
remote: Counting objects:  30% (803/2674)   remote: Counting objects:  
31% (829/2674)   remote: Counting objects:  32% (856/2674)   
remote: Counting objects:  33% (883/2674)   remote: Counting objects:  
34% (910/2674)   remote: Counting objects:  35% (936/2674)   
remote: Counting objects:  36% (963/2674)   remote: Counting objects:  
37% (990/2674)   remote: Counting objects:  38% (1017/2674)   
remote: Counting objects:  39% (1043/2674)   remote: Counting objects:  
40% (1070/2674)   remote: Counting objects:  41% (1097/2674)   
remote: Counting objects:  42% (1124/2674)   remote: Counting objects:  
43% (1150/2674)   remote: Counting objects:  44% (1177/2674)   
remote: Counting objects:  45% (1204/2674)   remote: Counting objects:  
46% (1231/2674)   remote: Counting objects:  47% (1257/2674)   
remote: Counting objects:  48% (1284/2674)   remote: Counting objects:  
49% (1311/2674)   remote: Counting objects:  50% (1337/2674)   
remote: Counting objects:  51% (1364/2674)   remote: Counting objects:  
52% (1391/2674)   remote: Counting objects:  53% (1418/2674)   
remote: Counting objects:  54% (1444/2674)   remote: Counting objects:  
55% 

Build failed in Jenkins: kafka-trunk-jdk10 #528

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 379211134740268b570fc8edd59ae78df0dffee9
error: Could not read 57d7f11e38e41892191f6fe87faae8f23aa0362e
remote: Enumerating objects: 3631, done.
remote: Counting objects:   0% (1/3631)   remote: Counting objects:   
1% (37/3631)   remote: Counting objects:   2% (73/3631)   
remote: Counting objects:   3% (109/3631)   remote: Counting objects:   
4% (146/3631)   remote: Counting objects:   5% (182/3631)   
remote: Counting objects:   6% (218/3631)   remote: Counting objects:   
7% (255/3631)   remote: Counting objects:   8% (291/3631)   
remote: Counting objects:   9% (327/3631)   remote: Counting objects:  
10% (364/3631)   remote: Counting objects:  11% (400/3631)   
remote: Counting objects:  12% (436/3631)   remote: Counting objects:  
13% (473/3631)   remote: Counting objects:  14% (509/3631)   
remote: Counting objects:  15% (545/3631)   remote: Counting objects:  
16% (581/3631)   remote: Counting objects:  17% (618/3631)   
remote: Counting objects:  18% (654/3631)   remote: Counting objects:  
19% (690/3631)   remote: Counting objects:  20% (727/3631)   
remote: Counting objects:  21% (763/3631)   remote: Counting objects:  
22% (799/3631)   remote: Counting objects:  23% (836/3631)   
remote: Counting objects:  24% (872/3631)   remote: Counting objects:  
25% (908/3631)   remote: Counting objects:  26% (945/3631)   
remote: Counting objects:  27% (981/3631)   remote: Counting objects:  
28% (1017/3631)   remote: Counting objects:  29% (1053/3631)   
remote: Counting objects:  30% (1090/3631)   remote: Counting objects:  
31% (1126/3631)   remote: Counting objects:  32% (1162/3631)   
remote: Counting objects:  33% (1199/3631)   remote: Counting objects:  
34% (1235/3631)   remote: Counting objects:  35% (1271/3631)   
remote: Counting objects:  36% (1308/3631)   remote: Counting objects:  
37% (1344/3631)   remote: Counting objects:  38% (1380/3631)   
remote: Counting objects:  39% (1417/3631)   remote: Counting objects:  
40% (1453/3631)   remote: Counting objects:  41% (1489/3631)   
remote: Counting objects:  42% (1526/3631)   remote: Counting objects:  
43% (1562/3631)   remote: Counting objects:  44% (1598/3631)   
remote: Counting objects:  45% (1634/3631)   remote: Counting objects:  
46% (1671/3631)   remote: Counting objects:  47% (1707/3631)   
remote: Counting objects:  48% (1743/3631)   remote: Counting objects:  
49% (1780/3631)   remote: Counting objects:  50% (1816/3631)   
remote: Counting objects:  51% (1852/3631)   remote: Counting objects:  
52% (1889/3631)   remote: Counting objects:  53% (1925/3631)   
remote: Counting objects:  54% (1961/3631)   remote: Counting objects:  
55% 

[jira] [Created] (KAFKA-7457) AbstractCoordinator struck in discover

[Joseph Aliase (JIRA)]

Joseph Aliase created KAFKA-7457:


 Summary: AbstractCoordinator struck in discover
 Key: KAFKA-7457
 URL: https://issues.apache.org/jira/browse/KAFKA-7457
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.1.1
 Environment: Linux
Reporter: Joseph Aliase


AbstractCoordinator in kafka-client is stuck in discover and never rejoins the 
group. Post restart application is able to join the consumer group and consume 
from the topic.

We see below logs every 10 minute. The sequence of events are:

a) NetworkClient complains that connection is idle and closes the connection.

b) Consumer client tries to determine co-ordinator by sending request to Node 2.

c) Node 2 responds by saying Node 3 is group co-ordinator.

d) Consumer client connects to group co-ordinator.

e) There is radio silence for 10 minutes and the sequence gets repeated. 

 

2018-09-28 16:35:59.771 TRACE org.apache.kafka.common.network.Selector 
[pool-4-thread-50] [active] [wc] About to close the idle connection from 
2147483644 due to being idle for 540140 millis
2018-09-28 16:35:59.771 DEBUG org.apache.kafka.clients.NetworkClient 
[pool-4-thread-50] [active] [wc] Node 2147483644 disconnected.
2018-09-28 16:35:59.771 INFO 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator 
[pool-4-thread-50] [active] [wc] Marking the coordinator kafka03-wc.net:9092 
(id: 2147483644 rack: null) dead for group test
2018-09-28 16:35:59.771 DEBUG 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator 
[pool-4-thread-50] [active] [wc] Sending coordinator request for group test to 
broker kafka02-wc.net:9092 (id: 2 rack: null)
2018-09-28 16:35:59.771 DEBUG org.apache.kafka.clients.NetworkClient 
[pool-4-thread-50] [active] [wc] Sending metadata request \{topics=[address]} 
to node 2
2018-09-28 16:35:59.796 DEBUG org.apache.kafka.clients.Metadata 
[pool-4-thread-50] [active] [wc] Updated cluster metadata version 401 to 
Cluster(id = oC0BPXfOT42WqN7-v6b5Gw, nodes = [kafka02-wc.net:9092 (id: 2 rack: 
null), kafka03-wc.net:9092 (id: 3 rack: null), kafka05-wc.net:9092 (id: 5 rack: 
null), kafka01-wc.net:9092 (id: 1 rack: null), kafka04-wc.net:9092 (id: 4 rack: 
null)], partitions = [Partition(topic = address, partition = 2, leader = 1, 
replicas = [1,5,4,], isr = [5,1,4,]), Partition(topic = address, partition = 1, 
leader = 5, replicas = [5,3,4,], isr = [5,3,4,]), Partition(topic = address, 
partition = 0, leader = 4, replicas = [2,3,4,], isr = [4,3,2,]), 
Partition(topic = address, partition = 6, leader = 5, replicas = [1,5,4,], isr 
= [5,1,4,]), Partition(topic = address, partition = 5, leader = 4, replicas = 
[5,3,4,], isr = [5,4,3,]), Partition(topic = address, partition = 4, leader = 
3, replicas = [1,2,3,], isr = [1,3,2,]), Partition(topic = address, partition = 
3, leader = 2, replicas = [1,5,2,], isr = [5,1,2,]), Partition(topic = address, 
partition = 16, leader = 5, replicas = [5,2,3,], isr = [5,3,2,]), 
Partition(topic = address, partition = 15, leader = 4, replicas = [1,2,4,], isr 
= [1,4,2,]), Partition(topic = address, partition = 10, leader = 4, replicas = 
[1,5,4,], isr = [5,1,4,]), Partition(topic = address, partition = 9, leader = 
3, replicas = [2,3,4,], isr = [3,4,2,]), Partition(topic = address, partition = 
8, leader = 2, replicas = [1,2,3,], isr = [1,3,2,]), Partition(topic = address, 
partition = 7, leader = 1, replicas = [1,5,2,], isr = [5,1,2,]), 
Partition(topic = address, partition = 14, leader = 3, replicas = [5,3,4,], isr 
= [5,4,3,]), Partition(topic = address, partition = 13, leader = 2, replicas = 
[2,3,4,], isr = [3,4,2,]), Partition(topic = address, partition = 12, leader = 
1, replicas = [1,2,3,], isr = [1,3,2,]), Partition(topic = address, partition = 
11, leader = 5, replicas = [1,5,2,], isr = [5,1,2,])])
2018-09-28 16:35:59.797 DEBUG 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator 
[pool-4-thread-50] [active] [wc] Received group coordinator response 
ClientResponse(receivedTimeMs=1538152559797, disconnected=false, 
request=ClientRequest(expectResponse=true, 
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@2531ff5f,
 
request=RequestSend(header=\{api_key=10,api_version=0,correlation_id=8354,client_id=active-wc-1-256},
 body=\{group_id=test}), createdTimeMs=1538152559771, 
sendTimeMs=1538152559771), 
responseBody=\{error_code=0,coordinator={node_id=3,host=kafka03-wc.net,port=9092}})
2018-09-28 16:35:59.797 INFO 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator 
[pool-4-thread-50] [active] [wc] Discovered coordinator kafka03-wc.net:9092 
(id: 2147483644 rack: null) for group test.
2018-09-28 16:35:59.797 INFO 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator 
[pool-4-thread-50] [active] [wc] Marking the coordinator kafka03-wc.net:9092 
(id: 

Build failed in Jenkins: kafka-trunk-jdk8 #2998

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2674, done.
remote: Counting objects:   0% (1/2674)   remote: Counting objects:   
1% (27/2674)   remote: Counting objects:   2% (54/2674)   
remote: Counting objects:   3% (81/2674)   remote: Counting objects:   
4% (107/2674)   remote: Counting objects:   5% (134/2674)   
remote: Counting objects:   6% (161/2674)   remote: Counting objects:   
7% (188/2674)   remote: Counting objects:   8% (214/2674)   
remote: Counting objects:   9% (241/2674)   remote: Counting objects:  
10% (268/2674)   remote: Counting objects:  11% (295/2674)   
remote: Counting objects:  12% (321/2674)   remote: Counting objects:  
13% (348/2674)   remote: Counting objects:  14% (375/2674)   
remote: Counting objects:  15% (402/2674)   remote: Counting objects:  
16% (428/2674)   remote: Counting objects:  17% (455/2674)   
remote: Counting objects:  18% (482/2674)   remote: Counting objects:  
19% (509/2674)   remote: Counting objects:  20% (535/2674)   
remote: Counting objects:  21% (562/2674)   remote: Counting objects:  
22% (589/2674)   remote: Counting objects:  23% (616/2674)   
remote: Counting objects:  24% (642/2674)   remote: Counting objects:  
25% (669/2674)   remote: Counting objects:  26% (696/2674)   
remote: Counting objects:  27% (722/2674)   remote: Counting objects:  
28% (749/2674)   remote: Counting objects:  29% (776/2674)   
remote: Counting objects:  30% (803/2674)   remote: Counting objects:  
31% (829/2674)   remote: Counting objects:  32% (856/2674)   
remote: Counting objects:  33% (883/2674)   remote: Counting objects:  
34% (910/2674)   remote: Counting objects:  35% (936/2674)   
remote: Counting objects:  36% (963/2674)   remote: Counting objects:  
37% (990/2674)   remote: Counting objects:  38% (1017/2674)   
remote: Counting objects:  39% (1043/2674)   remote: Counting objects:  
40% (1070/2674)   remote: Counting objects:  41% (1097/2674)   
remote: Counting objects:  42% (1124/2674)   remote: Counting objects:  
43% (1150/2674)   remote: Counting objects:  44% (1177/2674)   
remote: Counting objects:  45% (1204/2674)   remote: Counting objects:  
46% (1231/2674)   remote: Counting objects:  47% (1257/2674)   
remote: Counting objects:  48% (1284/2674)   remote: Counting objects:  
49% (1311/2674)   remote: Counting objects:  50% (1337/2674)   
remote: Counting objects:  51% (1364/2674)   remote: Counting objects:  
52% (1391/2674)   remote: Counting objects:  53% (1418/2674)   
remote: Counting objects:  54% (1444/2674)   remote: Counting objects:  
55% 

Build failed in Jenkins: kafka-trunk-jdk10 #527

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 379211134740268b570fc8edd59ae78df0dffee9
error: Could not read 57d7f11e38e41892191f6fe87faae8f23aa0362e
remote: Enumerating objects: 3631, done.
remote: Counting objects:   0% (1/3631)   remote: Counting objects:   
1% (37/3631)   remote: Counting objects:   2% (73/3631)   
remote: Counting objects:   3% (109/3631)   remote: Counting objects:   
4% (146/3631)   remote: Counting objects:   5% (182/3631)   
remote: Counting objects:   6% (218/3631)   remote: Counting objects:   
7% (255/3631)   remote: Counting objects:   8% (291/3631)   
remote: Counting objects:   9% (327/3631)   remote: Counting objects:  
10% (364/3631)   remote: Counting objects:  11% (400/3631)   
remote: Counting objects:  12% (436/3631)   remote: Counting objects:  
13% (473/3631)   remote: Counting objects:  14% (509/3631)   
remote: Counting objects:  15% (545/3631)   remote: Counting objects:  
16% (581/3631)   remote: Counting objects:  17% (618/3631)   
remote: Counting objects:  18% (654/3631)   remote: Counting objects:  
19% (690/3631)   remote: Counting objects:  20% (727/3631)   
remote: Counting objects:  21% (763/3631)   remote: Counting objects:  
22% (799/3631)   remote: Counting objects:  23% (836/3631)   
remote: Counting objects:  24% (872/3631)   remote: Counting objects:  
25% (908/3631)   remote: Counting objects:  26% (945/3631)   
remote: Counting objects:  27% (981/3631)   remote: Counting objects:  
28% (1017/3631)   remote: Counting objects:  29% (1053/3631)   
remote: Counting objects:  30% (1090/3631)   remote: Counting objects:  
31% (1126/3631)   remote: Counting objects:  32% (1162/3631)   
remote: Counting objects:  33% (1199/3631)   remote: Counting objects:  
34% (1235/3631)   remote: Counting objects:  35% (1271/3631)   
remote: Counting objects:  36% (1308/3631)   remote: Counting objects:  
37% (1344/3631)   remote: Counting objects:  38% (1380/3631)   
remote: Counting objects:  39% (1417/3631)   remote: Counting objects:  
40% (1453/3631)   remote: Counting objects:  41% (1489/3631)   
remote: Counting objects:  42% (1526/3631)   remote: Counting objects:  
43% (1562/3631)   remote: Counting objects:  44% (1598/3631)   
remote: Counting objects:  45% (1634/3631)   remote: Counting objects:  
46% (1671/3631)   remote: Counting objects:  47% (1707/3631)   
remote: Counting objects:  48% (1743/3631)   remote: Counting objects:  
49% (1780/3631)   remote: Counting objects:  50% (1816/3631)   
remote: Counting objects:  51% (1852/3631)   remote: Counting objects:  
52% (1889/3631)   remote: Counting objects:  53% (1925/3631)   
remote: Counting objects:  54% (1961/3631)   remote: Counting objects:  
55% 

Re: [VOTE] KIP-372: Naming Repartition Topics for Joins and Grouping

[Bill Bejeck]

All,

I've made a slight adjustment to the KIP and changed the "Joined.name"  to
"Joined.named"

Thanks,
Bill

On Thu, Sep 20, 2018 at 4:49 PM Bill Bejeck  wrote:

> Hi All,
>
> KIP-372 is now accepted with:
> - 3 Binding +1 (Damian, Guozhang, Matthias)
> - 3 Non-binding +1 (Dongjin, John, Bill)
>
> Thanks to everyone for the votes.
>
> -Bill
>
> On Thu, Sep 20, 2018 at 7:30 AM Damian Guy  wrote:
>
>> +1 (binding)
>>
>> On Tue, 18 Sep 2018 at 16:33 Bill Bejeck  wrote:
>>
>> > All,
>> >
>> > In starting work on the PR for KIP-372, the Grouped interface needed
>> some
>> > method renaming to be more consistent with the other configuration
>> classes
>> > (Joined, Produced, etc.).  As such I've updated the Grouped code
>> section of
>> > the KIP.
>> >
>> > As these changes address a comment from Matthias on the initial draft of
>> > the KIP and don't change any of the existing behavior already
>> outlined,  I
>> > don't think a re-vote is required.
>> >
>> > Thanks,
>> > Bill
>> >
>> > On Tue, Sep 18, 2018 at 10:09 AM John Roesler 
>> wrote:
>> >
>> > > +1 (non-binding)
>> > >
>> > > Thanks!
>> > >
>> > > On Mon, Sep 17, 2018 at 7:29 PM Dongjin Lee 
>> wrote:
>> > >
>> > > > Great improvements. +1. (Non-binding)
>> > > >
>> > > > On Tue, Sep 18, 2018 at 5:14 AM Matthias J. Sax <
>> matth...@confluent.io
>> > >
>> > > > wrote:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > -Matthias
>> > > > >
>> > > > > On 9/17/18 1:12 PM, Guozhang Wang wrote:
>> > > > > > +1 from me, thanks Bill !
>> > > > > >
>> > > > > > On Mon, Sep 17, 2018 at 12:43 PM, Bill Bejeck <
>> bbej...@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > >> All,
>> > > > > >>
>> > > > > >> I'd like to start the voting process for KIP-372.  Here's the
>> link
>> > > to
>> > > > > the
>> > > > > >> updated proposal
>> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > >> 372%3A+Naming+Repartition+Topics+for+Joins+and+Grouping
>> > > > > >>
>> > > > > >> I'll start with my own +1.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Bill
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > *Dongjin Lee*
>> > > >
>> > > > *A hitchhiker in the mathematical world.*
>> > > >
>> > > > *github:  github.com/dongjinleekr
>> > > > linkedin:
>> > > kr.linkedin.com/in/dongjinleekr
>> > > > slideshare:
>> > > > www.slideshare.net/dongjinleekr
>> > > > *
>> > > >
>> > >
>> >
>>
>

[jira] [Created] (KAFKA-7456) Serde Inheritance in Streams DSL

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-7456:


 Summary: Serde Inheritance in Streams DSL
 Key: KAFKA-7456
 URL: https://issues.apache.org/jira/browse/KAFKA-7456
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 2.1.0


This is a prerequisite for further topology optimization in the Streams DSL: we 
should let different operators inside the DSL to be able to pass along key and 
value serdes if they are not explicitly specified by users. The serde 
specification precedence should generally be:

1) Overridden values via control objects (e.g. Materialized, Serialized, 
Consumed, etc)
2) Serdes that can be inferred from the operator itself (e.g. 
groupBy().count(), where value serde can default to `LongSerde`).
3) Serde inherited from parent operator if possible (note if the key / value 
types have been changed, then the corresponding serde cannot be inherited).
4) Default serde specified in the config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7455) JmxTool cannot connect to an SSL-enabled JMX RMI port

[Attila Sasvari (JIRA)]

Attila Sasvari created KAFKA-7455:
-

 Summary: JmxTool cannot connect to an SSL-enabled JMX RMI port
 Key: KAFKA-7455
 URL: https://issues.apache.org/jira/browse/KAFKA-7455
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Attila Sasvari


When JmxTool tries to connect to an SSL-enabled JMX RMI port with 
JMXConnectorFactory'connect(), the connection attempt results in a 
"java.rmi.ConnectIOException: non-JRMP server at remote endpoint":

{code}
$ export 
KAFKA_OPTS="-Djavax.net.ssl.trustStore=/tmp/kafka.server.truststore.jks 
-Djavax.net.ssl.trustStorePassword=test"

$ bin/kafka-run-class.sh kafka.tools.JmxTool --object-name 
"kafka.server:type=kafka-metrics-count"  --jmx-url 
service:jmx:rmi:///jndi/rmi://localhost:9393/jmxrmi

ConnectIOException: non-JRMP server at remote endpoint].
java.io.IOException: Failed to retrieve RMIServer stub: 
javax.naming.CommunicationException [Root exception is 
java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
at 
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at 
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at kafka.tools.JmxTool$.main(JmxTool.scala:120)
at kafka.tools.JmxTool.main(JmxTool.scala)
{code}

The problem is that {{JmxTool}} does not specify {{SslRMIClientSocketFactory}} 
when it tries to connect
https://github.com/apache/kafka/blob/70d90c371833b09cf934c8c2358171433892a085/core/src/main/scala/kafka/tools/JmxTool.scala#L120
{code}  
  jmxc = JMXConnectorFactory.connect(url, null)
{code}
To connect to a secured RMI port, it should pass an envionrment map that 
contains a {{("com.sun.jndi.rmi.factory.socket", new 
SslRMIClientSocketFactory)}} entry.

More info:
- https://docs.oracle.com/cd/E19698-01/816-7609/security-35/index.html
- https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-339: Create a new IncrementalAlterConfigs API

[Colin McCabe]

Hi all,

Thanks for the discussion.  I'm going to close the vote later today if there 
are no more comments.

cheers,
Colin


On Mon, Sep 24, 2018, at 22:33, Colin McCabe wrote:
> +1 (binding)
> 
> Colin
> 
> 
> On Mon, Sep 24, 2018, at 17:49, Ismael Juma wrote:
> > Thanks Colin. I think this is much needed and I'm +1 (binding)
> > on fixing> this issue. However, I have a few minor suggestions:
> >
> > 1. Overload alterConfigs instead of creating a new method name. This
> >gives> us both the short name and a path for removal of the deprecated
> > overload.> 2. Did we consider Add/Remove instead of Append/Subtract?
> >
> > Ismael
> >
> > On Mon, Sep 24, 2018 at 10:29 AM Colin McCabe
> >  wrote:>
> > > Hi all,
> > >
> > > I would like to start voting on KIP-339, which creates a new
> > > IncrementalAlterConfigs API.
> > >
> > > The KIP is described here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+ModifyConfigs+API>
> > >  >
> > > Previous discussion:
> > > https://sematext.com/opensee/m/Kafka/uyzND1OYRKh2RrGba1
> > >
> > > best,
> > > Colin
> > >
> 

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

[Viktor Somogyi-Vass]

Hi All,

I made an update to the KIP. Just in short:
Currently KafkaAdminClient.describeTopics() and
KafkaAdminClient.listTopics() uses the Metadata protocol to acquire topic
information. The returned response however won't contain the topics that
are under deletion but couldn't complete yet (for instance because of some
replicas offline), therefore it is not possible to implement the current
command's "marked for deletion" feature. To get around this I introduced
some changes in the Metadata protocol.

Thanks,
Viktor

On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass 
wrote:

> Hi Mickael,
>
> Thanks for the feedback, I also think that many customers wanted this for
> a long time.
>
> Cheers,
> Viktor
>
> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison 
> wrote:
>
>> Hi Viktor,
>> Thanks for taking this task!
>> This is a very nice change as it will allow users to use this tool in
>> many Cloud environments where direct zookeeper access is not
>> available.
>>
>>
>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
>>  wrote:
>> >
>> > Hi All,
>> >
>> > This is the continuation of the old KIP-375 with the same title:
>> >
>> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
>> >
>> > The problem there was that two KIPs were created around the same time
>> and I
>> > chose to reorganize mine a bit and give it a new number to avoid
>> > duplication.
>> >
>> > The KIP summary here once again:
>> >
>> > I wrote up a relatively simple KIP about improving the Kafka protocol
>> and
>> > the TopicCommand tool to support the new Java based AdminClient and
>> > hopefully to deprecate the Zookeeper side of it.
>> >
>> > I would be happy to receive some opinions about this. In general I think
>> > this would be an important addition as this is one of the few left but
>> > important tools that still uses direct Zookeeper connection.
>> >
>> > Here is the link for the KIP:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
>> >
>> > Cheers,
>> > Viktor
>>
>

[jira] [Created] (KAFKA-7454) Use lazy allocation for SslTransportLayer buffers

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-7454:
-

 Summary: Use lazy allocation for SslTransportLayer buffers
 Key: KAFKA-7454
 URL: https://issues.apache.org/jira/browse/KAFKA-7454
 Project: Kafka
  Issue Type: Improvement
  Components: security
Affects Versions: 2.0.0, 1.1.1, 1.0.2, 0.11.0.3
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.1.0


At the moment, three heap buffers are allocated for SslTransportLayers at the 
time when the instance is created (before establishing the connection on the 
client-side and when accepting the connection on the broker-side). When there 
are a large number of connections and the broker is overloaded, this can result 
in unnecessary memory pressure on the broker due to client reconnections since 
buffers may be allocated unnecessarily for client connections whose handshake 
is never processed. It will be better to lazily allocate buffers to reduce 
memory pressure. On the broker-side, buffers will be allocated when the first 
packet is received from the client, starting handshake. On the client-side, 
buffers will be allocated once the connection is established when the client 
initiates handshake.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

[Viktor Somogyi-Vass]

Hi Mickael,

Thanks for the feedback, I also think that many customers wanted this for a
long time.

Cheers,
Viktor

On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison 
wrote:

> Hi Viktor,
> Thanks for taking this task!
> This is a very nice change as it will allow users to use this tool in
> many Cloud environments where direct zookeeper access is not
> available.
>
>
> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
>  wrote:
> >
> > Hi All,
> >
> > This is the continuation of the old KIP-375 with the same title:
> >
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
> >
> > The problem there was that two KIPs were created around the same time
> and I
> > chose to reorganize mine a bit and give it a new number to avoid
> > duplication.
> >
> > The KIP summary here once again:
> >
> > I wrote up a relatively simple KIP about improving the Kafka protocol and
> > the TopicCommand tool to support the new Java based AdminClient and
> > hopefully to deprecate the Zookeeper side of it.
> >
> > I would be happy to receive some opinions about this. In general I think
> > this would be an important addition as this is one of the few left but
> > important tools that still uses direct Zookeeper connection.
> >
> > Here is the link for the KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
> >
> > Cheers,
> > Viktor
>

[jira] [Created] (KAFKA-7453) Enable idle expiry of connections which are never selected

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-7453:
-

 Summary: Enable idle expiry of connections which are never selected
 Key: KAFKA-7453
 URL: https://issues.apache.org/jira/browse/KAFKA-7453
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 2.0.0, 1.1.1, 1.0.2, 0.11.0.3, 0.10.2.2
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.1.0


We add connections to Selector#channels when a connection is registered, but we 
start idle expiry of connections only when the connection is first  selected. 
In some environments where the channel may never get selected, this could leak 
memory and sockets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #2997

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk10 #526

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-7409; Validate message format version before creating topics or

--
[...truncated 2.15 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

[Mickael Maison]

Hi Viktor,
Thanks for taking this task!
This is a very nice change as it will allow users to use this tool in
many Cloud environments where direct zookeeper access is not
available.


On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass
 wrote:
>
> Hi All,
>
> This is the continuation of the old KIP-375 with the same title:
> https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E
>
> The problem there was that two KIPs were created around the same time and I
> chose to reorganize mine a bit and give it a new number to avoid
> duplication.
>
> The KIP summary here once again:
>
> I wrote up a relatively simple KIP about improving the Kafka protocol and
> the TopicCommand tool to support the new Java based AdminClient and
> hopefully to deprecate the Zookeeper side of it.
>
> I would be happy to receive some opinions about this. In general I think
> this would be an important addition as this is one of the few left but
> important tools that still uses direct Zookeeper connection.
>
> Here is the link for the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient
>
> Cheers,
> Viktor

Build failed in Jenkins: kafka-trunk-jdk8 #2996

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2654, done.
remote: Counting objects:   0% (1/2654)   remote: Counting objects:   
1% (27/2654)   remote: Counting objects:   2% (54/2654)   
remote: Counting objects:   3% (80/2654)   remote: Counting objects:   
4% (107/2654)   remote: Counting objects:   5% (133/2654)   
remote: Counting objects:   6% (160/2654)   remote: Counting objects:   
7% (186/2654)   remote: Counting objects:   8% (213/2654)   
remote: Counting objects:   9% (239/2654)   remote: Counting objects:  
10% (266/2654)   remote: Counting objects:  11% (292/2654)   
remote: Counting objects:  12% (319/2654)   remote: Counting objects:  
13% (346/2654)   remote: Counting objects:  14% (372/2654)   
remote: Counting objects:  15% (399/2654)   remote: Counting objects:  
16% (425/2654)   remote: Counting objects:  17% (452/2654)   
remote: Counting objects:  18% (478/2654)   remote: Counting objects:  
19% (505/2654)   remote: Counting objects:  20% (531/2654)   
remote: Counting objects:  21% (558/2654)   remote: Counting objects:  
22% (584/2654)   remote: Counting objects:  23% (611/2654)   
remote: Counting objects:  24% (637/2654)   remote: Counting objects:  
25% (664/2654)   remote: Counting objects:  26% (691/2654)   
remote: Counting objects:  27% (717/2654)   remote: Counting objects:  
28% (744/2654)   remote: Counting objects:  29% (770/2654)   
remote: Counting objects:  30% (797/2654)   remote: Counting objects:  
31% (823/2654)   remote: Counting objects:  32% (850/2654)   
remote: Counting objects:  33% (876/2654)   remote: Counting objects:  
34% (903/2654)   remote: Counting objects:  35% (929/2654)   
remote: Counting objects:  36% (956/2654)   remote: Counting objects:  
37% (982/2654)   remote: Counting objects:  38% (1009/2654)   
remote: Counting objects:  39% (1036/2654)   remote: Counting objects:  
40% (1062/2654)   remote: Counting objects:  41% (1089/2654)   
remote: Counting objects:  42% (1115/2654)   remote: Counting objects:  
43% (1142/2654)   remote: Counting objects:  44% (1168/2654)   
remote: Counting objects:  45% (1195/2654)   remote: Counting objects:  
46% (1221/2654)   remote: Counting objects:  47% (1248/2654)   
remote: Counting objects:  48% (1274/2654)   remote: Counting objects:  
49% (1301/2654)   remote: Counting objects:  50% (1327/2654)   
remote: Counting objects:  51% (1354/2654)   remote: Counting objects:  
52% (1381/2654)   remote: Counting objects:  53% (1407/2654)   
remote: Counting objects:  54% (1434/2654)   remote: Counting objects:  
55% 

Build failed in Jenkins: kafka-trunk-jdk8 #2995

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2654, done.
remote: Counting objects:   0% (1/2654)   remote: Counting objects:   
1% (27/2654)   remote: Counting objects:   2% (54/2654)   
remote: Counting objects:   3% (80/2654)   remote: Counting objects:   
4% (107/2654)   remote: Counting objects:   5% (133/2654)   
remote: Counting objects:   6% (160/2654)   remote: Counting objects:   
7% (186/2654)   remote: Counting objects:   8% (213/2654)   
remote: Counting objects:   9% (239/2654)   remote: Counting objects:  
10% (266/2654)   remote: Counting objects:  11% (292/2654)   
remote: Counting objects:  12% (319/2654)   remote: Counting objects:  
13% (346/2654)   remote: Counting objects:  14% (372/2654)   
remote: Counting objects:  15% (399/2654)   remote: Counting objects:  
16% (425/2654)   remote: Counting objects:  17% (452/2654)   
remote: Counting objects:  18% (478/2654)   remote: Counting objects:  
19% (505/2654)   remote: Counting objects:  20% (531/2654)   
remote: Counting objects:  21% (558/2654)   remote: Counting objects:  
22% (584/2654)   remote: Counting objects:  23% (611/2654)   
remote: Counting objects:  24% (637/2654)   remote: Counting objects:  
25% (664/2654)   remote: Counting objects:  26% (691/2654)   
remote: Counting objects:  27% (717/2654)   remote: Counting objects:  
28% (744/2654)   remote: Counting objects:  29% (770/2654)   
remote: Counting objects:  30% (797/2654)   remote: Counting objects:  
31% (823/2654)   remote: Counting objects:  32% (850/2654)   
remote: Counting objects:  33% (876/2654)   remote: Counting objects:  
34% (903/2654)   remote: Counting objects:  35% (929/2654)   
remote: Counting objects:  36% (956/2654)   remote: Counting objects:  
37% (982/2654)   remote: Counting objects:  38% (1009/2654)   
remote: Counting objects:  39% (1036/2654)   remote: Counting objects:  
40% (1062/2654)   remote: Counting objects:  41% (1089/2654)   
remote: Counting objects:  42% (1115/2654)   remote: Counting objects:  
43% (1142/2654)   remote: Counting objects:  44% (1168/2654)   
remote: Counting objects:  45% (1195/2654)   remote: Counting objects:  
46% (1221/2654)   remote: Counting objects:  47% (1248/2654)   
remote: Counting objects:  48% (1274/2654)   remote: Counting objects:  
49% (1301/2654)   remote: Counting objects:  50% (1327/2654)   
remote: Counting objects:  51% (1354/2654)   remote: Counting objects:  
52% (1381/2654)   remote: Counting objects:  53% (1407/2654)   
remote: Counting objects:  54% (1434/2654)   remote: Counting objects:  
55% 

Build failed in Jenkins: kafka-trunk-jdk10 #525

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 4866c33ac309ba5cc098a02948253f55a83666a3
error: Could not read 4866c33ac309ba5cc098a02948253f55a83666a3
error: Could not read 379211134740268b570fc8edd59ae78df0dffee9
error: Could not read f26377352d14af38af5d6cf42531b940fafe7236
error: Could not read 5b9889790e7233cb34af31b768a8494eb5ec56ca
error: Could not read 68b2f49ea75059df5527378e8ae771195029c98a
remote: Enumerating objects: 2532, done.
remote: Counting objects:   0% (1/2532)   remote: Counting objects:   
1% (26/2532)   remote: Counting objects:   2% (51/2532)   
remote: Counting objects:   3% (76/2532)   remote: Counting objects:   
4% (102/2532)   remote: Counting objects:   5% (127/2532)   
remote: Counting objects:   6% (152/2532)   remote: Counting objects:   
7% (178/2532)   remote: Counting objects:   8% (203/2532)   
remote: Counting objects:   9% (228/2532)   remote: Counting objects:  
10% (254/2532)   remote: Counting objects:  11% (279/2532)   
remote: Counting objects:  12% (304/2532)   remote: Counting objects:  
13% (330/2532)   remote: Counting objects:  14% (355/2532)   
remote: Counting objects:  15% (380/2532)   remote: Counting objects:  
16% (406/2532)   remote: Counting objects:  17% (431/2532)   
remote: Counting objects:  18% (456/2532)   remote: Counting objects:  
19% (482/2532)   remote: Counting objects:  20% (507/2532)   
remote: Counting objects:  21% (532/2532)   remote: Counting objects:  
22% (558/2532)   remote: Counting objects:  23% (583/2532)   
remote: Counting objects:  24% (608/2532)   remote: Counting objects:  
25% (633/2532)   remote: Counting objects:  26% (659/2532)   
remote: Counting objects:  27% (684/2532)   remote: Counting objects:  
28% (709/2532)   remote: Counting objects:  29% (735/2532)   
remote: Counting objects:  30% (760/2532)   remote: Counting objects:  
31% (785/2532)   remote: Counting objects:  32% (811/2532)   
remote: Counting objects:  33% (836/2532)   remote: Counting objects:  
34% (861/2532)   remote: Counting objects:  35% (887/2532)   
remote: Counting objects:  36% (912/2532)   remote: Counting objects:  
37% (937/2532)   remote: Counting objects:  38% (963/2532)   
remote: Counting objects:  39% (988/2532)   remote: Counting objects:  
40% (1013/2532)   remote: Counting objects:  41% (1039/2532)   
remote: Counting objects:  42% (1064/2532)   remote: Counting objects:  
43% (1089/2532)   remote: Counting objects:  44% (1115/2532)   
remote: Counting objects:  45% (1140/2532)   remote: Counting objects:  
46% (1165/2532)   remote: Counting objects:  47% (1191/2532)   
remote: Counting objects:  48% (1216/2532)   remote: Counting objects:  
49% (1241/2532)   remote: Counting objects:  50% (1266/2532)   
remote: 

[jira] [Created] (KAFKA-7452) Deleting snapshot files after check-pointing log recovery offsets can slow down replication when truncation happens

[Zhanxiang (Patrick) Huang (JIRA)]

Zhanxiang (Patrick) Huang created KAFKA-7452:


 Summary: Deleting snapshot files after check-pointing log recovery 
offsets can slow down replication when truncation happens
 Key: KAFKA-7452
 URL: https://issues.apache.org/jira/browse/KAFKA-7452
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.0.0, 1.1.1, 1.1.0, 1.0.1, 1.0.0
Reporter: Zhanxiang (Patrick) Huang


After KAFKA-5829, Kafka will try to iterate through all the partition dirs to 
delete useless snapshot files in "checkpointLogRecoveryOffsetsInDir". 
Currently, "checkpointLogRecoveryOffsetsInDir" is used in the following places:
 # Truncation
 # Log dir deletion and movement
 # Background thread checkpointing recovery offsets

In 2.0 deployment on a cluster with 10k partitions per broker, we found out 
that deleting useless snapshot files in the critical path of log truncation can 
significantly slow down followers to catch up with leader during rolling bounce 
(~2x slower than 0.11). The reason is that we basically do a "ls -R" for the 
whole data directory only to potentially delete the snapshot files in one 
partition directory because the way we identify snapshot files is to list the 
directories and check the filename suffix.

In our case, "listSnapshotFiles" takes ~1ms per partition directory so it takes 
~1ms * 10k = ~10s to just delete snapshot files in one partition after the 
truncation, which delays future fetches in the fetcher thread.

Here are the related code snippets:

 LogManager.scala

 
{code:java}
private def checkpointLogRecoveryOffsetsInDir(dir: File): Unit = {
  for {
partitionToLog <- logsByDir.get(dir.getAbsolutePath)
checkpoint <- recoveryPointCheckpoints.get(dir)
  } {
try {
  checkpoint.write(partitionToLog.mapValues(_.recoveryPoint))
  allLogs.foreach(_.deleteSnapshotsAfterRecoveryPointCheckpoint())
} catch {
  case e: IOException =>
logDirFailureChannel.maybeAddOfflineLogDir(dir.getAbsolutePath, s"Disk 
error while writing to recovery point " +
  s"file in directory $dir", e)
}
  }
}
{code}
 

 ProducerStateChangeManager.scala

 
{code:java}
private[log] def listSnapshotFiles(dir: File): Seq[File] = {
  if (dir.exists && dir.isDirectory) {
Option(dir.listFiles).map { files =>
  files.filter(f => f.isFile && isSnapshotFile(f)).toSeq
}.getOrElse(Seq.empty)
  } else Seq.empty
}


private def deleteSnapshotFiles(dir: File, predicate: Long => Boolean = _ => 
true) {
  listSnapshotFiles(dir).filter(file => 
predicate(offsetFromFile(file))).foreach { file =>
Files.deleteIfExists(file.toPath)
  }
}

{code}
 

There are a few things that can be optimized here:
 # We can have an in-memory cache for the snapshot files metadata (filename) in 
ProducerStateManager to avoid calling dir.listFiles in "deleteSnapshotFiles", 
"latestSnapshotFile" and "oldestSnapshotFile".
 # After truncation, we can only try to delete snapshot files for the truncated 
partitions (in replica fetcher thread, we truncate one partition at a time) 
instead of all partitions. Or maybe we don't even need to delete snapshot files 
in the critical path of truncation because the background 
log-recovery-offset-checkpoint-thread will do it periodically. This can also 
apply on log deletion/movement.
 # If we want to further optimize the actual snapshot file deletion, we can 
make it async. But I am not sure whether it is needed after we have 1) and 2).

Also, we notice that there is no way to disable transaction/exactly-once 
support in the broker-side given that it will bring in some extra overhead even 
though we have no clients using this feature. Not sure whether this is a common 
use case, but it is useful if we can have a switch to avoid the extra 
performance overhead.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7451) Missing JMX metrics

[Jork Zijlstra (JIRA)]

Jork Zijlstra created KAFKA-7451:


 Summary: Missing JMX metrics
 Key: KAFKA-7451
 URL: https://issues.apache.org/jira/browse/KAFKA-7451
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 1.0.1
Reporter: Jork Zijlstra
 Attachments: consumer-metrics.png, consumer-node-metrics.png

We are trying to use the jmx metrics that are being exported using the 
documentation located at: 
[https://docs.confluent.io/current/kafka/monitoring.html]

According to the docs there should be a "request-latency-avg" and 
"request-latency-max" available on "MBean: 
kafka.consumer:type=consumer-metrics,client-id=([-.w]+)". However what we see 
is that these metrics aren't there.

What I have noticed is that these are only available on 
"kafka.consumer:type=consumer-node-metrics,client-id=*,node-id=*". So the per 
node metrics, while according to the documentation they shouldn't exist there.

See attached screenshots.

 

So I was wondering if the documentation is wrong or are the metrics not 
exported properly?

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7450) kafka controller RequestSendThread stuck in infinite loop after SSL handshake failure with peer brokers

[Yu Yang (JIRA)]

Yu Yang created KAFKA-7450:
--

 Summary: kafka controller RequestSendThread stuck in infinite loop 
after SSL handshake failure with peer brokers
 Key: KAFKA-7450
 URL: https://issues.apache.org/jira/browse/KAFKA-7450
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 2.0.0
Reporter: Yu Yang


After updating security.inter.broker.protocol to SSL for our cluster, we 
observed that the controller can get into almost 100% cpu usage. 

{code}
listeners=PLAINTEXT://:9092,SSL://:9093
security.inter.broker.protocol=SSL
{code}

There is no obvious error in server.log. But in controller.log, there is 
repetitive SSL handshare failure error as below: 

{code}
[2018-09-28 05:53:10,821] WARN [RequestSendThread controllerId=6042] Controller 
6042's connection to broker datakafka06176.ec2.pin220.com:9093 (id: 6176 rack: 
null) was unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence 
violation, 2
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1487)
at 
sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at 
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:468)
at 
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
at 
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at 
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at 
org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
at 
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at 
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence 
violation, 2
at 
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:196)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at 
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at 
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
... 10 more

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2994

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
remote: Enumerating objects: 2654, done.
remote: Counting objects:   0% (1/2654)   remote: Counting objects:   
1% (27/2654)   remote: Counting objects:   2% (54/2654)   
remote: Counting objects:   3% (80/2654)   remote: Counting objects:   
4% (107/2654)   remote: Counting objects:   5% (133/2654)   
remote: Counting objects:   6% (160/2654)   remote: Counting objects:   
7% (186/2654)   remote: Counting objects:   8% (213/2654)   
remote: Counting objects:   9% (239/2654)   remote: Counting objects:  
10% (266/2654)   remote: Counting objects:  11% (292/2654)   
remote: Counting objects:  12% (319/2654)   remote: Counting objects:  
13% (346/2654)   remote: Counting objects:  14% (372/2654)   
remote: Counting objects:  15% (399/2654)   remote: Counting objects:  
16% (425/2654)   remote: Counting objects:  17% (452/2654)   
remote: Counting objects:  18% (478/2654)   remote: Counting objects:  
19% (505/2654)   remote: Counting objects:  20% (531/2654)   
remote: Counting objects:  21% (558/2654)   remote: Counting objects:  
22% (584/2654)   remote: Counting objects:  23% (611/2654)   
remote: Counting objects:  24% (637/2654)   remote: Counting objects:  
25% (664/2654)   remote: Counting objects:  26% (691/2654)   
remote: Counting objects:  27% (717/2654)   remote: Counting objects:  
28% (744/2654)   remote: Counting objects:  29% (770/2654)   
remote: Counting objects:  30% (797/2654)   remote: Counting objects:  
31% (823/2654)   remote: Counting objects:  32% (850/2654)   
remote: Counting objects:  33% (876/2654)   remote: Counting objects:  
34% (903/2654)   remote: Counting objects:  35% (929/2654)   
remote: Counting objects:  36% (956/2654)   remote: Counting objects:  
37% (982/2654)   remote: Counting objects:  38% (1009/2654)   
remote: Counting objects:  39% (1036/2654)   remote: Counting objects:  
40% (1062/2654)   remote: Counting objects:  41% (1089/2654)   
remote: Counting objects:  42% (1115/2654)   remote: Counting objects:  
43% (1142/2654)   remote: Counting objects:  44% (1168/2654)   
remote: Counting objects:  45% (1195/2654)   remote: Counting objects:  
46% (1221/2654)   remote: Counting objects:  47% (1248/2654)   
remote: Counting objects:  48% (1274/2654)   remote: Counting objects:  
49% (1301/2654)   remote: Counting objects:  50% (1327/2654)   
remote: Counting objects:  51% (1354/2654)   remote: Counting objects:  
52% (1381/2654)   remote: Counting objects:  53% (1407/2654)   
remote: Counting objects:  54% (1434/2654)   remote: Counting objects:  
55% 

Build failed in Jenkins: kafka-trunk-jdk10 #524

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 4866c33ac309ba5cc098a02948253f55a83666a3
error: Could not read 4866c33ac309ba5cc098a02948253f55a83666a3
error: Could not read 379211134740268b570fc8edd59ae78df0dffee9
error: Could not read f26377352d14af38af5d6cf42531b940fafe7236
error: Could not read 5b9889790e7233cb34af31b768a8494eb5ec56ca
error: Could not read 68b2f49ea75059df5527378e8ae771195029c98a
remote: Enumerating objects: 2532, done.
remote: Counting objects:   0% (1/2532)   remote: Counting objects:   
1% (26/2532)   remote: Counting objects:   2% (51/2532)   
remote: Counting objects:   3% (76/2532)   remote: Counting objects:   
4% (102/2532)   remote: Counting objects:   5% (127/2532)   
remote: Counting objects:   6% (152/2532)   remote: Counting objects:   
7% (178/2532)   remote: Counting objects:   8% (203/2532)   
remote: Counting objects:   9% (228/2532)   remote: Counting objects:  
10% (254/2532)   remote: Counting objects:  11% (279/2532)   
remote: Counting objects:  12% (304/2532)   remote: Counting objects:  
13% (330/2532)   remote: Counting objects:  14% (355/2532)   
remote: Counting objects:  15% (380/2532)   remote: Counting objects:  
16% (406/2532)   remote: Counting objects:  17% (431/2532)   
remote: Counting objects:  18% (456/2532)   remote: Counting objects:  
19% (482/2532)   remote: Counting objects:  20% (507/2532)   
remote: Counting objects:  21% (532/2532)   remote: Counting objects:  
22% (558/2532)   remote: Counting objects:  23% (583/2532)   
remote: Counting objects:  24% (608/2532)   remote: Counting objects:  
25% (633/2532)   remote: Counting objects:  26% (659/2532)   
remote: Counting objects:  27% (684/2532)   remote: Counting objects:  
28% (709/2532)   remote: Counting objects:  29% (735/2532)   
remote: Counting objects:  30% (760/2532)   remote: Counting objects:  
31% (785/2532)   remote: Counting objects:  32% (811/2532)   
remote: Counting objects:  33% (836/2532)   remote: Counting objects:  
34% (861/2532)   remote: Counting objects:  35% (887/2532)   
remote: Counting objects:  36% (912/2532)   remote: Counting objects:  
37% (937/2532)   remote: Counting objects:  38% (963/2532)   
remote: Counting objects:  39% (988/2532)   remote: Counting objects:  
40% (1013/2532)   remote: Counting objects:  41% (1039/2532)   
remote: Counting objects:  42% (1064/2532)   remote: Counting objects:  
43% (1089/2532)   remote: Counting objects:  44% (1115/2532)   
remote: Counting objects:  45% (1140/2532)   remote: Counting objects:  
46% (1165/2532)   remote: Counting objects:  47% (1191/2532)   
remote: Counting objects:  48% (1216/2532)   remote: Counting objects:  
49% (1241/2532)   remote: Counting objects:  50% (1266/2532)   
remote: