subject:"Re\: build failes for latest stable source tgz \(kafka

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-06 Thread Shlomi Hazan

just VPN'ed into my workstation:
the answer to 5 is [*yes*]
answer to 1,2:
the error's I see on the python client are first timeouts and then message
send failures, using sync send.

on the controller log:

ontroller.log.2014-08-26-13:[2014-08-26 13:40:44,317] ERROR
[Controller-1-to-broker-3-send-thread], Controller 1 epoch 3 failed to send
StopReplica request with correlation id 519 to broker
id:3,host:shlomi-kafka-broker-3,port:9092. Reconnecting to broker.
(kafka.controller.RequestSendThread)
controller.log.2014-08-26-13:[2014-08-26 13:40:44,319] ERROR
[Controller-1-to-broker-3-send-thread], Controller 1's connection to broker
id:3,host:shlomi-kafka-broker-3,port:9092 was unsuccessful
(kafka.controller.RequestSendThread)

on the server log (selected greps):
...
server.log.2014-08-27-01:[2014-08-27 01:44:23,143] ERROR
[ReplicaFetcherThread-4-2], Error for partition [vpq_android_gcm_h,270] to
broker 2:class kafka.common.NotLeaderForPartitionException
(kafka.server.ReplicaFetcherThread)
...
server.log.2014-08-27-12:[2014-08-27 12:08:34,638] ERROR Closing socket for
/10.184.150.54 because of error (kafka.network.Processor)

...
server.log.2014-08-28-07:[2014-08-28 07:57:35,944] ERROR [KafkaApi-1] Error
when processing fetch request for partition [vpq_android_gcm_h,184] offset
8798 from follower with correlation id 0 (kafka.server.KafkaApis)
...
erver.log.2014-09-03-15:[2014-09-03 15:46:18,220] ERROR
[ReplicaFetcherThread-2-3], Error in fetch Name: FetchRequest; Version: 0;
CorrelationId: 177593; ClientId: ReplicaFetcherThread-2-3; ReplicaId: 1;
MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [vpq_android_gcm_h,196]
- PartitionFetchInfo(65283,8388608),[vpq_android_gcm_h,76] -
PartitionFetchInfo(262787,8388608),[vpq_android_gcm_h,460] -
PartitionFetchInfo(285709,8388608),[vpq_android_gcm_h,100] -
PartitionFetchInfo(199405,8388608),[vpq_android_gcm_h,148] -
PartitionFetchInfo(339032,8388608),[vpq_android_gcm_h,436] -
PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,124] -
PartitionFetchInfo(484447,8388608),[vpq_android_gcm_h,484] -
PartitionFetchInfo(105945,8388608),[vpq_android_gcm_h,340] -
PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,388] -
PartitionFetchInfo(9,8388608),[vpq_android_gcm_h,316] -
PartitionFetchInfo(194766,8388608),[vpq_android_gcm_h,364] -
PartitionFetchInfo(139897,8388608),[vpq_android_gcm_h,292] -
PartitionFetchInfo(195408,8388608),[vpq_android_gcm_h,28] -
PartitionFetchInfo(329961,8388608),[vpq_android_gcm_h,172] -
PartitionFetchInfo(436959,8388608),[vpq_android_gcm_h,268] -
PartitionFetchInfo(59827,8388608),[vpq_android_gcm_h,244] -
PartitionFetchInfo(259731,8388608),[vpq_android_gcm_h,220] -
PartitionFetchInfo(61669,8388608),[vpq_android_gcm_h,412] -
PartitionFetchInfo(563609,8388608),[vpq_android_gcm_h,4] -
PartitionFetchInfo(360336,8388608),[vpq_android_gcm_h,52] -
PartitionFetchInfo(378533,8388608) (kafka.server.ReplicaFetcherThread)
...
server.log.2014-09-03-14:[2014-09-03 14:04:18,548] ERROR Error in acceptor
(kafka.network.Acceptor)
...



On Sat, Sep 6, 2014 at 5:48 PM, Shlomi Hazan shl...@viber.com wrote:

 Hi and sorry for the late response I just got into the weekend and still
 Satdurday here...
 Well, not at my desk but will answer what I can:
 1. what else on the logs? [*will vpn and check*]
 2. other broker failure reason? [**]
 3. other broker failure after taking leadership? [*how can I be sure? ask
 another to describe topic?*]
 4. how do I measure number of connections? [*ls -l /proc/pid/fd | grep
 socket | wc -l, also did watch on that*]
 5. is that number equals the number of {new Producer}?
 6. how many topics? [*1*] how many partitions [*504*]
 7. Are u using a partition key? [*yes, I use the python client with* ]









 *class ProducerIdPartitioner(Partitioner):Implements a
 partitioner which selects the target partition based on the sending
 producer IDdef partition(self, key, partitions):size =
 len(partitions)prod_id = int(key)idx = prod_id %
 sizereturn partitions[idx]*
 8. maybe running into over partitioned topic? [*producer instances is 6
 machines * 84 procs * 24 threads, but never got to start them all*,*b/c
 of errors*]
 9. r u running anything else? [*yes, zookeeper*]

 additional:
 do you want broker or other config? EC2 instance types? anythying else?
 Thanks,
 Shlomi


 On Thu, Sep 4, 2014 at 10:02 PM, Joe Stein joe.st...@stealth.ly wrote:

 I think it sounds more like another issue than your thinking...the broker
 should not be failing like that especially another broker being affected
 doesn't make sense.

 What else is in the logs on failure?  Is the other broker failing because
 of number of files too? Is it happening after it becomes the leader?

 How are you measuring number of connections? Is this how many producer
 connections you are opening up yourself (new Producer())?

 How many topics do you have? How many partitions?

 Are you using a partition key?  Maybe you are running into

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-06 Thread Joe Stein

If your creating a new producer for each send instead of reusing a pool of
producer connections for each send within your threads that could result in
the brokers surpassing your open file limit.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
/
On Sep 6, 2014 11:29 AM, Shlomi Hazan shl...@viber.com wrote:

 just VPN'ed into my workstation:
 the answer to 5 is [*yes*]
 answer to 1,2:
 the error's I see on the python client are first timeouts and then message
 send failures, using sync send.

 on the controller log:

 ontroller.log.2014-08-26-13:[2014-08-26 13:40:44,317] ERROR
 [Controller-1-to-broker-3-send-thread], Controller 1 epoch 3 failed to send
 StopReplica request with correlation id 519 to broker
 id:3,host:shlomi-kafka-broker-3,port:9092. Reconnecting to broker.
 (kafka.controller.RequestSendThread)
 controller.log.2014-08-26-13:[2014-08-26 13:40:44,319] ERROR
 [Controller-1-to-broker-3-send-thread], Controller 1's connection to broker
 id:3,host:shlomi-kafka-broker-3,port:9092 was unsuccessful
 (kafka.controller.RequestSendThread)

 on the server log (selected greps):
 ...
 server.log.2014-08-27-01:[2014-08-27 01:44:23,143] ERROR
 [ReplicaFetcherThread-4-2], Error for partition [vpq_android_gcm_h,270] to
 broker 2:class kafka.common.NotLeaderForPartitionException
 (kafka.server.ReplicaFetcherThread)
 ...
 server.log.2014-08-27-12:[2014-08-27 12:08:34,638] ERROR Closing socket for
 /10.184.150.54 because of error (kafka.network.Processor)

 ...
 server.log.2014-08-28-07:[2014-08-28 07:57:35,944] ERROR [KafkaApi-1]
 Error
 when processing fetch request for partition [vpq_android_gcm_h,184] offset
 8798 from follower with correlation id 0 (kafka.server.KafkaApis)
 ...
 erver.log.2014-09-03-15:[2014-09-03 15:46:18,220] ERROR
 [ReplicaFetcherThread-2-3], Error in fetch Name: FetchRequest; Version: 0;
 CorrelationId: 177593; ClientId: ReplicaFetcherThread-2-3; ReplicaId: 1;
 MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [vpq_android_gcm_h,196]
 - PartitionFetchInfo(65283,8388608),[vpq_android_gcm_h,76] -
 PartitionFetchInfo(262787,8388608),[vpq_android_gcm_h,460] -
 PartitionFetchInfo(285709,8388608),[vpq_android_gcm_h,100] -
 PartitionFetchInfo(199405,8388608),[vpq_android_gcm_h,148] -
 PartitionFetchInfo(339032,8388608),[vpq_android_gcm_h,436] -
 PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,124] -
 PartitionFetchInfo(484447,8388608),[vpq_android_gcm_h,484] -
 PartitionFetchInfo(105945,8388608),[vpq_android_gcm_h,340] -
 PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,388] -
 PartitionFetchInfo(9,8388608),[vpq_android_gcm_h,316] -
 PartitionFetchInfo(194766,8388608),[vpq_android_gcm_h,364] -
 PartitionFetchInfo(139897,8388608),[vpq_android_gcm_h,292] -
 PartitionFetchInfo(195408,8388608),[vpq_android_gcm_h,28] -
 PartitionFetchInfo(329961,8388608),[vpq_android_gcm_h,172] -
 PartitionFetchInfo(436959,8388608),[vpq_android_gcm_h,268] -
 PartitionFetchInfo(59827,8388608),[vpq_android_gcm_h,244] -
 PartitionFetchInfo(259731,8388608),[vpq_android_gcm_h,220] -
 PartitionFetchInfo(61669,8388608),[vpq_android_gcm_h,412] -
 PartitionFetchInfo(563609,8388608),[vpq_android_gcm_h,4] -
 PartitionFetchInfo(360336,8388608),[vpq_android_gcm_h,52] -
 PartitionFetchInfo(378533,8388608) (kafka.server.ReplicaFetcherThread)
 ...
 server.log.2014-09-03-14:[2014-09-03 14:04:18,548] ERROR Error in acceptor
 (kafka.network.Acceptor)
 ...



 On Sat, Sep 6, 2014 at 5:48 PM, Shlomi Hazan shl...@viber.com wrote:

  Hi and sorry for the late response I just got into the weekend and still
  Satdurday here...
  Well, not at my desk but will answer what I can:
  1. what else on the logs? [*will vpn and check*]
  2. other broker failure reason? [**]
  3. other broker failure after taking leadership? [*how can I be sure? ask
  another to describe topic?*]
  4. how do I measure number of connections? [*ls -l /proc/pid/fd | grep
  socket | wc -l, also did watch on that*]
  5. is that number equals the number of {new Producer}?
  6. how many topics? [*1*] how many partitions [*504*]
  7. Are u using a partition key? [*yes, I use the python client with* ]
 
 
 
 
 
 
 
 
 
  *class ProducerIdPartitioner(Partitioner):Implements a
  partitioner which selects the target partition based on the sending
  producer IDdef partition(self, key, partitions):size =
  len(partitions)prod_id = int(key)idx = prod_id %
  sizereturn partitions[idx]*
  8. maybe running into over partitioned topic? [*producer instances is 6
  machines * 84 procs * 24 threads, but never got to start them all*,*b/c
  of errors*]
  9. r u running anything else? [*yes, zookeeper*]
 
  additional:
  do you want broker or other config? EC2 instance types? anythying
 else?
  Thanks,
  Shlomi
 
 
  On Thu, Sep 4, 2014 at 10:02 PM, Joe Stein

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Harsha

Did you tried gradlew script in kafka source dir.
-Harsha

On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
 what gradle version is used to build kafka_2.9.2-0.8.1.1 ?
 
 tried with v2 and failed with :
 
 
 
 gradle --stacktrace clean
 
 FAILURE: Build failed with an exception.
 
 * Where:
 Build file
 '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
 line: 34
 
 * What went wrong:
 A problem occurred evaluating root project 'kafka-0.8.1.1-src'.
  Could not find method add() for arguments [licenseMain, class
 nl.javadude.gradle.plugins.license.License] on task set.
 
 * Try:
 Run with --info or --debug option to get more log output.
 
 * Exception is:
 org.gradle.api.GradleScriptException: A problem occurred evaluating root
 project 'kafka-0.8.1.1-src'.
 at
 org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
 at
 org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187)
 at
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
 at
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
 at
 org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
 at
 org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
 at
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470)
 at
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79)
 at
 org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31)
 at
 org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128)
 at
 org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105)
 at
 org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85)
 at
 org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81)
 at
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
 at
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
 at
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39)
 at
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29)
 at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
 at
 org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
 at
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
 at
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
 at
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
 at
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
 at
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
 at
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
 at
 org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
 at
 org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
 at org.gradle.launcher.Main.doAction(Main.java:33)
 at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
 at
 org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
 at
 org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
 at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
 Caused by: org.gradle.api.internal.MissingMethodException: Could not find
 method add() for arguments [licenseMain, class
 nl.javadude.gradle.plugins.license.License] on task set.
 at
 org.gradle.api.internal.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:68)
 at
 org.gradle.api.internal.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:56)
 at
 org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:172)
 at
 org.gradle.api.internal.tasks.DefaultTaskContainer_Decorated.invokeMethod(Unknown
 Source)
 at
 nl.javadude.gradle.plugins.license.LicensePlugin$_configureSourceSetRule_closure6_closure18.doCall(LicensePlugin.groovy:117)
 at
 org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59)
 at
 org.gradle.listener.ActionBroadcast.execute(ActionBroadcast.java:39)
 at

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Joe Stein

When building you need to use the ./gradelw script as Harsha said.  Please
take a look at the README for specific commands and how to run them.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/


On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan shl...@viber.com wrote:

 it failed with JDK 8 so I hoped a newer gradle will maybe do the magic, and
 stepped into this other problem.
 I assume you will say :
 install JDK 7 and build with our gradle 1.6.
 is it so?
 Shlomi


 On Thu, Sep 4, 2014 at 5:41 PM, Harsha ka...@harsha.io wrote:

  Did you tried gradlew script in kafka source dir.
  -Harsha
 
  On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
   what gradle version is used to build kafka_2.9.2-0.8.1.1 ?
  
   tried with v2 and failed with :
  
  
  
   gradle --stacktrace clean
  
   FAILURE: Build failed with an exception.
  
   * Where:
   Build file
  
 
 '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
   line: 34
  
   * What went wrong:
   A problem occurred evaluating root project 'kafka-0.8.1.1-src'.
Could not find method add() for arguments [licenseMain, class
   nl.javadude.gradle.plugins.license.License] on task set.
  
   * Try:
   Run with --info or --debug option to get more log output.
  
   * Exception is:
   org.gradle.api.GradleScriptException: A problem occurred evaluating
 root
   project 'kafka-0.8.1.1-src'.
   at
  
 
 org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
   at
  
 
 org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187)
   at
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
   at
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
   at
  
 
 org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
   at
  
 
 org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
   at
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470)
   at
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79)
   at
  
 
 org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31)
   at
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128)
   at
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105)
   at
  
 
 org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85)
   at
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81)
   at
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
   at
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
   at
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39)
   at
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29)
   at
 org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
   at
  
 
 org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
   at
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
   at
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
   at
  
 
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
   at
  
 
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
   at
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
   at
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
   at
  
 
 org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
   at
  
 
 org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
   at org.gradle.launcher.Main.doAction(Main.java:33)
   at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
   at
  
 
 org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
   at
  
 
 org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
   at

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Shlomi Hazan

./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. (error: error
while loading CharSequence, class file
'/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' is
broken)
I understand there's no escape from installing JDK 7?
10x
Shlomi


On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein joe.st...@stealth.ly wrote:

 When building you need to use the ./gradelw script as Harsha said.  Please
 take a look at the README for specific commands and how to run them.

 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /


 On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan shl...@viber.com wrote:

  it failed with JDK 8 so I hoped a newer gradle will maybe do the magic,
 and
  stepped into this other problem.
  I assume you will say :
  install JDK 7 and build with our gradle 1.6.
  is it so?
  Shlomi
 
 
  On Thu, Sep 4, 2014 at 5:41 PM, Harsha ka...@harsha.io wrote:
 
   Did you tried gradlew script in kafka source dir.
   -Harsha
  
   On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
what gradle version is used to build kafka_2.9.2-0.8.1.1 ?
   
tried with v2 and failed with :
   
   
   
gradle --stacktrace clean
   
FAILURE: Build failed with an exception.
   
* Where:
Build file
   
  
 
 '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
line: 34
   
* What went wrong:
A problem occurred evaluating root project 'kafka-0.8.1.1-src'.
 Could not find method add() for arguments [licenseMain, class
nl.javadude.gradle.plugins.license.License] on task set.
   
* Try:
Run with --info or --debug option to get more log output.
   
* Exception is:
org.gradle.api.GradleScriptException: A problem occurred evaluating
  root
project 'kafka-0.8.1.1-src'.
at
   
  
 
 org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
at
   
  
 
 org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187)
at
   
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
at
   
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
at
   
  
 
 org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
at
   
  
 
 org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
at
   
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470)
at
   
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79)
at
   
  
 
 org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31)
at
   
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128)
at
   
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105)
at
   
  
 
 org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85)
at
   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81)
at
   
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
at
   
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
at
   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39)
at
   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29)
at
  org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
at
   
  
 
 org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
at
   
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
at
   
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
at
   
  
 
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
at
   
  
 
 org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
at
   
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
at
   
  
 
 org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
at

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Joe Stein

Have you tried using a binary release http://kafka.apache.org/downloads.html
this way you don't have to-do a build?

We build using JDK 6 you should be able to run in 8 (I know for sure 6  7
work honestly never tried 8).

I just did a quick test with a broker running on 8 and produced/consumed a
few message didn't run into issues...

As for building in JDK 8 = I reproduced your issue and created a ticket
https://issues.apache.org/jira/browse/KAFKA-1624 patches are welcomed.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/


On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan shl...@viber.com wrote:

 ./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. (error: error
 while loading CharSequence, class file
 '/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' is
 broken)
 I understand there's no escape from installing JDK 7?
 10x
 Shlomi


 On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein joe.st...@stealth.ly wrote:

  When building you need to use the ./gradelw script as Harsha said.
 Please
  take a look at the README for specific commands and how to run them.
 
  /***
   Joe Stein
   Founder, Principal Consultant
   Big Data Open Source Security LLC
   http://www.stealth.ly
   Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
  /
 
 
  On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan shl...@viber.com wrote:
 
   it failed with JDK 8 so I hoped a newer gradle will maybe do the magic,
  and
   stepped into this other problem.
   I assume you will say :
   install JDK 7 and build with our gradle 1.6.
   is it so?
   Shlomi
  
  
   On Thu, Sep 4, 2014 at 5:41 PM, Harsha ka...@harsha.io wrote:
  
Did you tried gradlew script in kafka source dir.
-Harsha
   
On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
 what gradle version is used to build kafka_2.9.2-0.8.1.1 ?

 tried with v2 and failed with :



 gradle --stacktrace clean

 FAILURE: Build failed with an exception.

 * Where:
 Build file

   
  
 
 '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
 line: 34

 * What went wrong:
 A problem occurred evaluating root project 'kafka-0.8.1.1-src'.
  Could not find method add() for arguments [licenseMain, class
 nl.javadude.gradle.plugins.license.License] on task set.

 * Try:
 Run with --info or --debug option to get more log output.

 * Exception is:
 org.gradle.api.GradleScriptException: A problem occurred evaluating
   root
 project 'kafka-0.8.1.1-src'.
 at

   
  
 
 org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
 at

   
  
 
 org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187)
 at

   
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
 at

   
  
 
 org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
 at

   
  
 
 org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
 at

   
  
 
 org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
 at

   
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470)
 at

   
  
 
 org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79)
 at

   
  
 
 org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31)
 at

   
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128)
 at

   
  
 
 org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105)
 at

   
  
 
 org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85)
 at

   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81)
 at

   
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
 at

   
  
 
 org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
 at

   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39)
 at

   
  
 
 org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29)
 at

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Shlomi Hazan

I sure did. the reason I am building is trying to patch some. specifically
this : KAFKA-1623.
actually if I felt more confident about scala, I would happily send you a
patch.
If you don't care screening, just tell me how to prep it for ya and i will.
The bigger problem is running into too many open files when ramping up to
several thousand connection.
going above 4k is hard and around 6K the broker says goodbye. sometimes
taking a broker friend for a ride
This was what led me to the acceptor and it's being slow..
Have you got an idea why 10K connection should pose a problem. ulimit
checked. not that. ??
Shlomi

On Thu, Sep 4, 2014 at 7:00 PM, Joe Stein joe.st...@stealth.ly wrote:

Have you tried using a binary release
http://kafka.apache.org/downloads.html
this way you don't have to-do a build?

We build using JDK 6 you should be able to run in 8 (I know for sure 6 7
work honestly never tried 8).

I just did a quick test with a broker running on 8 and produced/consumed a
few message didn't run into issues...

As for building in JDK 8 = I reproduced your issue and created a ticket
https://issues.apache.org/jira/browse/KAFKA-1624 patches are welcomed.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan shl...@viber.com wrote:

./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. (error: error
while loading CharSequence, class file
'/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' is
broken)
I understand there's no escape from installing JDK 7?
10x
Shlomi

On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein joe.st...@stealth.ly wrote:

When building you need to use the ./gradelw script as Harsha said.
Please
take a look at the README for specific commands and how to run them.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan shl...@viber.com
wrote:

it failed with JDK 8 so I hoped a newer gradle will maybe do the
magic,
and
stepped into this other problem.
I assume you will say :
install JDK 7 and build with our gradle 1.6.
is it so?
Shlomi

On Thu, Sep 4, 2014 at 5:41 PM, Harsha ka...@harsha.io wrote:

Did you tried gradlew script in kafka source dir.
-Harsha

On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
what gradle version is used to build kafka_2.9.2-0.8.1.1 ?

tried with v2 and failed with :

gradle --stacktrace clean

FAILURE: Build failed with an exception.

* Where:
Build file

'/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
line: 34

* What went wrong:
A problem occurred evaluating root project 'kafka-0.8.1.1-src'.
Could not find method add() for arguments [licenseMain, class
nl.javadude.gradle.plugins.license.License] on task set.

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.GradleScriptException: A problem occurred
evaluating
root
project 'kafka-0.8.1.1-src'.
at

org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
at

org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187)
at

org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
at

org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
at

org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
at

org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55)
at

org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470)
at

org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79)
at

org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31)
at

org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

2014-09-04 Thread Joe Stein

I think it sounds more like another issue than your thinking...the broker
should not be failing like that especially another broker being affected
doesn't make sense.

What else is in the logs on failure? Is the other broker failing because
of number of files too? Is it happening after it becomes the leader?

How are you measuring number of connections? Is this how many producer
connections you are opening up yourself (new Producer())?

How many topics do you have? How many partitions?

Are you using a partition key? Maybe you are running into an over
partitioned topic type of scenario where your # of producer instances
multiplied by # topics is the problem (where the producer will open up it's
own connection over time to brokers for each partition leader).

Are you running anything else besides kafka broker on the box?

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Sep 4, 2014 at 2:06 PM, Shlomi Hazan shl...@viber.com wrote:

On Thu, Sep 4, 2014 at 7:00 PM, Joe Stein joe.st...@stealth.ly wrote:

Have you tried using a binary release
http://kafka.apache.org/downloads.html
this way you don't have to-do a build?

We build using JDK 6 you should be able to run in 8 (I know for sure 6
7
work honestly never tried 8).

I just did a quick test with a broker running on 8 and produced/consumed
a
few message didn't run into issues...

As for building in JDK 8 = I reproduced your issue and created a ticket
https://issues.apache.org/jira/browse/KAFKA-1624 patches are welcomed.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan shl...@viber.com wrote:

./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. (error:
error
while loading CharSequence, class file
'/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' is
broken)
I understand there's no escape from installing JDK 7?
10x
Shlomi

On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein joe.st...@stealth.ly
wrote:

When building you need to use the ./gradelw script as Harsha said.
Please
take a look at the README for specific commands and how to run them.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan shl...@viber.com
wrote:

it failed with JDK 8 so I hoped a newer gradle will maybe do the
magic,
and
stepped into this other problem.
I assume you will say :
install JDK 7 and build with our gradle 1.6.
is it so?
Shlomi

On Thu, Sep 4, 2014 at 5:41 PM, Harsha ka...@harsha.io wrote:

Did you tried gradlew script in kafka source dir.
-Harsha

On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote:
what gradle version is used to build kafka_2.9.2-0.8.1.1 ?

tried with v2 and failed with :

gradle --stacktrace clean

FAILURE: Build failed with an exception.

* Where:
Build file

'/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle'
line: 34

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.GradleScriptException: A problem occurred
evaluating
root
project 'kafka-0.8.1.1-src'.
at

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

Re: build failes for latest stable source tgz (kafka_2.9.2-0.8.1.1)

8 matches

Site Navigation

Mail list logo

Footer information