Hi and sorry for the late response I just got into the weekend and still
Satdurday here...
Well, not at my desk but will answer what I can:
1. what else on the logs? [*will vpn and check*]
2. other broker failure reason? [*"*]
3. other broker failure after taking leadership? [*how can I be sure? ask
another to describe topic?*]
4. how do I measure number of connections? [*ls -l /proc/<pid>/fd | grep
socket | wc -l, also did watch on that*]
5. is that number equals the number of {new Producer}?
6. how many topics? [*1*] how many partitions [*504*]
7. Are u using a partition key? [*yes, I use the python client with* ]*class ProducerIdPartitioner(Partitioner): """ Implements a partitioner which selects the target partition based on the sending producer ID """ def partition(self, key, partitions): size = len(partitions) prod_id = int(key) idx = prod_id % size return partitions[idx]* 8. maybe running into over partitioned topic? [*producer instances is 6 machines * 84 procs * 24 threads, but never got to start them all*,*b/c of errors*] 9. r u running anything else? [*yes, zookeeper*] additional: do you want broker or other config? EC2 instance types? anythying else?.... Thanks, Shlomi On Thu, Sep 4, 2014 at 10:02 PM, Joe Stein <[email protected]> wrote: > I think it sounds more like another issue than your thinking...the broker > should not be failing like that especially another broker being affected > doesn't make sense. > > What else is in the logs on failure? Is the other broker failing because > of number of files too? Is it happening after it becomes the leader? > > How are you measuring number of connections? Is this how many producer > connections you are opening up yourself (new Producer())? > > How many topics do you have? How many partitions? > > Are you using a partition key? Maybe you are running into an over > partitioned topic type of scenario where your # of producer instances > multiplied by # topics is the problem (where the producer will open up it's > own connection over time to brokers for each partition leader). > > Are you running anything else besides kafka broker on the box? > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > > On Thu, Sep 4, 2014 at 2:06 PM, Shlomi Hazan <[email protected]> wrote: > > > I sure did. the reason I am building is trying to patch some. > specifically > > this : KAFKA-1623. > > actually if I felt more confident about scala, I would happily send you a > > patch. > > If you don't care screening, just tell me how to prep it for ya and i > will. > > The bigger problem is running into too many open files when ramping up to > > several thousand connection. > > going above 4k is hard and around 6K the broker says goodbye. sometimes > > taking a broker friend for a ride.... > > This was what led me to the acceptor and it's being slow.. > > Have you got an idea why 10K connection should pose a problem. ulimit > > checked. not that. ?? > > Shlomi > > > > > > On Thu, Sep 4, 2014 at 7:00 PM, Joe Stein <[email protected]> wrote: > > > > > Have you tried using a binary release > > > http://kafka.apache.org/downloads.html > > > this way you don't have to-do a build? > > > > > > We build using JDK 6 you should be able to run in 8 (I know for sure 6 > & > > 7 > > > work honestly never tried 8). > > > > > > I just did a quick test with a broker running on 8 and > produced/consumed > > a > > > few message didn't run into issues... > > > > > > As for building in JDK 8 => I reproduced your issue and created a > ticket > > > https://issues.apache.org/jira/browse/KAFKA-1624 patches are welcomed. > > > > > > /******************************************* > > > Joe Stein > > > Founder, Principal Consultant > > > Big Data Open Source Security LLC > > > http://www.stealth.ly > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > > ********************************************/ > > > > > > > > > On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan <[email protected]> > wrote: > > > > > > > ./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. (error: > > error > > > > while loading CharSequence, class file > > > > '/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' > is > > > > broken) > > > > I understand there's no escape from installing JDK 7? > > > > 10x > > > > Shlomi > > > > > > > > > > > > On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein <[email protected]> > > wrote: > > > > > > > > > When building you need to use the ./gradelw script as Harsha said. > > > > Please > > > > > take a look at the README for specific commands and how to run > them. > > > > > > > > > > /******************************************* > > > > > Joe Stein > > > > > Founder, Principal Consultant > > > > > Big Data Open Source Security LLC > > > > > http://www.stealth.ly > > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop > > > > > > > ********************************************/ > > > > > > > > > > > > > > > On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan <[email protected]> > > > wrote: > > > > > > > > > > > it failed with JDK 8 so I hoped a newer gradle will maybe do the > > > magic, > > > > > and > > > > > > stepped into this other problem. > > > > > > I assume you will say : > > > > > > install JDK 7 and build with our gradle 1.6. > > > > > > is it so? > > > > > > Shlomi > > > > > > > > > > > > > > > > > > On Thu, Sep 4, 2014 at 5:41 PM, Harsha <[email protected]> wrote: > > > > > > > > > > > > > Did you tried "gradlew" script in kafka source dir. > > > > > > > -Harsha > > > > > > > > > > > > > > On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote: > > > > > > > > what gradle version is used to build kafka_2.9.2-0.8.1.1 ? > > > > > > > > > > > > > > > > tried with v2 and failed with : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > gradle --stacktrace clean > > > > > > > > > > > > > > > > FAILURE: Build failed with an exception. > > > > > > > > > > > > > > > > * Where: > > > > > > > > Build file > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle' > > > > > > > > line: 34 > > > > > > > > > > > > > > > > * What went wrong: > > > > > > > > A problem occurred evaluating root project > 'kafka-0.8.1.1-src'. > > > > > > > > > Could not find method add() for arguments [licenseMain, > class > > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > > > > > > > > > > > > > > > > * Try: > > > > > > > > Run with --info or --debug option to get more log output. > > > > > > > > > > > > > > > > * Exception is: > > > > > > > > org.gradle.api.GradleScriptException: A problem occurred > > > evaluating > > > > > > root > > > > > > > > project 'kafka-0.8.1.1-src'. > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29) > > > > > > > > at > > > > > > > org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22) > > > > > > > > at org.gradle.launcher.Main.doAction(Main.java:33) > > > > > > > > at > > > > > org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35) > > > > > > > > at > org.gradle.launcher.GradleMain.main(GradleMain.java:23) > > > > > > > > Caused by: org.gradle.api.internal.MissingMethodException: > > Could > > > > not > > > > > > find > > > > > > > > method add() for arguments [licenseMain, class > > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:68) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:56) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:172) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.tasks.DefaultTaskContainer_Decorated.invokeMethod(Unknown > > > > > > > > Source) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nl.javadude.gradle.plugins.license.LicensePlugin$_configureSourceSetRule_closure6_closure18.doCall(LicensePlugin.groovy:117) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > > > > > > > > at > > > > > > > > > > > > > > > > org.gradle.listener.ActionBroadcast.execute(ActionBroadcast.java:39) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.DefaultDomainObjectCollection.doAdd(DefaultDomainObjectCollection.java:164) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.DefaultDomainObjectCollection.add(DefaultDomainObjectCollection.java:159) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.DefaultNamedDomainObjectCollection.add(DefaultNamedDomainObjectCollection.java:71) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:58) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:40) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.gradle.api.plugins.JavaPlugin.configureSourceSets(JavaPlugin.java:80) > > > > > > > > at > > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:68) > > > > > > > > at > > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:46) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultPluginContainer.providePlugin(DefaultPluginContainer.java:110) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultPluginContainer.addPluginInternal(DefaultPluginContainer.java:69) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultPluginContainer.apply(DefaultPluginContainer.java:35) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyPlugin(DefaultObjectConfigurationAction.java:117) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$200(DefaultObjectConfigurationAction.java:36) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$3.run(DefaultObjectConfigurationAction.java:85) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:130) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:41) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > build_6i3s72vsr5r4u0gb7m8oo0piuq$_run_closure2.doCall(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:34) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > > > > > > > > at > > > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:130) > > > > > > > > at > > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:91) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:785) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:790) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.project.AbstractProject.subprojects(AbstractProject.java:773) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:225) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:129) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > org.gradle.groovy.scripts.BasicScript.methodMissing(BasicScript.java:79) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > build_6i3s72vsr5r4u0gb7m8oo0piuq.run(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:33) > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:52) > > > > > > > > ... 31 more > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 10x > > > > > > > > Shlomi > > > > > > > > > > > > > > > > > > > > > > > > > > > >
