I can see that in general but why would that limit be several K's ? The whole setup is designed to measure just that among the rest: e.g.: how many open files / connections can kafka handle? limits are far above and the question is do you see any limit on kafka's side to the number of connections?
On Sat, Sep 6, 2014 at 7:59 PM, Joe Stein <joe.st...@stealth.ly> wrote: > If your creating a new producer for each send instead of reusing a pool of > producer connections for each send within your threads that could result in > the brokers surpassing your open file limit. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop > ********************************************/ > On Sep 6, 2014 11:29 AM, "Shlomi Hazan" <shl...@viber.com> wrote: > > > just VPN'ed into my workstation: > > the answer to 5 is [*yes*] > > answer to 1,2: > > the error's I see on the python client are first timeouts and then > message > > send failures, using sync send. > > > > on the controller log: > > > > ontroller.log.2014-08-26-13:[2014-08-26 13:40:44,317] ERROR > > [Controller-1-to-broker-3-send-thread], Controller 1 epoch 3 failed to > send > > StopReplica request with correlation id 519 to broker > > id:3,host:shlomi-kafka-broker-3,port:9092. Reconnecting to broker. > > (kafka.controller.RequestSendThread) > > controller.log.2014-08-26-13:[2014-08-26 13:40:44,319] ERROR > > [Controller-1-to-broker-3-send-thread], Controller 1's connection to > broker > > id:3,host:shlomi-kafka-broker-3,port:9092 was unsuccessful > > (kafka.controller.RequestSendThread) > > > > on the server log (selected greps): > > ... > > server.log.2014-08-27-01:[2014-08-27 01:44:23,143] ERROR > > [ReplicaFetcherThread-4-2], Error for partition [vpq_android_gcm_h,270] > to > > broker 2:class kafka.common.NotLeaderForPartitionException > > (kafka.server.ReplicaFetcherThread) > > ... > > server.log.2014-08-27-12:[2014-08-27 12:08:34,638] ERROR Closing socket > for > > /10.184.150.54 because of error (kafka.network.Processor) > > > > ... > > server.log.2014-08-28-07:[2014-08-28 07:57:35,944] ERROR [KafkaApi-1] > > Error > > when processing fetch request for partition [vpq_android_gcm_h,184] > offset > > 8798 from follower with correlation id 0 (kafka.server.KafkaApis) > > ... > > erver.log.2014-09-03-15:[2014-09-03 15:46:18,220] ERROR > > [ReplicaFetcherThread-2-3], Error in fetch Name: FetchRequest; Version: > 0; > > CorrelationId: 177593; ClientId: ReplicaFetcherThread-2-3; ReplicaId: 1; > > MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [vpq_android_gcm_h,196] > > -> PartitionFetchInfo(65283,8388608),[vpq_android_gcm_h,76] -> > > PartitionFetchInfo(262787,8388608),[vpq_android_gcm_h,460] -> > > PartitionFetchInfo(285709,8388608),[vpq_android_gcm_h,100] -> > > PartitionFetchInfo(199405,8388608),[vpq_android_gcm_h,148] -> > > PartitionFetchInfo(339032,8388608),[vpq_android_gcm_h,436] -> > > PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,124] -> > > PartitionFetchInfo(484447,8388608),[vpq_android_gcm_h,484] -> > > PartitionFetchInfo(105945,8388608),[vpq_android_gcm_h,340] -> > > PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,388] -> > > PartitionFetchInfo(9,8388608),[vpq_android_gcm_h,316] -> > > PartitionFetchInfo(194766,8388608),[vpq_android_gcm_h,364] -> > > PartitionFetchInfo(139897,8388608),[vpq_android_gcm_h,292] -> > > PartitionFetchInfo(195408,8388608),[vpq_android_gcm_h,28] -> > > PartitionFetchInfo(329961,8388608),[vpq_android_gcm_h,172] -> > > PartitionFetchInfo(436959,8388608),[vpq_android_gcm_h,268] -> > > PartitionFetchInfo(59827,8388608),[vpq_android_gcm_h,244] -> > > PartitionFetchInfo(259731,8388608),[vpq_android_gcm_h,220] -> > > PartitionFetchInfo(61669,8388608),[vpq_android_gcm_h,412] -> > > PartitionFetchInfo(563609,8388608),[vpq_android_gcm_h,4] -> > > PartitionFetchInfo(360336,8388608),[vpq_android_gcm_h,52] -> > > PartitionFetchInfo(378533,8388608) (kafka.server.ReplicaFetcherThread) > > ... > > server.log.2014-09-03-14:[2014-09-03 14:04:18,548] ERROR Error in > acceptor > > (kafka.network.Acceptor) > > ... > > > > > > > > On Sat, Sep 6, 2014 at 5:48 PM, Shlomi Hazan <shl...@viber.com> wrote: > > > > > Hi and sorry for the late response I just got into the weekend and > still > > > Satdurday here... > > > Well, not at my desk but will answer what I can: > > > 1. what else on the logs? [*will vpn and check*] > > > 2. other broker failure reason? [*"*] > > > 3. other broker failure after taking leadership? [*how can I be sure? > ask > > > another to describe topic?*] > > > 4. how do I measure number of connections? [*ls -l /proc/<pid>/fd | > grep > > > socket | wc -l, also did watch on that*] > > > 5. is that number equals the number of {new Producer}? > > > 6. how many topics? [*1*] how many partitions [*504*] > > > 7. Are u using a partition key? [*yes, I use the python client with* ] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *class ProducerIdPartitioner(Partitioner): """ Implements a > > > partitioner which selects the target partition based on the sending > > > producer ID """ def partition(self, key, partitions): > size = > > > len(partitions) prod_id = int(key) idx = prod_id % > > > size return partitions[idx]* > > > 8. maybe running into over partitioned topic? [*producer instances is 6 > > > machines * 84 procs * 24 threads, but never got to start them all*,*b/c > > > of errors*] > > > 9. r u running anything else? [*yes, zookeeper*] > > > > > > additional: > > > do you want broker or other config? EC2 instance types? anythying > > else?.... > > > Thanks, > > > Shlomi > > > > > > > > > On Thu, Sep 4, 2014 at 10:02 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > >> I think it sounds more like another issue than your thinking...the > > broker > > >> should not be failing like that especially another broker being > affected > > >> doesn't make sense. > > >> > > >> What else is in the logs on failure? Is the other broker failing > > because > > >> of number of files too? Is it happening after it becomes the leader? > > >> > > >> How are you measuring number of connections? Is this how many producer > > >> connections you are opening up yourself (new Producer())? > > >> > > >> How many topics do you have? How many partitions? > > >> > > >> Are you using a partition key? Maybe you are running into an over > > >> partitioned topic type of scenario where your # of producer instances > > >> multiplied by # topics is the problem (where the producer will open up > > >> it's > > >> own connection over time to brokers for each partition leader). > > >> > > >> Are you running anything else besides kafka broker on the box? > > >> > > >> /******************************************* > > >> Joe Stein > > >> Founder, Principal Consultant > > >> Big Data Open Source Security LLC > > >> http://www.stealth.ly > > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > >> ********************************************/ > > >> > > >> > > >> On Thu, Sep 4, 2014 at 2:06 PM, Shlomi Hazan <shl...@viber.com> > wrote: > > >> > > >> > I sure did. the reason I am building is trying to patch some. > > >> specifically > > >> > this : KAFKA-1623. > > >> > actually if I felt more confident about scala, I would happily send > > you > > >> a > > >> > patch. > > >> > If you don't care screening, just tell me how to prep it for ya and > i > > >> will. > > >> > The bigger problem is running into too many open files when ramping > up > > >> to > > >> > several thousand connection. > > >> > going above 4k is hard and around 6K the broker says goodbye. > > sometimes > > >> > taking a broker friend for a ride.... > > >> > This was what led me to the acceptor and it's being slow.. > > >> > Have you got an idea why 10K connection should pose a problem. > ulimit > > >> > checked. not that. ?? > > >> > Shlomi > > >> > > > >> > > > >> > On Thu, Sep 4, 2014 at 7:00 PM, Joe Stein <joe.st...@stealth.ly> > > wrote: > > >> > > > >> > > Have you tried using a binary release > > >> > > http://kafka.apache.org/downloads.html > > >> > > this way you don't have to-do a build? > > >> > > > > >> > > We build using JDK 6 you should be able to run in 8 (I know for > sure > > >> 6 & > > >> > 7 > > >> > > work honestly never tried 8). > > >> > > > > >> > > I just did a quick test with a broker running on 8 and > > >> produced/consumed > > >> > a > > >> > > few message didn't run into issues... > > >> > > > > >> > > As for building in JDK 8 => I reproduced your issue and created a > > >> ticket > > >> > > https://issues.apache.org/jira/browse/KAFKA-1624 patches are > > >> welcomed. > > >> > > > > >> > > /******************************************* > > >> > > Joe Stein > > >> > > Founder, Principal Consultant > > >> > > Big Data Open Source Security LLC > > >> > > http://www.stealth.ly > > >> > > Twitter: @allthingshadoop < > http://www.twitter.com/allthingshadoop> > > >> > > ********************************************/ > > >> > > > > >> > > > > >> > > On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan <shl...@viber.com> > > >> wrote: > > >> > > > > >> > > > ./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. > > (error: > > >> > error > > >> > > > while loading CharSequence, class file > > >> > > > > > >> '/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' > is > > >> > > > broken) > > >> > > > I understand there's no escape from installing JDK 7? > > >> > > > 10x > > >> > > > Shlomi > > >> > > > > > >> > > > > > >> > > > On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein <joe.st...@stealth.ly > > > > >> > wrote: > > >> > > > > > >> > > > > When building you need to use the ./gradelw script as Harsha > > said. > > >> > > > Please > > >> > > > > take a look at the README for specific commands and how to run > > >> them. > > >> > > > > > > >> > > > > /******************************************* > > >> > > > > Joe Stein > > >> > > > > Founder, Principal Consultant > > >> > > > > Big Data Open Source Security LLC > > >> > > > > http://www.stealth.ly > > >> > > > > Twitter: @allthingshadoop < > > >> http://www.twitter.com/allthingshadoop> > > >> > > > > ********************************************/ > > >> > > > > > > >> > > > > > > >> > > > > On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan < > shl...@viber.com > > > > > >> > > wrote: > > >> > > > > > > >> > > > > > it failed with JDK 8 so I hoped a newer gradle will maybe do > > the > > >> > > magic, > > >> > > > > and > > >> > > > > > stepped into this other problem. > > >> > > > > > I assume you will say : > > >> > > > > > install JDK 7 and build with our gradle 1.6. > > >> > > > > > is it so? > > >> > > > > > Shlomi > > >> > > > > > > > >> > > > > > > > >> > > > > > On Thu, Sep 4, 2014 at 5:41 PM, Harsha <ka...@harsha.io> > > wrote: > > >> > > > > > > > >> > > > > > > Did you tried "gradlew" script in kafka source dir. > > >> > > > > > > -Harsha > > >> > > > > > > > > >> > > > > > > On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote: > > >> > > > > > > > what gradle version is used to build > kafka_2.9.2-0.8.1.1 ? > > >> > > > > > > > > > >> > > > > > > > tried with v2 and failed with : > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > gradle --stacktrace clean > > >> > > > > > > > > > >> > > > > > > > FAILURE: Build failed with an exception. > > >> > > > > > > > > > >> > > > > > > > * Where: > > >> > > > > > > > Build file > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle' > > >> > > > > > > > line: 34 > > >> > > > > > > > > > >> > > > > > > > * What went wrong: > > >> > > > > > > > A problem occurred evaluating root project > > >> 'kafka-0.8.1.1-src'. > > >> > > > > > > > > Could not find method add() for arguments > [licenseMain, > > >> class > > >> > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > > >> > > > > > > > > > >> > > > > > > > * Try: > > >> > > > > > > > Run with --info or --debug option to get more log > output. > > >> > > > > > > > > > >> > > > > > > > * Exception is: > > >> > > > > > > > org.gradle.api.GradleScriptException: A problem occurred > > >> > > evaluating > > >> > > > > > root > > >> > > > > > > > project 'kafka-0.8.1.1-src'. > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29) > > >> > > > > > > > at > > >> > > > > > > > >> org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22) > > >> > > > > > > > at org.gradle.launcher.Main.doAction(Main.java:33) > > >> > > > > > > > at > > >> > > > > > org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35) > > >> > > > > > > > at > > >> org.gradle.launcher.GradleMain.main(GradleMain.java:23) > > >> > > > > > > > Caused by: > org.gradle.api.internal.MissingMethodException: > > >> > Could > > >> > > > not > > >> > > > > > find > > >> > > > > > > > method add() for arguments [licenseMain, class > > >> > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:68) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:56) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:172) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.tasks.DefaultTaskContainer_Decorated.invokeMethod(Unknown > > >> > > > > > > > Source) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > nl.javadude.gradle.plugins.license.LicensePlugin$_configureSourceSetRule_closure6_closure18.doCall(LicensePlugin.groovy:117) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > >> > org.gradle.listener.ActionBroadcast.execute(ActionBroadcast.java:39) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.DefaultDomainObjectCollection.doAdd(DefaultDomainObjectCollection.java:164) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.DefaultDomainObjectCollection.add(DefaultDomainObjectCollection.java:159) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.DefaultNamedDomainObjectCollection.add(DefaultNamedDomainObjectCollection.java:71) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:58) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:40) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> > > org.gradle.api.plugins.JavaPlugin.configureSourceSets(JavaPlugin.java:80) > > >> > > > > > > > at > > >> > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:68) > > >> > > > > > > > at > > >> > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:46) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultPluginContainer.providePlugin(DefaultPluginContainer.java:110) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultPluginContainer.addPluginInternal(DefaultPluginContainer.java:69) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultPluginContainer.apply(DefaultPluginContainer.java:35) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyPlugin(DefaultObjectConfigurationAction.java:117) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$200(DefaultObjectConfigurationAction.java:36) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$3.run(DefaultObjectConfigurationAction.java:85) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:130) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:41) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > build_6i3s72vsr5r4u0gb7m8oo0piuq$_run_closure2.doCall(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:34) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > > >> > > > > > > > at > > >> > > > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:130) > > >> > > > > > > > at > > >> > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:91) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:785) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:790) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.project.AbstractProject.subprojects(AbstractProject.java:773) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:225) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:129) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > >> > > > > > >> > > > org.gradle.groovy.scripts.BasicScript.methodMissing(BasicScript.java:79) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > build_6i3s72vsr5r4u0gb7m8oo0piuq.run(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:33) > > >> > > > > > > > at > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:52) > > >> > > > > > > > ... 31 more > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > 10x > > >> > > > > > > > Shlomi > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
