Hi Telles, I replayed your problem and think I figured out why CLASSPATH does not work. Because in our script bin/run-class.sh, we have the line "CLASSPATH=$HADOOP_CONF_DIR", which actually ingores your setting.
So a simple solution is to copy the hadoop-hdfs.jar to your samza lib directory. Then run bin/run-job ----config-factory=... --config-path=... . Let me know how it goes. Thank you. Cheers, Fang, Yan [email protected] +1 (206) 849-4108 On Mon, Aug 11, 2014 at 4:07 PM, Telles Nobrega <[email protected]> wrote: > Sure, thanks. > > > On Mon, Aug 11, 2014 at 6:22 PM, Yan Fang <[email protected]> wrote: > > > Hi Telles, > > > > I am not sure whether exporting the CLASSPATH works. (sometimes it does > not > > work for me...) My suggestion is to include the hdfs jar explicitly in > the > > package that you upload to hdfs. Also , remember to put the jar into your > > local samza (which is deploy/samza/lib if you go with the hello-samza > > tutorial) Let me know if that works. > > > > Cheers, > > > > Fang, Yan > > [email protected] > > +1 (206) 849-4108 > > > > > > On Mon, Aug 11, 2014 at 2:04 PM, Chris Riccomini < > > [email protected]> wrote: > > > > > Hey Telles, > > > > > > Hmm. I'm out of ideas. If Zhijie is around, he'd probably be of use, > but > > I > > > haven't heard from him in a while. > > > > > > I'm afraid your best bet is probably to email the YARN dev mailing > list, > > > since this is a YARN config issue. > > > > > > Cheers, > > > Chris > > > > > > On 8/11/14 1:58 PM, "Telles Nobrega" <[email protected]> wrote: > > > > > > >I exported export > > > > > > >CLASSPATH=$CLASSPATH:hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-2.3.0.jar > > > >and still happened the same problem. > > > > > > > > > > > >On Mon, Aug 11, 2014 at 5:35 PM, Chris Riccomini < > > > >[email protected]> wrote: > > > > > > > >> Hey Telles, > > > >> > > > >> It sounds like either the HDFS jar is missing from the classpath, or > > the > > > >> hdfs file system needs to be configured: > > > >> > > > >> <property> > > > >> <name>fs.hdfs.impl</name> > > > >> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> > > > >> <description>The FileSystem for hdfs: uris.</description> > > > >> </property> > > > >> > > > >> > > > >> (from > > > >> > > > >> > > > > > > https://groups.google.com/a/cloudera.org/forum/#!topic/scm-users/lyho8ptA > > > >>zE > > > >> 0) > > > >> > > > >> I believe this will need to be configured for your NM. > > > >> > > > >> Cheers, > > > >> Chris > > > >> > > > >> On 8/11/14 1:31 PM, "Telles Nobrega" <[email protected]> > wrote: > > > >> > > > >> >Yes, it is like this: > > > >> > > > > >> ><configuration> > > > >> > <property> > > > >> > <name>dfs.datanode.data.dir</name> > > > >> > <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/datanode</value> > > > >> > <description>Comma separated list of paths on the local > > filesystem > > > >>of > > > >> >a > > > >> >DataNode where it should store its blocks.</description> > > > >> > </property> > > > >> > > > > >> > <property> > > > >> > <name>dfs.namenode.name.dir</name> > > > >> > <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/namenode</value> > > > >> > <description>Path on the local filesystem where the NameNode > > stores > > > >> >the > > > >> >namespace and transaction logs persistently.</description> > > > >> > </property> > > > >> ></configuration> > > > >> >~ > > > >> > > > > >> >I saw some report that this may be a classpath problem. Does this > > > >>sounds > > > >> >right to you? > > > >> > > > > >> > > > > >> >On Mon, Aug 11, 2014 at 5:25 PM, Yan Fang <[email protected]> > > > wrote: > > > >> > > > > >> >> Hi Telles, > > > >> >> > > > >> >> It looks correct. Did you put the hdfs-site.xml into your > > > >> >>HADOOP_CONF_DIR > > > >> >> ？(such as ~/.samza/conf) > > > >> >> > > > >> >> Fang, Yan > > > >> >> [email protected] > > > >> >> +1 (206) 849-4108 > > > >> >> > > > >> >> > > > >> >> On Mon, Aug 11, 2014 at 1:02 PM, Telles Nobrega > > > >> >><[email protected]> > > > >> >> wrote: > > > >> >> > > > >> >> > Hi Yan Fang, > > > >> >> > > > > >> >> > I was able to deploy the file to hdfs, I can see them in all my > > > >>nodes > > > >> >>but > > > >> >> > when I tried running I got this error: > > > >> >> > > > > >> >> > Exception in thread "main" java.io.IOException: No FileSystem > for > > > >> >>scheme: > > > >> >> > hdfs > > > >> >> > at > > > >> >> > > > > > > >>org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421) > > > >> >> > at > > > >> >> > > > > >>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) > > > >> >> > at > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) > > > >> >> > at > > > >> >> > > > > >>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) > > > >> >> > at > > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) > > > >> >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > > > >> >> > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > > > >> >> > at > > > >> >> > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.s > > > >>>>ca > > > >> >>la:111) > > > >> >> > at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) > > > >> >> > at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) > > > >> >> > at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) > > > >> >> > at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) > > > >> >> > at org.apache.samza.job.JobRunner.main(JobRunner.scala) > > > >> >> > > > > >> >> > > > > >> >> > This is my yarn.package.path config: > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>yarn.package.path=hdfs://telles-master-samza:50070/samza-job-package-0 > > > >>>>.7 > > > >> >>.0-dist.tar.gz > > > >> >> > > > > >> >> > Thanks in advance > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > On Mon, Aug 11, 2014 at 3:00 PM, Yan Fang < > [email protected]> > > > >> >>wrote: > > > >> >> > > > > >> >> > > Hi Telles, > > > >> >> > > > > > >> >> > > In terms of "*I tried pushing the tar file to HDFS but I got > an > > > >> >>error > > > >> >> > from > > > >> >> > > hadoop saying that it couldn’t find core-site.xml file*.", I > > > >>guess > > > >> >>you > > > >> >> > set > > > >> >> > > the HADOOP_CONF_DIR variable and made it point to > > ~/.samza/conf. > > > >>You > > > >> >> can > > > >> >> > do > > > >> >> > > 1) make the HADOOP_CONF_DIR point to the directory where your > > > >>conf > > > >> >> files > > > >> >> > > are, such as /etc/hadoop/conf. Or 2) copy the config files to > > > >> >> > > ~/.samza/conf. Thank you, > > > >> >> > > > > > >> >> > > Cheer, > > > >> >> > > > > > >> >> > > Fang, Yan > > > >> >> > > [email protected] > > > >> >> > > +1 (206) 849-4108 > > > >> >> > > > > > >> >> > > > > > >> >> > > On Mon, Aug 11, 2014 at 7:40 AM, Chris Riccomini < > > > >> >> > > [email protected]> wrote: > > > >> >> > > > > > >> >> > > > Hey Telles, > > > >> >> > > > > > > >> >> > > > To get YARN working with the HTTP file system, you need to > > > >>follow > > > >> >>the > > > >> >> > > > instructions on: > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > > >> > > > >> > > > > > > http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node > > > >> >>-y > > > >> >> > > > arn.html > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > In the "Set Up Http Filesystem for YARN" section. > > > >> >> > > > > > > >> >> > > > You shouldn't need to compile anything (no Gradle, which is > > > >>what > > > >> >>your > > > >> >> > > > stack trace is showing). This setup should be done for all > of > > > >>the > > > >> >> NMs, > > > >> >> > > > since they will be the ones downloading your job's package > > > >>(from > > > >> >> > > > yarn.package.path). > > > >> >> > > > > > > >> >> > > > Cheers, > > > >> >> > > > Chris > > > >> >> > > > > > > >> >> > > > On 8/9/14 9:44 PM, "Telles Nobrega" < > [email protected] > > > > > > >> >>wrote: > > > >> >> > > > > > > >> >> > > > >Hi again, I tried installing the scala libs but the Http > > > >>problem > > > >> >> still > > > >> >> > > > >occurs. I realised that I need to compile incubator samza > in > > > >>the > > > >> >> > > machines > > > >> >> > > > >that I¹m going to run the jobs, but the compilation fails > > with > > > >> >>this > > > >> >> > huge > > > >> >> > > > >message: > > > >> >> > > > > > > > >> >> > > > ># > > > >> >> > > > ># There is insufficient memory for the Java Runtime > > > >>Environment > > > >> >>to > > > >> >> > > > >continue. > > > >> >> > > > ># Native memory allocation (malloc) failed to allocate > > > >>3946053632 > > > >> >> > bytes > > > >> >> > > > >for committing reserved memory. > > > >> >> > > > ># An error report file with more information is saved as: > > > >> >> > > > ># > > /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2506.log > > > >> >> > > > >Could not write standard input into: Gradle Worker 13. > > > >> >> > > > >java.io.IOException: Broken pipe > > > >> >> > > > > at java.io.FileOutputStream.writeBytes(Native > Method) > > > >> >> > > > > at > > > >> >>java.io.FileOutputStream.write(FileOutputStream.java:345) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > >>java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > > > >> >> > > > > at > > > >> >> > > > > > > >>java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut > > > >>>>>pu > > > >> >>>tH > > > >> >> > > > >andleRunner.java:53) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > > > >>>>>rI > > > >> >>>mp > > > >> >> > > > >l$1.run(DefaultExecutorFactory.java:66) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >1145) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>>>>ja > > > >> >>>va > > > >> >> > > > >:615) > > > >> >> > > > > at java.lang.Thread.run(Thread.java:744) > > > >> >> > > > >Process 'Gradle Worker 13' finished with non-zero exit > > value 1 > > > >> >> > > > >org.gradle.process.internal.ExecException: Process 'Gradle > > > >>Worker > > > >> >> 13' > > > >> >> > > > >finished with non-zero exit value 1 > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor > > > >>>>>ma > > > >> >>>lE > > > >> >> > > > >xitValue(DefaultExecHandle.java:362) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default > > > >>>>>Wo > > > >> >>>rk > > > >> >> > > > >erProcess.java:89) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor > > > >>>>>ke > > > >> >>>rP > > > >> >> > > > >rocess.java:33) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D > > > >>>>>ef > > > >> >>>au > > > >> >> > > > >ltWorkerProcess.java:55) > > > >> >> > > > > at > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > >> >> Method) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >57) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > > > >>>>>or > > > >> >>>Im > > > >> >> > > > >pl.java:43) > > > >> >> > > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > > > >>>>>sp > > > >> >>>at > > > >> >> > > > >ch.java:35) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > > > >>>>>sp > > > >> >>>at > > > >> >> > > > >ch.java:24) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > > > >>>>>81 > > > >> >>>) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > > > >>>>>30 > > > >> >>>) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati > > > >>>>>on > > > >> >>>Ha > > > >> >> > > > >ndler.invoke(ProxyDispatchAdapter.java:93) > > > >> >> > > > > at com.sun.proxy.$Proxy46.executionFinished(Unknown > > > >> >>Source) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE > > > >>>>>xe > > > >> >>>cH > > > >> >> > > > >andle.java:212) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand > > > >>>>>le > > > >> >>>.j > > > >> >> > > > >ava:309) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne > > > >>>>>r. > > > >> >>>ja > > > >> >> > > > >va:108) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java > > > >>>>>:8 > > > >> >>>8) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > > > >>>>>rI > > > >> >>>mp > > > >> >> > > > >l$1.run(DefaultExecutorFactory.java:66) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >1145) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>>>>ja > > > >> >>>va > > > >> >> > > > >:615) > > > >> >> > > > > at java.lang.Thread.run(Thread.java:744) > > > >> >> > > > >OpenJDK 64-Bit Server VM warning: INFO: > > > >> >> > > > >os::commit_memory(0x000000070a6c0000, 3946053632, 0) > failed; > > > >> >> > > > >error='Cannot allocate memory' (errno=12) > > > >> >> > > > ># > > > >> >> > > > ># There is insufficient memory for the Java Runtime > > > >>Environment > > > >> >>to > > > >> >> > > > >continue. > > > >> >> > > > ># Native memory allocation (malloc) failed to allocate > > > >>3946053632 > > > >> >> > bytes > > > >> >> > > > >for committing reserved memory. > > > >> >> > > > ># An error report file with more information is saved as: > > > >> >> > > > ># > > /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2518.log > > > >> >> > > > >Could not write standard input into: Gradle Worker 14. > > > >> >> > > > >java.io.IOException: Broken pipe > > > >> >> > > > > at java.io.FileOutputStream.writeBytes(Native > Method) > > > >> >> > > > > at > > > >> >>java.io.FileOutputStream.write(FileOutputStream.java:345) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > >>java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > > > >> >> > > > > at > > > >> >> > > > > > > >>java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut > > > >>>>>pu > > > >> >>>tH > > > >> >> > > > >andleRunner.java:53) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > > > >>>>>rI > > > >> >>>mp > > > >> >> > > > >l$1.run(DefaultExecutorFactory.java:66) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >1145) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>>>>ja > > > >> >>>va > > > >> >> > > > >:615) > > > >> >> > > > > at java.lang.Thread.run(Thread.java:744) > > > >> >> > > > >Process 'Gradle Worker 14' finished with non-zero exit > > value 1 > > > >> >> > > > >org.gradle.process.internal.ExecException: Process 'Gradle > > > >>Worker > > > >> >> 14' > > > >> >> > > > >finished with non-zero exit value 1 > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor > > > >>>>>ma > > > >> >>>lE > > > >> >> > > > >xitValue(DefaultExecHandle.java:362) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default > > > >>>>>Wo > > > >> >>>rk > > > >> >> > > > >erProcess.java:89) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor > > > >>>>>ke > > > >> >>>rP > > > >> >> > > > >rocess.java:33) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D > > > >>>>>ef > > > >> >>>au > > > >> >> > > > >ltWorkerProcess.java:55) > > > >> >> > > > > at > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > >> >> Method) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >57) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > > > >>>>>or > > > >> >>>Im > > > >> >> > > > >pl.java:43) > > > >> >> > > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > > > >>>>>sp > > > >> >>>at > > > >> >> > > > >ch.java:35) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > > > >>>>>sp > > > >> >>>at > > > >> >> > > > >ch.java:24) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > > > >>>>>81 > > > >> >>>) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > > > >>>>>30 > > > >> >>>) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati > > > >>>>>on > > > >> >>>Ha > > > >> >> > > > >ndler.invoke(ProxyDispatchAdapter.java:93) > > > >> >> > > > > at com.sun.proxy.$Proxy46.executionFinished(Unknown > > > >> >>Source) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE > > > >>>>>xe > > > >> >>>cH > > > >> >> > > > >andle.java:212) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand > > > >>>>>le > > > >> >>>.j > > > >> >> > > > >ava:309) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne > > > >>>>>r. > > > >> >>>ja > > > >> >> > > > >va:108) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java > > > >>>>>:8 > > > >> >>>8) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > > > >>>>>rI > > > >> >>>mp > > > >> >> > > > >l$1.run(DefaultExecutorFactory.java:66) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > >>>>>av > > > >> >>>a: > > > >> >> > > > >1145) > > > >> >> > > > > at > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>>>>ja > > > >> >>>va > > > >> >> > > > >:615) > > > >> >> > > > > at java.lang.Thread.r > > > >> >> > > > > > > > >> >> > > > >Do I need more memory for my machines? Each already has > > 4GB. I > > > >> >> really > > > >> >> > > > >need to have this running. I¹m not sure which way is best > > > >>http or > > > >> >> hdfs > > > >> >> > > > >which one you suggest and how can i solve my problem for > > each > > > >> >>case. > > > >> >> > > > > > > > >> >> > > > >Thanks in advance and sorry for bothering this much. > > > >> >> > > > >On 10 Aug 2014, at 00:20, Telles Nobrega > > > >> >><[email protected]> > > > >> >> > > wrote: > > > >> >> > > > > > > > >> >> > > > >> Hi Chris, now I have the tar file in my RM machine, and > > the > > > >> >>yarn > > > >> >> > path > > > >> >> > > > >>points to it. I changed the core-site.xml to use > > > >>HttpFileSystem > > > >> >> > instead > > > >> >> > > > >>of HDFS now it is failing with > > > >> >> > > > >> > > > >> >> > > > >> Application application_1407640485281_0001 failed 2 > times > > > >>due > > > >> >>to > > > >> >> AM > > > >> >> > > > >>Container for appattempt_1407640485281_0001_000002 exited > > > >>with > > > >> >> > > > >>exitCode:-1000 due to: java.lang.ClassNotFoundException: > > > >>Class > > > >> >> > > > >>org.apache.samza.util.hadoop.HttpFileSystem not found > > > >> >> > > > >> > > > >> >> > > > >> I think I can solve this just installing scala files > from > > > >>the > > > >> >> samza > > > >> >> > > > >>tutorial, can you confirm that? > > > >> >> > > > >> > > > >> >> > > > >> On 09 Aug 2014, at 08:34, Telles Nobrega > > > >> >><[email protected] > > > >> >> > > > > >> >> > > > >>wrote: > > > >> >> > > > >> > > > >> >> > > > >>> Hi Chris, > > > >> >> > > > >>> > > > >> >> > > > >>> I think the problem is that I forgot to update the > > > >> >> > yarn.job.package. > > > >> >> > > > >>> I will try again to see if it works now. > > > >> >> > > > >>> > > > >> >> > > > >>> I have one more question, how can I stop (command line) > > the > > > >> >>jobs > > > >> >> > > > >>>running in my topology, for the experiment that I will > > run, > > > >>I > > > >> >>need > > > >> >> > to > > > >> >> > > > >>>run the same job in 4 minutes intervals. So I need to > kill > > > >>it, > > > >> >> clean > > > >> >> > > > >>>the kafka topics and rerun. > > > >> >> > > > >>> > > > >> >> > > > >>> Thanks in advance. > > > >> >> > > > >>> > > > >> >> > > > >>> On 08 Aug 2014, at 12:41, Chris Riccomini > > > >> >> > > > >>><[email protected]> wrote: > > > >> >> > > > >>> > > > >> >> > > > >>>> Hey Telles, > > > >> >> > > > >>>> > > > >> >> > > > >>>>>> Do I need to have the job folder on each machine in > my > > > >> >> cluster? > > > >> >> > > > >>>> > > > >> >> > > > >>>> No, you should not need to do this. There are two ways > > to > > > >> >>deploy > > > >> >> > > your > > > >> >> > > > >>>> tarball to the YARN grid. One is to put it in HDFS, > and > > > >>the > > > >> >> other > > > >> >> > is > > > >> >> > > > >>>>to > > > >> >> > > > >>>> put it on an HTTP server. The link to running a Samza > > job > > > >>in > > > >> >>a > > > >> >> > > > >>>>multi-node > > > >> >> > > > >>>> YARN cluster describes how to do both (either HTTP > > server > > > >>or > > > >> >> > HDFS). > > > >> >> > > > >>>> > > > >> >> > > > >>>> In both cases, once the tarball is put in on the > > HTTP/HDFS > > > >> >> > > server(s), > > > >> >> > > > >>>>you > > > >> >> > > > >>>> must update yarn.package.path to point to it. From > > there, > > > >>the > > > >> >> YARN > > > >> >> > > NM > > > >> >> > > > >>>> should download it for you automatically when you > start > > > >>your > > > >> >> job. > > > >> >> > > > >>>> > > > >> >> > > > >>>> * Can you send along a paste of your job config? > > > >> >> > > > >>>> > > > >> >> > > > >>>> Cheers, > > > >> >> > > > >>>> Chris > > > >> >> > > > >>>> > > > >> >> > > > >>>> On 8/8/14 8:04 AM, "Claudio Martins" > > > >> >><[email protected]> > > > >> >> > > wrote: > > > >> >> > > > >>>> > > > >> >> > > > >>>>> Hi Telles, it looks to me that you forgot to update > the > > > >> >> > > > >>>>> "yarn.package.path" > > > >> >> > > > >>>>> attribute in your config file for the task. > > > >> >> > > > >>>>> > > > >> >> > > > >>>>> - Claudio Martins > > > >> >> > > > >>>>> Head of Engineering > > > >> >> > > > >>>>> MobileAware USA Inc. / www.mobileaware.com > > > >> >> > > > >>>>> office: +1 617 986 5060 / mobile: +1 617 480 5288 > > > >> >> > > > >>>>> linkedin: www.linkedin.com/in/martinsclaudio > > > >> >> > > > >>>>> > > > >> >> > > > >>>>> > > > >> >> > > > >>>>> On Fri, Aug 8, 2014 at 10:55 AM, Telles Nobrega > > > >> >> > > > >>>>><[email protected]> > > > >> >> > > > >>>>> wrote: > > > >> >> > > > >>>>> > > > >> >> > > > >>>>>> Hi, > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> this is my first time trying to run a job on a > > multinode > > > >> >> > > > >>>>>>environment. I > > > >> >> > > > >>>>>> have the cluster set up, I can see in the GUI that > all > > > >> >>nodes > > > >> >> are > > > >> >> > > > >>>>>> working. > > > >> >> > > > >>>>>> Do I need to have the job folder on each machine in > my > > > >> >> cluster? > > > >> >> > > > >>>>>> - The first time I tried running with the job on the > > > >> >>namenode > > > >> >> > > > >>>>>>machine > > > >> >> > > > >>>>>> and > > > >> >> > > > >>>>>> it failed saying: > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> Application application_1407509228798_0001 failed 2 > > > >>times > > > >> >>due > > > >> >> to > > > >> >> > > AM > > > >> >> > > > >>>>>> Container for appattempt_1407509228798_0001_000002 > > > >>exited > > > >> >>with > > > >> >> > > > >>>>>>exitCode: > > > >> >> > > > >>>>>> -1000 due to: File > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>>>>>>file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job- > > > >>>>>>>>>>pa > > > >> >>>>>>>>ck > > > >> >> > > > >>>>>>age- > > > >> >> > > > >>>>>> 0.7.0-dist.tar.gz > > > >> >> > > > >>>>>> does not exist > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> So I copied the folder to each machine in my cluster > > and > > > >> >>got > > > >> >> > this > > > >> >> > > > >>>>>>error: > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> Application application_1407509228798_0002 failed 2 > > > >>times > > > >> >>due > > > >> >> to > > > >> >> > > AM > > > >> >> > > > >>>>>> Container for appattempt_1407509228798_0002_000002 > > > >>exited > > > >> >>with > > > >> >> > > > >>>>>>exitCode: > > > >> >> > > > >>>>>> -1000 due to: Resource > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > > > >>>>>>>>>>file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job- > > > >>>>>>>>>>pa > > > >> >>>>>>>>ck > > > >> >> > > > >>>>>>age- > > > >> >> > > > >>>>>> 0.7.0-dist.tar.gz > > > >> >> > > > >>>>>> changed on src filesystem (expected 1407509168000, > was > > > >> >> > > 1407509434000 > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> What am I missing? > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> p.s.: I followed this > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>>< > > > >> >> > > > > > > >> >> > > https://github.com/yahoo/samoa/wiki/Executing-SAMOA-with-Apache-Samz > > > >> >> > > > >>>>>>a> > > > >> >> > > > >>>>>> tutorial > > > >> >> > > > >>>>>> and this > > > >> >> > > > >>>>>> < > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> > > > >> >> > > > > > > >> >> > > > http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi- > > > >> >> > > > >>>>>>node > > > >> >> > > > >>>>>> -yarn.html > > > >> >> > > > >>>>>>> > > > >> >> > > > >>>>>> to > > > >> >> > > > >>>>>> set up the cluster. > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> Help is much appreciated. > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> Thanks in advance. > > > >> >> > > > >>>>>> > > > >> >> > > > >>>>>> -- > > > >> >> > > > >>>>>> ------------------------------------------ > > > >> >> > > > >>>>>> Telles Mota Vidal Nobrega > > > >> >> > > > >>>>>> M.sc. Candidate at UFCG > > > >> >> > > > >>>>>> B.sc. in Computer Science at UFCG > > > >> >> > > > >>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > > > >> >> > > > >>>>>> > > > >> >> > > > >>>> > > > >> >> > > > >>> > > > >> >> > > > >> > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > -- > > > >> >> > ------------------------------------------ > > > >> >> > Telles Mota Vidal Nobrega > > > >> >> > M.sc. Candidate at UFCG > > > >> >> > B.sc. in Computer Science at UFCG > > > >> >> > Software Engineer at OpenStack Project - HP/LSD-UFCG > > > >> >> > > > > >> >> > > > >> > > > > >> > > > > >> > > > > >> >-- > > > >> >------------------------------------------ > > > >> >Telles Mota Vidal Nobrega > > > >> >M.sc. Candidate at UFCG > > > >> >B.sc. in Computer Science at UFCG > > > >> >Software Engineer at OpenStack Project - HP/LSD-UFCG > > > >> > > > >> > > > > > > > > > > > >-- > > > >------------------------------------------ > > > >Telles Mota Vidal Nobrega > > > >M.sc. Candidate at UFCG > > > >B.sc. in Computer Science at UFCG > > > >Software Engineer at OpenStack Project - HP/LSD-UFCG > > > > > > > > > > > > -- > ------------------------------------------ > Telles Mota Vidal Nobrega > M.sc. Candidate at UFCG > B.sc. in Computer Science at UFCG > Software Engineer at OpenStack Project - HP/LSD-UFCG >
