Hi, I copied hadoop-hdfs-2.3.0 to my-job/lib and it changed the error which is
good but the error is back to
Exception in thread "main" java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.hadoop.hdfs.DistributedFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2415)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at
org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.scala:111)
at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55)
at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48)
at org.apache.samza.job.JobRunner.run(JobRunner.scala:62)
at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37)
at org.apache.samza.job.JobRunner.main(JobRunner.scala)
Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.hdfs.DistributedFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
... 13 more
Do I need to have this lib in all nodes at the job folder or just to submit?
On 11 Aug 2014, at 20:11, Yan Fang <[email protected]> wrote:
> Hi Telles,
>
> I replayed your problem and think I figured out why CLASSPATH does not
> work. Because in our script bin/run-class.sh, we have the line
> "CLASSPATH=$HADOOP_CONF_DIR", which actually ingores your setting.
>
> So a simple solution is to copy the hadoop-hdfs.jar to your samza lib
> directory. Then run bin/run-job ----config-factory=... --config-path=... .
> Let me know how it goes. Thank you.
>
> Cheers,
>
> Fang, Yan
> [email protected]
> +1 (206) 849-4108
>
>
> On Mon, Aug 11, 2014 at 4:07 PM, Telles Nobrega <[email protected]>
> wrote:
>
>> Sure, thanks.
>>
>>
>> On Mon, Aug 11, 2014 at 6:22 PM, Yan Fang <[email protected]> wrote:
>>
>>> Hi Telles,
>>>
>>> I am not sure whether exporting the CLASSPATH works. (sometimes it does
>> not
>>> work for me...) My suggestion is to include the hdfs jar explicitly in
>> the
>>> package that you upload to hdfs. Also , remember to put the jar into your
>>> local samza (which is deploy/samza/lib if you go with the hello-samza
>>> tutorial) Let me know if that works.
>>>
>>> Cheers,
>>>
>>> Fang, Yan
>>> [email protected]
>>> +1 (206) 849-4108
>>>
>>>
>>> On Mon, Aug 11, 2014 at 2:04 PM, Chris Riccomini <
>>> [email protected]> wrote:
>>>
>>>> Hey Telles,
>>>>
>>>> Hmm. I'm out of ideas. If Zhijie is around, he'd probably be of use,
>> but
>>> I
>>>> haven't heard from him in a while.
>>>>
>>>> I'm afraid your best bet is probably to email the YARN dev mailing
>> list,
>>>> since this is a YARN config issue.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On 8/11/14 1:58 PM, "Telles Nobrega" <[email protected]> wrote:
>>>>
>>>>> I exported export
>>>>
>>>
>>> CLASSPATH=$CLASSPATH:hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-2.3.0.jar
>>>>> and still happened the same problem.
>>>>>
>>>>>
>>>>> On Mon, Aug 11, 2014 at 5:35 PM, Chris Riccomini <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hey Telles,
>>>>>>
>>>>>> It sounds like either the HDFS jar is missing from the classpath, or
>>> the
>>>>>> hdfs file system needs to be configured:
>>>>>>
>>>>>> <property>
>>>>>> <name>fs.hdfs.impl</name>
>>>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
>>>>>> <description>The FileSystem for hdfs: uris.</description>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> (from
>>>>>>
>>>>>>
>>>>
>>>
>> https://groups.google.com/a/cloudera.org/forum/#!topic/scm-users/lyho8ptA
>>>>>> zE
>>>>>> 0)
>>>>>>
>>>>>> I believe this will need to be configured for your NM.
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>> On 8/11/14 1:31 PM, "Telles Nobrega" <[email protected]>
>> wrote:
>>>>>>
>>>>>>> Yes, it is like this:
>>>>>>>
>>>>>>> <configuration>
>>>>>>> <property>
>>>>>>> <name>dfs.datanode.data.dir</name>
>>>>>>> <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/datanode</value>
>>>>>>> <description>Comma separated list of paths on the local
>>> filesystem
>>>>>> of
>>>>>>> a
>>>>>>> DataNode where it should store its blocks.</description>
>>>>>>> </property>
>>>>>>>
>>>>>>> <property>
>>>>>>> <name>dfs.namenode.name.dir</name>
>>>>>>> <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/namenode</value>
>>>>>>> <description>Path on the local filesystem where the NameNode
>>> stores
>>>>>>> the
>>>>>>> namespace and transaction logs persistently.</description>
>>>>>>> </property>
>>>>>>> </configuration>
>>>>>>> ~
>>>>>>>
>>>>>>> I saw some report that this may be a classpath problem. Does this
>>>>>> sounds
>>>>>>> right to you?
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 11, 2014 at 5:25 PM, Yan Fang <[email protected]>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi Telles,
>>>>>>>>
>>>>>>>> It looks correct. Did you put the hdfs-site.xml into your
>>>>>>>> HADOOP_CONF_DIR
>>>>>>>> ？(such as ~/.samza/conf)
>>>>>>>>
>>>>>>>> Fang, Yan
>>>>>>>> [email protected]
>>>>>>>> +1 (206) 849-4108
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 11, 2014 at 1:02 PM, Telles Nobrega
>>>>>>>> <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Yan Fang,
>>>>>>>>>
>>>>>>>>> I was able to deploy the file to hdfs, I can see them in all my
>>>>>> nodes
>>>>>>>> but
>>>>>>>>> when I tried running I got this error:
>>>>>>>>>
>>>>>>>>> Exception in thread "main" java.io.IOException: No FileSystem
>> for
>>>>>>>> scheme:
>>>>>>>>> hdfs
>>>>>>>>> at
>>>>>>>>
>>>>
>>>
>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
>>>>>>>>> at
>>>>>>>>
>>>>
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
>>>>>>>>> at
>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
>>>>>>>>> at
>>>>>>>>
>>>>
>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
>>>>>>>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
>>>>>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
>>>>>>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>> org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.s
>>>>>>>> ca
>>>>>>>> la:111)
>>>>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55)
>>>>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48)
>>>>>>>>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:62)
>>>>>>>>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37)
>>>>>>>>> at org.apache.samza.job.JobRunner.main(JobRunner.scala)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is my yarn.package.path config:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>> yarn.package.path=hdfs://telles-master-samza:50070/samza-job-package-0
>>>>>>>> .7
>>>>>>>> .0-dist.tar.gz
>>>>>>>>>
>>>>>>>>> Thanks in advance
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 11, 2014 at 3:00 PM, Yan Fang <
>> [email protected]>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Telles,
>>>>>>>>>>
>>>>>>>>>> In terms of "*I tried pushing the tar file to HDFS but I got
>> an
>>>>>>>> error
>>>>>>>>> from
>>>>>>>>>> hadoop saying that it couldn’t find core-site.xml file*.", I
>>>>>> guess
>>>>>>>> you
>>>>>>>>> set
>>>>>>>>>> the HADOOP_CONF_DIR variable and made it point to
>>> ~/.samza/conf.
>>>>>> You
>>>>>>>> can
>>>>>>>>> do
>>>>>>>>>> 1) make the HADOOP_CONF_DIR point to the directory where your
>>>>>> conf
>>>>>>>> files
>>>>>>>>>> are, such as /etc/hadoop/conf. Or 2) copy the config files to
>>>>>>>>>> ~/.samza/conf. Thank you,
>>>>>>>>>>
>>>>>>>>>> Cheer,
>>>>>>>>>>
>>>>>>>>>> Fang, Yan
>>>>>>>>>> [email protected]
>>>>>>>>>> +1 (206) 849-4108
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 11, 2014 at 7:40 AM, Chris Riccomini <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Telles,
>>>>>>>>>>>
>>>>>>>>>>> To get YARN working with the HTTP file system, you need to
>>>>>> follow
>>>>>>>> the
>>>>>>>>>>> instructions on:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>> http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node
>>>>>>>> -y
>>>>>>>>>>> arn.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In the "Set Up Http Filesystem for YARN" section.
>>>>>>>>>>>
>>>>>>>>>>> You shouldn't need to compile anything (no Gradle, which is
>>>>>> what
>>>>>>>> your
>>>>>>>>>>> stack trace is showing). This setup should be done for all
>> of
>>>>>> the
>>>>>>>> NMs,
>>>>>>>>>>> since they will be the ones downloading your job's package
>>>>>> (from
>>>>>>>>>>> yarn.package.path).
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>> On 8/9/14 9:44 PM, "Telles Nobrega" <
>> [email protected]
>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi again, I tried installing the scala libs but the Http
>>>>>> problem
>>>>>>>> still
>>>>>>>>>>>> occurs. I realised that I need to compile incubator samza
>> in
>>>>>> the
>>>>>>>>>> machines
>>>>>>>>>>>> that I¹m going to run the jobs, but the compilation fails
>>> with
>>>>>>>> this
>>>>>>>>> huge
>>>>>>>>>>>> message:
>>>>>>>>>>>>
>>>>>>>>>>>> #
>>>>>>>>>>>> # There is insufficient memory for the Java Runtime
>>>>>> Environment
>>>>>>>> to
>>>>>>>>>>>> continue.
>>>>>>>>>>>> # Native memory allocation (malloc) failed to allocate
>>>>>> 3946053632
>>>>>>>>> bytes
>>>>>>>>>>>> for committing reserved memory.
>>>>>>>>>>>> # An error report file with more information is saved as:
>>>>>>>>>>>> #
>>> /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2506.log
>>>>>>>>>>>> Could not write standard input into: Gradle Worker 13.
>>>>>>>>>>>> java.io.IOException: Broken pipe
>>>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native
>> Method)
>>>>>>>>>>>> at
>>>>>>>> java.io.FileOutputStream.write(FileOutputStream.java:345)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>
>>>>
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut
>>>>>>>>> pu
>>>>>>>>> tH
>>>>>>>>>>>> andleRunner.java:53)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto
>>>>>>>>> rI
>>>>>>>>> mp
>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 1145)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>>>> ja
>>>>>>>>> va
>>>>>>>>>>>> :615)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>> Process 'Gradle Worker 13' finished with non-zero exit
>>> value 1
>>>>>>>>>>>> org.gradle.process.internal.ExecException: Process 'Gradle
>>>>>> Worker
>>>>>>>> 13'
>>>>>>>>>>>> finished with non-zero exit value 1
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor
>>>>>>>>> ma
>>>>>>>>> lE
>>>>>>>>>>>> xitValue(DefaultExecHandle.java:362)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default
>>>>>>>>> Wo
>>>>>>>>> rk
>>>>>>>>>>>> erProcess.java:89)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor
>>>>>>>>> ke
>>>>>>>>> rP
>>>>>>>>>>>> rocess.java:33)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D
>>>>>>>>> ef
>>>>>>>>> au
>>>>>>>>>>>> ltWorkerProcess.java:55)
>>>>>>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>> Method)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 57)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>>>>>>>>> or
>>>>>>>>> Im
>>>>>>>>>>>> pl.java:43)
>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi
>>>>>>>>> sp
>>>>>>>>> at
>>>>>>>>>>>> ch.java:35)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi
>>>>>>>>> sp
>>>>>>>>> at
>>>>>>>>>>>> ch.java:24)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:
>>>>>>>>> 81
>>>>>>>>> )
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:
>>>>>>>>> 30
>>>>>>>>> )
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati
>>>>>>>>> on
>>>>>>>>> Ha
>>>>>>>>>>>> ndler.invoke(ProxyDispatchAdapter.java:93)
>>>>>>>>>>>> at com.sun.proxy.$Proxy46.executionFinished(Unknown
>>>>>>>> Source)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE
>>>>>>>>> xe
>>>>>>>>> cH
>>>>>>>>>>>> andle.java:212)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand
>>>>>>>>> le
>>>>>>>>> .j
>>>>>>>>>>>> ava:309)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne
>>>>>>>>> r.
>>>>>>>>> ja
>>>>>>>>>>>> va:108)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java
>>>>>>>>> :8
>>>>>>>>> 8)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto
>>>>>>>>> rI
>>>>>>>>> mp
>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 1145)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>>>> ja
>>>>>>>>> va
>>>>>>>>>>>> :615)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: INFO:
>>>>>>>>>>>> os::commit_memory(0x000000070a6c0000, 3946053632, 0)
>> failed;
>>>>>>>>>>>> error='Cannot allocate memory' (errno=12)
>>>>>>>>>>>> #
>>>>>>>>>>>> # There is insufficient memory for the Java Runtime
>>>>>> Environment
>>>>>>>> to
>>>>>>>>>>>> continue.
>>>>>>>>>>>> # Native memory allocation (malloc) failed to allocate
>>>>>> 3946053632
>>>>>>>>> bytes
>>>>>>>>>>>> for committing reserved memory.
>>>>>>>>>>>> # An error report file with more information is saved as:
>>>>>>>>>>>> #
>>> /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2518.log
>>>>>>>>>>>> Could not write standard input into: Gradle Worker 14.
>>>>>>>>>>>> java.io.IOException: Broken pipe
>>>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native
>> Method)
>>>>>>>>>>>> at
>>>>>>>> java.io.FileOutputStream.write(FileOutputStream.java:345)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>
>>>>
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut
>>>>>>>>> pu
>>>>>>>>> tH
>>>>>>>>>>>> andleRunner.java:53)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto
>>>>>>>>> rI
>>>>>>>>> mp
>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 1145)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>>>> ja
>>>>>>>>> va
>>>>>>>>>>>> :615)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>>>> Process 'Gradle Worker 14' finished with non-zero exit
>>> value 1
>>>>>>>>>>>> org.gradle.process.internal.ExecException: Process 'Gradle
>>>>>> Worker
>>>>>>>> 14'
>>>>>>>>>>>> finished with non-zero exit value 1
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor
>>>>>>>>> ma
>>>>>>>>> lE
>>>>>>>>>>>> xitValue(DefaultExecHandle.java:362)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default
>>>>>>>>> Wo
>>>>>>>>> rk
>>>>>>>>>>>> erProcess.java:89)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor
>>>>>>>>> ke
>>>>>>>>> rP
>>>>>>>>>>>> rocess.java:33)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D
>>>>>>>>> ef
>>>>>>>>> au
>>>>>>>>>>>> ltWorkerProcess.java:55)
>>>>>>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>> Method)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 57)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>>>>>>>>> or
>>>>>>>>> Im
>>>>>>>>>>>> pl.java:43)
>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi
>>>>>>>>> sp
>>>>>>>>> at
>>>>>>>>>>>> ch.java:35)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi
>>>>>>>>> sp
>>>>>>>>> at
>>>>>>>>>>>> ch.java:24)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:
>>>>>>>>> 81
>>>>>>>>> )
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:
>>>>>>>>> 30
>>>>>>>>> )
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati
>>>>>>>>> on
>>>>>>>>> Ha
>>>>>>>>>>>> ndler.invoke(ProxyDispatchAdapter.java:93)
>>>>>>>>>>>> at com.sun.proxy.$Proxy46.executionFinished(Unknown
>>>>>>>> Source)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE
>>>>>>>>> xe
>>>>>>>>> cH
>>>>>>>>>>>> andle.java:212)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand
>>>>>>>>> le
>>>>>>>>> .j
>>>>>>>>>>>> ava:309)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne
>>>>>>>>> r.
>>>>>>>>> ja
>>>>>>>>>>>> va:108)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java
>>>>>>>>> :8
>>>>>>>>> 8)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto
>>>>>>>>> rI
>>>>>>>>> mp
>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>>>>> av
>>>>>>>>> a:
>>>>>>>>>>>> 1145)
>>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>>>>> ja
>>>>>>>>> va
>>>>>>>>>>>> :615)
>>>>>>>>>>>> at java.lang.Thread.r
>>>>>>>>>>>>
>>>>>>>>>>>> Do I need more memory for my machines? Each already has
>>> 4GB. I
>>>>>>>> really
>>>>>>>>>>>> need to have this running. I¹m not sure which way is best
>>>>>> http or
>>>>>>>> hdfs
>>>>>>>>>>>> which one you suggest and how can i solve my problem for
>>> each
>>>>>>>> case.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance and sorry for bothering this much.
>>>>>>>>>>>> On 10 Aug 2014, at 00:20, Telles Nobrega
>>>>>>>> <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Chris, now I have the tar file in my RM machine, and
>>> the
>>>>>>>> yarn
>>>>>>>>> path
>>>>>>>>>>>>> points to it. I changed the core-site.xml to use
>>>>>> HttpFileSystem
>>>>>>>>> instead
>>>>>>>>>>>>> of HDFS now it is failing with
>>>>>>>>>>>>>
>>>>>>>>>>>>> Application application_1407640485281_0001 failed 2
>> times
>>>>>> due
>>>>>>>> to
>>>>>>>> AM
>>>>>>>>>>>>> Container for appattempt_1407640485281_0001_000002 exited
>>>>>> with
>>>>>>>>>>>>> exitCode:-1000 due to: java.lang.ClassNotFoundException:
>>>>>> Class
>>>>>>>>>>>>> org.apache.samza.util.hadoop.HttpFileSystem not found
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I can solve this just installing scala files
>> from
>>>>>> the
>>>>>>>> samza
>>>>>>>>>>>>> tutorial, can you confirm that?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 09 Aug 2014, at 08:34, Telles Nobrega
>>>>>>>> <[email protected]
>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think the problem is that I forgot to update the
>>>>>>>>> yarn.job.package.
>>>>>>>>>>>>>> I will try again to see if it works now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have one more question, how can I stop (command line)
>>> the
>>>>>>>> jobs
>>>>>>>>>>>>>> running in my topology, for the experiment that I will
>>> run,
>>>>>> I
>>>>>>>> need
>>>>>>>>> to
>>>>>>>>>>>>>> run the same job in 4 minutes intervals. So I need to
>> kill
>>>>>> it,
>>>>>>>> clean
>>>>>>>>>>>>>> the kafka topics and rerun.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 08 Aug 2014, at 12:41, Chris Riccomini
>>>>>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Telles,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Do I need to have the job folder on each machine in
>> my
>>>>>>>> cluster?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> No, you should not need to do this. There are two ways
>>> to
>>>>>>>> deploy
>>>>>>>>>> your
>>>>>>>>>>>>>>> tarball to the YARN grid. One is to put it in HDFS,
>> and
>>>>>> the
>>>>>>>> other
>>>>>>>>> is
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> put it on an HTTP server. The link to running a Samza
>>> job
>>>>>> in
>>>>>>>> a
>>>>>>>>>>>>>>> multi-node
>>>>>>>>>>>>>>> YARN cluster describes how to do both (either HTTP
>>> server
>>>>>> or
>>>>>>>>> HDFS).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In both cases, once the tarball is put in on the
>>> HTTP/HDFS
>>>>>>>>>> server(s),
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> must update yarn.package.path to point to it. From
>>> there,
>>>>>> the
>>>>>>>> YARN
>>>>>>>>>> NM
>>>>>>>>>>>>>>> should download it for you automatically when you
>> start
>>>>>> your
>>>>>>>> job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Can you send along a paste of your job config?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 8/8/14 8:04 AM, "Claudio Martins"
>>>>>>>> <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Telles, it looks to me that you forgot to update
>> the
>>>>>>>>>>>>>>>> "yarn.package.path"
>>>>>>>>>>>>>>>> attribute in your config file for the task.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Claudio Martins
>>>>>>>>>>>>>>>> Head of Engineering
>>>>>>>>>>>>>>>> MobileAware USA Inc. / www.mobileaware.com
>>>>>>>>>>>>>>>> office: +1 617 986 5060 / mobile: +1 617 480 5288
>>>>>>>>>>>>>>>> linkedin: www.linkedin.com/in/martinsclaudio
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 8, 2014 at 10:55 AM, Telles Nobrega
>>>>>>>>>>>>>>>> <[email protected]>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> this is my first time trying to run a job on a
>>> multinode
>>>>>>>>>>>>>>>>> environment. I
>>>>>>>>>>>>>>>>> have the cluster set up, I can see in the GUI that
>> all
>>>>>>>> nodes
>>>>>>>> are
>>>>>>>>>>>>>>>>> working.
>>>>>>>>>>>>>>>>> Do I need to have the job folder on each machine in
>> my
>>>>>>>> cluster?
>>>>>>>>>>>>>>>>> - The first time I tried running with the job on the
>>>>>>>> namenode
>>>>>>>>>>>>>>>>> machine
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> it failed saying:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Application application_1407509228798_0001 failed 2
>>>>>> times
>>>>>>>> due
>>>>>>>> to
>>>>>>>>>> AM
>>>>>>>>>>>>>>>>> Container for appattempt_1407509228798_0001_000002
>>>>>> exited
>>>>>>>> with
>>>>>>>>>>>>>>>>> exitCode:
>>>>>>>>>>>>>>>>> -1000 due to: File
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>>>>>>> file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job-
>>>>>>>>>>>>>> pa
>>>>>>>>>>>>>> ck
>>>>>>>>>>>>>>>>> age-
>>>>>>>>>>>>>>>>> 0.7.0-dist.tar.gz
>>>>>>>>>>>>>>>>> does not exist
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So I copied the folder to each machine in my cluster
>>> and
>>>>>>>> got
>>>>>>>>> this
>>>>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Application application_1407509228798_0002 failed 2
>>>>>> times
>>>>>>>> due
>>>>>>>> to
>>>>>>>>>> AM
>>>>>>>>>>>>>>>>> Container for appattempt_1407509228798_0002_000002
>>>>>> exited
>>>>>>>> with
>>>>>>>>>>>>>>>>> exitCode:
>>>>>>>>>>>>>>>>> -1000 due to: Resource
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>>>>>>>>>>> file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job-
>>>>>>>>>>>>>> pa
>>>>>>>>>>>>>> ck
>>>>>>>>>>>>>>>>> age-
>>>>>>>>>>>>>>>>> 0.7.0-dist.tar.gz
>>>>>>>>>>>>>>>>> changed on src filesystem (expected 1407509168000,
>> was
>>>>>>>>>> 1407509434000
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What am I missing?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> p.s.: I followed this
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>
>>> https://github.com/yahoo/samoa/wiki/Executing-SAMOA-with-Apache-Samz
>>>>>>>>>>>>>>>>> a>
>>>>>>>>>>>>>>>>> tutorial
>>>>>>>>>>>>>>>>> and this
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>> http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-
>>>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>>> -yarn.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> set up the cluster.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Help is much appreciated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>>>> Telles Mota Vidal Nobrega
>>>>>>>>>>>>>>>>> M.sc. Candidate at UFCG
>>>>>>>>>>>>>>>>> B.sc. in Computer Science at UFCG
>>>>>>>>>>>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ------------------------------------------
>>>>>>>>> Telles Mota Vidal Nobrega
>>>>>>>>> M.sc. Candidate at UFCG
>>>>>>>>> B.sc. in Computer Science at UFCG
>>>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ------------------------------------------
>>>>>>> Telles Mota Vidal Nobrega
>>>>>>> M.sc. Candidate at UFCG
>>>>>>> B.sc. in Computer Science at UFCG
>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ------------------------------------------
>>>>> Telles Mota Vidal Nobrega
>>>>> M.sc. Candidate at UFCG
>>>>> B.sc. in Computer Science at UFCG
>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> ------------------------------------------
>> Telles Mota Vidal Nobrega
>> M.sc. Candidate at UFCG
>> B.sc. in Computer Science at UFCG
>> Software Engineer at OpenStack Project - HP/LSD-UFCG
>>
