[cloudera@quickstart bin]$ sudo su yarn bash-4.1$ hadoop fs -chmod 777 -chmod: Not enough arguments: expected 2 but got 1 Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH... bash-4.1$
you understand? 2015-06-04 17:04 GMT+02:00 Robert Metzger <rmetz...@apache.org>: > It looks like the user "yarn" which is running Flink doesn't have > permission to access the files. > > Can you do "sudo su yarn" to become the "yarn" user. Then, you can do > "hadoop fs -chmod 777" to make the files accessible for everyone. > > > On Thu, Jun 4, 2015 at 4:59 PM, Pa Rö <paul.roewer1...@googlemail.com> > wrote: > >> okay, it's work, i get a exception: >> >> [cloudera@quickstart Desktop]$ cd flink-0.9-SNAPSHOT/bin/ >> [cloudera@quickstart bin]$ flink run /home/cloudera/Desktop/ma-flink.jar >> bash: flink: command not found >> [cloudera@quickstart bin]$ ./flink run >> /home/cloudera/Desktop/ma-flink.jar >> log4j:WARN No appenders could be found for logger >> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >> log4j:WARN Please initialize the log4j system properly. >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for >> more info. >> Found YARN properties file >> /home/cloudera/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-properties >> Using JobManager address from YARN properties quickstart.cloudera/ >> 127.0.0.1:53874 >> java.io.IOException: Mkdirs failed to create /user/cloudera/outputs >> at >> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438) >> at >> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424) >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905) >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886) >> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783) >> at >> mgm.tp.bigdata.ma_commons.commons.Seeding.randomSeeding(Seeding.java:21) >> at >> mgm.tp.bigdata.ma_flink.FlinkMain.getCentroidDataSet(FlinkMain.java:178) >> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:47) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >> at org.apache.flink.client.program.Client.run(Client.java:315) >> at >> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) >> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >> at >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880) >> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) >> org.apache.flink.client.program.ProgramInvocationException: The program >> execution failed: Failed to submit job 934743a5c49c6d5e31c9e8201452e36d >> (KMeans Flink) >> at org.apache.flink.client.program.Client.run(Client.java:412) >> at org.apache.flink.client.program.Client.run(Client.java:355) >> at org.apache.flink.client.program.Client.run(Client.java:348) >> at >> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63) >> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:70) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >> at org.apache.flink.client.program.Client.run(Client.java:315) >> at >> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) >> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >> at >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880) >> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) >> Caused by: org.apache.flink.runtime.client.JobExecutionException: Failed >> to submit job 934743a5c49c6d5e31c9e8201452e36d (KMeans Flink) >> at org.apache.flink.runtime.jobmanager.JobManager.org >> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:595) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:192) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) >> at >> org.apache.flink.yarn.ApplicationMasterActor$$anonfun$receiveYarnMessages$1.applyOrElse(ApplicationMasterActor.scala:99) >> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29) >> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:29) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> at >> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:94) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> Caused by: org.apache.flink.runtime.JobException: Creating the input >> splits caused an error: File /user/cloudera/outputs/seed-1 does not exist >> or the user running Flink ('yarn') has insufficient permissions to access >> it. >> at >> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:162) >> at >> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:471) >> at org.apache.flink.runtime.jobmanager.JobManager.org >> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:535) >> ... 21 more >> Caused by: java.io.FileNotFoundException: File >> /user/cloudera/outputs/seed-1 does not exist or the user running Flink >> ('yarn') has insufficient permissions to access it. >> at >> org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106) >> at >> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:390) >> at >> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:51) >> at >> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:146) >> ... 23 more >> >> >> how i must set the files in the hdfs? >> quickstart.cloudera:50075/home/cloudera/output? >> >> 2015-06-04 16:51 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >> >>> Once you've started the YARN session, you can submit a Flink job with >>> "./bin/flink run <pathToYourJar>". >>> >>> The jar file of your job doesn't need to be in HDFS. It has to be in the >>> local file system and flink will send it to all machines. >>> >>> On Thu, Jun 4, 2015 at 4:48 PM, Pa Rö <paul.roewer1...@googlemail.com> >>> wrote: >>> >>>> okay, now it run on my hadoop. >>>> how i can start my flink job? and where must the jar file save, at hdfs >>>> or as local file? >>>> >>>> 2015-06-04 16:31 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>> >>>>> Yes, you have to run these commands in the command line of the >>>>> Cloudera VM. >>>>> >>>>> On Thu, Jun 4, 2015 at 4:28 PM, Pa Rö <paul.roewer1...@googlemail.com> >>>>> wrote: >>>>> >>>>>> you mean run this command on terminal/shell and not define a hue job? >>>>>> >>>>>> 2015-06-04 16:25 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>> >>>>>>> It should be certainly possible to run Flink on a cloudera live VM >>>>>>> >>>>>>> I think these are the commands you need to execute: >>>>>>> >>>>>>> wget >>>>>>> http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>> cd flink-0.9-SNAPSHOT/ >>>>>>> *export HADOOP_CONF_DIR=/usr/lib/hadoop/etc/hadoop/* >>>>>>> ./bin/yarn-session.sh -n 1 -jm 1024 -tm 1024 >>>>>>> >>>>>>> If that is not working for you, please post the exact error message >>>>>>> you are getting and I can help you to get it to run. >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 4, 2015 at 4:18 PM, Pa Rö < >>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>> >>>>>>>> hi robert, >>>>>>>> >>>>>>>> i think the problem is the hue api, >>>>>>>> i had the same problem with spark submit script, >>>>>>>> but on the new hue release, they have a spark submit api. >>>>>>>> >>>>>>>> i asked the group for the same problem with spark, no reply. >>>>>>>> >>>>>>>> i want test my app on local cluster, before i run it on the big >>>>>>>> cluster, >>>>>>>> for that i use cloudera live. maybe it give an other way to test >>>>>>>> flink on a local cluster vm? >>>>>>>> >>>>>>>> 2015-06-04 16:12 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>>>> >>>>>>>>> Hi Paul, >>>>>>>>> >>>>>>>>> why did running Flink from the regular scripts not work for you? >>>>>>>>> >>>>>>>>> I'm not an expert on Hue, I would recommend asking in the Hue user >>>>>>>>> forum / mailing list: >>>>>>>>> https://groups.google.com/a/cloudera.org/forum/#!forum/hue-user. >>>>>>>>> >>>>>>>>> On Thu, Jun 4, 2015 at 4:09 PM, Pa Rö < >>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> now i want run my app on cloudera live vm single node, >>>>>>>>>> how i can define my flink job with hue? >>>>>>>>>> i try to run the flink script in the hdfs, it's not work. >>>>>>>>>> >>>>>>>>>> best regards, >>>>>>>>>> paul >>>>>>>>>> >>>>>>>>>> 2015-06-02 14:50 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>>>>>> >>>>>>>>>>> I would recommend using HDFS. >>>>>>>>>>> For that, you need to specify the paths like this: >>>>>>>>>>> hdfs:///path/to/data. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 2, 2015 at 2:48 PM, Pa Rö < >>>>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> nice, >>>>>>>>>>>> >>>>>>>>>>>> which file system i must use for the cluster? java.io or >>>>>>>>>>>> hadoop.fs or flink? >>>>>>>>>>>> >>>>>>>>>>>> 2015-06-02 14:29 GMT+02:00 Robert Metzger <rmetz...@apache.org> >>>>>>>>>>>> : >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> you can start Flink on YARN on the Cloudera distribution. >>>>>>>>>>>>> >>>>>>>>>>>>> See here for more: >>>>>>>>>>>>> http://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html >>>>>>>>>>>>> >>>>>>>>>>>>> These are the commands you need to execute >>>>>>>>>>>>> >>>>>>>>>>>>> wget >>>>>>>>>>>>> http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgzcd flink-0.9-SNAPSHOT/ >>>>>>>>>>>>> ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jun 2, 2015 at 2:03 PM, Pa Rö < >>>>>>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> hi community, >>>>>>>>>>>>>> >>>>>>>>>>>>>> i want test my flink k-means on a hadoop cluster. i use the >>>>>>>>>>>>>> cloudera live distribution. how i can run flink on this cluster? >>>>>>>>>>>>>> maybe only >>>>>>>>>>>>>> the java dependencies are engouth? >>>>>>>>>>>>>> >>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>> paul >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
