Hi Everyone I was able to run the example with Spark runner on files present in /tmp directory . What if I want to use the files from HDFS?
On Wed, May 25, 2016 at 2:04 PM, Punit Naik <[email protected]> wrote: > Hi Ismael > > I am using maven 3.3.9 > > On Wed, May 25, 2016 at 1:31 PM, Ismaël Mejía <[email protected]> wrote: > >> Hi Punit, >> >> Can you confirm the version of Maven are you using ? >> >> I ask this because I had some issues building the project with the native >> version of maven that came with debian/ubuntu, but since I moved to the >> 'real' one I have no problmems. And I just did git pull from master and >> everything seems to work ok. >> >> Ismaël >> >> >> On Wed, May 25, 2016 at 10:00 AM, Amit Sela <[email protected]> wrote: >> >>> I'll take a look first chance I get and get back to you. >>> >>> On Wed, May 25, 2016, 10:57 Punit Naik <[email protected]> wrote: >>> >>>> Hi Amit >>>> >>>> Thanks, it ran but got stuck at: >>>> >>>> [INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ spark --- >>>> log4j:WARN No appenders could be found for logger >>>> (org.apache.beam.sdk.options.PipelineOptionsFactory). >>>> log4j:WARN Please initialize the log4j system properly. >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>> for more info. >>>> >>>> >>>> On Wed, May 25, 2016 at 1:17 PM, Amit Sela <[email protected]> >>>> wrote: >>>> >>>>> I'm not near a computer but are you running from the parent project >>>>> dir or the runner ? You need to run from the runner dir. >>>>> >>>>> On Wed, May 25, 2016, 10:44 Punit Naik <[email protected]> wrote: >>>>> >>>>>> Hi Amit >>>>>> >>>>>> Just tried that and still got an error: >>>>>> >>>>>> [ERROR] Failed to execute goal >>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (default-cli) on project >>>>>> parent: The parameters 'executable' for goal >>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec are missing or invalid -> >>>>>> [Help 1] >>>>>> >>>>>> On Wed, May 25, 2016 at 1:01 PM, Amit Sela <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Would you try running with mvn exec:exec ? >>>>>>> See here: >>>>>>> https://github.com/apache/incubator-beam/blob/master/runners/spark/README.md >>>>>>> >>>>>>> On Wed, May 25, 2016, 10:14 Punit Naik <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Everyone >>>>>>>> >>>>>>>> I compiled beam project with the command "mvn clean install >>>>>>>> -DskipTests" and then ran the WordCount example with the following >>>>>>>> command: >>>>>>>> >>>>>>>> mvn compile exec:java -pl examples/java >>>>>>>> -Dexec.mainClass=org.apache.beam.examples.WordCount >>>>>>>> -Dexec.args="--runner=SparkPipelineRunner --sparkMaster=local >>>>>>>> --inputFile=/in --output=/beam-out" >>>>>>>> >>>>>>>> But it gave me an error saying: >>>>>>>> >>>>>>>> [ERROR] Failed to execute goal >>>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project >>>>>>>> java-examples-all: An exception occured while executing the Java class. >>>>>>>> null: InvocationTargetException: Unknown 'runner' specified >>>>>>>> 'SparkPipelineRunner', supported pipeline runners >>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner, >>>>>>>> DirectPipelineRunner] -> [Help 1] >>>>>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to >>>>>>>> execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java >>>>>>>> (default-cli) >>>>>>>> on project java-examples-all: An exception occured while executing the >>>>>>>> Java >>>>>>>> class. null >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) >>>>>>>> at >>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) >>>>>>>> at >>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) >>>>>>>> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) >>>>>>>> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) >>>>>>>> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) >>>>>>>> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>> at >>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) >>>>>>>> at >>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) >>>>>>>> at >>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) >>>>>>>> at >>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) >>>>>>>> Caused by: org.apache.maven.plugin.MojoExecutionException: An >>>>>>>> exception occured while executing the Java class. null >>>>>>>> at >>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:345) >>>>>>>> at >>>>>>>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) >>>>>>>> at >>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) >>>>>>>> ... 20 more >>>>>>>> Caused by: java.lang.reflect.InvocationTargetException >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>> at >>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' >>>>>>>> specified 'SparkPipelineRunner', supported pipeline runners >>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner, >>>>>>>> DirectPipelineRunner] >>>>>>>> at >>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1434) >>>>>>>> at >>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:107) >>>>>>>> at >>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:292) >>>>>>>> at org.apache.beam.examples.WordCount.main(WordCount.java:192) >>>>>>>> ... 6 more >>>>>>>> Caused by: java.lang.ClassNotFoundException: SparkPipelineRunner >>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>> at java.lang.Class.forName(Class.java:264) >>>>>>>> at >>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1421) >>>>>>>> ... 9 more >>>>>>>> >>>>>>>> What is the problem? >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thank You >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Punit Naik >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thank You >>>>>> >>>>>> Regards >>>>>> >>>>>> Punit Naik >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Thank You >>>> >>>> Regards >>>> >>>> Punit Naik >>>> >>> >> > > > -- > Thank You > > Regards > > Punit Naik > -- Thank You Regards Punit Naik
