[ https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648493#comment-15648493 ]
Kenneth Knowles commented on BEAM-944: -------------------------------------- Perhaps you can split it into two options classes, with the troublesome one not registered? > Spark runner causes an exception when creating pipeline options > --------------------------------------------------------------- > > Key: BEAM-944 > URL: https://issues.apache.org/jira/browse/BEAM-944 > Project: Beam > Issue Type: Bug > Components: runner-spark > Affects Versions: 0.3.0-incubating > Reporter: Davor Bonaci > Assignee: Amit Sela > Fix For: 0.4.0-incubating > > > Running any example with *any* runner throws an exception as soon as it tries > to construct PipelineOptions as long as SparkRunner is on the class path: > {code} > mvn compile exec:java > -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard > -Dexec.args="--runner=DirectRunner" > [INFO] Scanning for projects... > [INFO] > > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Tutorial 0.0.1-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > Tutorial --- > [WARNING] Using platform encoding (UTF-8 actually) to copy filtered > resources, i.e. build is platform dependent! > [INFO] skip non existing resourceDirectory > /Users/fjp/dev/mobile/src/main/resources > [INFO] > [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial --- > [INFO] No sources to compile > [INFO] > [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial --- > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > [WARNING] > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ExceptionInInitializerError > at > org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179) > ... 6 more > Caused by: java.lang.TypeNotPresentException: Type > org.apache.spark.streaming.api.java.JavaStreamingListener not present > at > sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117) > at > sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125) > at > sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) > at > sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68) > at > sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138) > at > sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) > at > sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94) > at > java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284) > at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282) > at > java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387) > at > java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653) > at > java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326) > at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:138) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587) > at > org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:536) > ... 7 more > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.streaming.api.java.JavaStreamingListener > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114) > ... 25 more > {code} > It is introduced by https://github.com/apache/incubator-beam/pull/1072 that > was added in mid-October, so it reproduces starting with 0.3.0-incubating, > but not earlier. > A possible workaround for users is to add this explicit dependency, which is > marked as provided in the Spark runner: > {code} > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-streaming_2.10</artifactId> > <version>${spark.version}</version> > </dependency> > {code} > On a quick glance, JavaStreamingListener is not a real pipeline option. > Perhaps it can be just removed from pipeline options, and passed around > differently. > Separately, we should perhaps re-think how this works with respect to > dependencies in general. > Marking blocking for 0.4.0-incubating since it affects Spark quickstart. > [~amitsela], what do you think? -- This message was sent by Atlassian JIRA (v6.3.4#6332)