[ https://issues.apache.org/jira/browse/BEAM-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757869#comment-16757869 ]
Kenneth Knowles commented on BEAM-6558: --------------------------------------- Yup, clearly BuiltinStringFunctions depends on commons-codec but it is not in our dependencies. This should have been a compile error. > Beam SQL transitive dependencies appear incomplete / broken > ----------------------------------------------------------- > > Key: BEAM-6558 > URL: https://issues.apache.org/jira/browse/BEAM-6558 > Project: Beam > Issue Type: Bug > Components: dsl-sql > Affects Versions: 2.9.0 > Reporter: Kenneth Knowles > Assignee: Kenneth Knowles > Priority: Blocker > Fix For: 2.10.0 > > > Reporting on the dev list: > ---- > Hi, all, > Newbie here trying to figure out how to use published > beam-sdks-java-extensions-sql-2.9.0 in my own project. > I tried to create a gradle project to use BeamSQL sdk libraries. Here is the > build.gradle I have: > {code:java} > plugins { > id 'java' > } > group 'com.mycompany.myproject' > version '1.0-SNAPSHOT' > sourceCompatibility = 1.8 > repositories { > mavenCentral() > } > apply plugin: 'java' > sourceSets.main.java.srcDirs = [ > 'src/main/java' > ] > dependencies { > compile 'org.apache.beam:beam-sdks-java-core:2.9.0' > compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0' > compile 'com.google.code.findbugs:jsr305:3.0.2' > runtime 'org.apache.beam:beam-runners-direct-java:2.9.0' > testCompile group: 'junit', name: 'junit', version: '4.12' > } > // Run basic SQL example > task runBasicExample(type: JavaExec) { > description = "Run basic SQL example" > main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample" > classpath = sourceSets.main.runtimeClasspath > args = ["--runner=DirectRunner"] > println classpath.getAsPath() > println args > } > {code} > The example BeamSQLExample is just copied from > [https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java]. > I was able to compile the example in JDK8. However, when I tried to run it, I > hit the following exception: > {code:java} > Exception in thread "main" java.util.ServiceConfigurationError: > org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider: > Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions > could not be instantiated > at java.util.ServiceLoader.fail(ServiceLoader.java:232) > at java.util.ServiceLoader.access$100(ServiceLoader.java:185) > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471) > at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357) > at > com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72) > Caused by: java.lang.NoClassDefFoundError: > org/apache/commons/codec/DecoderException > at java.lang.Class.getDeclaredConstructors0(Native Method) > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) > at java.lang.Class.getConstructor0(Class.java:3075) > at java.lang.Class.newInstance(Class.java:412) > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) > ... 9 more > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.codec.DecoderException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 14 more > {code} > When I traced into the code, it turns out that the failure occurs when the > classloader tries to get the default constructor w/o any parameters for > BuiltinStringFunctions.class. I double checked my local gradle cache and > confirmed that the jar is there: > {code:java} > SJCMAC91THJHD4:beamsql-demo ypan$ ls -l > ~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar > > -rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02 > /Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar > {code} > I also have tried to compile the sql sdk libraries in Beam's source repo and > just copying over the generated class files over to my runtime classpath. > Apparently, that did not work well since there are many shadowed libraries > now need explicit declaring dependencies. I tried to search for an example of > maven or gradle dependency declaration that users will use to create their > own application using the SQL SDKs but was not lucky enough to find one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)