Example project configuration (maven or gradle) for projects depending on BeamSQL sdk extensions

[Yi Pan]

Hi, all,

Newbie here trying to figure out how to use published
beam-sdks-java-extensions-sql-2.9.0 in my own project.

I tried to create a gradle project to use BeamSQL sdk libraries. Here is
the build.gradle I have:
{code}
plugins {
    id 'java'
}

group 'com.mycompany.myproject'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

apply plugin: 'java'

sourceSets.main.java.srcDirs = [
        'src/main/java'
]

dependencies {
    compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
    compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
    compile 'com.google.code.findbugs:jsr305:3.0.2'
    runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

// Run basic SQL example
task runBasicExample(type: JavaExec) {
  description = "Run basic SQL example"
  main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
  classpath = sourceSets.main.runtimeClasspath
  args = ["--runner=DirectRunner"]
  println classpath.getAsPath()
  println args
}
{code}

The example BeamSQLExample is just copied from
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
.

I was able to compile the example in JDK8. However, when I tried to run it,
I hit the following exception:
{code}
Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider:
Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions
could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
at
com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
Caused by: java.lang.NoClassDefFoundError:
org/apache/commons/codec/DecoderException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 9 more
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.codec.DecoderException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
{code}

When I traced into the code, it turns out that the failure occurs when the
classloader tries to get the default constructor w/o any parameters for
BuiltinStringFunctions.class. I double checked my local gradle cache and
confirmed that the jar is there:
{code}
SJCMAC91THJHD4:beamsql-demo ypan$ ls -l
~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
-rw-r--r--  1 ypan  192360288  12761025 Jan 29 18:02
/Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
{code}

I also have tried to compile the sql sdk libraries in Beam's source repo
and just copying over the generated class files over to my runtime
classpath. Apparently, that did not work well since there are many shadowed
libraries now need explicit declaring dependencies. I tried to search for
an example of maven or gradle dependency declaration that users will use to
create their own application using the SQL SDKs but was not lucky enough to
find one.

Given the amount of debugging and searching effort I have put it there, I
think that it is reasonable to ask the question to the community. Can
someone shed some light on what went wrong in my build project?

Thanks!

-Yi