Hi all!
I've started playing with Hop recently and do like quite a few things about it.
Thanks for all the effort!
Unfortunately, there's one thing that's holding me back now: A pipeline
containing the rest transform works fine when launched from the GUI using local
or Beam-Direct. But it fails (see stack trace below) when launched via
./hop-run.sh using Beam-Direct and it always fails for DataFlow – which is what
I want to use in the end.
For Beam-Direct, I've verified the failure on both my macOS machine and on a
debian bullseye VM, with these java versions:
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment Homebrew (build 11.0.15+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.15+0, mixed mode)
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed
mode)
After some digging, my guess is that the rest transform implementation is based
on jersey 1.19.4 but beam comes with jersey 2.3. I don't understand all the
classloading magic going on, so for now I'm giving up.
I'll create a ticket on Jira, but I still wanted to hear whether there's
something like an easy way out. Ideas very much welcome!
cheers
Fabian
2022/07/13 13:39:16 - General - Executing this pipeline using the Beam Pipeline
Engine with run configuration 'Beam-Direct'
2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: Error initializing transform
[GET PLT alerts]
2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: java.lang.NoClassDefFoundError:
javax/ws/rs/core/Link$Builder
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.Class.getConstructor0(Class.java:3342)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.Class.newInstance(Class.java:556)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:117)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
org.apache.hop.pipeline.transforms.rest.Rest.init(Rest.java:571)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.Thread.run(Thread.java:829)
2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by:
java.lang.ClassNotFoundException: Unable to load class
'javax.ws.rs.core.Link$Builder' in this classloader or in the parent
2022/07/13 13:39:18 - GET PLT alerts.0 - at
org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:96)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
2022/07/13 13:39:18 - GET PLT alerts.0 - ... 12 more
2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by:
java.lang.ClassNotFoundException: javax.ws.rs.core.Link$Builder
2022/07/13 13:39:18 - GET PLT alerts.0 - at
java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
org.apache.hop.core.plugins.HopURLClassLoader.loadClassFromThisLoader(HopURLClassLoader.java:66)
2022/07/13 13:39:18 - GET PLT alerts.0 - at
org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:91)
2022/07/13 13:39:18 - GET PLT alerts.0 - ... 13 more
2022/07/13 13:39:18 - ERROR: Transform [GET PLT alerts.0] failed to initialize!
2022/07/13 13:39:18 - General - ERROR: Error starting the Beam pipeline
2022/07/13 13:39:18 - General - ERROR:
org.apache.hop.core.exception.HopException:
2022/07/13 13:39:18 - General - Error executing pipeline with runner Direct
2022/07/13 13:39:18 - General - java.lang.RuntimeException: Error executing
TransformFn
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:254)
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:301)
2022/07/13 13:39:18 - General - at
java.base/java.lang.Thread.run(Thread.java:829)
2022/07/13 13:39:18 - General - Caused by:
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.RuntimeException: Error executing TransformFn
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:242)
2022/07/13 13:39:18 - General - ... 2 more
2022/07/13 13:39:18 - General - Caused by: java.lang.RuntimeException: Error
executing TransformFn
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:571)
2022/07/13 13:39:18 - General - Caused by:
org.apache.hop.core.exception.HopException:
2022/07/13 13:39:18 - General - We failed to initialize at least one transform.
Execution can not begin!
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General - at
org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1026)
2022/07/13 13:39:18 - General - at
org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:206)
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:441)
2022/07/13 13:39:18 - General - at
org.apache.hop.beam.core.transform.TransformTransform$TransformFn$DoFnInvoker.invokeProcessElement(Unknown
Source)
2022/07/13 13:39:18 - General - at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:228)
2022/07/13 13:39:18 - General - at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:184)
2022/07/13 13:39:18 - General - at
org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
2022/07/13 13:39:18 - General - at
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
2022/07/13 13:39:18 - General - at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2022/07/13 13:39:18 - General - at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2022/07/13 13:39:18 - General - at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2022/07/13 13:39:18 - General - at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2022/07/13 13:39:18 - General - at
java.base/java.lang.Thread.run(Thread.java:829)
