Hi Fabian, Thanks for testing things out, you have indeed found a situation where the base Beam libraries are conflicting with our transforms. When using the Hop engine we are able to keep the classpaths separate and each transform/plugin is isolated in its own classloader.
When running in Beam we make a combination of all our jars and they end up in the same classpath. We have already put quite some time in aligning libraries for core transforms so they work seamlessly in Beam. I'll work on upgrading the jersey libraries so these transforms can work in Beam too [1]. Cheers, Hans [1] https://issues.apache.org/jira/browse/HOP-4059 On Thu, 14 Jul 2022 at 13:03, Fabian Peters <[email protected]> wrote: > Hi all! > > I've started playing with Hop recently and do like quite a few things > about it. Thanks for all the effort! > > Unfortunately, there's one thing that's holding me back now: A pipeline > containing the rest transform works fine when launched from the GUI using > local or Beam-Direct. But it fails (see stack trace below) when launched > via ./hop-run.sh using Beam-Direct and it always fails for DataFlow – which > is what I want to use in the end. > > For Beam-Direct, I've verified the failure on both my macOS machine and on > a debian bullseye VM, with these java versions: > > openjdk version "11.0.15" 2022-04-19 > OpenJDK Runtime Environment Homebrew (build 11.0.15+0) > OpenJDK 64-Bit Server VM Homebrew (build 11.0.15+0, mixed mode) > > openjdk version "11.0.11" 2021-04-20 > OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9) > OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, > mixed mode) > > After some digging, my guess is that the rest transform implementation is > based on jersey 1.19.4 but beam comes with jersey 2.3. I don't understand > all the classloading magic going on, so for now I'm giving up. > > I'll create a ticket on Jira, but I still wanted to hear whether there's > something like an easy way out. Ideas very much welcome! > > cheers > > Fabian > > > 2022/07/13 13:39:16 - General - Executing this pipeline using the Beam > Pipeline Engine with run configuration 'Beam-Direct' > 2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: Error initializing > transform [GET PLT alerts] > 2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: > java.lang.NoClassDefFoundError: javax/ws/rs/core/Link$Builder > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.Class.getDeclaredConstructors0(Native Method) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.Class.getConstructor0(Class.java:3342) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.Class.newInstance(Class.java:556) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:117) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > org.apache.hop.pipeline.transforms.rest.Rest.init(Rest.java:571) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.Thread.run(Thread.java:829) > 2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by: > java.lang.ClassNotFoundException: Unable to load class > 'javax.ws.rs.core.Link$Builder' in this classloader or in the parent > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:96) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > 2022/07/13 13:39:18 - GET PLT alerts.0 - ... 12 more > 2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by: > java.lang.ClassNotFoundException: javax.ws.rs.core.Link$Builder > 2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.net > .URLClassLoader.findClass(URLClassLoader.java:471) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > org.apache.hop.core.plugins.HopURLClassLoader.loadClassFromThisLoader(HopURLClassLoader.java:66) > 2022/07/13 13:39:18 - GET PLT alerts.0 - at > org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:91) > 2022/07/13 13:39:18 - GET PLT alerts.0 - ... 13 more > 2022/07/13 13:39:18 - ERROR: Transform [GET PLT alerts.0] failed to > initialize! > 2022/07/13 13:39:18 - General - ERROR: Error starting the Beam pipeline > 2022/07/13 13:39:18 - General - ERROR: > org.apache.hop.core.exception.HopException: > 2022/07/13 13:39:18 - General - Error executing pipeline with runner Direct > 2022/07/13 13:39:18 - General - java.lang.RuntimeException: Error > executing TransformFn > 2022/07/13 13:39:18 - General - > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:254) > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:301) > 2022/07/13 13:39:18 - General - at > java.base/java.lang.Thread.run(Thread.java:829) > 2022/07/13 13:39:18 - General - Caused by: > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.RuntimeException: Error executing TransformFn > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218) > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:242) > 2022/07/13 13:39:18 - General - ... 2 more > 2022/07/13 13:39:18 - General - Caused by: java.lang.RuntimeException: > Error executing TransformFn > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:571) > 2022/07/13 13:39:18 - General - Caused by: > org.apache.hop.core.exception.HopException: > 2022/07/13 13:39:18 - General - We failed to initialize at least one > transform. Execution can not begin! > 2022/07/13 13:39:18 - General - > 2022/07/13 13:39:18 - General - > 2022/07/13 13:39:18 - General - at > org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1026) > 2022/07/13 13:39:18 - General - at > org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:206) > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:441) > 2022/07/13 13:39:18 - General - at > org.apache.hop.beam.core.transform.TransformTransform$TransformFn$DoFnInvoker.invokeProcessElement(Unknown > Source) > 2022/07/13 13:39:18 - General - at > org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:228) > 2022/07/13 13:39:18 - General - at > org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:184) > 2022/07/13 13:39:18 - General - at > org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165) > 2022/07/13 13:39:18 - General - at > org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129) > 2022/07/13 13:39:18 - General - at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > 2022/07/13 13:39:18 - General - at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > 2022/07/13 13:39:18 - General - at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > 2022/07/13 13:39:18 - General - at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > 2022/07/13 13:39:18 - General - at > java.base/java.lang.Thread.run(Thread.java:829)
