Hi team,
Could you please provide an update on the previous email? Additionally,
when I try to submit a pipeline from the Apache Hop Web UI to Flink
(running on the same server), I encounter a NullPointerException. I have
attached the detailed error logs from both Apache Hop and Flink below for
your reference.
Looking forward to your support.
On Mon, 10 Mar 2025 at 13:54, arjun s <[email protected]> wrote:
> Hi team,
>
> I’m trying to create a pipeline in Apache Hop (running on *Windows*) and
> submit it to a remote Flink cluster (running on *Linux*). However, I’m
> encountering a NULL POINTER EXCEPTION when attempting to execute it. After
> consulting the user mailing list, I learned that submitting to Flink
> remotely doesn’t seem to work. So, I am now trying to submit the pipeline
> on the same machine. Before proceeding, I have a few questions:
>
>
> - Does "same machine" mean that both Apache Hop and Flink should run
> on the same server(*linux*), or can they be on different servers(
> *linux*)?
> - If they need to be on the same machine, should Apache Hop run on the
> same server as the Flink master, or is it okay to submit from any Flink
> node?
> - I am also setting up Apache Hop Web using Docker on a Linux server,
> and I expect to submit pipelines to Flink from there. Is this the correct
> approach?
> - Is there any difference between using Apache Hop’s desktop
> application and Apache Hop Web?
> - Finally, can Apache Hop handle stopping and canceling jobs, or is it
> only capable of submitting jobs to Flink?
>
> Thanks in Advance
>
org.apache.flink.runtime.client.JobInitializationException: Could not start the
JobMaster.
at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException:
java.lang.NullPointerException
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.flink.api.common.ExecutionConfig.getNumberOfExecutionRetries(ExecutionConfig.java:486)
at
org.apache.flink.api.common.ExecutionConfig.getRestartStrategy(ExecutionConfig.java:459)
at
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:102)
at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
... 3 more
2025-03-10 12:56:16,525 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
701529550595ddd0bf5976792abdb1be has been registered for cleanup in the
JobResultStore after reaching a terminal state.2025/03/10 13:54:07 - MyfirstPipeline - ERROR: Error starting the Beam pipeline
2025/03/10 13:54:07 - MyfirstPipeline - ERROR:
org.apache.hop.core.exception.HopException:
2025/03/10 13:54:07 - MyfirstPipeline - Error executing pipeline with runner
Flink
2025/03/10 13:54:07 - MyfirstPipeline - Pipeline execution failed
2025/03/10 13:54:07 - MyfirstPipeline -
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:322)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.hop.beam.engines.BeamPipelineEngine.startThreads(BeamPipelineEngine.java:348)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.lambda$startThreads$17(HopGuiPipelineGraph.java:4587)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.lang.Thread.run(Thread.java:833)
2025/03/10 13:54:07 - MyfirstPipeline - Caused by: java.lang.RuntimeException:
Pipeline execution failed
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:312)
2025/03/10 13:54:07 - MyfirstPipeline - ... 3 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
while waiting for job to be initialized
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1061)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:958)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.executePipeline(FlinkPipelineExecutionEnvironment.java:145)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:104)
2025/03/10 13:54:07 - MyfirstPipeline - ... 4 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
while waiting for job to be initialized
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1056)
2025/03/10 13:54:07 - MyfirstPipeline - ... 7 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by: java.lang.RuntimeException:
Error while waiting for job to be initialized
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:144)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.lambda$execute$2(AbstractSessionClusterExecutor.java:87)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
java.util.concurrent.ExecutionException:
org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete
the operation. Number of retries has been exhausted.
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.lambda$null$1(AbstractSessionClusterExecutor.java:89)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:131)
2025/03/10 13:54:07 - MyfirstPipeline - ... 9 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete
the operation. Number of retries has been exhausted.
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:293)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.lang.Thread.run(Thread.java:833)
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
java.util.concurrent.CompletionException:
org.apache.flink.runtime.rest.util.RestClientException: Response was neither of
the expected type([simple type, class
org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody]) nor
an error.
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152)
2025/03/10 13:54:07 - MyfirstPipeline - ... 4 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
org.apache.flink.runtime.rest.util.RestClientException: Response was neither of
the expected type([simple type, class
org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody]) nor
an error.
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:550)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:514)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
2025/03/10 13:54:07 - MyfirstPipeline - ... 4 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by: java.io.IOException: Failed
to deserialize org.apache.flink.util.SerializedThrowable
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:53)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:106)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:50)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:523)
2025/03/10 13:54:07 - MyfirstPipeline - ... 6 more
2025/03/10 13:54:07 - MyfirstPipeline - Caused by:
java.lang.ClassNotFoundException: org.apache.flink.util.SerializedThrowable
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.lang.Class.forName0(Native Method)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.lang.Class.forName(Class.java:467)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2045)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1909)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2235)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)
2025/03/10 13:54:07 - MyfirstPipeline - at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
2025/03/10 13:54:07 - MyfirstPipeline - at
org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:49)
2025/03/10 13:54:07 - MyfirstPipeline - ... 18 more
