Hi,
Are those paths:
file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing)
file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar'
(missing)
accessible from the inside of your container?
bin/flink run --help
(…)
-C,--classpath <url> Adds a URL to each user code
classloader on all nodes in the
cluster. The paths must specify a
protocol (e.g. file://) and be
accessible on all nodes (e.g. by means
of a NFS share). You can use this
option multiple times for specifying
more than one URL. The protocol must
be supported by the {@link
java.net.URLClassLoader}.
Other nit, maybe the problem is with single slash after “file:”. You have
file:/home/...
While it might need to be
file://home/ <file://home/>...
Piotrek
> On 3 Aug 2018, at 13:03, Joshua Fan <[email protected]> wrote:
>
> Hi,
>
> I'd like to submit a job with dependency jars by flink run, but it failed.
>
> Here is the script,
>
> /usr/bin/hadoop/software/flink-1.4.2/bin/flink run \
> -m yarn-cluster -yn 1 -ys 8 -yjm 2148 -ytm 4096 -ynm jarsTest \
> -c StreamExample \
> -C file:/home/work/xxx/lib/commons-math3-3.5.jar \
> -C file:/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar \
> ...
> xxx-1.0.jar
>
> As described in
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html#usage
>
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html#usage>
> , "-C" means to provide the dependency jar.
>
> After I execute the command, the job succeed to submit, but can not run in
> flink cluster on yarn. Exceptions is like below:
>
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load
> user class: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
> ClassLoader info: URL ClassLoader:
> file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing)
> file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar'
> (missing)
> .......
> Class not resolvable through given classloader.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:232)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:95)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:231)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:745)
>
> It appears that the two dependency jar cannot be found in TaskManager, so I
> dig into the source code, from CliFrontend to PackagedProgram to
> ClusterClient to JobGraph. It seems like the dependency jars is put in
> classpath and userCodeClassLoader in PackagedProgram, but never upload to the
> BlobServer in JobGraph where the xxx-1.0.jar is uploaded.
>
> Am I missing something? In Flink 1.4.2, dependency jar is not supported?
>
> Hope someone can give me some hint.
>
> Appreciate it very mush.
>
>
> Yours Sincerely
>
> Joshua
>
>
>
>
