Zhou Parker created FLINK-21289: ----------------------------------- Summary: Application mode on kubernetes deployment support run PackagedProgram.main with pipeline.classpaths Key: FLINK-21289 URL: https://issues.apache.org/jira/browse/FLINK-21289 Project: Flink Issue Type: Improvement Components: Client / Job Submission, Deployment / Kubernetes Affects Versions: 1.12.1, 1.11.2 Environment: flink: 1.11
kubernetes: 1.15 Reporter: Zhou Parker Attachments: 0001-IMP.patch 我尝试将flink作业以application mode方式提交到kubernetes上运行。但程序的依赖包并不完全存在于local:///opt/flink/usrlib/xxxx.jar中。导致找不到类。 在yarn上可以工作,是因为我们用 {color:#FF0000}-C [http://xxxx|http://xxxx/] {color}的方式,让依赖可以被URLClassloader加载。 但我发现,当实验提交到kubernetes时,-C只会在 configmap/flink-conf.yaml 中生成一个pipeline.classpaths 配置条目。我们的main函数可以执行,但是在加载外部依赖类的时候提示找不到类。 通过阅读源码,*我发现运行用户代码的类加载器实际并把 pipeline.classpaths 中的条目加入候选URL*,这导致了无法加载类的情况。从源码中,我也发现,可以通过将依赖包放在usrlib目录下(默认的userClassPath)可以解决问题。但我们的依赖可能是动态的,不合适一次性打到镜像里面。 我提议可以改进这个过程,将pipeline.classpaths也加入到对应的类加载器。这个改动很小,我自己经过测试,可以完美解决问题。 I'm trying to submit flink job to kubernetes cluster with application mode, but throw ClassNotFoundException when dependency class is not shipped in kind of local:///opt/flink/usrlib/xxxx.jar. This works on yarn, since we use {color:#FF0000}-C http://xxxx{color} command line style that let dependency class can be load by URLClassloader. But i figure out that not works on kubernetes. When submit to kubernetes cluster, -C is only shipped as item "pipeline.classpaths" in configmap/flink-conf.yaml。 After read the source code, *i find out that the Classloader launching the "main" entry of user code without consider add pipeline.classpaths into candidates URLs*. from source code, i also learn that we can ship the dependency jar in the usrlib dir to solve the problem. But failed for me, we are not _preferred_ to ship dependencies in image at compile time, since they are known dynamically in runtime I proposed improving the process, let the Classloader consider usrlib as well as pipeline.classpaths, this is a quite little change. I test the solution and it works quite well -- This message was sent by Atlassian Jira (v8.3.4#803005)