Re: Application mode - Custom Flink docker image with Python user code
Hi Sumeet, It still has not provided special support to handle the dependencies for the Application mode in PyFlink. This means that the dependencies could be handled the same as the other deployment modes. However, it is indeed correct that the dependencies could be handled differently in application mode. For Python user code, I guess you could add them to the environment variable PYTHONPATH? Then it's not necessary to specify them again in the job. Regards, Dian On Tue, Oct 26, 2021 at 3:23 PM Sumeet Malhotra wrote: > Hi, > > I'm currently submitting my Python user code from my local machine to a > Flink cluster running in Session mode on Kubernetes. For this, I have a > custom Flink image with Python as per this reference [1]. > > Now, I'd like to move to using the Application mode with Native > Kubernetes, where the user code is embedded within the container image. For > Java, the process is described here [2]. This basically requires the user > application jar to be copied over to $FLINK_HOME/usrlib directory. I > couldn't find any documentation on where should Python user code live? Can > it be packaged at any location and invoked? > > Thanks, > Sumeet > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode >
Re: Application mode - Custom Flink docker image with Python user code
Hi Sumeet，
Actually, running pyflink jobs in application mode on kubernetes has been
supported since release 1.13.
To build a docker image with PyFlink installed, please refer to Enabling
Python[1]. In order to run the python code in application mode, you also
need to COPY the code files into the container image when building it. Then
you can submit the job by executing the following command:
$ ./bin/flink run-application \
--target kubernetes-application
-Dkubernetes.cluster-id={your-custom-cluster-id} \
-Dkubernetes.container.image = {your-custom-image-name} \
-pyfs {the python file path with the container image} \
-pym {the-entry-module-name} \
[the-entry-main-args](optional)
Hope this helps!
Best,
Shuiqiang
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python
Sumeet Malhotra 于2021年10月26日周二 下午3:23写道：
> Hi,
>
> I'm currently submitting my Python user code from my local machine to a
> Flink cluster running in Session mode on Kubernetes. For this, I have a
> custom Flink image with Python as per this reference [1].
>
> Now, I'd like to move to using the Application mode with Native
> Kubernetes, where the user code is embedded within the container image. For
> Java, the process is described here [2]. This basically requires the user
> application jar to be copied over to $FLINK_HOME/usrlib directory. I
> couldn't find any documentation on where should Python user code live? Can
> it be packaged at any location and invoked?
>
> Thanks,
> Sumeet
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode
>
