Let me ask a related question: We are building our own base Flink docker image. We will be deploying both JVM and python apps via flink-kubernetes-operator.
Is there any reason not to install Flink in this image via `pip install apache-flink` and use it for JVM apps? -Andrew Otto Wikimedia Foundation On Tue, Jan 24, 2023 at 4:26 PM Andrew Otto <o...@wikimedia.org> wrote: > Hello, > > I'm having quite a bit of trouble running pyflink from the default flink > distribution tarballs. I'd expect the python examples to work as long as > python is installed, and we've got the distribution. Some python > dependencies are not included in the flink distribution tarballs: > cloudpickle, py4j and pyflink are in opt/python. Others are not, e.g. > protobuf. > > Now that I'm looking, I see that the pyflink installation instructions > <https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/installation/> > are > to install via pip. > > I'm doing this in Docker for use with the flink-kubernetes-operator. In > the Using Flink Python on Docker > <https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker> > instructions, > there is a pip3 install apache-flink step. I find this strange, since I'd > expect the 'FROM flink:1.15.2' part to be sufficient. > > By pip installing apache-flink, this docker image will have the flink > distro installed at /opt/flink and FLINK_HOME set to /opt/flink > <https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/Dockerfile>. > BUT ALSO flink lib jars will be installed at e.g. > /usr/local/lib/python3.7/dist-packages/pyflink/lib! > So, by following those instructions, flink is effectively installed twice > into the docker image. > > Am I correct or am I missing something? > > Is using pyflink from the flink distribution tarball (without pip) not a > supported way to use pyflink? > > Thanks! > -Andrew Otto > Wikimedia Foundation > >
