Thanks for the detailed info Xingbo, I am pretty new to the PyFlink module
so was mainly trying to experiment a little :)

I think supporting it in Flink 1.16 is more than enough, it would probably
be an overkill to try to backport this. The M1 issue only really occurs in
local development/testing.

Gyula

On Fri, Jul 8, 2022 at 5:51 AM Xingbo Huang <hxbks...@gmail.com> wrote:

> Hi Gyula,
>
> According to the log, we can see that you downloaded the source package of
> pemja, not the wheel package of pemja[1]. I guess you are using the m1
> machine. If you install pemja from the source package, you need to have
> JDK, gcc tools and CPython with Numpy in the environment. I believe this
> can be solved after you prepared those tools, but other dependencies of
> pyflink 1.15 do not support m1, which makes PyFlink 1.15 unable to install
> and use in m1.
>
> We have supported m1 in release 1.16[2]. If a large number of m1 users
> have big demand for PyFlink 1.15, I think we need to consider whether it is
> necessary to backport this support to 1.15, which will break our
> compatibility issues between minor versions.
> Best,
> Xingbo
>
> [1] https://pypi.org/project/pemja/0.1.4/
> [2] https://issues.apache.org/jira/browse/FLINK-25188
>
> Gyula Fóra <gyula.f...@gmail.com> 于2022年7月6日周三 13:56写道:
>
>> Here it is, copied from the docs essentially:
>>
>> FROM flink:1.15.0
>>
>>
>> # install python3: it has updated Python to 3.9 in Debian 11 and so
>> install Python 3.7 from source
>> # it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink
>> officially.
>>
>> RUN apt-get update -y && \
>> apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev
>> libffi-dev git && \
>> wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz && \
>> tar -xvf Python-3.7.9.tgz && \
>> cd Python-3.7.9 && \
>> ./configure --without-tests --enable-shared && \
>> make -j6 && \
>> make install && \
>> ldconfig /usr/local/lib && \
>> cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \
>> ln -s /usr/local/bin/python3 /usr/local/bin/python && \
>> apt-get clean && \
>> rm -rf /var/lib/apt/lists/*
>>
>> # install PyFlink
>> RUN pip3 install apache-flink==1.15.0
>>
>> And I am running:
>> docker build --tag pyflink:latest .
>>
>> This gives the following error:
>>
>>
>> *#6 64.12          cwd: /tmp/pip-install-9_farey_/pemja/#6 64.12
>> Complete output (1 lines):#6 64.12     Include folder should be at
>> '/usr/local/openjdk-11/include' but doesn't exist. Please check you've
>> installed the JDK properly.*
>>
>> A side note:
>> The Dockerfile in the docs is missing git so initially I got the
>> following error:
>>
>> *#7 57.73         raise OSError("%r was not found" % name)#7 57.73
>> OSError: 'git' was not found *
>>
>> @Weihua Hu <huweihua....@gmail.com> can you please send your working
>> Dockerfile?
>>
>> Gyula
>>
>> On Wed, Jul 6, 2022 at 4:16 AM Weihua Hu <huweihua....@gmail.com> wrote:
>>
>>> Hi Gyula,
>>>
>>> I can build pyFlink image successfully by following this guide. Did you
>>> add a dependency outside of the documentation? And could you provide your
>>> Dockerfile
>>>
>>>
>>> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Tue, Jul 5, 2022 at 11:40 PM Gyula Fóra <gyula.f...@gmail.com> wrote:
>>>
>>>> Well in any case either the official image is incorrect (maybe we
>>>> should include JDK by default not JRE) or we should update the
>>>> documentation regarding the python docker build because it simply doesn't
>>>> work at the moment.
>>>>
>>>> I am still looking for a full working example that adds the required
>>>> Python packages on top of a Flink 1.15.0 base image :)
>>>>
>>>> Gyula
>>>>
>>>> On Tue, Jul 5, 2022 at 5:36 PM Weihua Hu <huweihua....@gmail.com>
>>>> wrote:
>>>>
>>>>> In addition, you can try providing the Dockerfile
>>>>>
>>>>> Best,
>>>>> Weihua
>>>>>
>>>>>
>>>>> On Tue, Jul 5, 2022 at 11:24 PM Weihua Hu <huweihua....@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The base image flink:1.15.0 is built from openjdk:11-jre, and this
>>>>>> image only installs jre but not jdk.
>>>>>> It looks like the package you want to install (pemja) depends on jdk.
>>>>>> you need install openjdk-11-jdk in dockerfile,
>>>>>> take a look to how it is installed in the official image:
>>>>>>
>>>>>>
>>>>>> https://hub.docker.com/layers/openjdk/library/openjdk/11-jdk/images/sha256-bc0af19c7c4f492fe6ff0c1d1c8c0e5dd90ab801385b220347bb91dbe2b4094f?context=explore
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Weihua
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 5, 2022 at 3:50 PM Gyula Fóra <gyula.f...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All!
>>>>>>>
>>>>>>> I have been trying to experiment with the Flink python support on
>>>>>>> Kubernetes but I got stuck creating a custom image with all the 
>>>>>>> necessary
>>>>>>> python libraries.
>>>>>>>
>>>>>>> I found this guide in the docs:
>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker
>>>>>>>
>>>>>>> However when I try to build a custom image using it, I get the
>>>>>>> following error:
>>>>>>>
>>>>>>> #7 131.7 Collecting pemja==0.1.4
>>>>>>> #7 131.8   Downloading pemja-0.1.4.tar.gz (32 kB)
>>>>>>> #7 131.9     ERROR: Command errored out with exit status 255:
>>>>>>> #7 131.9      command: /usr/local/bin/python3.7 -c 'import sys,
>>>>>>> setuptools, tokenize; sys.argv[0] =
>>>>>>> '"'"'/tmp/pip-install-y6o6djs1/pemja/setup.py'"'"';
>>>>>>> __file__='"'"'/tmp/pip-install-y6o6djs1/pemja/setup.py'"'"';f=getattr(tokenize,
>>>>>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
>>>>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
>>>>>>> egg_info --egg-base /tmp/pip-pip-egg-info-47iid6nm
>>>>>>> #7 131.9          cwd: /tmp/pip-install-y6o6djs1/pemja/
>>>>>>> #7 131.9     Complete output (1 lines):
>>>>>>> #7 131.9     Include folder should be at
>>>>>>> '/usr/local/openjdk-11/include' but doesn't exist. Please check you've
>>>>>>> installed the JDK properly.
>>>>>>>
>>>>>>> Has anyone seen this before and knows the solution?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Gyula
>>>>>>>
>>>>>>

Reply via email to