William Ashley created FLINK-32136:
--------------------------------------
Summary: Pyflink gateway server launch fails when purelib !=
platlib
Key: FLINK-32136
URL: https://issues.apache.org/jira/browse/FLINK-32136
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.13.3
Reporter: William Ashley
On distros where python's {{purelib}} is different than {{platlib}} (e.g.
Amazon Linux 2, but from my research it's all of the Redhat-based ones), you
wind up with components of packages being installed across two different
locations (e.g. {{/usr/local/lib/python3.7/site-packages/pyflink}} and
{{{}/usr/local/lib64/python3.7/site-packages/pyflink{}}}).
{{_find_flink_home}}
[handles|https://github.com/apache/flink/blob/06688f345f6793a8964ec00002175f44cda13c33/flink-python/pyflink/find_flink_home.py#L58C63-L60]
this, and in flink releases <= 1.13.2 its setting of the {{FLINK_LIB_DIR}}
environment variable was the one being used. However, from 1.13.3, a
refactoring of launch_gateway_server_process
([1.13.2,|https://github.com/apache/flink/blob/release-1.13.2/flink-python/pyflink/pyflink_gateway_server.py#L200]
[1.13.3|https://github.com/apache/flink/blob/release-1.13.3/flink-python/pyflink/pyflink_gateway_server.py#L280])
re-ordered some method calls. prepare_environment_variable's non-awareness of
multiple homes and setting of {{FLINK_LIB_DIR}} now is the one that matters,
and it is the incorrect location.
I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not
exist on, for example, Ubuntu 20 and 22 (for which {{platlib}} ==
{{{}purelib{}}}).
Repro steps on Amazon Linux 2
{quote}{{yum -y install python3 java-11}}
{{pip3 install apache-flink==1.13.3}}
{{python3 -c 'from pyflink.table import EnvironmentSettings ;
EnvironmentSettings.new_instance()'}}
{quote}
The resulting error is
{quote}{{The flink-python jar is not found in the opt folder of the FLINK_HOME:
/usr/local/lib64/python3.7/site-packages/pyflink}}
{{Error: Could not find or load main class
org.apache.flink.client.python.PythonGatewayServer}}
{{Caused by: java.lang.ClassNotFoundException:
org.apache.flink.client.python.PythonGatewayServer}}
{{Traceback (most recent call last):}}
{{ File "<string>", line 1, in <module>}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 214, in new_instance}}
{{ return EnvironmentSettings.Builder()}}
{{ File
"/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py",
line 48, in __init__}}
{{ gateway = get_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 62, in get_gateway}}
{{ _gateway = launch_gateway()}}
{{ File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py",
line 112, in launch_gateway}}
{{ raise Exception("Java gateway process exited before sending its port
number")}}
{{Exception: Java gateway process exited before sending its port number}}
{quote}
The flink home under /lib64/ does not contain the jar, but it is in the /lib/
location
{quote}{{bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name
"flink-python*.jar"}}
{{/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar}}
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
