Re: arm，centos7下部署pyflink
Hi, 肯定不是的，这个只是python shell的启动脚本。你想想看你mvn编译的只是java的代码，里面都没有pyflink的python代码。你写的python job那些Import 的pyflink的包python都认识不了，肯定跑不起来呀。你如果想玩python shell，可以参考文档[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/python_shell.html Best, Xingbo 琴师 <1129656...@qq.com> 于2020年8月6日周四 上午9:11写道： > Hi: > 我用mvn clean package > -DskipTest完整的编译了flink-1.11.1-src.tgz，我看到flink-1.11.1/build-target/bin/pyflink-shell.sh，请问这个是做什么用的，是不是用这个就可以相当于pyflink了？ > > > --原始邮件-- > 发件人: > "琴师" > < > 1129656...@qq.com; > 发送时间:2020年8月4日(星期二) 上午10:19 > 收件人:"user-zh" > 主题:回复： arm，centos7下部署pyflink > > > > 确实，我尝试了编译pyarrow，但是没有完成，而且不止这一个第三方包。 > 真是遗憾。 > > > > > -- 原始邮件 -- > 发件人: > "user-zh" > < > hxbks...@gmail.com; > 发送时间:2020年8月4日(星期二) 上午10:09 > 收件人:"user-zh" > 主题:Re: arm，centos7下部署pyflink > > > > Hello, > > 我们pyflink会依赖pyarrow的包，而pyarrow并没有发布对应在arm下的wheel包，所以，你安装的话，会去下载pyarrow的源码包，然后尝试去编译源码包，所以出现你报错信息中的cmake失败的内容。 > > 现在可行的方式是你得尝试自己手动编译安装pyarrow，不过pyarrow并没有在arm架构下做测试，而且在高版本的pyarrow包甚至不发布源码包了，所以我很怀疑你能否顺利编译安装成功pyarrow。 > > Best, > Xingbo > > 琴师 <1129656...@qq.com 于2020年8月4日周二 上午9:57写道： > > 你好： > nbsp; nbsp; > > 我尝试在arm架构下centos系统内部署pyflink，python版本是3.5.9（3.7.1也尝试过），均不能完成部署，卡在pyarrow这一步，请问，这是系统不支持，还是我部署的问题？如果是我部署的问题，请问有解决方案么？ > nbsp;ERROR: Command errored out with exit status 1: > nbsp; nbsp;command: /usr/local/python3/bin/python3.5 > > /usr/local/python3/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py > build_wheel /tmp/tmpfda_qhew > nbsp; nbsp; nbsp; nbsp;cwd: > /tmp/pip-install-r4b5m7u0/pyarrow > nbsp; Complete output (428 lines): > nbsp; running bdist_wheel > nbsp; running build > nbsp; running build_py > nbsp; creating build > nbsp; creating build/lib.linux-aarch64-3.5 > nbsp; creating build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/jvm.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/benchmark.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/_generated_version.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/plasma.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/hdfs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/cuda.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/orc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/filesystem.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/parquet.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/flight.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/types.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/ipc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/fs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/compat.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/json.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/__init__.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/csv.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/util.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/pandas_compat.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/serialization.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/feather.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; creating build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_jvm.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_deprecations.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_pandas.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_feather.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_gandiva.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_fs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_json.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_ipc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/te
Re: arm，centos7下部署pyflink
Hi, 是的，pyarrow的编译安装依赖很多与机器相关的c的库，特别难安装上，所以高版本的pyarrow直接就不发源码包，省的你去编译报错安不上。 Best, Xingbo 琴师 <1129656...@qq.com> 于2020年8月4日周二 上午10:20写道： > 确实，我尝试了编译pyarrow，但是没有完成，而且不止这一个第三方包。 > 真是遗憾。 > > > > > --原始邮件-- > 发件人: > "user-zh" > < > hxbks...@gmail.com; > 发送时间:2020年8月4日(星期二) 上午10:09 > 收件人:"user-zh" > 主题:Re: arm，centos7下部署pyflink > > > > Hello, > > 我们pyflink会依赖pyarrow的包，而pyarrow并没有发布对应在arm下的wheel包，所以，你安装的话，会去下载pyarrow的源码包，然后尝试去编译源码包，所以出现你报错信息中的cmake失败的内容。 > > 现在可行的方式是你得尝试自己手动编译安装pyarrow，不过pyarrow并没有在arm架构下做测试，而且在高版本的pyarrow包甚至不发布源码包了，所以我很怀疑你能否顺利编译安装成功pyarrow。 > > Best, > Xingbo > > 琴师 <1129656...@qq.com 于2020年8月4日周二 上午9:57写道： > > 你好： > nbsp; nbsp; > > 我尝试在arm架构下centos系统内部署pyflink，python版本是3.5.9（3.7.1也尝试过），均不能完成部署，卡在pyarrow这一步，请问，这是系统不支持，还是我部署的问题？如果是我部署的问题，请问有解决方案么？ > nbsp;ERROR: Command errored out with exit status 1: > nbsp; nbsp;command: /usr/local/python3/bin/python3.5 > > /usr/local/python3/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py > build_wheel /tmp/tmpfda_qhew > nbsp; nbsp; nbsp; nbsp;cwd: > /tmp/pip-install-r4b5m7u0/pyarrow > nbsp; Complete output (428 lines): > nbsp; running bdist_wheel > nbsp; running build > nbsp; running build_py > nbsp; creating build > nbsp; creating build/lib.linux-aarch64-3.5 > nbsp; creating build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/jvm.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/benchmark.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/_generated_version.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/plasma.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/hdfs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/cuda.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/orc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/filesystem.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/parquet.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/flight.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/types.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/ipc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/fs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/compat.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/json.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/__init__.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/csv.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/util.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/pandas_compat.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/serialization.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; copying pyarrow/feather.py -gt; > build/lib.linux-aarch64-3.5/pyarrow > nbsp; creating build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_jvm.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_deprecations.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_pandas.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_feather.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_gandiva.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_fs.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_json.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_ipc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_plasma_tf_op.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_misc.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_scalars.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/conftest.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/strategies.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/pandas_examples.py -gt; > build/lib.linux-aarch64-3.5/pyarrow/tests > nbsp; copying pyarrow/tests/test_cuda.py -gt; > build/lib.linux-aarch64-3.5/pyarro
Re: arm，centos7下部署pyflink
Hello, 我们pyflink会依赖pyarrow的包，而pyarrow并没有发布对应在arm下的wheel包，所以，你安装的话，会去下载pyarrow的源码包，然后尝试去编译源码包，所以出现你报错信息中的cmake失败的内容。 现在可行的方式是你得尝试自己手动编译安装pyarrow，不过pyarrow并没有在arm架构下做测试，而且在高版本的pyarrow包甚至不发布源码包了，所以我很怀疑你能否顺利编译安装成功pyarrow。 Best, Xingbo 琴师 <1129656...@qq.com> 于2020年8月4日周二 上午9:57写道： > 你好： > > 我尝试在arm架构下centos系统内部署pyflink，python版本是3.5.9（3.7.1也尝试过），均不能完成部署，卡在pyarrow这一步，请问，这是系统不支持，还是我部署的问题？如果是我部署的问题，请问有解决方案么？ > ERROR: Command errored out with exit status 1: > command: /usr/local/python3/bin/python3.5 > /usr/local/python3/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py > build_wheel /tmp/tmpfda_qhew >cwd: /tmp/pip-install-r4b5m7u0/pyarrow > Complete output (428 lines): > running bdist_wheel > running build > running build_py > creating build > creating build/lib.linux-aarch64-3.5 > creating build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/jvm.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/benchmark.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/_generated_version.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/plasma.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/hdfs.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/cuda.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/orc.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/filesystem.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/parquet.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/flight.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/types.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/ipc.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/fs.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/compat.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/json.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/__init__.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/csv.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/util.py - build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/pandas_compat.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/serialization.py - > build/lib.linux-aarch64-3.5/pyarrow > copying pyarrow/feather.py - build/lib.linux-aarch64-3.5/pyarrow > creating build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_jvm.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_deprecations.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_pandas.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_feather.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_gandiva.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_fs.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_json.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_ipc.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_plasma_tf_op.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_misc.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_scalars.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/conftest.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/strategies.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/pandas_examples.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_cuda.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_sparse_tensor.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_compute.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_strategies.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_tensor.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_array.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_hdfs.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_cuda_numba_interop.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_serialization.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/__init__.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_memory.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/deserialize_buffer.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/util.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_plasma.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_cython.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_filesystem.py - > build/lib.linux-aarch64-3.5/pyarrow/tests > copying pyarrow/tests/test_io.py - >
