That's a tough question because we haven't done extensive benchmarks. At
an earlier stage we had determined a 5-10% overhead for the language
portability. I can only say that this depends heavily on the type of
work load. Also there are some optimizations with regards to state and
timers which could boost the performance.
I would be curious to hear about any results you might have during your
experiments.
Thanks,
Max
On 27.05.19 13:10, 青雉（祁明良） wrote:
Thanks max, it is clear to me now.
BTW, I would like to ask about the performance of python runner on Flink. As I
remember, when Flink first introduce python support(maybe around 2015), it was
5-10 slower than scala. For now, what is the performance difference of scala /
python with Beam on Flink?
Since we would like to use tensorflow transform with Beam, python may probably
be the better choice over JVM based language.
Cheers,
Mingliang
On 27 May 2019, at 6:53 PM, Maximilian Michels <m...@apache.org> wrote:
Hi Mingliang,
The environment is created for each TaskManager.
For docker, will it create one docker per flink taskmanager?
Yes.
For process, does it mean start a python process to run the user code? And it seems
"command" should be set in the environment config, but what should it be?
You will have to start the same Python SDK Harness which would run inside a
Docker container if you had chosen Docker. This is a more manual approach which
should only be chosen if you cannot use Docker.
For external(loopback), does it mean let flink operator to call an external
service and by default set to the place where I submit the beam job? This looks
like all the data will be shift to a single machine and processed there.
This intended for a long-running SDK Harness which is already running when you
run your pipeline. Thus, you only provide the address to the already running
SDK Harness.
Cheers,
Max
On 26.05.19 13:51, 青雉（祁明良） wrote:
Hi All,
I'm currently trying python portable runner with Flink. I see there are 3 kinds of
environment_type available "docker/process/external(loopback)" when submit a
job. But I didn't find any material explain more.
1. For docker, will it create one docker per flink taskmanager?
2. For process, does it mean start a python process to run the user
code? And it seems "command" should be set in the environment
config, but what should it be?
3. For external(loopback), does it mean let flink operator to call an
external service and by default set to the place where I submit the
beam job? This looks like all the data will be shift to a single
machine and processed there.
Thanks,
Mingliang
本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）
本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！
This communication may contain privileged or other confidential information of
Red. If you have received it in error, please advise the sender by reply e-mail
and immediately delete the message and any attachments without copying or
disclosing the contents. Thank you.
本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This communication may contain privileged or other confidential information of
Red. If you have received it in error, please advise the sender by reply e-mail
and immediately delete the message and any attachments without copying or
disclosing the contents. Thank you.
