Thanks max, it is clear to me now. BTW, I would like to ask about the performance of python runner on Flink. As I remember, when Flink first introduce python support(maybe around 2015), it was 5-10 slower than scala. For now, what is the performance difference of scala / python with Beam on Flink? Since we would like to use tensorflow transform with Beam, python may probably be the better choice over JVM based language.
Cheers, Mingliang > On 27 May 2019, at 6:53 PM, Maximilian Michels <[email protected]> wrote: > > Hi Mingliang, > > The environment is created for each TaskManager. > >> For docker, will it create one docker per flink taskmanager? > > Yes. > >> For process, does it mean start a python process to run the user code? >> And it seems "command" should be set in the environment config, but what >> should it be? > > You will have to start the same Python SDK Harness which would run inside a > Docker container if you had chosen Docker. This is a more manual approach > which should only be chosen if you cannot use Docker. > >> For external(loopback), does it mean let flink operator to call an >> external service and by default set to the place where I submit the beam >> job? This looks like all the data will be shift to a single machine and >> processed there. > > This intended for a long-running SDK Harness which is already running when > you run your pipeline. Thus, you only provide the address to the already > running SDK Harness. > > Cheers, > Max > > On 26.05.19 13:51, 青雉（祁明良） wrote: >> Hi All, >> I'm currently trying python portable runner with Flink. I see there are 3 >> kinds of environment_type available "docker/process/external(loopback)" when >> submit a job. But I didn't find any material explain more. >> 1. For docker, will it create one docker per flink taskmanager? >> 2. For process, does it mean start a python process to run the user >> code? And it seems "command" should be set in the environment >> config, but what should it be? >> 3. For external(loopback), does it mean let flink operator to call an >> external service and by default set to the place where I submit the >> beam job? This looks like all the data will be shift to a single >> machine and processed there. >> Thanks, >> Mingliang >> 本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发） >> 本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ >> This communication may contain privileged or other confidential information >> of Red. If you have received it in error, please advise the sender by reply >> e-mail and immediately delete the message and any attachments without >> copying or disclosing the contents. Thank you. 本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.
