dianfu commented on a change in pull request #18748: URL: https://github.com/apache/flink/pull/18748#discussion_r806768683
########## File path: docs/content.zh/docs/dev/python/python_execution_mode.md ########## @@ -0,0 +1,101 @@ +--- +title: "执行模式" +weight: 40 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Execution Mode + +The Python API supports different runtime execution modes from which you can choose depending on the +requirements of your use case and the characteristics of your job. The Python runtime execution mode +defines how to execute your customized Python functions. + +Prior to release-1.15, there is the only execution mode called `PROCESS` execution mode. The `PROCESS` +mode means that the Python user-defined functions will be executed in separate Python process. Review comment: ```suggestion mode means that the Python user-defined functions will be executed in separate Python processes. ``` ########## File path: docs/content.zh/docs/dev/python/python_execution_mode.md ########## @@ -0,0 +1,101 @@ +--- +title: "执行模式" +weight: 40 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Execution Mode + +The Python API supports different runtime execution modes from which you can choose depending on the +requirements of your use case and the characteristics of your job. The Python runtime execution mode +defines how to execute your customized Python functions. + +Prior to release-1.15, there is the only execution mode called `PROCESS` execution mode. The `PROCESS` +mode means that the Python user-defined functions will be executed in separate Python process. + +In release-1.15, there are another two execution modes called `MULTI-THREAD` execution mode and +`SUB-INTERPRETER` execution mode. The `MULTI-THREAD` mode means that the Python user-defined functions +will be executed in the same thread as Java Operator, but it will be affected by GIL performance. +The `SUB-INTERPRETER` mode means that the Python user-defined functions will be executed in Python +different sub-interpreters rather than different threads of one interpreter, which can largely overcome +the effects of the GIL, but it may fail in some CPython extensions libraries, such as numpy, tensorflow. Review comment: ```suggestion the effects of the GIL, but some CPython extensions libraries doesn't support it, such as numpy, tensorflow, etc. ``` ########## File path: docs/content.zh/docs/dev/python/python_execution_mode.md ########## @@ -0,0 +1,101 @@ +--- +title: "执行模式" +weight: 40 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Execution Mode + +The Python API supports different runtime execution modes from which you can choose depending on the +requirements of your use case and the characteristics of your job. The Python runtime execution mode +defines how to execute your customized Python functions. + +Prior to release-1.15, there is the only execution mode called `PROCESS` execution mode. The `PROCESS` +mode means that the Python user-defined functions will be executed in separate Python process. + +In release-1.15, there are another two execution modes called `MULTI-THREAD` execution mode and Review comment: ```suggestion In release-1.15, it has introduced another two execution modes called `MULTI-THREAD` execution mode and ``` ########## File path: docs/content.zh/docs/dev/python/python_execution_mode.md ########## @@ -0,0 +1,101 @@ +--- +title: "执行模式" +weight: 40 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Execution Mode + +The Python API supports different runtime execution modes from which you can choose depending on the +requirements of your use case and the characteristics of your job. The Python runtime execution mode +defines how to execute your customized Python functions. + +Prior to release-1.15, there is the only execution mode called `PROCESS` execution mode. The `PROCESS` +mode means that the Python user-defined functions will be executed in separate Python process. + +In release-1.15, there are another two execution modes called `MULTI-THREAD` execution mode and +`SUB-INTERPRETER` execution mode. The `MULTI-THREAD` mode means that the Python user-defined functions +will be executed in the same thread as Java Operator, but it will be affected by GIL performance. +The `SUB-INTERPRETER` mode means that the Python user-defined functions will be executed in Python +different sub-interpreters rather than different threads of one interpreter, which can largely overcome +the effects of the GIL, but it may fail in some CPython extensions libraries, such as numpy, tensorflow. + +## When can/should I use MULTI-THREAD execution mode or SUB-INTERPRETER execution mode? + +The purpose of the introduction of `MULTI-THREAD` mode and `SUB-INTERPRETER` mode is to overcome the +overhead of serialization/deserialization and network communication caused in `PROCESS` mode. +So if performance is not your concern, or the computing logic of your customized Python functions is +the performance bottleneck of the job, `PROCESS` mode will be the best choice as `PROCESS` mode provides +the best isolation compared to `MULTI-THREAD` mode and `SUB-INTERPRETER` mode. + +Compared to `MULTI-THREAD` execution mode, `SUB-INTERPRETER` execution mode can largely overcome the +effects of the GIL, so it can get better performance. However, `SUB-INTERPRETER` may fail in some CPython Review comment: ```suggestion effects of the GIL, so you can get better performance usually. However, `SUB-INTERPRETER` may fail in some CPython ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
