HuangXingBo commented on a change in pull request #13230: URL: https://github.com/apache/flink/pull/13230#discussion_r480232415
########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- Review comment: Rename the file to operators.md ? ########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section give a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function and +pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implement a MapFunction that return plus one value of input value. Review comment: return -> returns ########## File path: docs/dev/python/user-guide/datastream/operations.zh.md ########## @@ -0,0 +1,77 @@ +--- Review comment: Rename the file to operators.zh.md ? ########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section give a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function and +pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implement a MapFunction that return plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` that data will be in a form of byte array generated by Review comment: ```suggestion defined, the output type will be `Types.PICKLED_BYTE_ARRAY` so that data will be in a form of byte array generated by ``` ########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section give a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function and +pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implement a MapFunction that return plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` that data will be in a form of byte array generated by +pickle seriallizer. For more detail, please refer to [DataTypes]({% link dev/python/user-guide/datastream/data_types.md Review comment: ```suggestion pickle seriallizer. For more details on the `Pickle Serialization` , you can refer to [DataTypes]({% link dev/python/user-guide/datastream/data_types.md ``` ########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section give a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function and +pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implement a MapFunction that return plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` that data will be in a form of byte array generated by +pickle seriallizer. For more detail, please refer to [DataTypes]({% link dev/python/user-guide/datastream/data_types.md + %}). + +## Lambda Functions +As shown in previous examples, all operations can also accept a lambda function for describing the operation: +<p> +{% highlight python %} +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(lambda x: x + 1, output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> Operations ConnectedStream.map() and ConnectedStream.flat_map() are Review comment: ```suggestion <span class="label label-info">Note</span> The operations of map and flat_map in ConnectedStream only support accepting function interface ``` ########## File path: docs/dev/python/user-guide/datastream/operations.md ########## @@ -0,0 +1,87 @@ +--- +title: "Operators" +nav-parent_id: python_datastream_api +nav-pos: 20 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + + +Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into +sophisticated dataflow topologies. + +This section give a description of the basic transformations Python DataStream API provides, the effective physical +partitioning after applying those as well as insights into Flink's operator chaining. + +* This will be replaced by the TOC +{:toc} + +# DataStream Transformations + +DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, +filtering, reducing). Please see [operators]({% link dev/stream/operators/index.md %} +?code_tab=python) for an overview of the available stream transformations in Python DataStream API. + +# Functions +Most operators require a user-defined function. The following will describe different ways of how they can be specified. + +## Implementing Function Interfaces +Function interfaces for different operations are provided in Python DataStream API. Users can implement a Function and +pass it to the corresponding operation. Take MapFunction for instance: +<p> +{% highlight python %} +# Implement a MapFunction that return plus one value of input value. +class MyMapFunction(MapFunction): + + def map(value): + return value + 1 + +data_stream = env.from_collection([1, 2, 3, 4, 5],type_info=Types.INT()) +mapped_stream = data_stream.map(MyMapFunction(), output_type=Types.INT()) +{% endhighlight %} +</p> +<span class="label label-info">Note</span> In Python DataStream API, users are able to defined the output type information of the operation. If not +defined, the output type will be `Types.PICKLED_BYTE_ARRAY` that data will be in a form of byte array generated by +pickle seriallizer. For more detail, please refer to [DataTypes]({% link dev/python/user-guide/datastream/data_types.md + %}). Review comment: ```suggestion %}#pickle-serialization). ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org