[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r281010551
##
File path:
flink-python/src/main/java/org/apache/flink/api/python/PythonGatewayServer.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.python;
+
+import py4j.GatewayServer;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.InetAddress;
+
+/**
+ * The Py4j Gateway Server provides RPC service for user's python process.
+ */
+public class PythonGatewayServer {
Review comment:
Agreed. we should consider this potential dependency problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r281002536
##
File path: flink-python/pyflink/table/table.py
##
@@ -0,0 +1,117 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from py4j.java_gateway import get_method
+
+__all__ = ['Table']
+
+
+class Table(object):
+
+"""
+A :class:`Table` is the core component of the Table API.
+Similar to how the batch and streaming APIs have DataSet and DataStream,
+the Table API is built around :class:`Table`.
+
+Use the methods of :class:`Table` to transform data.
+
+Example:
+::
+>>> t_config =
TableConfig.Builder().as_streaming_execution().set_parallelism(1).build()
+>>> t_env = TableEnvironment.get_table_environment(t_config)
+>>> ...
+>>> t = t_env.scan("source")
+>>> t.select(...)
+...
+>>> t.insert_into("print")
+>>> t_env.execute()
+
+Operations such as :func:`~pyflink.table.Table.join`,
:func:`~pyflink.table.Table.select`,
+:func:`~pyflink.table.Table.where` and
:func:`~pyflink.table.Table.group_by`
+take arguments in an expression string. Please refer to the documentation
for
+the expression syntax.
+"""
+
+def __init__(self, j_table):
+self._j_table = j_table
+
+def select(self, fields):
+"""
+Performs a selection operation. Similar to a SQL SELECT statement. The
field expressions
+can contain complex expressions.
+
+Example:
+::
+>>> t = tab.select("key, value + 'hello'")
Review comment:
For this point it makes sense. I'll adjust these document to be consistent
with scala documents.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279640426 ## File path: flink-python/pyflink/table/table_environment.py ## @@ -0,0 +1,160 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from abc import ABCMeta + +from pyflink.java_gateway import get_gateway +from pyflink.table import Table +from pyflink.util import type_utils, utils + +__all__ = [ +'BatchTableEnvironment', +'StreamTableEnvironment', +'TableEnvironment' +] + + +class TableEnvironment(object): +""" +The abstract base class for batch and stream TableEnvironments. +""" + +__metaclass__ = ABCMeta + +def __init__(self, j_tenv): +self._j_tenv = j_tenv + +def from_table_source(self, table_source): +""" +Creates a table from a table source. + +:param table_source: table source used as table +:return: result table +""" +return Table(self._j_tenv.fromTableSource(table_source._j_table_source)) + +def register_table(self, name, table): +""" +Registers a :class:`Table` under a unique name in the TableEnvironment's catalog. +Registered tables can be referenced in SQL queries. + +:param name: The name under which the table will be registered. +:param table: The table to register. +""" +self._j_tenv.registerTable(name, table._java_table) + +def register_table_source(self, name, table_source): +""" +Registers an external :class:`TableSource` in this :class:`TableEnvironment`'s catalog. +Registered tables can be referenced in SQL queries. + +:param name: The name under which the :class:`TableSource` is registered. +:param table_source: The :class:`TableSource` to register. +""" +self._j_tenv.registerTableSource(name, table_source._j_table_source) + +def register_table_sink(self, name, field_names, field_types, table_sink): +""" +Registers an external :class:`TableSink` with given field names and types in this +:class:`TableEnvironment`\ 's catalog. Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279640287
##
File path: flink-python/pyflink/table/table.py
##
@@ -0,0 +1,117 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from py4j.java_gateway import get_method
+
+__all__ = ['Table']
+
+
+class Table(object):
+
+"""
+A :class:`Table` is the core component of the Table API.
+Similar to how the batch and streaming APIs have DataSet and DataStream,
+the Table API is built around :class:`Table`.
+
+Use the methods of :class:`Table` to transform data.
+
+Example:
+::
+>>> t_config =
TableConfig.Builder().as_streaming_execution().set_parallelism(1).build()
+>>> t_env = TableEnvironment.get_table_environment(t_config)
+>>> ...
+>>> t = t_env.scan("source")
+>>> t.select(...)
+...
+>>> t.insert_into("print")
+>>> t_env.execute()
+
+Operations such as :func:`~pyflink.table.Table.join`,
:func:`~pyflink.table.Table.select`,
+:func:`~pyflink.table.Table.where` and
:func:`~pyflink.table.Table.group_by`
+take arguments in an expression string. Please refer to the documentation
for
+the expression syntax.
+"""
+
+def __init__(self, j_table):
+self._j_table = j_table
+
+def select(self, fields):
+"""
+Performs a selection operation. Similar to a SQL SELECT statement. The
field expressions
+can contain complex expressions.
+
+Example:
+::
+>>> t = tab.select("key, value + 'hello'")
Review comment:
I think using a variable to store the select result is more appropriate
because select operation won't change the table itself. If we just write
`tab.select("key, value + 'hello'")` and don't store its result, we will lose
this select result.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279640357 ## File path: flink-python/pyflink/table/table_environment.py ## @@ -0,0 +1,160 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from abc import ABCMeta + +from pyflink.java_gateway import get_gateway +from pyflink.table import Table +from pyflink.util import type_utils, utils + +__all__ = [ +'BatchTableEnvironment', +'StreamTableEnvironment', +'TableEnvironment' +] + + +class TableEnvironment(object): +""" +The abstract base class for batch and stream TableEnvironments. +""" + +__metaclass__ = ABCMeta + +def __init__(self, j_tenv): +self._j_tenv = j_tenv + +def from_table_source(self, table_source): +""" +Creates a table from a table source. + +:param table_source: table source used as table +:return: result table +""" Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279638825
##
File path: flink-python/pyflink/table/table.py
##
@@ -0,0 +1,117 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from py4j.java_gateway import get_method
+
+__all__ = ['Table']
+
+
+class Table(object):
+
+"""
+A :class:`Table` is the core component of the Table API.
+Similar to how the batch and streaming APIs have DataSet and DataStream,
+the Table API is built around :class:`Table`.
+
+Use the methods of :class:`Table` to transform data.
+
+Example:
+::
+>>> t_config =
TableConfig.Builder().as_streaming_execution().set_parallelism(1).build()
+>>> t_env = TableEnvironment.get_table_environment(t_config)
+>>> ...
+>>> t = t_env.scan("source")
+>>> t.select(...)
+...
+>>> t.insert_into("print")
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279638781 ## File path: flink-python/pyflink/table/table.py ## @@ -0,0 +1,117 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from py4j.java_gateway import get_method + +__all__ = ['Table'] + + +class Table(object): + +""" +A :class:`Table` is the core component of the Table API. +Similar to how the batch and streaming APIs have DataSet and DataStream, +the Table API is built around :class:`Table`. + +Use the methods of :class:`Table` to transform data. + +Example: +:: +>>> t_config = TableConfig.Builder().as_streaming_execution().set_parallelism(1).build() +>>> t_env = TableEnvironment.get_table_environment(t_config) +>>> ... Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279638728 ## File path: flink-dist/src/main/flink-bin/bin/pyflink2.sh ## @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# = +bin=`dirname "$0"` Review comment: Agreed. I renamed it in the new commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279638523 ## File path: flink-python/pyflink/find_flink_home.py ## @@ -0,0 +1,44 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function +import os +import sys + + +def _find_flink_home(): +""" +Find the FLINK_HOME. Review comment: IMO the util function shouldn't echo messages like this to stdout. Maybe we can log this after introducing the log module? I printed the value of FLINK_HOME in the end to end test in the new commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279636240 ## File path: flink-python/README.md ## @@ -0,0 +1,22 @@ +# Apache Flink Python API + +Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities. + +Learn more about Flink at [http://flink.apache.org/](http://flink.apache.org/) + +This packaging is currently a very initial version and will change in future versions. + +## Installation + +In order to use PyFlink, you need to install Flink on your device and set the value of the environment variable FLINK_HOME to the root directory of Flink. Review comment: Agreed, I rewrote this in the new commit to make it clearer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279636291 ## File path: flink-python/README.md ## @@ -0,0 +1,22 @@ +# Apache Flink Python API + +Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities. + +Learn more about Flink at [http://flink.apache.org/](http://flink.apache.org/) + +This packaging is currently a very initial version and will change in future versions. + +## Installation + +In order to use PyFlink, you need to install Flink on your device and set the value of the environment variable FLINK_HOME to the root directory of Flink. +Then enter the directory where this README.md file is located and execute `python setup.py install` to install PyFlink on your device. + +## Running Tests + +Currently you can perform an end-to-end test of PyFlink in the directory where this file is located with the following command: + +PYTHONPATH=./ python ./pyflink/table/tests/test_end_to_end.py Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279635500 ## File path: flink-python/pyflink/java_gateway.py ## @@ -0,0 +1,106 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import shutil +import signal +import struct +import tempfile +import time +from subprocess import Popen, PIPE +from threading import RLock + +from py4j.java_gateway import java_import, JavaGateway, GatewayParameters +from pyflink.find_flink_home import _find_flink_home + + +_gateway = None +_lock = RLock() + + +def get_gateway(): +# type: () -> JavaGateway +global _gateway +global _lock +with _lock: +if _gateway is None: +# if Java Gateway is already running +if 'PYFLINK_GATEWAY_PORT' in os.environ: +gateway_port = int(os.environ['PYFLINK_GATEWAY_PORT']) +gateway_param = GatewayParameters(port=gateway_port, auto_convert=True) +_gateway = JavaGateway(gateway_parameters=gateway_param) +else: +_gateway = launch_gateway() +return _gateway + + +def launch_gateway(): +# type: () -> JavaGateway +""" +launch jvm gateway +""" + +FLINK_HOME = _find_flink_home() +# TODO windows support Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279635443 ## File path: .gitignore ## @@ -14,7 +14,9 @@ tmp *.iml *.swp *.jar +*.zip *.log +*.pyc Review comment: Agreed. These directories appeared after execute `python ./setup.py xxx`. We should ignore them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279634928
##
File path: flink-python/setup.py
##
@@ -0,0 +1,63 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import os
+import sys
+from setuptools import setup
+
+if sys.version_info < (2, 7):
+print("Python versions prior to 2.7 are not supported for PyFlink.",
+ file=sys.stderr)
+sys.exit(-1)
+
+this_directory = os.path.abspath(os.path.dirname(__file__))
+version_file = os.path.join(this_directory, 'pyflink/version.py')
+
+try:
+exec(open(version_file).read())
+except IOError:
+print("Failed to load PyFlink version file for packaging. " +
+ "'%s' not found!" % version_file,
+ file=sys.stderr)
+sys.exit(-1)
+VERSION = __version__ # noqa
+
+with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279634867 ## File path: flink-python/README.md ## @@ -0,0 +1,22 @@ +# Apache Flink Python API + +Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities. Review comment: Agreed. I added some introduction for Python in the new commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279634557
##
File path: docs/dev/table/tableApi.zh.md
##
@@ -297,6 +297,81 @@ val result = orders.where('b === "red")
+
+
+
+
+
+
+ Operators
+ Description
+
+
+
+
+
+Scan
+Batch Streaming
+
+
+Similar to the FROM clause in a SQL query. Performs a scan of a
registered table.
Review comment:
This makes sense. I translated them into Chinese in the new commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279634579 ## File path: flink-python/pom.xml ## @@ -0,0 +1,201 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd;> + + 4.0.0 + + + flink-parent + org.apache.flink + 1.9-SNAPSHOT + .. + + + flink-python + flink-python + + jar + + + + net.sf.py4j + py4j + 0.10.8.1 + + + + + + + org.apache.maven.plugins + maven-enforcer-plugin + + + dependency-convergence + + enforce + + + true + + + + + + + + com.github.siom79.japicmp + japicmp-maven-plugin + + + + org.apache.maven.plugins + maven-surefire-plugin + + + false + + + + + + org.apache.maven.plugins + maven-eclipse-plugin + 2.8 + + true + + org.scala-ide.sdt.core.scalanature + org.eclipse.jdt.core.javanature + + + org.scala-ide.sdt.core.scalabuilder + + + org.scala-ide.sdt.launching.SCALA_CONTAINER + + org.eclipse.jdt.launching.JRE_CONTAINER + + + + org.scala-lang:scala-library + org.scala-lang:scala-compiler + + + **/*.scala Review comment: There is no scala code in our project. I removed this plugin in the new commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279634616 ## File path: flink-python/README.md ## @@ -0,0 +1,22 @@ +# Apache Flink Python API + +Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities. Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279633966 ## File path: flink-python/README.md ## @@ -0,0 +1,22 @@ +# Apache Flink Python API + +Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities. + +Learn more about Flink at [http://flink.apache.org/](http://flink.apache.org/) + +This packaging is currently a very initial version and will change in future versions. + +## Installation + +In order to use PyFlink, you need to install Flink on your device and set the value of the environment variable FLINK_HOME to the root directory of Flink. +Then enter the directory where this README.md file is located and execute `python setup.py install` to install PyFlink on your device. + +## Running Tests + +Currently you can perform an end-to-end test of PyFlink in the directory where this file is located with the following command: + +PYTHONPATH=./ python ./pyflink/table/tests/test_end_to_end.py + +## Python Requirements + +PyFlink currently depends on `Py4J 0.10.8.1`. Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279633908
##
File path: flink-dist/src/main/flink-bin/bin/pyflink2.sh
##
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# =
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/config.sh
+
+FLINK_CLASSPATH=`constructFlinkClassPath`
+
+ARGS=()
+
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+-c|--class)
+DRIVER=$2
+shift
+shift
+;;
+*)
+ ARGS+=("$1")
+ shift
+ ;;
+esac
+done
+
+PYTHON_JAR_PATH=`echo "$FLINK_ROOT_DIR"/opt/flink-python-*.jar`
+TABLE_JAR_PATH=`echo "$FLINK_ROOT_DIR"/opt/flink-table*.jar`
+exec $JAVA_RUN $JVM_ARGS -cp
${FLINK_CLASSPATH}:${TABLE_JAR_PATH}:${PYTHON_JAR_PATH} ${DRIVER} ${ARGS[@]}
Review comment:
IMO it is unnecessary, because this file is used internally.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279633441
##
File path: flink-dist/src/main/flink-bin/bin/pyflink2.sh
##
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# =
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/config.sh
+
+FLINK_CLASSPATH=`constructFlinkClassPath`
+
+ARGS=()
+
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+-c|--class)
+DRIVER=$2
+shift
+shift
+;;
+*)
+ ARGS+=("$1")
+ shift
+ ;;
+esac
+done
+
Review comment:
Yes, this makes sense. I added log parameters in the new commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279632921
##
File path: docs/dev/table/tableApi.zh.md
##
@@ -297,6 +297,81 @@ val result = orders.where('b === "red")
+
+
+
+
+
+
+ Operators
+ Description
+
+
+
+
+
+Scan
+Batch Streaming
+
+
+Similar to the FROM clause in a SQL query. Performs a scan of a
registered table.
+{% highlight python %}
+orders = table_env.scan("Orders");
+{% endhighlight %}
+
+
+
+
+Select
+Batch Streaming
+
+
+Similar to a SQL SELECT statement. Performs a select operation.
+{% highlight python %}
+orders = tableEnv.scan("Orders");
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279632956
##
File path: flink-dist/src/main/assemblies/opt.xml
##
@@ -177,5 +177,20 @@
flink-streaming-python_${scala.binary.version}-${project.version}.jar
0644
+
+
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279632895
##
File path: docs/dev/table/tableApi.md
##
@@ -297,6 +297,81 @@ val result = orders.where('b === "red")
+
+
+
+
+
+
+ Operators
+ Description
+
+
+
+
+
+Scan
+Batch Streaming
+
+
+Similar to the FROM clause in a SQL query. Performs a scan of a
registered table.
+{% highlight python %}
+orders = table_env.scan("Orders");
+{% endhighlight %}
+
+
+
+
+Select
+Batch Streaming
+
+
+Similar to a SQL SELECT statement. Performs a select operation.
+{% highlight python %}
+orders = tableEnv.scan("Orders");
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279230381 ## File path: flink-python/flink-python-table/src/main/python/pyflink/table/__init__.py ## @@ -0,0 +1,37 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from pyflink.table.table import Table +from pyflink.table.table_config import TableConfig +from pyflink.table.table_environment import TableEnvironment, StreamTableEnvironment, BatchTableEnvironment +from pyflink.table.table_sink import TableSink, CsvTableSink +from pyflink.table.table_source import TableSource, CsvTableSource +from pyflink.table.types import DataTypes + Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279230371 ## File path: flink-python/flink-python-table/src/main/python/setup.py ## @@ -0,0 +1,32 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from distutils.core import setup + +setup( Review comment: Fixed. Now the setup.py will check whether the python version is above 2.7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279230215 ## File path: flink-python/flink-python-table/src/main/python/setup.py ## @@ -0,0 +1,32 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from distutils.core import setup + +setup( +name='pyflink', +version='1.0', Review comment: Yes, it makes sense to me. Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support URL: https://github.com/apache/flink/pull/8267#discussion_r279230115 ## File path: flink-python/flink-python-table/src/main/python/setup.py ## @@ -0,0 +1,32 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from distutils.core import setup + +setup( +name='pyflink', +version='1.0', +packages=['pyflink', + 'pyflink.table', + 'pyflink.util'], +url='http://flink.apache.org', +license='Licensed under the Apache License, Version 2.0', +author='Flink Developers', +author_email='d...@flink.apache.org', +install_requires=['py4j==0.10.8.1'], +description='Flink Python API' Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #8267: [FLINK-12311][table][python] Add base python framework and Add Scan, Projection, and Filter operator support
WeiZhong94 commented on a change in pull request #8267:
[FLINK-12311][table][python] Add base python framework and Add Scan,
Projection, and Filter operator support
URL: https://github.com/apache/flink/pull/8267#discussion_r279230110
##
File path: docs/dev/table/tableApi.md
##
@@ -297,6 +297,81 @@ val result = orders.where('b === "red")
+
Review comment:
I agree that a README.md file is necessary. I added it in the new commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
