Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon closed pull request #45741: [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases URL: https://github.com/apache/spark/pull/45741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on PR #45741: URL: https://github.com/apache/spark/pull/45741#issuecomment-2026332334 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on PR #45741: URL: https://github.com/apache/spark/pull/45741#issuecomment-2024504624 seems like we should add the implementation for Scala side as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
gene-db commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542249535
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
Review Comment:
removed.
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
Review Comment:
done.
##
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##
@@ -6594,6 +6594,24 @@ object functions {
fnWithOptions("from_json", options, e, schema)
}
+ /**
+ * Parses a JSON string and constructs a Variant value.
+ *
+ * @param json a JSON string.
+ *
+ * @since 4.0.0
+ */
+ def parse_json(json: String): Column = parse_json(lit(json))
Review Comment:
removed.
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
+>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+>>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Removed alias.
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
Review Comment:
removed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
zhengruifeng commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542253499
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
+>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+>>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
let's do not use alias to also verify the default column name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190834
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
Review Comment:
should be listed in `python/docs/source/reference/pyspark.sql/functions.rst`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190297
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
+>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+>>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Let's probably use `show()`. instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190658
##
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##
@@ -6594,6 +6594,24 @@ object functions {
fnWithOptions("from_json", options, e, schema)
}
+ /**
+ * Parses a JSON string and constructs a Variant value.
+ *
+ * @param json a JSON string.
+ *
+ * @since 4.0.0
+ */
+ def parse_json(json: String): Column = parse_json(lit(json))
Review Comment:
I would remove this String type signature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190297
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
+>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+>>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Let's probably use `show()`. instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190216
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
+
+Returns
+---
+:class:`~pyspark.sql.Column`
+a new column of VariantType.
+
+Examples
+
+>>> from pyspark.sql.types import *
Review Comment:
this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190154
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema,
_options_to_str(options))
+@_try_remote_functions
+def parse_json(
+col: "ColumnOrName",
+) -> Column:
+"""
+Parses a column containing a JSON string into a :class:`VariantType`.
+
+.. versionadded:: 4.0.0
+
+Parameters
+--
+col : :class:`~pyspark.sql.Column` or str
+a column or column name JSON formatted strings
+
+.. # noqa
Review Comment:
I think we don't need this (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
