[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219974#comment-17219974
]
Apache Spark commented on SPARK-32084:
--
User 'zero323' has created a pull request for this issue:
https://github.com/apache/spark/pull/30143
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219973#comment-17219973
]
Apache Spark commented on SPARK-32084:
--
User 'zero323' has created a pull request for this issue:
https://github.com/apache/spark/pull/30143
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218011#comment-17218011
]
Hyukjin Kwon commented on SPARK-32084:
--
Thank you [~zero323] so much.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217914#comment-17217914
]
Maciej Szymkiewicz commented on SPARK-32084:
I got that part and I fully agree. At the end of the day majority of work goes
into docstrings, not the implementation.
I'll try to start working on this by the end of this week.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217412#comment-17217412
]
Hyukjin Kwon commented on SPARK-32084:
--
Just for a bit of more context, I think we should have a better docstring wth
many examples. Also I am thinking about switching to numpydoc style
(SPARK-32085).
I can't think of a good benefit of keeping the functions as dynamically defined
.. it makes hard to write docstrings and the code deduplication is very small.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217407#comment-17217407
]
Hyukjin Kwon commented on SPARK-32084:
--
Yeah, that's not valid anymore. But I think still we should better just have
the regular functions at least to have docstrings more easily.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217339#comment-17217339
]
Maciej Szymkiewicz commented on SPARK-32084:
[~hyukjin.kwon] I can work on that, though I believe that the primary
motivation is no longer of concern, now when we have annotations in place.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217263#comment-17217263
]
Hyukjin Kwon commented on SPARK-32084:
--
[~zero323] are you interested in this?
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187365#comment-17187365
]
Hyukjin Kwon commented on SPARK-32084:
--
Hey [~nchammas], thanks for taking a look. I basically just mean:
{code}
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index f01bdb0165f..039050ab4ea 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -116,15 +116,7 @@ def _create_window_function(name, doc=''):
def _options_to_str(options):
return {key: to_str(value) for (key, value) in options.items()}
-_lit_doc = """
-Creates a :class:`Column` of literal value.
-
->>> df.select(lit(5).alias('height')).withColumn('spark_user',
lit(True)).take(1)
-[Row(height=5, spark_user=True)]
-"""
_functions = {
-'lit': _lit_doc,
-'col': 'Returns a :class:`Column` based on the given column name.',
'column': 'Returns a :class:`Column` based on the given column name.',
'asc': 'Returns a sort expression based on the ascending order of the
given column name.',
'desc': 'Returns a sort expression based on the descending order of the
given column name.',
@@ -316,6 +308,23 @@ for _name, _doc in _functions_2_4.items():
del _name, _doc
+@since(1.3)
+def lit(col):
+"""
+Creates a :class:`Column` of literal value.
+
+>>> df.select(lit(5).alias('height')).withColumn('spark_user',
lit(True)).take(1)
+[Row(height=5, spark_user=True)]
+"""
+return since(1.3)(_create_function("lit", _doc))(col)
+
+@since(1.3)
+def col(name):
+"""
+Returns a :class:`Column` based on the given column name.
+"""
+return since(1.3)(_create_function(_name, _doc))(name)
+
@since(1.3)
def approxCountDistinct(col, rsd=None):
"""
{code}
There might be a bit of small cleanups and refactoring but this was basically
what I had in my mind.
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py
[
https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187364#comment-17187364
]
Nicholas Chammas commented on SPARK-32084:
--
Can you share a couple of examples of what you are referring to in this ticket?
> Replace dictionary-based function definitions to proper functions in
> functions.py
> -
>
> Key: SPARK-32084
> URL: https://issues.apache.org/jira/browse/SPARK-32084
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 3.1.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently some functions in {{functions.py}} are defined by a dictionary. It
> programmatically defines the functions to the module; however, it makes some
> IDEs such as PyCharm don't detect.
> Also, it makes hard to add proper examples into the docstrings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
