Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
Looks like v2.2 hasn't been branched off yet. We're still using the
dickens out of this feature locally. It's also a good step forward for Python
API parity. Is there anyone with some time
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
I'm sure everyone was busy over the holidays and with the release of v2.1
(congrats).
Anecdotally, we've been using this functionality a *lot* in our internal
fork, so I think it could
Github user shea-parkes commented on a diff in the pull request:
https://github.com/apache/spark/pull/16094#discussion_r94609053
--- Diff: python/pyspark/sql/column.py ---
@@ -298,19 +299,34 @@ def isin(self, *cols):
isNotNull = _unary_op("isNotNull", "True
Github user shea-parkes commented on a diff in the pull request:
https://github.com/apache/spark/pull/16094#discussion_r90748611
--- Diff: python/pyspark/sql/column.py ---
@@ -298,19 +299,34 @@ def isin(self, *cols):
isNotNull = _unary_op("isNotNull", "True
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
There we go. I found other examples of the `versionchanged` directive
syntax and added a corresponding entry to `pyspark.sql.Column.alias`. I think
this should be good to look over again
Github user shea-parkes commented on a diff in the pull request:
https://github.com/apache/spark/pull/16094#discussion_r90748330
--- Diff: python/pyspark/sql/column.py ---
@@ -298,19 +299,34 @@ def isin(self, *cols):
isNotNull = _unary_op("isNotNull", "True
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
Alright @marmbrus , the tests are at least passing now (including the pesky
python2.6). Let me know if you'd like any further changes.
---
If your project is set up for it, you can reply
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
That last failure was a syntax screw-up in my intentionally failed unit
tests; I'm too used to `py.test` style tests. I've hopefully got the syntax
correct now (and went ahead and switch
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
Alright, I've pushed up a new commit that should make this compatible with
python 2.6. I don't have a convenient python 2.6 environment to test in, but
it still is valid in my python 3 one
Github user shea-parkes commented on the issue:
https://github.com/apache/spark/pull/16094
Gah, I'm sorry. I forgot python 2.6 compatibility. I'll switch this over
shortly to use the old-school style. Likely do something like this:
```python
def alias(self, *alias
GitHub user shea-parkes opened a pull request:
https://github.com/apache/spark/pull/16094
[SPARK-18541][Python]Add metadata parameter to pyspark.sql.Column.alias()
## What changes were proposed in this pull request?
Add a `metadata` keyword parameter
11 matches
Mail list logo
