Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
My impression so far was that we note things at migration notes when they
are improvements (not bugs), and non-trivial and related to backward
compatibility.
Shall we clarify what
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
BTW, it's closer to bug rather then improvement tho. `from_json` should
have default name `from_json` rather then `jsontostructs` - end users would
have no idea why it's called `jsontostructs
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
That's the exact issue I raised before and we ended up with not keeping the
compatibility in column names. @cloud-fan and @hvanhovell
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22795
Thanks @viirya!!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22816#discussion_r228012237
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -114,7 +114,7 @@ private[spark] abstract class BasePythonRunner
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22816#discussion_r228012035
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -114,7 +114,7 @@ private[spark] abstract class BasePythonRunner
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22237
Thanks all!!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22795
Thanks, @BryanCutler.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
Adding it as a known issue sounds reasonable to me as well.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22807#discussion_r227784077
--- Diff: python/pyspark/sql/tests.py ---
@@ -4961,6 +4961,31 @@ def foofoo(x, y):
).collect
)
+def
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22730
adding @cloud-fan since accumulator version 2 was added by you.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22795#discussion_r227634476
--- Diff: python/pyspark/sql/functions.py ---
@@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None,
functionType=None
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22237
https://github.com/apache/spark/pull/22237/files#r223707899 makes sense to
me. Addressed. LGTM from my side as well
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
> According to the policy, we don't have to block the current release
because of i
@cloud-fan, BTW, would you mind if I ask to share what you read? I want to
be aware of the pol
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22514
@cloud-fan, is this a performance regression that affects users that use
Hive serde tables as well?
---
-
To unsubscribe, e
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
Wait wait .. do we only care about regressions as blockers for the last
release (2.3)? I'm asking this because I really don't know. If so, I'm okay
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22237
Ah, yea I have a direct access to this branch. Let me just rebase/address
the comment tomorrow.
---
-
To unsubscribe, e
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
For instance,
https://groups.google.com/forum/?utm_medium=email_source=footer#!msg/sketches-user/GmH4-OlHP9g/MW-J7Hg4BwAJ
this discussion thread was started almost one year ago
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
It does try to fix a backward compatibility issue. It is found later now
but still it is true we found a breaking issue to fix. Broken backward
compatibility that potentially affects a set
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22237
Oh wait you left a sign-off. Let me rebase it within tomorrow - wouldn't be
a big job.
---
-
To unsubscribe, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22237
If @gengliangwang find some time to work on this, yea please go ahead.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22047#discussion_r227367500
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/AnyAgg.scala
---
@@ -0,0 +1,64 @@
+/*
+ * Licensed
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22803
Yea, similar opinion. If it does not fix an actual problem, I wouldn't
encourage to fix too ..
---
-
To unsubscribe, e
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22144
If we were going for 3.0, then I would definitely leave +1 and I agree that
we should rather focus on Spark itself as a higher priority - we should do that
when we go 3.0 and rather drop
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r227336895
--- Diff: bin/docker-image-tool.sh ---
@@ -79,7 +79,7 @@ function build {
fi
# Verify that Spark has actually been built
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r227336318
--- Diff: bin/docker-image-tool.sh ---
@@ -79,7 +79,7 @@ function build {
fi
# Verify that Spark has actually been built
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22775#discussion_r227251515
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -770,8 +776,17 @@ case class SchemaOfJson
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22775#discussion_r227239707
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -770,8 +776,17 @@ case class SchemaOfJson
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22775#discussion_r227239024
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -770,8 +776,17 @@ case class SchemaOfJson
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22730
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
@cloud-fan, mind taking a look please?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22795#discussion_r227199025
--- Diff: python/pyspark/sql/functions.py ---
@@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None,
functionType=None
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22795#discussion_r227198050
--- Diff: python/pyspark/sql/functions.py ---
@@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None,
functionType=None
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r227173103
--- Diff: bin/docker-image-tool.sh ---
@@ -79,7 +79,7 @@ function build {
fi
# Verify that Spark has actually been built
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22795#discussion_r227026198
--- Diff: python/pyspark/sql/functions.py ---
@@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None,
functionType=None
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
It's chaotic ...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22795
cc @viirya, @BryanCutler and @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22655
Hey @viirya, I happened to find some times to work on it - I submitted a PR
https://github.com/apache/spark/pull/22795
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22795
[SPARK-25798][PYTHON] Internally document type conversion between Pandas
data and SQL types in Pandas UDFs
## What changes were proposed in this pull request?
We are facing some
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
github looks buggy for now. Let me clean up my comments if they got messed.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Yea, looks we should better fix the comments.
LGTM otherwise.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Yea looks good as we discussed. Should we maybe better update the migration
guide too while we are here
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/spark/pull/22783
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22783
[WIP][BUILD] Fix errors of log4j when pip sanity checking
## What changes were proposed in this pull request?
PIP sanity checking produces some errors about log4j. I have some
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22662
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22776
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22782
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22782
pip packaging tests got passed. Let me merge this one since it blocks
almost every PR.
---
-
To unsubscribe, e-mail
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r226833951
--- Diff: bin/docker-image-tool.sh ---
@@ -79,7 +79,7 @@ function build {
fi
# Verify that Spark has actually been built
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r226833113
--- Diff: python/pyspark/__init__.py ---
@@ -16,7 +16,7 @@
#
"""
-PySpark is the Python API for Spark.
+PySpar
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22782#discussion_r226833051
--- Diff: dev/run-tests.py ---
@@ -551,7 +551,8 @@ def main():
if not changed_files or any(f.endswith(".
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22501
Yup, I made a fix https://github.com/apache/spark/pull/22782
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22748
@vanzin, the test failure was related. don't merge if the tests are failed.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22782
[WIP][HOTFIX] PIP failure fix
## What changes were proposed in this pull request?
## How was this patch tested?
Jenkins
You can merge this pull request into a Git
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22501
Thanks. It might rather more be related to external factors.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22501
I guess it's related with pip packaging tho.
```
Traceback (most recent call last):
File "", line 1, in
File
"/home/je
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22776
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21157
The workaround is to use CloudPickler btw. Technically we many cases that
normal pickler does not support. This one specific case (namedtuple) was
allowed by this weird hack
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Nope, it will work for both 2.11.8 and 2.11.12. I manually checked. This
change only uses the methods existing in both 2.11.8 and 2.11.12 at Scala.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
Ah.. let me rebase and sync the tests
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22781#discussion_r226816661
--- Diff: docs/building-spark.md ---
@@ -12,7 +12,7 @@ redirect_from: "building-with-maven.html"
## Apache Maven
The Maven-b
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22666#discussion_r226814727
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3886,6 +3886,31 @@ object functions {
withExpr(new CsvToStructs
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
Thank you @viirya and @dongjoon-hyun.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
Yup, will fix.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
Other JIRAs have different fixed versions. Let me create a new JIRA then.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22776
[SPARK-25779][SQL][TESTS] Remove SQL query tests for function documentation
by DESCRIBE FUNCTION at SQLQueryTestSuite
Currently, there are some tests testing function descriptions
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
Should be ready for a look now. Would you mind taking a look please
@cloud-fan and @gatorsmile?
---
-
To unsubscribe, e
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22775
This should be targeted to 2.4 .. otherwise we should describe the
behaviour change at migration note.
---
-
To unsubscribe
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22775
[SPARK-24709][SQL][FOLLOW-UP] Make schema_of_json's input json as literal
only
## What changes were proposed in this pull request?
The main purpose of `schema_of_json` is the usage
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22773
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22761
Merged to master and branch-2.4.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22773
[MINOR][SQL] Add prettyNames for from_json, to_json, from_csv, and
schema_of_json
## What changes were proposed in this pull request?
This PR adds `prettyNames` for `from_json
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536773
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -840,12 +840,19 @@ abstract class DDLSuite extends
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536735
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -207,6 +207,14 @@ class SessionCatalog
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536585
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -2370,4 +2370,17 @@ class HiveDDLSuite
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536555
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -2370,4 +2370,17 @@ class HiveDDLSuite
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536442
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -840,12 +840,19 @@ abstract class DDLSuite extends
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536456
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -840,12 +840,19 @@ abstract class DDLSuite extends
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22466#discussion_r226536304
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -840,12 +840,19 @@ abstract class DDLSuite extends
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22466
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22772
It's okay. the doc fix was huge and there should likely be some mistakes. I
will read it closely too this weekends
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22666
This is a WIP.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
I am able to address his comments for his vacation. Please keep reviewing
this.
---
-
To unsubscribe, e-mail: reviews
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22503
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22503#discussion_r226524439
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -220,6 +221,17 @@ class CSVSuite extends
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22576
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
801 - 900 of 12680 matches
Mail list logo
