Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21339
Sorry. I will start the progress now
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21310
To summarize our discussion in this pr:
Spark-avro is now merged into Spark as a built-in data source. Upstream
community is not merging the AvroEncoder to support Avro types in Dataset
Github user fangshil closed the pull request at:
https://github.com/apache/spark/pull/21310
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21339
thanks for the follow up. i will rebase this patch to latest master
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21310
@HyukjinKwon thanks for the update. What do you mean by "Avro was now in
Spark"? The PR I mentioned is
https://github.com/databricks/spark-avro/pull/215/files. I have been
maintaini
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21310
@viirya thanks for the feedback. We internally customized the AvroEncoder
based on the open source PR, since it never gets merged into spark-avro. we
propose this feature since it should apply
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/21339
[SPARK-24287][Core] Spark -packages option should support classifier,
no-transitive, and custom conf
## What changes were proposed in this pull request?
We should extend Spark's -package
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21310
@viirya @cloud-fan before I add test, could you guys take a look and
advise if the approach taken in this patch is acceptable
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21310
I will investigate how can we add test for this. thoughts are welcomed
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/21310
[SPARK-24256][SQL] SPARK-24256: ExpressionEncoder should support
user-defined types as fields of Scala case class and tuple
## What changes were proposed in this pull request?
Right now
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21276
made a few changes:
since getSimpleName is a method of Class, it makes sense for
Utils.getSimpleName to takes a Class param
Utils.getSimpleName will try to use the Class's getSimpleName
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187540911
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2715,6 +2715,66 @@ private[spark] object Utils extends Logging
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187540472
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -1168,6 +1168,35 @@ class UtilsSuite extends SparkFunSuite
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187540452
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2715,6 +2715,66 @@ private[spark] object Utils extends Logging
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187518454
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2715,6 +2715,66 @@ private[spark] object Utils extends Logging
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21276
>> Do we have other places in Spark calling getSimpleName which also need
this change?
there are a few other places that may have similar issue. will update
to
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187271712
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -1168,6 +1168,32 @@ class UtilsSuite extends SparkFunSuite
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187263297
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala
---
@@ -109,7 +110,8 @@ trait
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/21276#discussion_r187263253
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -1168,6 +1168,23 @@ class UtilsSuite extends SparkFunSuite
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21276
updating
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/21276
[SPARK-24216][SQL] Spark TypedAggregateExpression uses getSimpleName that
is not safe in scala
## What changes were proposed in this pull request?
When we create a aggregator object
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/20931#discussion_r179912635
--- Diff:
core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
@@ -186,7 +186,9 @@ class
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/20931#discussion_r179672722
--- Diff:
core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
@@ -186,7 +186,9 @@ class
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/20931#discussion_r179517200
--- Diff:
core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
@@ -186,7 +186,9 @@ class
Github user fangshil commented on a diff in the pull request:
https://github.com/apache/spark/pull/20931#discussion_r179346911
--- Diff:
core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
@@ -186,7 +186,9 @@ class
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/20974
[SPARK-23862][SQL] Spark ExpressionEncoder should support java enum type in
scala
## What changes were proposed in this pull request?
In SPARK-21255, spark upstream adds support
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/20931
@sujithjay the test cases added in SPARK-20236 already covers the
scenario(multi partitioning) in this patch. The added unit tests work fine with
local file system, and the bug we reported only
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/20931
[SPARK-23815][Core]Spark writer dynamic partition overwrite mode may fail
to write output on multi level partition
## What changes were proposed in this pull request?
Spark
28 matches
Mail list logo
