Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r208770874
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -78,6 +79,25 @@ class AvroSuite extends QueryTest
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206746980
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +182,118 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206740630
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +182,118 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206737762
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +186,117 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206691928
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -120,7 +133,7 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206691906
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -87,17 +87,30 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r206607145
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +182,118 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on the issue:
https://github.com/apache/spark/pull/21847
RE Performance regression... The test case I used performs the same before
the changes in this patch and after, so this doesn't appear related to this
change. I'll investigate further and file
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912718
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -87,17 +88,33 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912517
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,112 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912461
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,112 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912386
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,112 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205912358
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -148,7 +165,8 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on the issue:
https://github.com/apache/spark/pull/21847
@gengliangwang I think I agree with your take on the multi-type unions in
principal. The only issue is that avro itself does support that as a valid use
case; however, when I think about
Github user lindblombr commented on the issue:
https://github.com/apache/spark/pull/21847
FYI Doing some additional testing around performance and have found a
pretty gnarly regression with a particular type of schema. I'll try to track
down what's causing
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205592450
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,100 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205591619
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -40,12 +41,40 @@ import org.apache.spark.sql.types._
class
Github user lindblombr commented on the issue:
https://github.com/apache/spark/pull/21847
@MaxGekk thanks for the feedback! Should be addressed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205591400
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -165,16 +183,100 @@ class AvroSerializer(rootCatalystType
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205568259
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -40,12 +41,40 @@ import org.apache.spark.sql.types._
class
Github user lindblombr commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r205505006
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -41,7 +42,7 @@ class AvroSerializer(rootCatalystType
GitHub user lindblombr opened a pull request:
https://github.com/apache/spark/pull/21847
[SPARK-24855][SQL][EXTERNAL][WIP]: Built-in AVRO support should support
specified schema on write
## What changes were proposed in this pull request?
Allows `avroSchema` option
22 matches
Mail list logo
