Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227437560
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1788,79 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227420673
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,72 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227398926
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,72 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227388793
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,75 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227387167
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,72 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227358174
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,75 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227358278
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,75 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227353342
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,75 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227350272
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -1787,3 +1787,75 @@ case class
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227109203
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -278,24 +278,20 @@ object JavaTypeInference
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r227016344
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -278,24 +278,20 @@ object JavaTypeInference
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r226965648
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -26,7 +26,7 @@ import scala.language.existentials
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r226877628
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -26,7 +26,7 @@ import scala.language.existentials
Github user vofque closed the pull request at:
https://github.com/apache/spark/pull/22767
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user vofque closed the pull request at:
https://github.com/apache/spark/pull/22768
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22768#discussion_r226456443
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software
GitHub user vofque opened a pull request:
https://github.com/apache/spark/pull/22768
[SPARK-21402][SQL][BACKPORT-2.2] Fix java array of structs deserialization
This PR is to backport #22708 to branch 2.2.
## What changes were proposed in this pull request
GitHub user vofque opened a pull request:
https://github.com/apache/spark/pull/22767
[SPARK-21402][SQL][BACKPORT-2.3] Fix java array of structs deserialization
This PR is to backport #22708 to branch 2.3.
## What changes were proposed in this pull request
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22745
Added these classes:
_UnresolvedGetArrayFromMap_ in unresolved.scala - an unresolved
substitution.
_GetArrayFromMap_ in complexTypeExtractors.scala - an extraction algorithm
(not sure
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r226187880
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -509,3 +509,24 @@ case class UnresolvedOrdinal(ordinal
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22708
@dongjoon-hyun, sure, I'll create equal pull requests.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225840666
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithMapSuite.java ---
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225826366
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22708
Fixed all issues in test class. Thanks a lot for your help and patience.
Rebase was probably a bad idea, sorry
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225813918
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22745#discussion_r225810882
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithMapSuite.java ---
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225810177
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225800282
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanWithArraySuite.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225799822
--- Diff: sql/core/src/test/resources/test-data/with-array-fields ---
@@ -0,0 +1,3 @@
+{ "id": 1, "intervals": [{ "startT
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225505036
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -271,32 +272,41 @@ object JavaTypeInference
GitHub user vofque opened a pull request:
https://github.com/apache/spark/pull/22745
[SPARK-21402][SQL][FOLLOWUP] Fix java map of structs deserialization
When deserializing values of MapType with struct keys/values in java beans,
fields of structs get mixed up.
I suggest using
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22708
Corrected all issues.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225451652
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -30,6 +30,7 @@ import
Github user vofque commented on a diff in the pull request:
https://github.com/apache/spark/pull/22708#discussion_r225121081
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
---
@@ -271,32 +272,41 @@ object JavaTypeInference
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22708
In a nutshell:
First we calculate field ordinals of array elements using java bean field
information (which doesn't guarantee any particular field order).
Then we apply these ordinals
Github user vofque commented on the issue:
https://github.com/apache/spark/pull/22708
The original problem is described here:
https://issues.apache.org/jira/browse/SPARK-21402
I'll try to explain what happens in detail.
Let's consider this data structure:
root
GitHub user vofque opened a pull request:
https://github.com/apache/spark/pull/22708
[Spark 21402] Fix java array/map of structs deserialization
When deserializing values of ArrayType with struct elements or MapType with
struct keys/values in java beans, fields of structs get mixed
37 matches
Mail list logo
