Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19810
@maropu , this looks rather cold :sunglasses: , but extremely interesting
and relevant.
---
-
To unsubscribe, e-mail: reviews
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19810#discussion_r218743585
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19810#discussion_r218716111
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19810#discussion_r218714493
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/CachedColumnarRDD.scala
---
@@ -0,0 +1,99 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19810#discussion_r218705219
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala
---
@@ -193,38 +195,68 @@ case class
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19810#discussion_r218704390
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala
---
@@ -52,6 +52,68 @@ object InMemoryRelation
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217902092
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217901988
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/17400#discussion_r211565064
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
---
@@ -321,6 +321,58 @@ case class EnsureRequirements
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/17400
thanks @maropu I appreciate this.
must say I'm pretty surprised a bug like that lives so long...
---
-
To unsubscribe, e-mail
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/17400
in my use case, I aggregate a dataset, the use select to align columns with
a case-class. I later try to join the resulting dataset based on the same
columns used for aggregattion.
the join
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/17400
@maropu , yes it does :-)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/17400#discussion_r209909106
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
---
@@ -321,6 +321,58 @@ case class EnsureRequirements
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/17400
@maropu , any reason why this is on hold for so long?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r209499005
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
@hvanhovell ,thanks for picking this up :sunglasses:
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r198816624
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r192631230
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r191655937
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r191077231
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190829341
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,7 +415,106 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
@cloud-fan , i guess we can copy the test util from scala as long as it
doesnt violet licensing or credits, please advice on the proper way of
doingbthat and where do u want to place the code? Does
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190542635
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190375595
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190372506
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190371425
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +591,25 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190370842
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r190370765
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189921783
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189919617
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +591,25 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189919031
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -267,7 +273,7 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
@advancedxy , using jvisualvm+heap dump I could see that the second
introduced test case ("drop all references to the underlying map once the
iterator is exhausted") eliminated all
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189794281
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +591,24 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189794046
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -305,8 +310,8 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189794097
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +591,24 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
well, I took the time trying to figure out how's the iterator is eventually
being used,
(most of) it boils down to
`org.apache.spark.scheduler.ShuffleMapTask#runTask` which does
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189452094
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189438351
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/21369
@lianhuiwang, @davies, @hvanhovell can you please have a look?
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user eyalfa opened a pull request:
https://github.com/apache/spark/pull/21369
[SPARK-22713][CORE] ExternalAppendOnlyMap leaks when spilled during
iteration
## What changes were proposed in this pull request?
This PR solves
[SPARK-22713](https://issues.apache.org/jira
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19054#discussion_r165861581
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
---
@@ -220,45 +220,99 @@ case class EnsureRequirements
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19054#discussion_r165860433
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
---
@@ -220,45 +220,99 @@ case class EnsureRequirements
Github user eyalfa closed the pull request at:
https://github.com/apache/spark/pull/19481
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19481
@hvanhovell , @juliuszsompolski
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user eyalfa opened a pull request:
https://github.com/apache/spark/pull/19481
[SPARK-21907][CORE][BACKPORT 2.2] oom during spill
back-port #19181 to branch-2.2.
1. a test reproducing
[SPARK-21907](https://issues.apache.org/jira/browse/SPARK-21907)
2. a fix
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@hvanhovell , thanks :+1:
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r142005517
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java
---
@@ -503,6 +511,39 @@ public void
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@hvanhovell ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@hvanhovell , PTAL
let me now if there's anything that requires fixing here.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r139594335
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java
---
@@ -503,6 +511,39 @@ public void
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
I agree, but 'it is what it is'ð
We can probably come up with some mechanism that detects such scenarios and
avoids invoking the spill method on an object whose already 'on the stack
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@juliuszsompolski,
if you comment the few added lines in the reset() method, you'd see that
the test fails with a stack frame very similar to the one you pasted in JIRA
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@hvanhovell, @juliuszsompolski,
failure seem unrelated to my work, can you please request a retest?
---
-
To unsubscribe, e
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r138751839
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java
---
@@ -503,6 +511,47 @@ public void
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r138751518
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
---
@@ -85,7 +85,7 @@
private final LinkedList
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/19181#discussion_r138373142
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
---
@@ -170,6 +170,10 @@ public void free
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/19181
@ericl , @davies ,
you guys seem to be the last ones to edit this area of the code, I'd
appreciate if you could take a look
GitHub user eyalfa opened a pull request:
https://github.com/apache/spark/pull/19181
[SPARK-21907][CORE] oom during spill
## What changes were proposed in this pull request?
1. a test reproducing
[SPARK-21907](https://issues.apache.org/jira/browse/SPARK-21907)
2. a fix
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@cloud-fan , @vanzin ,
any idea what happened to this build? seem environment issue after a
successful build (0 failed tests, 'Build step 'Execute shell' marked build as
failure')
can one
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
Funny enough, that's the approach I've chosen.
On Aug 15, 2017 19:17, "Marcelo Vanzin" <notificati...@github.com> wrote:
> *@vanzin* commented o
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r133180495
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r133144224
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r133141988
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r133135659
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r132978316
--- Diff: project/SparkBuild.scala ---
@@ -790,7 +790,7 @@ object TestSettings {
javaOptions in Test ++=
System.getProperties.asScala.filter(_._1
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@cloud-fan I think I found the sbt setting that controlled max heap size
for forked tests, I've increased it from 3g to 6g.
cc: @srowen, @vanzin and @a-roberts you guys seem to be the last ones
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
Agreed, though I can't really understand it...
Ran SBT locally with --mem 12000 and provided the magical jvm flag that
prints jvm provided CLI args, it seems SBT is indeed running with 12gb
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@cloud-fan, any idea how much memory is allocated for running sbt?
currently one of my newly introduced tests fails on OOM during kryo
serialization...
it's actually a bit weird
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@cloud-fan , can you please give the 'ok to test'?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131914883
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131913940
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131912886
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131912438
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131758308
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131713763
--- Diff: core/src/test/scala/org/apache/spark/storage/DiskStoreSuite.scala
---
@@ -92,6 +92,31 @@ class DiskStoreSuite extends SparkFunSuite
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131712834
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@HyukjinKwon , interesting reading but I couldn't find a concrete reason or
solution to the issue.
@cloud-fan, I've encountered this bug when working with a disk persisted
RDD, it turned out
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@kiszk , fixed styling+readability according to your comments.
BTW, any idea why JIRA didn't associate this PR with SPARK-3151?
---
If your project is set up for it, you can reply to this email
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543591
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/18855#discussion_r131543602
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -165,6 +147,62 @@ private[spark] class DiskStore
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/18855
@rxin, @JoshRosen , @cloud-fan ,
you seem to be the last guys to touch this class, can you please review?
---
If your project is set up for it, you can reply to this email and have your
reply
GitHub user eyalfa opened a pull request:
https://github.com/apache/spark/pull/18855
[Spark 3151][Block Manager] DiskStore.getBytes fails for files larger than
2GB
## What changes were proposed in this pull request?
introduced `DiskBlockData`, a new implementation of `BlockData
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r99475332
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
---
@@ -0,0 +1,499 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r99475106
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
---
@@ -0,0 +1,499 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r99475096
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
---
@@ -0,0 +1,499 @@
+/*
+ * Licensed
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/16043
@HyukjinKwon, @hvanhovell, are you familiar with this build failure? seems
to be unrelated to my specific build...
---
If your project is set up for it, you can reply to this email and have your
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98660310
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,166 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98660085
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -293,6 +293,12 @@ object SimplifyConditionals extends
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/16043
@hvanhovell can you figure out what fail the build? seems all tests passed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/16043
@hvanhovel, currently in the process of refactoring,luckily I've introduced
some tests in the initial sprint so I've cought my mistakes and refreshed
my mind around the assumptions this code
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98222460
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98216632
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98193617
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98192298
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
---
@@ -0,0 +1,482 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98192112
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,131 @@
+/*
+ * Licensed
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/16043#discussion_r98191785
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed
1 - 100 of 193 matches
Mail list logo
