[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450927238 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450777350 OK, so you are suggesting increasing the heap size there just because it currently fails sometimes? that's fine too, I can also make that change separately. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450776101 Oh right I understand now. Well, I think I'm going to revert that change now anyway. When I made a change to avoid DirectByteBuffer without a Cleaner, it passed again with 3g on Java 11. I will merge this then when it passes again and follow up with my fix separately. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450772125 Following my comment at https://github.com/apache/spark/pull/23419#discussion_r244610615 -- I don't think that's the explanation. The test itself fails on allocating a big int array, which itself is not related to any Java 11 ByteBuffer changes. It may be that this test is just where a huge allocation takes place and where the higher memory usage causes a problem. I am worried that being unable to set a `Cleaner` on `DirectByteBuffer` is potentially causing a memory leak, and need to revisit the change I mentioned above. This change itself ought to be safe and fine even if we later change the memory limit back down, but I'm first going to try some other changes locally to confirm/deny. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450769651 OK that's good news; maybe the tests really all do pass now :) We'll know better when JDK 11 is available on Jenkins. @felixcheung Looks like `-Xmx4g` works locally for the TimSort test, so I pushed that here (obviously it was going to pass here either way). For reference this was the failure it fixes: ``` SorterSuite: - equivalent to Arrays.sort - KVArraySorter *** RUN ABORTED *** java.lang.OutOfMemoryError: Java heap space at org.apache.spark.util.collection.TestTimSort.createArray(TestTimSort.java:56) at org.apache.spark.util.collection.TestTimSort.getTimSortBugTestSet(TestTimSort.java:43) at org.apache.spark.util.collection.SorterSuite.$anonfun$new$8(SorterSuite.scala:70) at org.apache.spark.util.collection.SorterSuite$$Lambda$2204/0x000800a5f840.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11
srowen commented on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450738213 @dongjoon-hyun by the way, how did you test manually? I find we still can't build Spark with Java 11 (Scala itself and some build tools don't seem to be ready yet) but can run Spark tests with Java 11. (We will need to continue to build with Java 8 I think to ensure it runs on Java 8, so that's not something we have to 'fix') I stopped at getting all tests up to but not including SQL working, as I was getting some possibly-flaky test failures there. Did you see it run SQL or catalyst tests successfully? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
