[GitHub] spark pull request #22729: [SPARK-25737][CORE] Remove JavaSparkContextVararg...

2018-10-15 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22729 [SPARK-25737][CORE] Remove JavaSparkContextVarargsWorkaround ## What changes were proposed in this pull request? Remove JavaSparkContextVarargsWorkaround ## How was this patch

[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22727#discussion_r225179121 --- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala --- @@ -71,6 +71,12 @@ object

[GitHub] spark issue #22714: [SPARK-25720][WEBUI] Support auto refresh page for the W...

2018-10-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22714 Shouldn't it go in `commonHeaderNodes`? Looks like this was added waaay back in https://github.com/JoshRosen/spark/commit/6aa08c39cf30fa5c4ed97f4fff16371b9030a2e6 by @tdas but never

[GitHub] spark issue #22706: [SPARK-25716][SQL][MINOR] remove unnecessary collection ...

2018-10-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22706 It makes some sense, but how much difference does it make, performance-wise? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22705: [SPARK-25704][CORE][WIP] Allocate a bit less than...

2018-10-13 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22705#discussion_r224965160 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -195,7 +196,11 @@ object ChunkedByteBuffer { val is = new

[GitHub] spark pull request #22383: [SPARK-25362][JavaAPI] Replace Spark Optional cla...

2018-10-13 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22383#discussion_r224964998 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,8 @@ object MimaExcludes { // Exclude rules for 3.0.x lazy val v30excludes

[GitHub] spark pull request #22383: [SPARK-25362][JavaAPI] Replace Spark Optional cla...

2018-10-13 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22383#discussion_r224965080 --- Diff: project/MimaExcludes.scala --- @@ -36,9 +36,11 @@ object MimaExcludes { // Exclude rules for 3.0.x lazy val v30excludes

[GitHub] spark pull request #22383: [SPARK-25362][JavaAPI] Replace Spark Optional cla...

2018-10-13 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22383#discussion_r224965113 --- Diff: project/MimaExcludes.scala --- @@ -36,9 +36,11 @@ object MimaExcludes { // Exclude rules for 3.0.x lazy val v30excludes

[GitHub] spark pull request #22703: [SPARK-25705][BUILD][STREAMING] Remove Kafka 0.8 ...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22703#discussion_r224936431 --- Diff: docs/streaming-kafka-0-10-integration.md --- @@ -3,7 +3,11 @@ layout: global title: Spark Streaming + Kafka Integration Guide (Kafka broker

[GitHub] spark issue #22414: [SPARK-25424][SQL] Window duration and slide duration wi...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22414 Yeah, the test that failed here asserts that it's an `AnalysisException`. I guess it could be removed. The thing is, many other cases are still handled as `AnalysisException`. Maybe it'

[GitHub] spark issue #22690: [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMapValues...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22690 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22670: [SPARK-25631][SPARK-25632][SQL][TEST] Improve the test r...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22670 I don't so much mean that much refactoring. I wonder if there are 1-2 other places where common Kafka params are set in tests that we could add this to for now, that kind of thing. This change

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & formatting for JavaDoc

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22322: [SPARK-25312][Documentation, Spark Core] Add description...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22322 Ping @npoberezkin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22670: [SPARK-25631][SPARK-25632][SQL][TEST] Improve the test r...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22670 @dilipbiswal I like this change too. The suite goes from 4:34 to 0:53. I wonder if we can make this change elsewhere in general Kafka test config? This kind of setting seems useful everywhere

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r224874828 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,30 @@ private[spark] class MemoryStore

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r224875899 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1930,6 +1930,18 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request #21322: [SPARK-24225][CORE] Support closing AutoClosable ...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21322#discussion_r224875111 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -384,15 +385,30 @@ private[spark] class MemoryStore

[GitHub] spark pull request #22703: [SPARK-25705][BUILD][STREAMING] Remove Kafka 0.8 ...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22703#discussion_r224870517 --- Diff: python/pyspark/streaming/tests.py --- @@ -1047,259 +1046,6 @@ def check_output(n): self.ssc.stop(True, True) -class

[GitHub] spark issue #22689: [SPARK-25697][CORE]When zstd compression enabled, InProg...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22689 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22383: [SPARK-25362][JavaAPI] Replace Spark Optional cla...

2018-10-12 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22383#discussion_r224866566 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,8 @@ object MimaExcludes { // Exclude rules for 3.0.x lazy val v30excludes

[GitHub] spark issue #22700: [SPARK-25712][Core][Minor] Improve usage message of star...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22700 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22678: [SPARK-25685][BUILD] Allow running tests in Jenkins in e...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22678 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22657: [SPARK-25670][TEST] Reduce number of tested timezones in...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22657 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22645: [SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22645 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-12 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21588 I know this is probably just reviving an old thread elsewhere, but, we don't know how to update our 1.2.1 Hive fork anyway, it seems? if so, and the fork is undesirable, seems like time to dr

[GitHub] spark pull request #22703: [SPARK-25705][BUILD][STREAMING] Remove Kafka 0.8 ...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22703#discussion_r224621015 --- Diff: python/pyspark/streaming/tests.py --- @@ -1047,259 +1046,6 @@ def check_output(n): self.ssc.stop(True, True) -class

[GitHub] spark pull request #22703: [SPARK-25705][BUILD][STREAMING] Remove Kafka 0.8 ...

2018-10-11 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22703 [SPARK-25705][BUILD][STREAMING] Remove Kafka 0.8 integration ## What changes were proposed in this pull request? Remove Kafka 0.8 integration ## How was this patch tested

[GitHub] spark pull request #22701: [SPARK-25690][SQL] Analyzer rule HandleNullInputs...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22701#discussion_r224620279 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2150,8 +2150,10 @@ class Analyzer

[GitHub] spark issue #22692: [SPARK-25598][STREAMING][BUILD][test-maven] Remove flume...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22692 Merged to master so I can get on with removing Kafka 0.8 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22594: [SPARK-25674][SQL] If the records are incremented by mor...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22594 Merged to master/2.4/2.3 as a clean simple bug fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224607076 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark pull request #22701: [SPARK-25690][SQL] Analyzer rule HandleNullInputs...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22701#discussion_r224606888 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -351,8 +351,8 @@ class AnalysisSuite extends

[GitHub] spark issue #22692: [SPARK-25598][STREAMING][BUILD][test-maven] Remove flume...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22692 Passes with Maven and SBT, and sounds like broad support. It's clean to remove, so I'll go ahead for 3.0 --- - To unsu

[GitHub] spark issue #22671: [SPARK-25615][SQL][TEST] Improve the test runtime of Kaf...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22671 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22691: [SPARK-24109][CORE] Remove class SnappyOutputStreamWrapp...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22691 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22701: [SPARK-25690][SQL] Analyzer rule HandleNullInputs...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22701#discussion_r224547656 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -351,8 +351,8 @@ class AnalysisSuite extends

[GitHub] spark issue #22699: [SPARK-25711][Core] Allow start-history-server.sh to sho...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22699 Agree that's a good idea --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224540341 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224540330 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224534401 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Huh, I'm really confused why this would fail, or at least, start failing right now. We use these HTML tags elsewhere. You could try updating the unidoc plugin version, but I think it's a

[GitHub] spark issue #22657: [SPARK-25670][TEST] Reduce number of tested timezones in...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22657 Yeah, I could see the argument both ways for keeping all the tests in CastSuite or just checking a subset. We already got the test down considerably, though it's still like 24 seconds. Is ther

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Hm, that's a weird error. Big javadoc failures from unrelated classes. This looks like errors you get when you run javadoc on translated Scala classes. No idea why it's just popped up.

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224512333 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark issue #22695: [MINOR][SQL]remove Redundant semicolons

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22695 OK, but this kind of thing isn't worth opening a PR for. If you can maybe get some related minor changes together, that

[GitHub] spark issue #22383: [SPARK-25362][JavaAPI] Replace Spark Optional class with...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22383 Yeah, you'll have to add this to the 3.0 excludes section of project/MimaExcludes: `ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.api.jav

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-11 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224506066 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22318 Oh I see there was indeed more discussion on this, and it does relate to resolving columns to joined DataFrames. I don't know enough to bless this change, but it seems reasonable. @maropu app

[GitHub] spark issue #22383: [SPARK-25395][JavaAPI] Replace Spark Optional class with...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22383 Oops, I mean SPARK-25362. SPARK-25395 was a duplicate. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22690: [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMapValues...

2018-10-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22690 Yes, for 3.0. it's an old API mistake --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional com

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-10-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r224263116 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF

[GitHub] spark pull request #22692: [SPARK-25598][STREAMING][BUILD] Remove flume conn...

2018-10-10 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22692 [SPARK-25598][STREAMING][BUILD] Remove flume connector in Spark 3 ## What changes were proposed in this pull request? Removes all vestiges of Flume in the build, for Spark 3. I don&#

[GitHub] spark pull request #22691: [SPARK-24109][CORE] Remove class SnappyOutputStre...

2018-10-10 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22691 [SPARK-24109][CORE] Remove class SnappyOutputStreamWrapper ## What changes were proposed in this pull request? Remove SnappyOutputStreamWrapper and other workaround now that new Snappy

[GitHub] spark pull request #22690: [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMa...

2018-10-10 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22690 [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMapValues requires function returning Iterable, not Iterator ## What changes were proposed in this pull request? Fix old oversight in API

[GitHub] spark pull request #22383: [SPARK-25395][JavaAPI] Replace Spark Optional cla...

2018-10-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22383#discussion_r224228243 --- Diff: core/src/test/java/test/org/apache/spark/JavaAPISuite.java --- @@ -476,10 +476,10 @@ public void leftOuterJoin() { new Tuple2<>

[GitHub] spark issue #22689: [SPARK-25697][CORE]When zstd compression enabled, InProg...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22689 I guess that doing nothing is better than an error screen. Is it possible to just skip reading incomplete files here? I don't know this code well. That sounds b

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 Merged to master. Note that the master hadoop 2.6 job will fail immediately now, so ignore it. On the upside ... this job already wont' take much of any time from the Jenkins cl

[GitHub] spark issue #22689: [SPARK-25697][CORE]When zstd compression enabled, InProg...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22689 Should the Event Log be available for running apps? Or if it's not going to work, disable it where it can't be shown, but I suppose that could be difficult. This just silently sends you b

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 I tried a release build that causes `--pip` and `--r` to be set, and the result looked OK. Both pyspark and R packages built and seemed normal. The source build worked too and comes before binary

[GitHub] spark pull request #22657: [SPARK-25670][TEST] Reduce number of tested timez...

2018-10-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22657#discussion_r224151216 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -106,4 +107,14 @@ abstract class SparkFunSuite

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Although it's the Scala API, it's callable from Java just as well. There's no Java-specific API here. So, yeah, actually it makes sense to have javadoc and scaladoc for this. And I

[GitHub] spark pull request #22594: [SPARK-25674][SQL] If the records are incremented...

2018-10-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22594#discussion_r224146203 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala --- @@ -70,6 +70,8 @@ class FileScanRDD

[GitHub] spark issue #22641: [SPARK-25611][SPARK-25612][SQL][TESTS] Improve test run ...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22641 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22671: [SPARK-25615][SQL][TEST] Improve the test runtime of Kaf...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22671 I like the change. This test is down to 5 seconds now. Unfortunately I don't see speedup in other Kafka tests, but I think we should leave th

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...

2018-10-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Have a look at `Column.scala` and `Dataset.scala` in `org.apache.spark.sql`. But, on a second look, this is how I see the lists render: https://user-images.githubusercontent.com/822522

[GitHub] spark pull request #21816: [SPARK-24794][CORE] Driver launched through rest ...

2018-10-10 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21816#discussion_r224128493 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala --- @@ -83,6 +83,26 @@ class StandaloneRestSubmitSuite extends

[GitHub] spark pull request #22414: [SPARK-25424][SQL] Window duration and slide dura...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22414#discussion_r223861364 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimeWindow.scala --- @@ -137,16 +139,44 @@ object TimeWindow

[GitHub] spark issue #21524: [SPARK-24212][ML][doc] Add the example and user guide fo...

2018-10-09 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21524 Ping @tengpeng to update or close --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21858: [SPARK-24899][SQL][DOC] Add example of monotonically_inc...

2018-10-09 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21858 Ping @jaceklaskowski to update or close --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22671: [SPARK-25615][SQL][TEST] Improve the test runtime...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22671#discussion_r223776123 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSinkSuite.scala --- @@ -332,7 +332,9 @@ class KafkaSinkSuite extends

[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...

2018-10-09 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22593 Ping @niofire to update or close --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22641: [SPARK-25611][SPARK-25612][SQL][TESTS] Improve te...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22641#discussion_r223756260 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/CompressionCodecSuite.scala --- @@ -262,7 +261,10 @@ class CompressionCodecSuite extends

[GitHub] spark pull request #22671: [SPARK-25615][SQL][TEST] Improve the test runtime...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22671#discussion_r223752722 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSinkSuite.scala --- @@ -332,7 +332,9 @@ class KafkaSinkSuite extends

[GitHub] spark pull request #22654: [SPARK-25660][SQL] Fix for the backward slash as ...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22654#discussion_r223751904 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtils.scala --- @@ -97,23 +97,22 @@ object CSVUtils

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-09 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 @felixcheung regarding building PIP and R in one release, yeah I was wondering that too. Ideally it would just be one. If the build changes only affect the source release, that's OK, as th

[GitHub] spark pull request #22615: [SPARK-25016][BUILD][CORE] Remove support for Had...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22615#discussion_r223729888 --- Diff: docs/index.md --- @@ -30,9 +30,6 @@ Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark {{s uses Scala

[GitHub] spark pull request #22615: [SPARK-25016][BUILD][CORE] Remove support for Had...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22615#discussion_r223729095 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -71,7 +71,7 @@ class HadoopTableReader( // Hadoop honors

[GitHub] spark pull request #22675: [SPARK-25347][ML][DOC] Spark datasource for image...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22675#discussion_r223728031 --- Diff: docs/ml-datasource.md --- @@ -0,0 +1,51 @@ +--- +layout: global +title: Data sources +displayTitle: Data sources

[GitHub] spark pull request #22654: [SPARK-25660][SQL] Fix for the backward slash as ...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22654#discussion_r223726757 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtils.scala --- @@ -97,23 +97,22 @@ object CSVUtils

[GitHub] spark pull request #22654: [SPARK-25660][SQL] Fix for the backward slash as ...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22654#discussion_r223724099 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -1826,4 +1826,13 @@ class CSVSuite extends QueryTest

[GitHub] spark pull request #22676: [SPARK-25684][SQL] Organize header related codes ...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22676#discussion_r223722902 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVHeaderChecker.scala --- @@ -0,0 +1,131 @@ +/* + * Licensed to

[GitHub] spark pull request #22675: [SPARK-25347][ML][DOC] Spark datasource for image...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22675#discussion_r223720371 --- Diff: docs/ml-datasource.md --- @@ -0,0 +1,51 @@ +--- +layout: global +title: Data sources +displayTitle: Data sources

[GitHub] spark pull request #22675: [SPARK-25347][ML][DOC] Spark datasource for image...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22675#discussion_r223720808 --- Diff: docs/ml-datasource.md --- @@ -0,0 +1,51 @@ +--- +layout: global +title: Data sources +displayTitle: Data sources

[GitHub] spark pull request #22675: [SPARK-25347][ML][DOC] Spark datasource for image...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22675#discussion_r223720105 --- Diff: docs/ml-datasource.md --- @@ -0,0 +1,51 @@ +--- +layout: global +title: Data sources +displayTitle: Data sources

[GitHub] spark pull request #22675: [SPARK-25347][ML][DOC] Spark datasource for image...

2018-10-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22675#discussion_r223719759 --- Diff: docs/ml-datasource.md --- @@ -0,0 +1,51 @@ +--- +layout: global +title: Data sources +displayTitle: Data sources

[GitHub] spark issue #22659: [SPARK-25623][SPARK-25624][SPARK-25625][TEST] Reduce tes...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22659 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 Yeah this does need to be in a public repo. apache/spark-jenkins-configurations or something. We can ask INFRA to create them. But, I'm not against just putting them in dev/ or something in the

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 I guess we've just pinged @shaneknapp ! But I figured the jobs would simply fail and could be removed at leisure. Yes, this mechanism is a little weird but may be the simplest thing he

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223491937 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile var

[GitHub] spark issue #21755: Doc fix: The Imputer is an Estimator

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21755 Minor stuff: we usually tag this with `[MINOR]` in the title to be clear there's no JIRA. Also ideal to batch together small related changes but I don't know that there was anything else

[GitHub] spark issue #22672: [SPARK-25605][TESTS] Alternate take. Run cast string to ...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22672 The original change in #22631 made the test time go down from about 2:30 to 0:17. See build 96945: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96945/testReport/junit

[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22649 This kind of stuff did fail when we were updating for 2.12 and we had to make a lot of similar changes to the Java code for this reason, yeah

[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22649 The pull request builder runs Scala 2.11, and this only becomes ambiguous in 2.12 (long story). For now 2.12 is still a secondary build. I suspect we'll switch it to be the primary scala ve

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22623 Just to check my understanding, `exitedCleanly` is `false` even when the expected exception is thrown? OK that makes sense

[GitHub] spark issue #22595: [SPARK-25577][Web UI] Add an on-off switch to display th...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22595 CC @jerryshao for https://github.com/apache/spark/pull/14617 where this was added. It looks like the display is on purpose, but can you clarify? I don't think a "show additional co

[GitHub] spark issue #21816: [SPARK-24794][CORE] Driver launched through rest should ...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21816 Ping @srinathshankar @ericl again for comments? I don't know this well, but seems like a low risk change at worst. --- ---

[GitHub] spark pull request #22466: [SPARK-25464][SQL] Create Database to the locatio...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22466#discussion_r223395695 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -207,6 +207,16 @@ class SessionCatalog

[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22637 OK, trying this again. Tests have definitely run this time and we've had another good pass at small review changes. --- ---

[GitHub] spark pull request #22641: [SPARK-25611][SPARK-25612][SQL][TESTS] Improve te...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22641#discussion_r223392700 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/CompressionCodecSuite.scala --- @@ -262,7 +261,10 @@ class CompressionCodecSuite extends

<    2   3   4   5   6   7   8   9   10   11   >