[GitHub] spark issue #22838: [SPARK-25835][K8s] Create kubernetes-tests profile and u...

2018-10-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22838 I merged to master/2.4 . The key parts here already passed individually. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22852#discussion_r228536180 --- Diff: docs/security.md --- @@ -6,7 +6,20 @@ title: Security * This will become a table of contents (this text will be scraped). {:toc

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22852#discussion_r228536280 --- Diff: docs/security.md --- @@ -6,7 +6,20 @@ title: Security * This will become a table of contents (this text will be scraped). {:toc

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22852#discussion_r228535892 --- Diff: docs/security.md --- @@ -6,7 +6,20 @@ title: Security * This will become a table of contents (this text will be scraped). {:toc

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21588 So, let's say we decide to only support Hive 2.3.x+, as a precursor to this. We could already eliminate a lot of the Hive tests, right? that might be useful in its own right as they take tim

[GitHub] spark pull request #22815: [SPARK-25821][SQL] Remove SQLContext methods depr...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22815#discussion_r228396856 --- Diff: R/pkg/R/SQLContext.R --- @@ -434,6 +388,7 @@ read.orc <- function(path, ...) { #' Loads a Parquet file, returning the res

[GitHub] spark pull request #22838: [SPARK-25835][K8s] Create kubernetes-tests profil...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22838#discussion_r228363727 --- Diff: resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --- @@ -103,4 +104,4 @@ then properties=( ${properties

[GitHub] spark issue #22838: [SPARK-25835][K8s] Create kubernetes-tests profile and u...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22838 I'll spin it one more time just to try to get a green light, but I agree, this change really isn't even testable by the exis

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22840 Yeah ... I read that as being about the source tree from the repo. But it does say only "... like the ones distributed by the project". OK I think this is fin

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22840 Hm, I'm not dead-set against it, though it seems a little deceptive; you won't actually get the exact binary distro out, and that's what the script purports to do. (Arguably, why d

[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22840 Hm, but then you haven't created a completely valid binary release; it's missing licenses. As I say, I don't know that the binary release tarball must be exactly recreatable from the

[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...

2018-10-25 Thread srowen
Github user srowen closed the pull request at: https://github.com/apache/spark/pull/22829 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22838: [SPARK-25835][K8s] Create kubernetes-tests profil...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22838#discussion_r228311880 --- Diff: pom.xml --- @@ -2654,6 +2654,16 @@ kubernetes + +resource-managers/kubernetes/core

[GitHub] spark pull request #22838: [SPARK-25835][K8s] Create kubernetes-tests profil...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22838#discussion_r228316418 --- Diff: pom.xml --- @@ -2654,6 +2654,16 @@ kubernetes + +resource-managers/kubernetes/core

[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22829#discussion_r228285511 --- Diff: pom.xml --- @@ -2656,7 +2656,8 @@ kubernetes resource-managers/kubernetes/core -resource-managers

[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...

2018-10-25 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22829 [SPARK-25836][BUILD][K8S] For now disable kubernetes-integration-tests ## What changes were proposed in this pull request? For now make building and running kubernetes-integration-tests

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22723 BinaryFileRDD uses the `minPartitions` input from the user. See https://issues.apache.org/jira/browse/SPARK-22357 I think the logic is more complex and already takes some account of

[GitHub] spark pull request #22815: [SPARK-25821][SQL] Remove SQLContext methods depr...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22815#discussion_r228233269 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -54,6 +54,7 @@ import org.apache.spark.sql.util.ExecutionListenerManager

[GitHub] spark pull request #22815: [SPARK-25821][SQL] Remove SQLContext methods depr...

2018-10-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22815#discussion_r228232993 --- Diff: R/pkg/R/SQLContext.R --- @@ -434,6 +388,7 @@ read.orc <- function(path, ...) { #' Loads a Parquet file, returning the res

[GitHub] spark pull request #22826: [SPARK-25760][DOCS][FOLLOWUP] Add note about AddJ...

2018-10-25 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22826 [SPARK-25760][DOCS][FOLLOWUP] Add note about AddJar return value change in migration guide ## What changes were proposed in this pull request? Add note about AddJar return value change in

[GitHub] spark issue #22690: [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMapValues...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22690 Actually sorry for the ignorant question @HyukjinKwon but is there a migration guide for things outside SQL and MLlib? those are the two I've found. This one isn't specific to those two

[GitHub] spark issue #21816: [SPARK-24794][CORE] Driver launched through rest should ...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21816 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22690: [SPARK-19287][CORE][STREAMING] JavaPairRDD flatMapValues...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22690 Yeah let me go back and add a note about several recent changes like this. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22747: [SPARK-25760][SQL] Set AddJarCommand return empty

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22747 It seems like a bug fix more than anything, and I assume we wouldn't document every single one, but don't object to mentioning it if that's

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22803 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22803 Whoa, never seen a JVM crash like that! ``` [CodeBlob (0x7f6fc018add0)] Framesize: 2 Runtime Stub (0x7f6fc018add0): handle_exception_from_callee Runtime1 stub Could

[GitHub] spark issue #21933: [SPARK-24917][CORE] make chunk size configurable

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21933 Yeah this can be closed; we updated to 4.1.30 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22819: [BUILD] Close stale PRs

2018-10-24 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22819 [BUILD] Close stale PRs Closes #22567 Closes #18457 Closes #21517 Closes #21858 Closes #22383 Closes #19219 Closes #22401 Closes #22811 You can merge this pull request

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22803 OK, I ran an SBT test earlier and indeed it's kind of a glaring warning. I think I've changed my mind, if it does harmonize versions and avoid a wa

[GitHub] spark issue #22663: [SPARK-25490][SQL][TEST] Fix OOM of KryoBenchmark due to...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22663 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22815: [SPARK-25821][SQL] Remove SQLContext methods depr...

2018-10-24 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22815#discussion_r227972751 --- Diff: R/pkg/R/SQLContext.R --- @@ -343,7 +343,6 @@ setMethod("toDF", signature(x = "RDD"), #' path <- "path/to/f

[GitHub] spark issue #22729: [SPARK-25737][CORE] Remove JavaSparkContextVarargsWorkar...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22729 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22815: [SPARK-25821][SQL] Remove SQLContext methods depr...

2018-10-24 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22815 [SPARK-25821][SQL] Remove SQLContext methods deprecated in 1.4 ## What changes were proposed in this pull request? Remove SQLContext methods deprecated in 1.4 ## How was this patch

[GitHub] spark issue #18390: [SPARK-21178][ML] Add support for label specific metrics...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18390 If the label-specific metrics are already exposed in the multiclass metrics output, why do we need this too? --- - To

[GitHub] spark issue #22723: [SPARK-25729][CORE]It is better to replace `minPartition...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22723 I don't think that's necessarily true. This forces the default to be the minimum, which is a behavior change and not obviously what the

[GitHub] spark pull request #22729: [SPARK-25737][CORE] Remove JavaSparkContextVararg...

2018-10-24 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22729#discussion_r227818211 --- Diff: project/MimaExcludes.scala --- @@ -55,9 +59,12 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem

[GitHub] spark issue #22730: [SPARK-16775][CORE] Remove deprecated accumulator v1 API...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22730 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22144 @tgravescs yes, I didn't say it was normal or good, but that it's not forbidden. Ex: no more Java 7 support in Spark 2.3. The point was: if this change were on purpose, then no it's n

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22144 @markhamstra I do think there's an unspoken but legitimate consideration here, and that's that there is also a cost to not shipping the N thousand other things users are waiting on in th

[GitHub] spark issue #22668: [SPARK-25675] [Spark Job History] Job UI page does not s...

2018-10-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22668 (I did the same thing 2 weeks ago) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22144 So, some particular functionality worked in Spark 2.1, but didn't work starting in 2.2. If anything, that was the breaking change. (I wonder if it was on purpose, or documented, but, what'

[GitHub] spark issue #22668: [SPARK-25675] [Spark Job History] Job UI page does not s...

2018-10-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22668 @shivusondur @felixcheung looks like this test fails: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/5070/testReport

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227431599 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/docker/DockerForDesktopBackend.scala

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227431063 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/docker/DockerForDesktopBackend.scala

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227429686 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/docker/DockerForDesktopBackend.scala

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227429795 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227430300 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/docker/DockerForDesktopBackend.scala

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22803 What does this affect, though? those are just nullability annotations and don't result in behavior changes in the code, at all right? I don't know if it's worth managing them to t

[GitHub] spark issue #22796: [SPARK-25800][Build] Remove all instances of netty 3.x i...

2018-10-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22796 Yeah, maybe the better question is, what brings in this dependency? I took out the declaration and all exclusions for `io.netty:netty` and checked dependencies and it's basically Zookeeper (

[GitHub] spark pull request #22754: [SPARK-25776][CORE][MINOR]The disk write buffer s...

2018-10-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22754#discussion_r227168292 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java --- @@ -62,6 +62,8 @@ public UnsafeSorterSpillWriter

[GitHub] spark issue #22796: [SPARK-25800][Build] Remove all instances of netty 3.x i...

2018-10-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22796 Well I think we want to remove the direct dependency on 3.x right @dongjoon-hyun ? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22662: [SPARK-25627][TEST] Reduce test time for ContinuousStres...

2018-10-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22662 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22796: [SPARK-25800][Build] Remove all instances of nett...

2018-10-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22796#discussion_r227046597 --- Diff: dev/deps/spark-deps-hadoop-2.7 --- @@ -148,7 +148,7 @@ metrics-graphite-3.1.5.jar metrics-json-3.1.5.jar metrics-jvm-3.1.5.jar

[GitHub] spark issue #22784: [SPARK-25790][MLLIB] PCA: Support more than 65535 column...

2018-10-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22784 Hm, as a general comment, is this going to scale? This is making a potentially huge sparse data set dense, and computing a PCA via SVD. I get the idea that it's better to have some option than

[GitHub] spark pull request #22425: [SPARK-23367][Build] Include python document styl...

2018-10-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22425#discussion_r227001346 --- Diff: dev/tox.ini --- @@ -14,6 +14,8 @@ # limitations under the License. [pycodestyle] -ignore=E226,E241,E305,E402,E722,E731,E741

[GitHub] spark pull request #22425: [SPARK-23367][Build] Include python document styl...

2018-10-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22425#discussion_r227001114 --- Diff: dev/lint-python --- @@ -99,6 +104,29 @@ else echo "flake8 checks passed." fi +# Check python document style, ski

[GitHub] spark issue #22383: [SPARK-25362][JavaAPI] Replace Spark Optional class with...

2018-10-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22383 Yeah, I think we just can't do this unfortunately. It was worth looking into. --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

2018-10-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22765 That's probably fine. As an aside, I think we can remove all instances of Netty 3.x in the code base, if any, now that Flume is

[GitHub] spark issue #22757: [SPARK-24601][FOLLOWUP] Update Jackson to 2.9.6 in Kines...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22757 Merging this to master as a 'hotfix' for a pretty optional component --- - To unsubscribe, e-mail: review

[GitHub] spark pull request #22756: [SPARK-25758][ML] Deprecate computeCost on Bisect...

2018-10-17 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22756#discussion_r226023867 --- Diff: python/pyspark/ml/clustering.py --- @@ -335,20 +335,6 @@ def clusterCenters(self): """Get the cluster centers, repres

[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22732 At this point I think you both know more than I do about this, so go ahead. To the limits of my understanding it sounds reasonable

[GitHub] spark issue #22757: [SPARK-24601][FOLLOWUP] Update Jackson to 2.9.6 in Kines...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22757 CC @Fokko - unfortunately the build didn't catch this one because it didn't know enough to trigger Kinesis tests. --- --

[GitHub] spark pull request #22757: [SPARK-24601][FOLLOWUP] Update Jackson to 2.9.6 i...

2018-10-17 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22757 [SPARK-24601][FOLLOWUP] Update Jackson to 2.9.6 in Kinesis ## What changes were proposed in this pull request? Also update Kinesis SDK's Jackson to match Spark's ## Ho

[GitHub] spark issue #22729: [SPARK-25737][CORE] Remove JavaSparkContextVarargsWorkar...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22729 I think it's a legit failure. It touches a Kinesis test, and that has triggered what I assume is an existing problem in the Kinesis integration. It looks like it didn't like the fact that

[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22727 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22744: [SPARK-25741][WebUI] Long URLs are not rendered properly...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22744 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22732 That's a good point, but that was already an issue right? it isn't introduced by this change at least? --- - To unsu

[GitHub] spark issue #22753: [SPARK-25754][DOC] Change CDN for MathJax

2018-10-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22753 Merged to master/2.4/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-16 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225764876 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -81,11 +81,11 @@ case class UserDefinedFunction

[GitHub] spark issue #22745: [SPARK-21402][SQL][FOLLOW-UP] Fix java map of structs de...

2018-10-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22745 Is this a separate PR because this part is pretty separable, and you think could be considered separately? if it's all part of one logical change that should go in together or not at all, the

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-16 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225735714 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -81,11 +81,11 @@ case class UserDefinedFunction

[GitHub] spark issue #22670: [SPARK-25631][SPARK-25632][SQL][TEST] Improve the test r...

2018-10-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22670 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-16 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225693558 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -81,11 +81,11 @@ case class UserDefinedFunction

[GitHub] spark issue #22383: [SPARK-25362][JavaAPI] Replace Spark Optional class with...

2018-10-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22383 Oh, hm: ``` Serialization stack: - object not serializable (class: java.util.Optional, value: Optional[x]) - field (class: scala.Tuple2, name: _2, type: class

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-16 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225602236 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -73,27 +73,27 @@ case class UserDefinedFunction

[GitHub] spark issue #22747: [MINOR][SQL] Set AddJarCommand return empty

2018-10-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22747 Interesting, this was changed a long time ago, with the claim that it was needed to match Hive: https://github.com/apache/spark/pull/4586#discussion_r28394029 Maybe it's changed again? CC @a

[GitHub] spark issue #22747: [MINOR][SQL] Set AddJarCommand return empty

2018-10-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22747 I tend to agree, but I wonder if it returns a row with 0 to emulate something else like Hive? --- - To unsubscribe, e-mail

[GitHub] spark pull request #21816: [SPARK-24794][CORE] Driver launched through rest ...

2018-10-16 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21816#discussion_r225546659 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala --- @@ -83,6 +83,26 @@ class StandaloneRestSubmitSuite extends

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225393919 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -39,29 +40,29 @@ import

[GitHub] spark issue #22731: [SPARK-25674][FOLLOW-UP] Update the stats for each Colum...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22731 I'm going to merge this back to 2.3, as I had merged the original change back to 2.3 --- - To unsubscribe, e-mail: re

[GitHub] spark issue #22730: [SPARK-16775][CORE][WIP] Remove deprecated accumulator v...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22730 Hm, am I right that SparkR and Pyspark only use the accumulator v2 API? then indeed I think this all works. --- - To unsubscribe

[GitHub] spark pull request #22731: [SPARK-25674][FOLLOW-UP] Update the stats for eac...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22731#discussion_r225337806 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala --- @@ -106,15 +106,16 @@ class FileScanRDD

[GitHub] spark issue #22733: [SPARK-25738][SQL] Fix LOAD DATA INPATH for hdfs port

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22733 CC @sujith71955 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22731: [SPARK-25674][FOLLOW-UP] Update the stats for eac...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22731#discussion_r225325506 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala --- @@ -106,15 +106,16 @@ class FileScanRDD

[GitHub] spark pull request #22730: [SPARK-16775][CORE][WIP] Remove deprecated accumu...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22730#discussion_r225286458 --- Diff: core/src/test/java/test/org/apache/spark/JavaAPISuite.java --- @@ -186,7 +184,7 @@ public void randomSplit() { long s1 = splits[1].count

[GitHub] spark pull request #22730: [SPARK-16775][CORE][WIP] Remove deprecated accumu...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22730#discussion_r225286762 --- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala --- @@ -256,7 +110,7 @@ private[spark] object AccumulatorSuite

[GitHub] spark issue #22414: [SPARK-25424][SQL] Window duration and slide duration wi...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22414 ... look at the test failure at https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4374/testReport/org.apache.spark.sql.catalyst.analysis/AnalysisErrorSuite

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225281705 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2137,36 +2137,27 @@ class Analyzer

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225280838 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2137,36 +2137,27 @@ class Analyzer

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225280977 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2137,36 +2137,27 @@ class Analyzer

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225278704 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -39,29 +40,29 @@ import

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225279364 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -314,24 +314,24 @@ class AnalysisSuite extends

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225280125 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -73,27 +73,27 @@ case class UserDefinedFunction

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225280432 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -124,8 +124,10 @@ class UDFRegistration private[sql] (functionRegistry

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22732#discussion_r225278847 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -31,6 +31,7 @@ import

[GitHub] spark issue #22414: [SPARK-25424][SQL] Window duration and slide duration wi...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22414 I get it, but this becomes inconsistent, right? other invalid window values aren't handled the same way. --- - To unsubs

[GitHub] spark issue #22730: [SPARK-16775][CORE][WIP] Remove deprecated accumulator v...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22730 Note this doesn't yet figure out what has to change in Pyspark, SparkR. --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark pull request #22731: [SPARK-25674][FOLLOW-UP] Update the stats for eac...

2018-10-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22731#discussion_r225262559 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala --- @@ -106,15 +106,16 @@ class FileScanRDD

[GitHub] spark issue #22703: [SPARK-25705][BUILD][STREAMING][test-maven] Remove Kafka...

2018-10-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22703 So far looking good to those who have looked, and it passed Maven and SBT tests. I think this will help reduce complexity a bit (and test time in some cases), so will go for it tomorrow

[GitHub] spark pull request #22730: [SPARK-16775][CORE] Remove deprecated accumulator...

2018-10-15 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22730 [SPARK-16775][CORE] Remove deprecated accumulator v1 APIs ## What changes were proposed in this pull request? Remove deprecated accumulator v1 ## How was this patch tested

<    1   2   3   4   5   6   7   8   9   10   >