[GitHub] spark pull request #23178: [SPARK-26216][SQL] Do not use case class as publi...

2018-11-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23178#discussion_r237521153 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -38,114 +38,108 @@ import

[GitHub] spark pull request #23178: [SPARK-26216][SQL] Do not use case class as publi...

2018-11-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23178#discussion_r237522078 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -38,114 +38,108 @@ import

[GitHub] spark pull request #23126: [SPARK-26158] [MLLIB] fix covariance accuracy pro...

2018-11-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23126#discussion_r237520303 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -128,6 +128,82 @@ class RowMatrix @Since("

[GitHub] spark issue #23148: [SPARK-26177] Automated formatting for Scala code

2018-11-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23148 Merged to master. @shaneknapp I think you can update the PR builder to call this now, if we're ready to give it a spin

[GitHub] spark issue #23122: [MINOR][ML] add missing params to Instr

2018-11-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23122 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23152: [SPARK-26181][SQL] the `hasMinMaxStats` method of `Colum...

2018-11-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23152 We should still make this change right, even if it doesn't manifest as an actual bug right now? does the test need an update

[GitHub] spark issue #23167: [SPARK-26024][FOLLOWUP][MINOR] Follow-up to remove extra...

2018-11-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23167 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r237333561 --- Diff: docs/ml-clustering.md --- @@ -265,3 +265,44 @@ Refer to the [R API docs](api/R/spark.gaussianMixture.html) for more details

[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r237330636 --- Diff: docs/ml-clustering.md --- @@ -265,3 +265,44 @@ Refer to the [R API docs](api/R/spark.gaussianMixture.html) for more details

[GitHub] spark issue #23168: [SPARK-26207][doc]add PowerIterationClustering (PIC) doc...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23168 Pardon, was this also already added to master? I don't see it but I could be missing something. --- - To unsubscribe, e-mail

[GitHub] spark pull request #23052: [SPARK-26081][SQL] Prevent empty files for empty ...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23052#discussion_r237287908 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala --- @@ -169,13 +169,18 @@ private[csv] class

[GitHub] spark pull request #23052: [SPARK-26081][SQL] Prevent empty files for empty ...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23052#discussion_r237210777 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala --- @@ -169,13 +169,18 @@ private[csv] class

[GitHub] spark issue #23150: [SPARK-26178][SQL] Use java.time API for parsing timesta...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23150 Looks like that kind of fallback logic was there for compatibility with Spark 1.x and 2.0; see some comments about 'backwards compatibility' in for example https://github.com/apache/spark/blame

[GitHub] spark pull request #23167: [SPARK-26024][FOLLOWUP][MINOR] Follow-up to remov...

2018-11-28 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/23167 [SPARK-26024][FOLLOWUP][MINOR] Follow-up to remove extra blank lines in R function descriptions ## What changes were proposed in this pull request? Follow-up to remove extra blank lines

[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23025#discussion_r237174135 --- Diff: R/pkg/R/DataFrame.R --- @@ -767,6 +767,14 @@ setMethod("repartition", #' using \code{spark.sql.shuffle.

[GitHub] spark issue #23150: [SPARK-26178][SQL] Use java.time API for parsing timesta...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23150 The code looks good at a glance. So the flag lets people select the old behavior; that keeps it pretty safe. Are there any other behavior changes with the new code, besides being able to parse

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r237164158 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r237135103 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark issue #22991: [SPARK-25989][ML] OneVsRestModel handle empty outputCols...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22991 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23126: [SPARK-26158] [MLLIB] fix covariance accuracy pro...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23126#discussion_r237130345 --- Diff: mllib/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java --- @@ -67,7 +66,7 @@ public void testPCA() { JavaRDD dataRDD

[GitHub] spark pull request #23126: [SPARK-26158] [MLLIB] fix covariance accuracy pro...

2018-11-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23126#discussion_r237133038 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -128,6 +128,69 @@ class RowMatrix @Since("

[GitHub] spark issue #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to hold wea...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22995 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23102: [SPARK-26137][CORE] Use Java system property "file.separ...

2018-11-28 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23102 Merged to master/2.4/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r236840320 --- Diff: dev/create-release/release-build.sh --- @@ -110,16 +110,18 @@ fi # Depending on the version being built, certain extra profiles need

[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23025#discussion_r236837158 --- Diff: R/pkg/R/DataFrame.R --- @@ -767,6 +767,14 @@ setMethod("repartition", #' using \code{spark.sql.shuffle.

[GitHub] spark pull request #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to h...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22995#discussion_r236726776 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -92,8 +95,15 @@ private[spark] class TorrentBroadcast[T: ClassTag

[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r236726267 --- Diff: dev/create-release/release-build.sh --- @@ -110,16 +110,18 @@ fi # Depending on the version being built, certain extra profiles need

[GitHub] spark pull request #23130: [SPARK-26161][SQL] Ignore empty files in load

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23130#discussion_r236721863 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala --- @@ -142,4 +143,15 @@ class SaveLoadSuite extends DataSourceTest

[GitHub] spark pull request #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to h...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22995#discussion_r236721318 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -92,8 +95,15 @@ private[spark] class TorrentBroadcast[T: ClassTag

[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r236714704 --- Diff: R/pkg/R/sparkR.R --- @@ -269,7 +269,7 @@ sparkR.sparkContext <- function( #' sparkR.session("yarn-client", "Spa

[GitHub] spark pull request #23130: [SPARK-26161][SQL] Ignore empty files in load

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23130#discussion_r236705916 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala --- @@ -142,4 +143,15 @@ class SaveLoadSuite extends DataSourceTest

[GitHub] spark pull request #23130: [SPARK-26161][SQL] Ignore empty files in load

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23130#discussion_r236705666 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala --- @@ -142,4 +143,15 @@ class SaveLoadSuite extends DataSourceTest

[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...

2018-11-27 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22683 Ping @httfighter --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23102: [SPARK-26137][CORE] Use Java system property "fil...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23102#discussion_r236703765 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -19,6 +19,7 @@ package org.apache.spark.deploy import

[GitHub] spark pull request #23126: [SPARK-26158] [MLLIB] fix covariance accuracy pro...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23126#discussion_r236656961 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala --- @@ -266,6 +266,16 @@ class RowMatrixSuite extends

[GitHub] spark pull request #23126: [SPARK-26158] [MLLIB] fix covariance accuracy pro...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23126#discussion_r236658063 --- Diff: mllib/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java --- @@ -67,7 +66,7 @@ public void testPCA() { JavaRDD dataRDD

[GitHub] spark pull request #23052: [SPARK-26081][SQL] Prevent empty files for empty ...

2018-11-27 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23052#discussion_r236652952 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala --- @@ -169,13 +169,18 @@ private[csv] class

[GitHub] spark pull request #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23100#discussion_r236471495 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala --- @@ -17,126 +17,512 @@ package org.apache.spark.ml.feature

[GitHub] spark pull request #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to h...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22995#discussion_r236471154 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -92,8 +95,15 @@ private[spark] class TorrentBroadcast[T: ClassTag

[GitHub] spark pull request #23148: [SPARK-26177] Automated formatting for Scala code

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23148#discussion_r236421304 --- Diff: .scalafmt.conf --- @@ -0,0 +1,24 @@ +# --- End diff -- Can this file live in dev/? I tried to stuff as many things in there as I

[GitHub] spark pull request #23148: [SPARK-26177] Automated formatting for Scala code

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23148#discussion_r236421583 --- Diff: pom.xml --- @@ -156,6 +156,10 @@ 3.2.2 2.12.7 2.12 +1.5.1 --- End diff -- If these are all just

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236405278 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to h...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22995#discussion_r236394496 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -93,7 +96,14 @@ private[spark] class TorrentBroadcast[T: ClassTag

[GitHub] spark pull request #23102: [SPARK-26137][CORE] Use Java system property "fil...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23102#discussion_r236347084 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -61,11 +61,13 @@ private[deploy] object DependencyUtils extends Logging

[GitHub] spark pull request #23102: [SPARK-26137][CORE] Use Java system property "fil...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23102#discussion_r236343465 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -61,11 +61,13 @@ private[deploy] object DependencyUtils extends Logging

[GitHub] spark issue #23126: [SPARK-26158] [MLLIB] fix covariance accuracy problem fo...

2018-11-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23126 Is there a simple test case you can add to cover that too? that would really prove this change. --- - To unsubscribe, e-mail

[GitHub] spark issue #22759: [MINOR][SQL][DOC] Correct parquet nullability documentat...

2018-11-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22759 @dima-asana can you rebase, and add a simple test case or else find one that does demonstrate the behavior here

[GitHub] spark issue #22184: [SPARK-25132][SQL][DOC] Add migration doc for case-insen...

2018-11-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22184 @seancxmao is this PR still live? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #17586: [SPARK-20249][ML][PYSPARK] Add summary for Linear...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/17586#discussion_r236239347 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala --- @@ -287,6 +290,27 @@ class LinearSVCModel private[classification

[GitHub] spark issue #21412: [SPARK-18805][DStream] Avoid StackOverflowError while ge...

2018-11-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21412 Agree, this PR should be closed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236236422 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236236327 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -150,8 +150,9 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #22991: [SPARK-25989][ML] OneVsRestModel handle empty out...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22991#discussion_r236230624 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -209,6 +215,9 @@ final class OneVsRestModel private[ml

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236228669 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236228862 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -150,8 +150,9 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #23130: [SPARK-26161][SQL] Ignore empty files in load

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23130#discussion_r236226380 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -388,7 +388,7 @@ case class FileSourceScanExec

[GitHub] spark issue #23123: [SPARK-26153][ML] GBT & RandomForest avoid unnecessary `...

2018-11-26 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23123 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23122: [MINOR][ML] add missing params to Instr

2018-11-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23122#discussion_r236219489 --- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala --- @@ -671,7 +671,7 @@ class ALS(@Since("1.4.0") override val u

[GitHub] spark issue #23121: [SPARK-24553][UI][FOLLOWUP][2.4 Backport] Fix unnecessar...

2018-11-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23121 @jerryshao you can close this now --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23100#discussion_r236097273 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala --- @@ -17,126 +17,512 @@ package org.apache.spark.ml.feature

[GitHub] spark pull request #23100: [SPARK-26133][ML] Remove deprecated OneHotEncoder...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23100#discussion_r236097295 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala --- @@ -17,126 +17,512 @@ package org.apache.spark.ml.feature

[GitHub] spark pull request #23052: [SPARK-26081][SQL] Prevent empty files for empty ...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23052#discussion_r236097176 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala --- @@ -169,13 +169,18 @@ private[csv] class

[GitHub] spark issue #23129: [MINOR] Update all DOI links to preferred resolver

2018-11-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23129 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236092947 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236085936 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236085919 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236086094 --- Diff: core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala --- @@ -77,6 +77,30 @@ class AppStatusStoreSuite extends SparkFunSuite

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236085876 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236086033 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -222,29 +223,20 @@ private[spark] class AppStatusStore( val

[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...

2018-11-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23088#discussion_r236086063 --- Diff: core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala --- @@ -77,6 +77,30 @@ class AppStatusStoreSuite extends SparkFunSuite

[GitHub] spark issue #23134: [SPARK-25504][Docs] Update doc about retained tasks, job...

2018-11-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23134 I think the existing docs are fine. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #23131: [SPARK-25908][SQL][FOLLOW-UP] Add back unionAll

2018-11-24 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23131#discussion_r236057261 --- Diff: docs/sql-migration-guide-upgrade.md --- @@ -341,8 +341,6 @@ displayTitle: Spark SQL Upgrading Guide APIs. Instead, `DataFrame` remains

[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...

2018-11-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22683 Yeah, there are going to be several more tests that fail because they are expecting a string like 'KB'. Hopefully easy to fix. https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder

[GitHub] spark issue #23125: [SPARK-26156][WebUI] Revise summary section of stage pag...

2018-11-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23125 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22779: [SPARK-25786][CORE]If the ByteBuffer.hasArray is false ,...

2018-11-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22779 Merged to master, but I'm also going to try to back port to 2.4 and 2.3 as a bug fix. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...

2018-11-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22683 I believe most of the Spark code really uses units like KiB, multiples of 1024, as you see here. However it's often referred to as KB, incorrectly, and so on. It seems like the UI is the only place

[GitHub] spark pull request #22991: [SPARK-25989][ML] OneVsRestModel handle empty out...

2018-11-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22991#discussion_r235929179 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -219,14 +225,20 @@ final class OneVsRestModel private[ml

[GitHub] spark issue #23109: [SPARK-26069][TESTS][FOLLOWUP]Add another possible error...

2018-11-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23109 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #23125: [SPARK-26156][WebUI] Revise summary section of st...

2018-11-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23125#discussion_r235924136 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -79,6 +79,9 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark issue #16732: [SPARK-19368][MLlib] BlockMatrix.toIndexedRowMatrix() op...

2018-11-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16732 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22875: [SPARK-25867][ML] Remove KMeans computeCost

2018-11-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22875 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #23102: [SPARK-26137][CORE] Use Java system property "fil...

2018-11-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23102#discussion_r235821628 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -61,11 +62,12 @@ private[deploy] object DependencyUtils extends Logging

[GitHub] spark issue #23112: [GraphX] Remove unused variables left over by previous r...

2018-11-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23112 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #18784: [SPARK-21559][Mesos] remove mesos fine-grained mo...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18784#discussion_r235563442 --- Diff: docs/running-on-mesos.md --- @@ -180,10 +180,7 @@ Note that jars or python files that are passed to spark-submit should be URIs re

[GitHub] spark pull request #18784: [SPARK-21559][Mesos] remove mesos fine-grained mo...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18784#discussion_r235563605 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -81,8 +79,8

[GitHub] spark pull request #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary t...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/17654#discussion_r235562839 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -935,7 +935,8 @@ class

[GitHub] spark pull request #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary t...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/17654#discussion_r235562982 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala --- @@ -444,31 +444,21 @@ class LinearRegressionModel private[ml

[GitHub] spark pull request #17654: [SPARK-20351] [ML] Add trait hasTrainingSummary t...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/17654#discussion_r235563050 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/HasTrainingSummary.scala --- @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #21066: [SPARK-23977][CLOUD][WIP] Add commit protocol binding to...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21066 If we're considering only supporting Hadoop 3 in Spark 3 -- and I think we should -- this could even go into the main source tree

[GitHub] spark issue #23093: [SPARK-26127][ML] Remove deprecated setters from tree re...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23093 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23107: small question in Spillable class

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23107 Please send questions to u...@spark.apache.org; this should be closed. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #23102: [SPARK-26137][CORE] Use Java system property "fil...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23102#discussion_r235546231 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -65,7 +65,7 @@ private[deploy] object DependencyUtils extends Logging

[GitHub] spark issue #23062: [SPARK-8288][SQL] ScalaReflection can use companion obje...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23062 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23088 When evaluating these things I use git blame (or visual tools here and in an IDE) to figure out if there was a previous change that explicitly tried to change the behavior, and whether

[GitHub] spark issue #23081: [SPARK-26109][WebUI]Duration in the task summary metrics...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23081 Merged to master/2.4/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22822: [SPARK-25678] Requesting feedback regarding a prototype ...

2018-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22822 Yeah, I can't imagine merging support for any other resource manager I know of now; it's just way too much to maintain. I have not heard of this one myself. It should be implemented outside Spark

[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...

2018-11-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r235418193 --- Diff: bin/load-spark-env.cmd --- @@ -21,37 +21,42 @@ rem This script loads spark-env.cmd if it exists, and ensures it is only loaded rem spark

[GitHub] spark issue #23087: [SPARK-26124][BUILD] Update plugins to latest versions

2018-11-20 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23087 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23097: [SPARK-26043][HOTFIX] Hotfix a change to SparkHadoopUtil...

2018-11-20 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23097 I'm going to merge this as it's solving a compile problem, and that much clearly works now. --- - To unsubscribe, e-mail

[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...

2018-11-20 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r235198196 --- Diff: R/pkg/R/sparkR.R --- @@ -269,7 +269,7 @@ sparkR.sparkContext <- function( #' sparkR.session("yarn-client", "Spa

<    1   2   3   4   5   6   7   8   9   10   >