[GitHub] spark pull request #22485: [SPARK-18364][YARN] Expose metrics for YarnShuffl...

2018-09-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/22485#discussion_r219916796 --- Diff: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java --- @@ -168,6 +170,15 @@ protected void serviceInit

[GitHub] spark issue #22485: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2018-09-20 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/22485 Sorry for not following through on getting this into Apache. FWIW, it's been in the Palantir fork of Spark for over a year: https://github.com/palantir/spark/search?q=SPARK-18364_q=SPARK

[GitHub] spark issue #21334: [minor][SQL]Improve ParseError stop location when offend...

2018-05-15 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/21334 Hi @rubenfiszel thanks for the contribution! Can you please take a glance through http://spark.apache.org/contributing.html to see the best way to get your change merged into Apache Spark

[GitHub] spark issue #20372: Improved block merging logic for partitions

2018-01-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/20372 Tagging folks who have touched this code recently: @vgankidi @ericl @davies This seems to provide a more compact packing in every scenario, which should improve execution times. One risk

[GitHub] spark issue #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/20372 Please fix the scala style checks -- ``` Running Scala style checks Scalastyle checks failed at following

[GitHub] spark pull request #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/20372#discussion_r163464207 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -445,16 +445,25 @@ case class FileSourceScanExec

[GitHub] spark issue #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/20372 Jenkins, this is ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/20372#discussion_r163419745 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala --- @@ -142,15 +142,16 @@ class

[GitHub] spark pull request #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/20372#discussion_r163424784 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -445,16 +445,25 @@ case class FileSourceScanExec

[GitHub] spark pull request #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/20372#discussion_r163419675 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala --- @@ -142,15 +142,16 @@ class

[GitHub] spark pull request #20372: Improved block merging logic for partitions

2018-01-23 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/20372#discussion_r163424415 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -445,16 +445,25 @@ case class FileSourceScanExec

[GitHub] spark pull request #19917: Add failing test for select with a splatted strea...

2017-12-06 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19917 Add failing test for select with a splatted stream ## What changes were proposed in this pull request? Add additional test. ## How was this patch tested? Additional test

[GitHub] spark issue #19257: [SPARK-22042] [SQL] ReorderJoinPredicates can break when...

2017-11-29 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19257 @cloud-fan @gatorsmile any more changes needed on this PR before merging? I don't see any un-addressed comments left

[GitHub] spark issue #19829: [WIP]Upgrade Netty to 4.1.17

2017-11-27 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19829 Looks like a fix for https://issues.apache.org/jira/browse/SPARK-19552 -- should that be reopened now that netty is deprecating 4.0.x so we can't do it "Later&quo

[GitHub] spark pull request #19702: [SPARK-10365][SQL] Support Parquet logical type T...

2017-11-16 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19702#discussion_r151569727 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala --- @@ -372,23 +381,18 @@ private[parquet

[GitHub] spark pull request #19708: [SPARK-22479][SQL] Exclude credentials from Savei...

2017-11-14 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19708#discussion_r151010772 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommandSuite.scala --- @@ -0,0 +1,48

[GitHub] spark issue #19708: [SPARK-22479][SQL] Exclude credentials from SaveintoData...

2017-11-09 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19708 Jenkins, this is ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19694: [SPARK-22470][DOC][SQL] functions.hash is also us...

2017-11-08 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19694 [SPARK-22470][DOC][SQL] functions.hash is also used internally for shuffle and bucketing ## What changes were proposed in this pull request? Add clarifying documentation to the scaladoc

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-10-25 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r147021138 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,229

[GitHub] spark pull request #19574: [SPARK-21991][LAUNCHER][FOLLOWUP] Fix java lint

2017-10-25 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19574 [SPARK-21991][LAUNCHER][FOLLOWUP] Fix java lint ## What changes were proposed in this pull request? Fix java lint ## How was this patch tested? Run `./dev/lint-java` You

[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

2017-10-25 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19217 https://github.com/apache/spark/pull/19574 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

2017-10-25 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19217 ``` Running Java style checks Using

[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

2017-10-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19217 @nivox can you please update the PR title when you get the chance? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

2017-10-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19217 How about `[SPARK-21991][LAUNCHER] Fix race condition in LauncherServer#acceptConnections` ? --- - To unsubscribe, e-mail

[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

2017-10-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19217#discussion_r146493105 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -232,20 +232,20 @@ public void run

[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

2017-10-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19217#discussion_r146492057 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -232,20 +232,20 @@ public void run

[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

2017-10-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19217#discussion_r146493161 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -232,20 +232,20 @@ public void run

[GitHub] spark issue #19486: [SPARK-22268][BUILD] Fix lint-java

2017-10-19 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19486 Updated --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-17 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145296142 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Command.scala --- @@ -0,0 +1,114

[GitHub] spark pull request #19486: [SPARK-22268][BUILD] Fix lint-java

2017-10-12 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19486 [SPARK-22268][BUILD] Fix lint-java ## What changes were proposed in this pull request? Fix java style issues ## How was this patch tested? Run `./dev/lint-java` locally

[GitHub] spark issue #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Sc...

2017-10-10 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19468 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class

2017-09-08 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/19131 A check for unused imports should be added to scalastyle to prevent these from creeping back in. If this PR was accompanied with that check (failing before, now passing) I think the merge conflicts

[GitHub] spark pull request #19164: [SPARK-21953] Show both memory and disk bytes spi...

2017-09-08 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19164 [SPARK-21953] Show both memory and disk bytes spilled if either is present You can merge this pull request into a Git repository by running: $ git pull https://github.com/ash211/spark patch-3

[GitHub] spark pull request #19136: [DO NOT MERGE][SPARK-15689][SQL] data source v2

2017-09-07 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19136#discussion_r137469790 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/ReadTask.java --- @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19136: [DO NOT MERGE][SPARK-15689][SQL] data source v2

2017-09-07 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/19136#discussion_r137471056 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/upward/StatisticsSupport.java --- @@ -0,0 +1,26 @@ +/* + * Licensed

[GitHub] spark pull request #19153: SPARK-21941 Stop storing unused attemptId in SQLT...

2017-09-07 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19153 SPARK-21941 Stop storing unused attemptId in SQLTaskMetrics ## What changes were proposed in this pull request? In a driver heap dump containing 390,105 instances of SQLTaskMetrics

[GitHub] spark pull request #19088: [SPARK-21875][BUILD] Fix Java style bugs

2017-08-30 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/19088 [SPARK-21875][BUILD] Fix Java style bugs ## What changes were proposed in this pull request? Fix Java code style so `./dev/lint-java` succeeds ## How was this patch tested

[GitHub] spark pull request #18996: [MINOR][TYPO] Fix typos: runnning and Excecutors

2017-08-18 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/18996 [MINOR][TYPO] Fix typos: runnning and Excecutors ## What changes were proposed in this pull request? Fix typos ## How was this patch tested? Existing tests You can merge

[GitHub] spark pull request #18913: [SPARK-21563][CORE] Fix race condition when seria...

2017-08-11 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/18913#discussion_r132721021 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1792,6 +1796,9 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark pull request #18913: [SPARK-21563][CORE] Fix race condition when seria...

2017-08-10 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/18913 [SPARK-21563][CORE] Fix race condition when serializing TaskDescriptions and adding jars ## What changes were proposed in this pull request? Fix the race condition when serializing

[GitHub] spark pull request #18877: [SPARK-17742][core] Handle child process exit in ...

2017-08-10 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/18877#discussion_r132583520 --- Diff: launcher/src/main/java/org/apache/spark/launcher/ChildProcAppHandle.java --- @@ -118,14 +116,40 @@ void setChildProc(Process childProc, String

[GitHub] spark pull request #18877: [SPARK-17742][core] Handle child process exit in ...

2017-08-10 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/18877#discussion_r132583553 --- Diff: launcher/src/main/java/org/apache/spark/launcher/ChildProcAppHandle.java --- @@ -166,4 +185,15 @@ private synchronized void fireEvent(boolean

[GitHub] spark issue #18789: SPARK-20433 Bump jackson from 2.6.5 to 2.6.7.1

2017-08-09 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18789 @srowen sorry for not picking up on this -- thanks for pushing it over the finish line in your PR! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #18789: Bump jackson from 2.6.5 to 2.6.7.1 (#241)

2017-07-31 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/18789 Bump jackson from 2.6.5 to 2.6.7.1 (#241) This brings in a security fix for CVE-2017-7525 in the jackson-databind library, which Spark uses. When releasing this patch, upstream released

[GitHub] spark issue #18658: [SPARK-20871][SQL] only log Janino code at debug level

2017-07-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18658 FYI for future reviewers as well, we've been running an [extremely similar patch](https://github.com/palantir/spark/pull/181) to PJ's on our distribution of Spark for the past several months and had

[GitHub] spark pull request #18658: [SPARK-20871][SQL] only log Janino code at debug ...

2017-07-17 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/18658#discussion_r127819158 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -1037,24 +1037,22 @@ object

[GitHub] spark issue #18658: [SPARK-20871] only log Janino code at debug level

2017-07-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18658 Jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18621: [SPARK-21400][SQL] Don't overwrite output committers on ...

2017-07-13 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18621 jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18581: [SPARK-21289][SQL][ML] Supports custom line separ...

2017-07-10 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/18581#discussion_r126534985 --- Diff: mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala --- @@ -41,11 +41,15 @@ private[libsvm] class LibSVMOptions(@transient

[GitHub] spark issue #18176: [SPARK-20952] Make TaskContext an InheritableTheadLocal

2017-06-27 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18176 Jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18406: [SPARK-21195] Automatically register new metrics from so...

2017-06-26 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18406 @robert3005 looks like a bunch of tests are failing with `java.lang.IllegalArgumentException: A metric named local-1498509661743.driver.HiveExternalCatalog.fileCacheHits already exists

[GitHub] spark issue #18427: [SPARK-21219][scheduler] Fix race condition between addi...

2017-06-26 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18427 Jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18406: [SPARK-21195] Automatically register new metrics from so...

2017-06-26 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18406 Jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18209: [SPARK-20992][Scheduler] Add support for Nomad as a sche...

2017-06-22 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18209 @manojlds I'm a part of the Spark-on-k8s team that's currently building k8s integration for Spark outside of the Apache Spark repo. You can follow our work at https://github.com/apache-spark-on-k8s

[GitHub] spark issue #17935: [SPARK-20690][SQL] Subqueries in FROM should have alias ...

2017-06-05 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17935 @JoshRosen what was the other type of database you were using? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18176: [SPARK-20952] Make TaskContext an InheritableTheadLocal

2017-06-02 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/18176 Jenkins this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17680: [SPARK-20364][SQL] Support Parquet predicate pushdown on...

2017-05-09 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17680 Are there any comments on this PR or is it ready to be merged? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17680: [SPARK-20364][SQL] Support Parquet predicate pushdown on...

2017-04-28 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17680 Any further thoughts on this? It was quite surprising for one of our users so I wanted to make sure it was fixed in a future Apache release --- If your project is set up for it, you can reply

[GitHub] spark pull request #17680: [SPARK-20364][SQL] Support Parquet predicate push...

2017-04-20 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17680#discussion_r112531312 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -536,4 +537,43 @@ class

[GitHub] spark pull request #17680: [SPARK-20364][SQL] Support Parquet predicate push...

2017-04-20 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17680#discussion_r112529697 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala --- @@ -487,6 +487,20 @@ class

[GitHub] spark pull request #17680: [SPARK-20364][SQL] Support Parquet predicate push...

2017-04-20 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17680#discussion_r112530989 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -536,4 +537,43 @@ class

[GitHub] spark pull request #17680: [SPARK-20364][SQL] Support Parquet predicate push...

2017-04-19 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17680#discussion_r112285677 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -536,4 +537,43 @@ class

[GitHub] spark pull request #17680: [SPARK-20364][SQL] Support Parquet predicate push...

2017-04-19 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17680#discussion_r112285883 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -536,4 +537,43 @@ class

[GitHub] spark pull request #17667: Failing test for parquet predicate pushdown on co...

2017-04-19 Thread ash211
Github user ash211 closed the pull request at: https://github.com/apache/spark/pull/17667 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on columns w...

2017-04-19 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17667 Agreed, will close for now until there's a fix to go along with the test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17667: Failing test for parquet predicate pushdown on dots with...

2017-04-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17667 @HyukjinKwon thanks for looking at this! Please feel free to open a Jira so we can begin discussing a fix. I haven't started working on a patch yet, only have the test case at this point. So

[GitHub] spark pull request #17667: Failing test for parquet predicate pushdown on do...

2017-04-17 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/17667 Failing test for parquet predicate pushdown on dots with columns // checking against Jenkins to make sure this is still live on master You can merge this pull request into a Git repository

[GitHub] spark pull request #17664: Typo fix: distitrbuted -> distributed

2017-04-17 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/17664 Typo fix: distitrbuted -> distributed ## What changes were proposed in this pull request? Typo fix: distitrbuted -> distributed ## How was this patch tested? Ex

[GitHub] spark issue #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2017-03-28 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17401 @jerryshao ready for re-review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffl...

2017-03-28 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17401#discussion_r108562943 --- Diff: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java --- @@ -0,0 +1,123 @@ +/* + * Licensed

[GitHub] spark issue #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2017-03-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17401 Ready for further review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17411: logging improvements

2017-03-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17411 Thanks for the contribution @juanrh ! I'm happy to see contributions no matter how small. For larger changes you would need to file a Jira ticket, but this is small enough that it's not necessary

[GitHub] spark issue #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2017-03-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17401 Thanks again for the comments @jerryshao ! I've now added some tests to verify that the metrics get converted in the expected way to the collector, and camel-cased shuffleService --- If your

[GitHub] spark pull request #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffl...

2017-03-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17401#discussion_r107840148 --- Diff: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java --- @@ -0,0 +1,118 @@ +/* + * Licensed

[GitHub] spark pull request #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffl...

2017-03-24 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/17401#discussion_r107840109 --- Diff: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java --- @@ -166,6 +170,23 @@ protected void serviceInit

[GitHub] spark issue #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffleServic...

2017-03-23 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17401 Thanks for taking a look @jerryshao ! I've reformatted to two-space indentation and run `./dev/lint-java` to make sure this code passes the linter. `src/main/java/org/apache/spark/sql

[GitHub] spark pull request #17401: [SPARK-18364][YARN] Expose metrics for YarnShuffl...

2017-03-23 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/17401 [SPARK-18364][YARN] Expose metrics for YarnShuffleService Registers the shuffle server's metrics with the Hadoop Node Manager's DefaultMetricsSystem. ## What changes were proposed

[GitHub] spark issue #17399: Update functions.scala

2017-03-23 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/17399 Thanks for contributing to Spark @roxannemoslehi ! I think Sean just means updating the title to something more like `[DOCS] Clarify round mode in format_number function`. It doesn't feel

[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...

2017-03-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/14615 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...

2017-03-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/14615 @robert3005 looks like this has unit test failures on `org.apache.spark.sql.hive.orc.OrcSourceSuite.SPARK-19459/SPARK-18220: read char/varchar column written by Hive` -- is that a flake

[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...

2017-03-17 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/14615 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-03-02 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16959 Any last changes before merging? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-28 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103576994 --- Diff: core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala --- @@ -195,6 +195,17 @@ class OutputCommitCoordinatorSuite

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-24 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16959 @vanzin are you right person to review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-17 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r101762924 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -48,25 +48,28 @@ private[spark] class OutputCommitCoordinator

[GitHub] spark issue #16575: [SPARK-19213] DatasourceScanExec uses runtime sparksessi...

2017-01-20 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16575 @hvanhovell does that description make sense? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16503: [SPARK-18113] Use ask to replace askWithRetry in canComm...

2017-01-13 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16503 Making this idempotent looks great. I think there's a separate issue with this code still not handling poorly-timed preemption, but let's deal with that in a separate ticket / PR. Good

[GitHub] spark issue #16575: [SPARK-19213] DatasourceScanExec uses runtime sparksessi...

2017-01-13 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16575 @hvanhovell the need is also explained in the Jira ticket: https://issues.apache.org/jira/browse/SPARK-19213 Does that code snippet make sense? --- If your project is set up for it, you

[GitHub] spark issue #16503: [SPARK-18113] Use ask to replace askWithRetry in canComm...

2017-01-12 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16503 @jinxing64 can you please fix the failing Scala style tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16503: [SPARK-18113] Use ask to replace askWithRetry in canComm...

2017-01-12 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16503 You covered my concerns! I think this will fix some parts of this problem for sure, not sure if it covers every possible case though. --- If your project is set up for it, you can reply

[GitHub] spark pull request #16503: [SPARK-18113] Use ask to replace askWithRetry in ...

2017-01-12 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/16503#discussion_r95660339 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -165,9 +167,14 @@ private[spark] class OutputCommitCoordinator

[GitHub] spark pull request #16503: [SPARK-18113] Use ask to replace askWithRetry in ...

2017-01-12 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/16503#discussion_r95659921 --- Diff: core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala --- @@ -221,6 +227,22 @@ private case class

[GitHub] spark pull request #16503: [SPARK-18113] Use ask to replace askWithRetry in ...

2017-01-12 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/16503#discussion_r95659396 --- Diff: core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala --- @@ -189,6 +188,13 @@ class OutputCommitCoordinatorSuite

[GitHub] spark pull request #16558: Fix missing close-parens for In filter's toString

2017-01-11 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/16558 Fix missing close-parens for In filter's toString Otherwise the open parentheses isn't closed in query plan descriptions of batch scans. PushedFilters: [In(COL_A

[GitHub] spark issue #16281: [SPARK-13127][SQL] Update Parquet to 1.9.0

2016-12-15 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16281 What are the specific patches to parquet that folks are proposing should be included in a parquet 1.8.1-spark1 ? Or what would be desired in a parquet-released 1.8.2 ? --- If your project is set

[GitHub] spark issue #16092: [SPARK-18662] Move resource managers to separate directo...

2016-12-01 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16092 With a k8s backend on the way I do think it adds a nice organization for these 3 clearly grouped modules --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/16061 Another external scheduler backend I'm aware of is Two Sigma's scheduler backend for the system they've created called [Cook](https://github.com/twosigma/Cook). See

[GitHub] spark issue #15932: [SPARK-18448][CORE] SparkSession should implement java.l...

2016-11-18 Thread ash211
Github user ash211 commented on the issue: https://github.com/apache/spark/pull/15932 Yep that's precisely what I was envisioning. Thanks @srowen ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #15835: [SPARK-17059][SQL] Allow FileFormat to specify pa...

2016-11-12 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/15835#discussion_r87703118 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala --- @@ -703,6 +705,81 @@ class

[GitHub] spark pull request #15486: Typo: form -> from

2016-10-14 Thread ash211
GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/15486 Typo: form -> from ## What changes were proposed in this pull request? Minor typo fix ## How was this patch tested? Existing unit tests on Jenkins You can merge this p

  1   2   3   4   >