[GitHub] spark issue #19487: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-13 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19487 I will change from `test:` to `::invalid::` to explicitly indicate an invalid path (I picked the first path which gave me a parse error :) ). On the question of whether `path` constructor

[GitHub] spark issue #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support f...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19385 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support f...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82744/ Test PASSed. ---

[GitHub] spark pull request #19487: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-10-13 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19487#discussion_r144631350 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -60,15 +71,6 @@ class

[GitHub] spark issue #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support f...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19385 **[Test build #82744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82744/testReport)** for PR 19385 at commit

[GitHub] spark issue #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support f...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19385 **[Test build #82744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82744/testReport)** for PR 19385 at commit

[GitHub] spark pull request #19481: [SPARK-21907][CORE][BACKPORT 2.2] oom during spil...

2017-10-13 Thread eyalfa
Github user eyalfa closed the pull request at: https://github.com/apache/spark/pull/19481 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support f...

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19385 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19494: [SPARK-22249][SQL] isin with empty list throws ex...

2017-10-13 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19494#discussion_r144622642 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -104,7 +104,8 @@ case class

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19448 > But, if I were working on a Spark distribution at a vendor, this is something I would definitely include because it's such a useful feature. I concur :) ---

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19452 **[Test build #82743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82743/testReport)** for PR 19452 at commit

[GitHub] spark pull request #19494: [SPARK-22249][SQL] isin with empty list throws ex...

2017-10-13 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19494#discussion_r144621674 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -104,7 +104,8 @@ case class

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread maryannxue
Github user maryannxue commented on the issue: https://github.com/apache/spark/pull/19488 @cloud-fan Please see CheckAnalysis.scala:170. It checks the input expression of each aggregate expression to make sure that they are not another aggregate function and are deterministic. ---

[GitHub] spark pull request #19452: [SPARK-22136][SS] Evaluate one-sided conditions e...

2017-10-13 Thread joseph-torres
Github user joseph-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/19452#discussion_r144620005 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSymmetricHashJoinHelperSuite.scala --- @@ -0,0 +1,118 @@ +/* + *

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 ``` $ ldd linux/amd64/libzstd-jni.so ldd: warning: you do not have execution permission for `linux/amd64/libzstd-jni.so' linux/amd64/libzstd-jni.so: /lib64/libc.so.6: version

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144618995 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144618498 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,37 @@ import

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19451 If we have to do this all over again i'd put all rules in their own files. Replace isn't really a great high level category because all rules at some level replace something. ---

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 Good news is that I can reproduce it on the amplab machine, so I'll try to play around with the zstd-jni code a bit. --- - To

[GitHub] spark pull request #19493: [SPARK-22252][SQL][followup] Command should not b...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19493 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19493 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144617310 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,87 @@ class FileSuite extends SparkFunSuite with LocalSparkContext {

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144617011 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,87 @@ class FileSuite extends SparkFunSuite with LocalSparkContext {

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19493 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19493 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19493 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82739/ Test PASSed. ---

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19493 **[Test build #82739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82739/testReport)** for PR 19493 at commit

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 Yeah but that would also cause it to fail locally if it were the cause, and it passes for me. I can't really figure out from the rest of the logs if something obvious is wrong, so I guess the best

[GitHub] spark issue #19354: [SPARK-20992][Scheduler] Add links in documentation to N...

2017-10-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19354 That's a reasonable suggestion, though the K8S integration is intended to come back into Spark soon. Hence doing nothing here is also about the right thing in the near term, even if it's not

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18805 This seems to be caused by a issue in the `zstd-jni` library. It probably uses the wrong `ClassLoader` to load the native library, and as a result it cannot find the library & load it. ---

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19448 Thank you :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82742/testReport)** for PR 19419 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread krishna-pandey
Github user krishna-pandey commented on the issue: https://github.com/apache/spark/pull/19419 @jerryshao removed Whitespace at end of line 440 in package.scala. ok to test. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82738/ Test PASSed. ---

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 Turns out that's caused by SparkContext failing to clean up after itself when the `UnsatisfiedLinkError` happens, so those errors are red herrings... ---

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82738/testReport)** for PR 19269 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19419 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82741/ Test FAILed. ---

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82741/testReport)** for PR 19419 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19419 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82741/testReport)** for PR 19419 at commit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19448 Sure, I will and let me note it ahead next time. I made a mistake while trying to think of reasons for this backport. --- -

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144609180 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,37 @@ import

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19451 @rxin I think it would be better to keep all the rules of the "Replace Operators" batch in a single file. So if you prefer to keep the rule in a new file, we can move all the replace operator

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 I haven't been able to reproduce the issue locally, but looking at the jenkins logs I see a bunch of exceptions like these: ``` 17/10/13 06:53:26.609 dispatcher-event-loop-15 ERROR

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 (I'll file a bug and send a PR for it separately, btw.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19451 @gatorsmile > Could you please add an end-to-end testsuite except.sql of SQLQueryTestSuite.scala? Please verify `except.sql ` and `except.sql.out` files are enough for the

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19451 **[Test build #82740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82740/testReport)** for PR 19451 at commit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19448 I am not really worried about this particular change. It's already merged and it seems a small and safe change. I am not planning to revert it. But, in general, let's avoid of merging changes

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144604878 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,87 @@ class FileSuite extends SparkFunSuite with LocalSparkContext {

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144603628 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,37 @@ import

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18460 Thank you, @cloud-fan , @gatorsmile , and @viirya !!! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144602775 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala --- @@ -84,25 +84,45 @@ private[sql] trait ColumnarBatchScan

[GitHub] spark issue #19354: [SPARK-20992][Scheduler] Add links in documentation to N...

2017-10-13 Thread barnardb
Github user barnardb commented on the issue: https://github.com/apache/spark/pull/19354 I totally understand the reluctance to have non-ASF projects in a list headed by "The system currently supports…". Looking at the [Powered By](https://spark.apache.org/powered-by.html) page, it

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18460 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144601470 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18460 LGTM, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/19448 I have a lot of sympathy for the argument that infrastructure software shouldn't have too many backports and that those should be generally bug fixes. But, if I were working on a Spark distribution

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144600676 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19448 Okay. I am sorry for this trouble. Should we revert this if you guys feel strongly about it? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144599111 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144598051 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144597485 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #19494: [SPARK-22249][SQL] isin with empty list throws exception...

2017-10-13 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19494 @srowen do you mean replacing `contains` with `exists`? If so, might you please explain me why `exists` is a better option? Thanks. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18979 Could you resolve the conflicts again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTas...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18979#discussion_r144595826 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala --- @@ -44,20 +47,32 @@ case class

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19448 @HyukjinKwon branch-2.2 is in a maintenance branch, I am not sure it is appropriate to merge this change to branch-2.2 since it is not really a bug fix. If the doc is not accurate, we should fix the

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144594930 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #19494: [SPARK-22249][SQL] isin with empty list throws exception...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19494 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82735/ Test FAILed. ---

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19494: [SPARK-22249][SQL] isin with empty list throws ex...

2017-10-13 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/19494 [SPARK-22249][SQL] isin with empty list throws exception on cached DataFrame ## What changes were proposed in this pull request? As pointed out in the JIRA, there is a bug which causes an

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19448 @steveloughran Thanks for your inputs. Totally agree on your opinions. Spark is an infrastructure software. We have to be very careful when backporting the PRs. ---

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19480 **[Test build #82735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82735/testReport)** for PR 19480 at commit

[GitHub] spark issue #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19470 BTW, @cloud-fan . Could you review #18460 , too? I think we need your final approval. :) --- - To unsubscribe, e-mail:

[GitHub] spark issue #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19470 Thank you so much, @cloud-fan , @gatorsmile , and @viirya ! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...

2017-10-13 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19480#discussion_r144588780 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,35 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82731/testReport)** for PR 18979 at commit

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82733/testReport)** for PR 19488 at commit

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82731/ Test PASSed. ---

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-13 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19476 @jerryshao Thanks a lot for ping. I left comments by my understanding. Not sure if it's helpful :) --- - To unsubscribe,

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82733/ Test PASSed. ---

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-13 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144577910 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -355,11 +355,21 @@ package object config { .doc("The

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-13 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144586111 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -355,11 +355,21 @@ package object config { .doc("The

[GitHub] spark pull request #19476: [SPARK-22062][CORE] Spill large block to disk in ...

2017-10-13 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/19476#discussion_r144585860 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1552,4 +1582,65 @@ private[spark] object BlockManager {

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r144584455 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -205,14 +205,17 @@ object PhysicalAggregation {

[GitHub] spark issue #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark s...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19470 thanks, merging to master/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use ...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19470 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...

2017-10-13 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19480#discussion_r144581126 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,35 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19493 **[Test build #82739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82739/testReport)** for PR 19493 at commit

[GitHub] spark pull request #19487: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19487#discussion_r144580099 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -48,6 +49,16 @@ class

[GitHub] spark issue #19493: [SPARK-22252][SQL][followup] Command should not be a Lea...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19493 cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19493: [SPARK-22252][SQL][followup] Command should not b...

2017-10-13 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/19493 [SPARK-22252][SQL][followup] Command should not be a LeafNode ## What changes were proposed in this pull request? #19474 partially reverted #18064 but accidentally introduced a behavior

[GitHub] spark issue #19487: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19487 "" can come in via configuration files; I'd treat that the same as null. Things which aren't valid URIs though, that's something you want to bounce ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82732/ Test PASSed. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82732/testReport)** for PR 18979 at commit

[GitHub] spark issue #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19470 Now, it's passed again. :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

<    1   2   3   4   >