[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-10-13 Thread amankothari04
Github user amankothari04 commented on the issue: https://github.com/apache/spark/pull/16578 @viirya did you get a chance to review this ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19492: [SPARK-22228][SQL] Add support for array...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19492 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19487: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19487 Looking a bit more at this. I see it handles """ as well as empty, and also other forms of invalid URI which Path can't handle today ("multiple colons except with file:// on windows, etc).

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82730/ Test PASSed. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82730/testReport)** for PR 18979 at commit

[GitHub] spark pull request #19452: [SPARK-22136][SS] Evaluate one-sided conditions e...

2017-10-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/19452#discussion_r144532481 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSymmetricHashJoinHelperSuite.scala --- @@ -0,0 +1,118 @@ +/* + * Licensed

[GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...

2017-10-13 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19480#discussion_r144541081 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala --- @@ -201,6 +201,23 @@ class

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82725/ Test PASSed. ---

[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82726/ Test PASSed. ---

[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82726/testReport)** for PR 19464 at commit

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82731/testReport)** for PR 18979 at commit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19448 Thanks for reviewing this/getting it in. Personally, I had it in the "improvement" category rather than bug fix. If it wasn't for that line in the docs, there'd be no ambiguity about

[GitHub] spark issue #19487: [SPARK-21549][CORE] Respect OutputFormats with no/invali...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19487 LGTM. I'm going stick out today a slight roll of my PathOutputCommitter class which is one layer above FileOutputCommitter : lets people write committers without output & work paths, yet

[GitHub] spark issue #17290: [SPARK-16599][CORE] java.util.NoSuchElementException: No...

2017-10-13 Thread aphasingnirvana
Github user aphasingnirvana commented on the issue: https://github.com/apache/spark/pull/17290 @srowen I believe the bug still persists. Shouldn't we reopen it? --- - To unsubscribe, e-mail:

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82730/testReport)** for PR 18979 at commit

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19476 **[Test build #82727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82727/testReport)** for PR 19476 at commit

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19476 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19476: [SPARK-22062][CORE] Spill large block to disk in BlockMa...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19476 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82727/ Test PASSed. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82732/testReport)** for PR 18979 at commit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19448 I guess we wouldn't change the docs in branch-2.2 alone as we have a safe fix here for this mismatch anyway. I think I just wanted to say this backport can be justified. ---

[GitHub] spark issue #19491: [SPARK-22273][SQL] Fix key/value schema field names in H...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19491 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82728/ Test PASSed. ---

[GitHub] spark issue #19491: [SPARK-22273][SQL] Fix key/value schema field names in H...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19491 **[Test build #82728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82728/testReport)** for PR 19491 at commit

[GitHub] spark issue #19488: [SPARK-22266][SQL] The same aggregate function was evalu...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19488 **[Test build #82733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82733/testReport)** for PR 19488 at commit

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19480 **[Test build #82734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82734/testReport)** for PR 19480 at commit

[GitHub] spark issue #17290: [SPARK-16599][CORE] java.util.NoSuchElementException: No...

2017-10-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17290 I don't think this change was the ultimate fix, and it caused another problem, so no I don't think this PR should be reopened. ---

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18805 **[Test build #82729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82729/testReport)** for PR 18805 at commit

[GitHub] spark pull request #19487: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-10-13 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/19487#discussion_r144545827 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -60,15 +71,6 @@ class

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18805 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19452: [SPARK-22136][SS] Evaluate one-sided conditions e...

2017-10-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/19452#discussion_r144533029 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSymmetricHashJoinHelperSuite.scala --- @@ -0,0 +1,118 @@ +/* + * Licensed

[GitHub] spark pull request #19488: [SPARK-22266][SQL] The same aggregate function wa...

2017-10-13 Thread maryannxue
Github user maryannxue commented on a diff in the pull request: https://github.com/apache/spark/pull/19488#discussion_r144543377 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -205,14 +205,15 @@ object PhysicalAggregation {

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19448 PS, for people who are interested in dynamic committers, [MAPREDUCE-6823](https://issues.apache.org/jira/browse/MAPREDUCE-6823) is something to look at. It allows you to switch committers

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19494: [SPARK-22249][SQL] isin with empty list throws ex...

2017-10-13 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/19494 [SPARK-22249][SQL] isin with empty list throws exception on cached DataFrame ## What changes were proposed in this pull request? As pointed out in the JIRA, there is a bug which causes an

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19480 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82735/ Test FAILed. ---

[GitHub] spark issue #19494: [SPARK-22249][SQL] isin with empty list throws exception...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19494 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19494: [SPARK-22249][SQL] isin with empty list throws exception...

2017-10-13 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19494 @srowen do you mean replacing `contains` with `exists`? If so, might you please explain me why `exists` is a better option? Thanks. ---

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144597485 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19448 Okay. I am sorry for this trouble. Should we revert this if you guys feel strongly about it? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/19448 I have a lot of sympathy for the argument that infrastructure software shouldn't have too many backports and that those should be generally bug fixes. But, if I were working on a Spark distribution

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144602775 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala --- @@ -84,25 +84,45 @@ private[sql] trait ColumnarBatchScan

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82741/testReport)** for PR 19419 at commit

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 Turns out that's caused by SparkContext failing to clean up after itself when the `UnsatisfiedLinkError` happens, so those errors are red herrings... ---

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82738/ Test PASSed. ---

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19269 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19480 **[Test build #82735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82735/testReport)** for PR 19480 at commit

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19448 @steveloughran Thanks for your inputs. Totally agree on your opinions. Spark is an infrastructure software. We have to be very careful when backporting the PRs. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18979 Could you resolve the conflicts again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144599111 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18460 Thank you, @cloud-fan , @gatorsmile , and @viirya !!! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19448 I am not really worried about this particular change. It's already merged and it seems a small and safe change. I am not planning to revert it. But, in general, let's avoid of merging changes

[GitHub] spark pull request #19464: [SPARK-22233] [core] Allow user to filter out emp...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19464#discussion_r144604878 --- Diff: core/src/test/scala/org/apache/spark/FileSuite.scala --- @@ -510,4 +510,87 @@ class FileSuite extends SparkFunSuite with LocalSparkContext {

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19451 @rxin I think it would be better to keep all the rules of the "Replace Operators" batch in a single file. So if you prefer to keep the rule in a new file, we can move all the replace operator

[GitHub] spark issue #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82738/testReport)** for PR 19269 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82742/testReport)** for PR 19419 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82737/testReport)** for PR 19419 at commit

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18979 Hi, @steveloughran . > is the issue with ORC that if there's nothing to write, it doesn't generate a file (so avoiding that issue with sometimes you get 0-byte ORC files & things

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82732/ Test PASSed. ---

[GitHub] spark issue #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsT...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18979 **[Test build #82731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82731/testReport)** for PR 18979 at commit

[GitHub] spark issue #19470: [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark s...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19470 BTW, @cloud-fan . Could you review #18460 , too? I think we need your final approval. :) --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144594930 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144600676 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144601470 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark issue #18460: [SPARK-21247][SQL] Type comparison should respect case-s...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18460 LGTM, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19354: [SPARK-20992][Scheduler] Add links in documentation to N...

2017-10-13 Thread barnardb
Github user barnardb commented on the issue: https://github.com/apache/spark/pull/19354 I totally understand the reluctance to have non-ASF projects in a list headed by "The system currently supports…". Looking at the [Powered By](https://spark.apache.org/powered-by.html) page, it

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19451 **[Test build #82740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82740/testReport)** for PR 19451 at commit

[GitHub] spark issue #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-13 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19451 @gatorsmile > Could you please add an end-to-end testsuite except.sql of SQLQueryTestSuite.scala? Please verify `except.sql ` and `except.sql.out` files are enough for the

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 I haven't been able to reproduce the issue locally, but looking at the jenkins logs I see a bunch of exceptions like these: ``` 17/10/13 06:53:26.609 dispatcher-event-loop-15 ERROR

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19448 Sure, I will and let me note it ahead next time. I made a mistake while trying to think of reasons for this backport. --- -

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144609180 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,37 @@ import

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread krishna-pandey
Github user krishna-pandey commented on the issue: https://github.com/apache/spark/pull/19419 @jerryshao removed Whitespace at end of line 440 in package.scala. ok to test. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #18979: [SPARK-21762][SQL] FileFormatWriter/BasicWriteTas...

2017-10-13 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18979#discussion_r144595826 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala --- @@ -44,20 +47,32 @@ case class

[GitHub] spark issue #19448: [SPARK-22217] [SQL] ParquetFileFormat to support arbitra...

2017-10-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/19448 @HyukjinKwon branch-2.2 is in a maintenance branch, I am not sure it is appropriate to merge this change to branch-2.2 since it is not really a bug fix. If the doc is not accurate, we should fix the

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-13 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r144598051 --- Diff: python/pyspark/sql/session.py --- @@ -510,9 +511,43 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr

[GitHub] spark pull request #18460: [SPARK-21247][SQL] Type comparison should respect...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18460 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-13 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r144603628 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,37 @@ import

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18805 (I'll file a bug and send a PR for it separately, btw.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19419 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19419 **[Test build #82741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82741/testReport)** for PR 19419 at commit

[GitHub] spark issue #19419: [SPARK-22188] [CORE] Adding security headers for prevent...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19419 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82741/ Test FAILed. ---

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82746/ Test PASSed. ---

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18747: [SPARK-20822][SQL] Generate code to directly get value f...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82746/testReport)** for PR 18747 at commit

[GitHub] spark pull request #19435: [MINOR][SS] "keyWithIndexToNumValues" -> "keyWith...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19435 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19467: [SPARK-22238] Fix plan resolution bug caused by EnsureSt...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19467 **[Test build #82747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82747/testReport)** for PR 19467 at commit

[GitHub] spark issue #19467: [SPARK-22238] Fix plan resolution bug caused by EnsureSt...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19467 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82747/ Test FAILed. ---

[GitHub] spark pull request #19385: [SPARK-11034] [LAUNCHER] [MESOS] Launcher: add su...

2017-10-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19385 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19495: [SPARK-22278][SS] Expose current event time watermark an...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19495 **[Test build #82751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82751/testReport)** for PR 19495 at commit

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144681758 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +175,119 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680448 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -122,7 +126,8 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144681733 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +175,119 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680213 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala --- @@ -159,7 +160,8 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680478 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +175,119 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680608 --- Diff: docs/running-on-mesos.md --- @@ -522,6 +522,43 @@ See the [configuration page](configuration.html) for information on Spark config

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680489 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +175,119 @@

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-13 Thread ArtRand
Github user ArtRand commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r144680353 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -122,7 +126,8 @@

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19452 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19452: [SPARK-22136][SS] Evaluate one-sided conditions early in...

2017-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19452 **[Test build #82748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82748/testReport)** for PR 19452 at commit

  1   2   3   4   >