[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19531 **[Test build #82899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82899/testReport)** for PR 19531 at commit

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19514 **[Test build #82901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82901/consoleFull)** for PR 19514 at commit

[GitHub] spark pull request #19485: [SPARK-20055] [Docs] Added documentation for load...

2017-10-18 Thread jomach
Github user jomach closed the pull request at: https://github.com/apache/spark/pull/19485 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19534 **[Test build #82903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82903/testReport)** for PR 19534 at commit

[GitHub] spark pull request #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark ...

2017-10-18 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19459#discussion_r145603645 --- Diff: python/pyspark/sql/session.py --- @@ -414,6 +415,73 @@ def _createFromLocal(self, data, schema): data = [schema.toInternal(row) for

[GitHub] spark issue #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector to abs...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82897/ Test PASSed. ---

[GitHub] spark issue #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector to abs...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19508 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19521 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19521 Thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r145602871 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,72 @@ import

[GitHub] spark issue #19534: [SPARK-22312][CORE] Fix bug in Executor allocation manag...

2017-10-18 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/19534 cc - @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19534: [SPARK-22312][CORE] Fix bug in Executor allocatio...

2017-10-18 Thread sitalkedia
GitHub user sitalkedia opened a pull request: https://github.com/apache/spark/pull/19534 [SPARK-22312][CORE] Fix bug in Executor allocation manager in running… ## What changes were proposed in this pull request? We often see the issue of Spark jobs stuck because the

[GitHub] spark issue #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19269 **[Test build #82902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82902/testReport)** for PR 19269 at commit

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

2017-10-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19527 Benchmark against multi-column one hot encoder. Multi-Col, Multiple run: The first commit. Run multiple `treeAggregate` on columns. Multi-Col, Single Run: Run one `treeAggregate` on

[GitHub] spark pull request #18747: [SPARK-20822][SQL] Generate code to directly get ...

2017-10-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18747#discussion_r145601384 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -23,21 +23,72 @@ import

[GitHub] spark issue #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector to abs...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19508 **[Test build #82897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82897/testReport)** for PR 19508 at commit

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestam...

2017-10-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145597733 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -42,6 +43,13 @@ object ArrowUtils { case

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestam...

2017-10-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r145597432 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -42,6 +43,13 @@ object ArrowUtils { case

[GitHub] spark issue #19530: [SPARK-22309][ML] Remove unused param in `LDAModel.getTo...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19530 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82898/ Test PASSed. ---

[GitHub] spark issue #19512: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19512 @FRosner, this was backported into branch-2.2 but this can't be automatically closed for some reasons. Could you close this one manually please? ---

[GitHub] spark issue #19530: [SPARK-22309][ML] Remove unused param in `LDAModel.getTo...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19530 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19512: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19512 Thanks. Merged to branch-2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19530: [SPARK-22309][ML] Remove unused param in `LDAModel.getTo...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19530 **[Test build #82898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82898/testReport)** for PR 19530 at commit

[GitHub] spark issue #19512: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19512 Seems fine to backport into 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19514 **[Test build #82901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82901/consoleFull)** for PR 19514 at commit

[GitHub] spark issue #19512: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19512 @rxin, looks the PR was merged into master by you. Do you think it's okay to backport to other branches too? --- - To

[GitHub] spark issue #19514: [SPARK-21551][Python] Increase timeout for PythonRDD.ser...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19514 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19533: Merge pull request #1 from apache/master

2017-10-18 Thread BiggerBrain
Github user BiggerBrain closed the pull request at: https://github.com/apache/spark/pull/19533 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19533: Merge pull request #1 from apache/master

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19533 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19533: Merge pull request #1 from apache/master

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19533 @BiggerBrain, looks mistakenly open. Could you close this please? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19533: Merge pull request #1 from apache/master

2017-10-18 Thread BiggerBrain
Github user BiggerBrain commented on the issue: https://github.com/apache/spark/pull/19533 get one commit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19533: Merge pull request #1 from apache/master

2017-10-18 Thread BiggerBrain
GitHub user BiggerBrain opened a pull request: https://github.com/apache/spark/pull/19533 Merge pull request #1 from apache/master get ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch

[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...

2017-10-18 Thread nkronenfeld
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 nope, using lazy val initialization won't work - at the very least, UnsafeKryoSerializerSuite modifies conf before context construction ---

[GitHub] spark issue #19532: [CORE]stage api modify the description format, add versi...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19532 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19524: [SPARK-22302][INFRA] Remove manual backports for ...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19524#discussion_r145593044 --- Diff: dev/run-tests --- @@ -20,4 +20,10 @@ FWDIR="$(cd "`dirname $0`"/..; pwd)" cd "$FWDIR" +PYTHON_VERSION_CHECK=$(python -c

[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82895/ Test PASSed. ---

[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19532: [CORE]stage api modify the description format, ad...

2017-10-18 Thread guoxiaolongzte
GitHub user guoxiaolongzte opened a pull request: https://github.com/apache/spark/pull/19532 [CORE]stage api modify the description format, add version api, modify the duration real-time calculation ## What changes were proposed in this pull request? stage api

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19524 **[Test build #82900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82900/testReport)** for PR 19524 at commit

[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #82895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82895/testReport)** for PR 19529 at commit

[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19531 **[Test build #82899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82899/testReport)** for PR 19531 at commit

[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...

2017-10-18 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19531 cc @cloud-fan @gatorsmile @ron8hu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19531: [SPARK-22310] [SQL] Refactor join estimation to i...

2017-10-18 Thread wzhfy
GitHub user wzhfy opened a pull request: https://github.com/apache/spark/pull/19531 [SPARK-22310] [SQL] Refactor join estimation to incorporate estimation logic for different kinds of statistics ## What changes were proposed in this pull request? The current join

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145591347 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #19530: [SPARK-22309][ML] Remove unused param in `LDAModel.getTo...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19530 **[Test build #82898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82898/testReport)** for PR 19530 at commit

[GitHub] spark pull request #19530: [SPARK-22309][ML] Remove unused param in `LDAMode...

2017-10-18 Thread zhengruifeng
GitHub user zhengruifeng opened a pull request: https://github.com/apache/spark/pull/19530 [SPARK-22309][ML] Remove unused param in `LDAModel.getTopicDistributionMethod` & destory `nodeToFeaturesBc` in RandomForest ## What changes were proposed in this pull request? Remove

[GitHub] spark issue #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector to abs...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19508 **[Test build #82897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82897/testReport)** for PR 19508 at commit

[GitHub] spark issue #17819: [SPARK-20542][ML][SQL] Add an API to Bucketizer that can...

2017-10-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17819 @MLnick Any more comments or thoughts on this I need to address? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19508: [SPARK-20783][SQL][Follow-up] Create ColumnVector...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19508#discussion_r145588024 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala --- @@ -495,6 +474,8 @@ private[columnar]

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19459 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82894/ Test PASSed. ---

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19459 **[Test build #82894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82894/testReport)** for PR 19459 at commit

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145586905 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala --- @@ -0,0 +1,137 @@ +/* + * Licensed

[GitHub] spark issue #19357: [SPARK-21322][SQL][WIP] support histogram in filter card...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19357 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19357: [SPARK-21322][SQL][WIP] support histogram in filter card...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19357 **[Test build #82896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82896/testReport)** for PR 19357 at commit

[GitHub] spark issue #19357: [SPARK-21322][SQL][WIP] support histogram in filter card...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19357 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82896/ Test FAILed. ---

[GitHub] spark issue #19357: [SPARK-21322][SQL][WIP] support histogram in filter card...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19357 **[Test build #82896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82896/testReport)** for PR 19357 at commit

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-18 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145579513 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java --- @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145577058 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #13252: [SPARK-15473][SQL] CSV data source writes header for emp...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13252 It's notbresolved yet. I am not working on this for now. Please take over this if you are willing to do. --- - To

[GitHub] spark issue #19529: Support alternative unit testing styles in external appl...

2017-10-18 Thread nkronenfeld
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 There is one small hack in the way this was done, which is documented - see the comments and documentation on SharedSparkSession.initializeSession and SharedSparkContext.initializeContext. I

[GitHub] spark issue #19505: [WIP][SPARK-20396][SQL][PySpark][FOLLOW-UP] groupby().ap...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19505 I meant to ask if others agree with the current change as I could not see the ongoing discussion at that time. --- - To

[GitHub] spark issue #19529: Support alternative unit testing styles in external appl...

2017-10-18 Thread nkronenfeld
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 I made my original changes here by using git mv PlanTest.scala PlanTestBase.scala git mv SQLTestUnit.scala SQLTestUnitBase.scala git mv SharedSQLContext.scala

[GitHub] spark issue #19529: Support alternative unit testing styles in external appl...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #82895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82895/testReport)** for PR 19529 at commit

[GitHub] spark pull request #19529: Support alternative unit testing styles in extern...

2017-10-18 Thread nkronenfeld
GitHub user nkronenfeld opened a pull request: https://github.com/apache/spark/pull/19529 Support alternative unit testing styles in external applications ## What changes were proposed in this pull request? Support unit tests of external code (i.e., applications that use spark)

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-10-18 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/18805 Created https://github.com/luben/zstd-jni/issues/47. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145572702 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala --- @@ -0,0 +1,137 @@ +/* + *

[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19485 Could it be an option to leave a link back to the new page in the API doc to refer the options and remove the option list in API doc @gatorsmile and @liancheng? ---

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145569192 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #19459: [SPARK-20791][PYSPARK] Use Arrow to create Spark DataFra...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19459 **[Test build #82894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82894/testReport)** for PR 19459 at commit

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145569379 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19505: [WIP][SPARK-20396][SQL][PySpark][FOLLOW-UP] group...

2017-10-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19505#discussion_r145568265 --- Diff: python/pyspark/sql/functions.py --- @@ -2192,67 +2208,82 @@ def pandas_udf(f=None, returnType=StringType()): :param f: user-defined

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145567644 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #19524: [SPARK-22302][INFRA] Remove manual backports for subproc...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19524 Sure, thanks. Let me update soon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...

2017-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19485 My only worry is duplication and we would have another place to update the doc for options. Others sound okay to me too. ---

[GitHub] spark pull request #19269: [SPARK-22026][SQL] data source v2 write path

2017-10-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r145563004 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriter.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19505: [WIP][SPARK-20396][SQL][PySpark][FOLLOW-UP] group...

2017-10-18 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19505#discussion_r145552397 --- Diff: python/pyspark/sql/functions.py --- @@ -2192,67 +2208,82 @@ def pandas_udf(f=None, returnType=StringType()): :param f: user-defined

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82893/ Test PASSed. ---

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19439 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19439: [SPARK-21866][ML][PySpark] Adding spark image reader

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19439 **[Test build #82893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82893/testReport)** for PR 19439 at commit

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19383 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-10-18 Thread imaxxs
Github user imaxxs commented on the issue: https://github.com/apache/spark/pull/18784 I work with @ArtRand and @susanxhuynh Fine grained mode has been deprecated for a while. If it is standard procedure to wait till next release and if that is 3.0 we should wait till Spark 3.0

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19383 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82890/ Test PASSed. ---

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19383 **[Test build #82890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82890/testReport)** for PR 19383 at commit

[GitHub] spark issue #19528: [SPARK-20393] [Core] Existing patch applied to 1.6 branc...

2017-10-18 Thread ambauma
Github user ambauma commented on the issue: https://github.com/apache/spark/pull/19528 Understood. Working on porting to 2.0... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82888/ Test PASSed. ---

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19509 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19509: [SPARK-22290][core] Avoid creating Hive delegation token...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19509 **[Test build #82888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82888/testReport)** for PR 19509 at commit

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19521 Thank you, @rxin ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19521: [SPARK-22300][BUILD] Update ORC to 1.4.1

2017-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19521 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82892/ Test PASSed. ---

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18692 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18692 **[Test build #82892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82892/testReport)** for PR 18692 at commit

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18692 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82891/ Test PASSed. ---

[GitHub] spark issue #18692: [SPARK-21417][SQL] Infer join conditions using propagate...

2017-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18692 **[Test build #82891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82891/testReport)** for PR 18692 at commit

[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...

2017-10-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19485 Appreciate it. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...

2017-10-18 Thread jomach
Github user jomach commented on the issue: https://github.com/apache/spark/pull/19485 @gatorsmile will do --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...

2017-10-18 Thread MrBago
Github user MrBago commented on a diff in the pull request: https://github.com/apache/spark/pull/19527#discussion_r145531800 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...

2017-10-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19485 Just checked it with @liancheng Both think creating a separate page sounds good. Also cc @rxin --- - To

  1   2   3   4   >