[GitHub] spark pull request #20438: [SPARK-23272][SQL] add calendar interval type sup...

2018-01-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20438#discussion_r164717772 --- Diff: sql/core/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java --- @@ -195,6 +196,7 @@ * struct field. */ public

[GitHub] spark pull request #20438: [SPARK-23272][SQL] add calendar interval type sup...

2018-01-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20438#discussion_r164719793 --- Diff: sql/core/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java --- @@ -135,9 +135,7 @@ public UTF8String getUTF8String(int ordinal) {

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20434 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20434 **[Test build #86813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86813/testReport)** for PR 20434 at commit

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20436 **[Test build #86821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86821/testReport)** for PR 20436 at commit

[GitHub] spark issue #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86811/ Test PASSed. ---

[GitHub] spark issue #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20431 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20431 **[Test build #86811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86811/testReport)** for PR 20431 at commit

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20343 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86812/ Test PASSed. ---

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20343 **[Test build #86812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86812/testReport)** for PR 20343 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86820/testReport)** for PR 20435 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86820/ Test FAILed. ---

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/382/

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86820/testReport)** for PR 20435 at commit

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20378 > One question I have is, do the current violations cause significant document error? I think this is a good point. Maybe, we could enable ones fixing actual significant problems, at

[GitHub] spark pull request #20427: [SPARK-23260][SPARK-23262][SQL] several data sour...

2018-01-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20427 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20427: [SPARK-23260][SPARK-23262][SQL] several data source v2 n...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20427 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20438: [SPARK-23272][SQL] add calendar interval type support to...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20438 **[Test build #86819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86819/testReport)** for PR 20438 at commit

[GitHub] spark issue #20438: [SPARK-23272][SQL] add calendar interval type support to...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20438 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...

2018-01-30 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/20404 @HyukjinKwon I'm afraid that's not enough because we need to synchronize at all the places we touch the `defaultSession`, otherwise another thread can set the value between `defaultSession.get() eq

[GitHub] spark issue #20438: [SPARK-23272][SQL] add calendar interval type support to...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20438 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/381/

[GitHub] spark issue #20438: [SPARK-23272][SQL] add calendar interval type support to...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20438 cc @sameeragarwal @viirya @kiszk @ueshin @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20438: [SPARK-23272][SQL] add calendar interval type sup...

2018-01-30 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/20438 [SPARK-23272][SQL] add calendar interval type support to ColumnVector ## What changes were proposed in this pull request? `ColumnVector` is aimed to support all the data types, but

[GitHub] spark issue #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream Streami...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20437 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20432 **[Test build #86818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86818/testReport)** for PR 20432 at commit

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20432 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/380/

[GitHub] spark issue #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream Streami...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20437 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20433: [SPARK-23264][SQL] Support interval values without INTER...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20433 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86810/ Test FAILed. ---

[GitHub] spark issue #20433: [SPARK-23264][SQL] Support interval values without INTER...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20433 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20433: [SPARK-23264][SQL] Support interval values without INTER...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20433 **[Test build #86810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86810/testReport)** for PR 20433 at commit

[GitHub] spark issue #20427: [SPARK-23260][SPARK-23262][SQL] several data source v2 n...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86805/ Test PASSed. ---

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-30 Thread guoxiaolongzte
GitHub user guoxiaolongzte opened a pull request: https://github.com/apache/spark/pull/20437 [SPARK-23270][Streaming][WEB-UI]FileInputDStream Streaming UI 's records should not be set to the default value of 0, it should be the total number of rows of new files. ## What changes

[GitHub] spark issue #20427: [SPARK-23260][SPARK-23262][SQL] several data source v2 n...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20427 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20432 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20427: [SPARK-23260][SPARK-23262][SQL] several data source v2 n...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20427 **[Test build #86805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86805/testReport)** for PR 20427 at commit

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20432 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20432 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86803/ Test FAILed. ---

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20432 **[Test build #86803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86803/testReport)** for PR 20432 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86817 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86817/testReport)** for PR 20435 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86817/ Test FAILed. ---

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86817/testReport)** for PR 20435 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/379/

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86806/ Test FAILed. ---

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20422 **[Test build #86806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86806/testReport)** for PR 20422 at commit

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread ashashwat
Github user ashashwat commented on the issue: https://github.com/apache/spark/pull/20436 @HyukjinKwon Let me go ahead and check all the scripts for similar instances or typos. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19340: [SPARK-22119][ML] Add cosine distance to KMeans

2018-01-30 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19340 @Kevin-Ferret pointed out that both the input and the centers should be normalized to unit Euclidean length. Citing you, > the solution is also the arithmetic mean only if all vectors are

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20436 I would rather suggest to fix these together in the followup already open, 20432 or find out more typos while we are here. Can you take another look for other scripts and check if there are

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20436 **[Test build #86816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86816/testReport)** for PR 20436 at commit

[GitHub] spark pull request #20436: [SPARK-23174][DOC][PYTHON] python code style chec...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20436#discussion_r164706951 --- Diff: dev/lint-python --- @@ -60,9 +60,9 @@ export "PYLINT_HOME=$PYTHONPATH" export "PATH=$PYTHONPATH:$PATH" # There is no need to

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20436 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20408 **[Test build #86815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86815/testReport)** for PR 20408 at commit

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20436 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20436: [SPARK-23174][DOC][PYTHON] python code style checker upd...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20436 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20436: [SPARK-23174][DOC][PYTHON] python code style chec...

2018-01-30 Thread ashashwat
GitHub user ashashwat opened a pull request: https://github.com/apache/spark/pull/20436 [SPARK-23174][DOC][PYTHON] python code style checker update fix. ## What changes were proposed in this pull request? Consistency in style, grammar and removal of extraneous characters.

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164703291 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ConsoleWriterSuite.scala --- @@ -34,9 +33,9 @@ class

[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20404 How about something like this? ```diff --- a/python/pyspark/sql/session.py +++ b/python/pyspark/sql/session.py @@ -213,10 +213,10 @@ class SparkSession(object):

[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

2018-01-30 Thread Jiri-Kremser
Github user Jiri-Kremser commented on the issue: https://github.com/apache/spark/pull/19802 @srowen was Jenkins ok with the change? I can't see the results. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/378/

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...

2018-01-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20404 For perfectness, I think we should hold a lock with JVM instance but I wonder if it's easily possible. I roughly knew this but I think underestimated this because I believe that will quite

[GitHub] spark pull request #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hu...

2018-01-30 Thread rednaxelafx
Github user rednaxelafx commented on a diff in the pull request: https://github.com/apache/spark/pull/20434#discussion_r164687283 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -660,12 +660,10 @@ object SQLConf { val

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86814/ Test FAILed. ---

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86814/testReport)** for PR 20435 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20435 **[Test build #86814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86814/testReport)** for PR 20435 at commit

[GitHub] spark pull request #20435: [SPARK-23268][SQL]Reorganize packages in data sou...

2018-01-30 Thread gengliangwang
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/20435 [SPARK-23268][SQL]Reorganize packages in data source V2 ## What changes were proposed in this pull request? 1. create a new package for partitioning/distribution related classes.

[GitHub] spark issue #19340: [SPARK-22119][ML] Add cosine distance to KMeans

2018-01-30 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/19340 @mgaido91 @srowen I have the same concern as @Kevin-Ferret and @viirya I don't find the normailization of vectors before training, and the update of center seems incorrect. The

[GitHub] spark pull request #20361: [SPARK-23188][SQL] Make vectorized columar reader...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20361#discussion_r164685543 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -377,6 +377,12 @@ object SQLConf { .booleanConf

[GitHub] spark issue #20430: [SPARK-23263][SQL] Create table stored as parquet should...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20430 CC @wzhfy --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer.commit ...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20386 I like this change! It adds a missing feature which is required for migrating the file-based data source(which use `FileCommitProtocol` and has a callback for task commit), and also make it

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164681157 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ConsoleWriterSuite.scala --- @@ -34,9 +33,9 @@ class

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164680877 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkV2Suite.scala --- @@ -41,19 +41,22 @@ class MemorySinkV2Suite

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164680686 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memoryV2.scala --- @@ -135,14 +142,21 @@ class MemoryStreamWriter(val

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164680538 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ConsoleWriter.scala --- @@ -39,13 +41,20 @@ class

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164680632 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memoryV2.scala --- @@ -118,14 +118,21 @@ class MemoryWriter(sink:

[GitHub] spark pull request #20386: [SPARK-23202][SQL] Break down DataSourceV2Writer....

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20386#discussion_r164680081 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala --- @@ -54,10 +54,6 @@ case class

[GitHub] spark issue #18339: [SPARK-21094][PYTHON] Add popen_kwargs to launch_gateway

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18339 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suit...

2018-01-30 Thread sameeragarwal
Github user sameeragarwal closed the pull request at: https://github.com/apache/spark/pull/20231 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20427: [SPARK-23260][SPARK-23262][SQL] several data source v2 n...

2018-01-30 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20427 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-01-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20378 One question I have is, do the current violations cause significant document error? Overall this is a good idea. However, is it worth enforcedly applying this if we consider the effort of

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20434 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20427: [SPARK-23260][SPARK-23262][SQL] several data sour...

2018-01-30 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20427#discussion_r164676886 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -23,7 +23,7 @@ import

[GitHub] spark pull request #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hu...

2018-01-30 Thread sameeragarwal
Github user sameeragarwal commented on a diff in the pull request: https://github.com/apache/spark/pull/20434#discussion_r164676771 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -660,12 +660,10 @@ object SQLConf { val

[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...

2018-01-30 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/20404 @felixcheung I see, in that case, we should revert the last commit (cc4b8510c1445fb742c0d750958d352adfa84902) to check the default session is updated or not? ---

[GitHub] spark pull request #20378: [SPARK-11222][Build][Python] Python document styl...

2018-01-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20378#discussion_r164672962 --- Diff: dev/run-tests.py --- @@ -576,7 +576,10 @@ def main(): for f in changed_files): #

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20434 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/377/

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20434 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20434 cc @sameeragarwal @zsxwing @rxin @cloud-fan @rednaxelafx @yhuai --- - To unsubscribe, e-mail:

[GitHub] spark issue #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMetho...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20434 **[Test build #86813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86813/testReport)** for PR 20434 at commit

[GitHub] spark pull request #20434: [SPARK-23267] [SQL] Increase spark.sql.codegen.hu...

2018-01-30 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/20434 [SPARK-23267] [SQL] Increase spark.sql.codegen.hugeMethodLimit to 65535 ## What changes were proposed in this pull request? Still saw the performance regression introduced by

[GitHub] spark issue #20433: [SPARK-23264][SQL] Support interval values without INTER...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20433 **[Test build #86810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86810/testReport)** for PR 20433 at commit

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-01-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20343 **[Test build #86812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86812/testReport)** for PR 20343 at commit

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-01-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/376/

<    1   2   3   4   5   6   7   >