[GitHub] spark pull request #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if...

2018-07-31 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/21852#discussion_r206695251 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -416,6 +416,21 @@ object SimplifyConditionals

[GitHub] spark issue #21847: [SPARK-24855][SQL][EXTERNAL]: Built-in AVRO support shou...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21847 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries

2018-07-31 Thread jzhuge
Github user jzhuge commented on a diff in the pull request: https://github.com/apache/spark/pull/21911#discussion_r206693956 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala --- @@ -102,6 +104,39 @@ object ResolveHints { }

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21305 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93844/ Test FAILed. ---

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21305 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21305 **[Test build #93844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93844/testReport)** for PR 21305 at commit

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19449 **[Test build #4225 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4225/testReport)** for PR 19449 at commit

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21927 > Second thought: PartitionPruningRDD is just an implementation of RDD. Every user / developer can implement a similar one. Also this doesn't handle the case mentioned by @felixcheung :

[GitHub] spark pull request #21847: [SPARK-24855][SQL][EXTERNAL]: Built-in AVRO suppo...

2018-07-31 Thread lindblombr
Github user lindblombr commented on a diff in the pull request: https://github.com/apache/spark/pull/21847#discussion_r206691928 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala --- @@ -120,7 +133,7 @@ class AvroSerializer(rootCatalystType:

[GitHub] spark pull request #21847: [SPARK-24855][SQL][EXTERNAL]: Built-in AVRO suppo...

2018-07-31 Thread lindblombr
Github user lindblombr commented on a diff in the pull request: https://github.com/apache/spark/pull/21847#discussion_r206691906 --- Diff: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala --- @@ -87,17 +87,30 @@ class AvroSerializer(rootCatalystType:

[GitHub] spark issue #21915: [SPARK-24954][Core] Fail fast on job submit if run a bar...

2018-07-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21915 thanks @jiangxb1987, lgtm aside from @mengxr 's comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21847: [SPARK-24855][SQL][EXTERNAL]: Built-in AVRO support shou...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21847 **[Test build #93846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93846/testReport)** for PR 21847 at commit

[GitHub] spark issue #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if all th...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21852 **[Test build #93845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93845/testReport)** for PR 21852 at commit

[GitHub] spark pull request #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries

2018-07-31 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21911#discussion_r206689783 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala --- @@ -102,6 +104,39 @@ object ResolveHints { }

[GitHub] spark issue #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if all th...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce Hint for SQL Queries

2018-07-31 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21911 @jzhuge, I think it is confusing that this hint exposes the `shuffle` boolean flag. The Spark API makes a clear distinction between `repartition` and `coalesce` where `coalesce` means that Spark

[GitHub] spark issue #21852: [SPARK-24893] [SQL] Remove the entire CaseWhen if all th...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21852 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19449 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93841/ Test FAILed. ---

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19449 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19449 **[Test build #93841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93841/testReport)** for PR 19449 at commit

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21305 **[Test build #93844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93844/testReport)** for PR 21305 at commit

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21305 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21305 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21305: [SPARK-24251][SQL] Add AppendData logical plan.

2018-07-31 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21305 @cloud-fan, @gatorsmile, this has been ready for final review for a while. Do you think you'll have some time to look at it? ---

[GitHub] spark issue #21938: [SPARK-24982][SQL] UDAF resolution should not throw Asse...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21938 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21938: [SPARK-24982][SQL] UDAF resolution should not throw Asse...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21938 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21938: [SPARK-24982][SQL] UDAF resolution should not throw Asse...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21938 **[Test build #93843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93843/testReport)** for PR 21938 at commit

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r206682882 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,29 @@ class DAGScheduler( s"longer

[GitHub] spark issue #21938: [SPARK-24982][SQL] UDAF resolution should not throw Asse...

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21938 @rxin, thanks! I am a bot who has found some folks who might be able to help with the review:@gatorsmile, @cloud-fan and @yhuai ---

[GitHub] spark pull request #21934: [SPARK-24951][SQL] Table valued functions should ...

2018-07-31 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21934#discussion_r206682497 --- Diff: sql/core/src/test/resources/sql-tests/results/table-valued-functions.sql.out --- @@ -83,8 +83,13 @@ select * from range(1, null) --

[GitHub] spark pull request #21938: [SPARK-24982][SQL] UDAF resolution should not thr...

2018-07-31 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/21938 [SPARK-24982][SQL] UDAF resolution should not throw AssertionError ## What changes were proposed in this pull request? When user calls anUDAF with the wrong number of arguments, Spark previously

[GitHub] spark pull request #21934: [SPARK-24951][SQL] Table valued functions should ...

2018-07-31 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/21934#discussion_r206681479 --- Diff: sql/core/src/test/resources/sql-tests/results/table-valued-functions.sql.out --- @@ -83,8 +83,13 @@ select * from range(1, null) -- !query 6

[GitHub] spark pull request #21934: [SPARK-24951][SQL] Table valued functions should ...

2018-07-31 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21934#discussion_r206680796 --- Diff: sql/core/src/test/resources/sql-tests/results/table-valued-functions.sql.out --- @@ -83,8 +83,13 @@ select * from range(1, null) --

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15970 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15970 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #21925: [SPARK-24973][PYTHON]Add numIter to Python Cluste...

2018-07-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21925 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21925: [SPARK-24973][PYTHON]Add numIter to Python ClusteringSum...

2018-07-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21925 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21932: [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93833/ Test PASSed. ---

[GitHub] spark issue #21925: [SPARK-24973][PYTHON]Add numIter to Python ClusteringSum...

2018-07-31 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21925 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21488: [SPARK-18057][SS] Update Kafka client version fro...

2018-07-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21488 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21932: [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21932 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21932: [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21932 **[Test build #93833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93833/testReport)** for PR 21932 at commit

[GitHub] spark issue #17280: [SPARK-19939] [ML] Add support for association rules in ...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17280 **[Test build #93842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93842/testReport)** for PR 17280 at commit

[GitHub] spark issue #17280: [SPARK-19939] [ML] Add support for association rules in ...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17280 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #17280: [SPARK-19939] [ML] Add support for association rules in ...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17280 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 Thanks! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes no objections. There's no concrete problem, and there is upside to making the change. I think you're aware of the potential issues, and have thought through them in the context of deeper

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @srowen May I read your comment as "no objections"? The current PR looks good to me. If you don't have objections, I will go ahead and merge it. ---

[GitHub] spark issue #21934: [SPARK-24951][SQL] Table valued functions should throw A...

2018-07-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21934 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes, good argument. Having been burned in the past (actually, by Kafka and ZK changes, though that's in the past), I'm aware that even compiling against version B instead of A, with no library code

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Anyway, overall I think you should definitely make this change. Spark users are currently penalised heavily when running on clusters with the message format introduced in 0.11.0, which has important

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @ijuma I see. I was looking at 0.10 jar. Thanks for correcting me. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 @zsxwing `org.apache.kafka.clients.admin.AdminClient` is in the clients jar. It's not relevant for this PR, I am just mentioning it so if Spark decides to migrate to that eventually. I see code using

[GitHub] spark issue #21934: [SPARK-24951][SQL] Table valued functions should throw A...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21934 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93834/ Test FAILed. ---

[GitHub] spark issue #21934: [SPARK-24951][SQL] Table valued functions should throw A...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21934 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Yeah, the Java client libraries have been evolved in a compatible manner for the most part since 0.10.0. The set of broker versions supported by 0.10.0 and 2.0.0 is exactly the same. The

[GitHub] spark issue #21934: [SPARK-24951][SQL] Table valued functions should throw A...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21934 **[Test build #93834 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93834/testReport)** for PR 21934 at commit

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @srowen just to be clear, `AdminClient` is not in `kafka-clients` jar. The user has to add `kafka` jar as a dependency to use `AdminClient`. In addition, even if this upgrade has some

[GitHub] spark issue #21892: [SPARK-24945][SQL] Switching to uniVocity 2.7.2

2018-07-31 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21892 @jbax I got the following exception on **2.7.3-SNAPSHOT** (commit e51b0958a): ``` Internal state when error was thrown: line=20, column=20481, record=20, charIndex=82594, headers=[col0,...,

[GitHub] spark issue #21884: [SPARK-24960][Kubernetes] explicitly expose ports on dri...

2018-07-31 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21884 Actually can you also update the unit test? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21884: [SPARK-24960][Kubernetes] explicitly expose ports on dri...

2018-07-31 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21884 Can you quickly change `[Kubernetes`] to `[K8S]` in the PR title? Will merge after that is done. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19449 **[Test build #93841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93841/testReport)** for PR 19449 at commit

[GitHub] spark pull request #21885: [SPARK-24926][CORE] Ensure numCores is used consi...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21885#discussion_r206652789 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2668,17 +2668,24 @@ object SparkContext extends Logging { } /**

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19449 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #19449: [SPARK-22219][SQL] Refactor code to get a value for "spa...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19449 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21885: [SPARK-24926][CORE] Ensure numCores is used consi...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21885#discussion_r206652822 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2668,17 +2668,24 @@ object SparkContext extends Logging { } /**

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes, and that then impacts programs that depend on this module to use Kafka 0.10+ from Spark. They'd have to update `kafka-clients` too, or would be automatically updated really. I don't see a

[GitHub] spark issue #21937: [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21937 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21937: [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21937 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 @srowen I meant that the library is compatible, so if you just change the version in the pom, it's fine. If you changed the code to use some methods in AdminClient, then you'd have to be more

[GitHub] spark issue #21937: [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21937 **[Test build #93840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93840/testReport)** for PR 21937 at commit

[GitHub] spark issue #21937: [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion

2018-07-31 Thread holdensmagicalunicorn
Github user holdensmagicalunicorn commented on the issue: https://github.com/apache/spark/pull/21937 @kiszk, thanks! I am a bot who has found some folks who might be able to help with the review:@ueshin, @rxin and @colorant ---

[GitHub] spark pull request #21937: [WIP][SPARK-23914][SQL][follow-up] refactor Array...

2018-07-31 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/21937 [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion ## What changes were proposed in this pull request? This PR refactors `ArrayUnion` based on [this

[GitHub] spark issue #21885: [SPARK-24926][CORE] Ensure numCores is used consistently...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21885 **[Test build #4224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4224/testReport)** for PR 21885 at commit

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 @ijuma this changes the version of `kakfa-clients` in compile scope, too -- right? looks like it in the POM. That's not just a test change. It's probably OK if the client is backwards-compatible

[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....

2018-07-31 Thread ijuma
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Yes, Kafka clients 0.10.2 and higher support brokers from 0.10.0 and higher if the protocols being used are available in that version. Since we only changed test code in this PR, upgrading to 2.0.0

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206645632 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206645451 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206645799 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -156,6 +160,14 @@ private[yarn] class

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206645146 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206644083 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -249,6 +259,10 @@ private[spark] class Client(

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206644780 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206645302 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206644521 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,121 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206644203 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,121 @@ +/* + * Licensed to the

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206643906 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -86,6 +86,13 @@ private[spark] class Client(

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93831/ Test PASSed. ---

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21103 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21103 **[Test build #93831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93831/testReport)** for PR 21103 at commit

[GitHub] spark issue #21935: [SPARK-24773] Avro: support logical timestamp type with ...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21935 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21935: [SPARK-24773] Avro: support logical timestamp type with ...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21935 **[Test build #93838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93838/testReport)** for PR 21935 at commit

[GitHub] spark issue #21935: [SPARK-24773] Avro: support logical timestamp type with ...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21935 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93838/ Test FAILed. ---

[GitHub] spark pull request #21858: [SPARK-24899][SQL][DOC] Add example of monotonica...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21858#discussion_r206642272 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/MonotonicallyIncreasingID.scala --- @@ -80,7 +80,5 @@ case class

[GitHub] spark pull request #21858: [SPARK-24899][SQL][DOC] Add example of monotonica...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21858#discussion_r206643036 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -1150,16 +1150,48 @@ object functions { /** * A column

[GitHub] spark issue #21936: [SPARK-24981][Core] ShutdownHook timeout causes job to f...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21936 **[Test build #93839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93839/testReport)** for PR 21936 at commit

[GitHub] spark issue #21885: [SPARK-24926][CORE] Ensure numCores is used consistently...

2018-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21885 **[Test build #4224 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4224/testReport)** for PR 21885 at commit

[GitHub] spark pull request #21885: [SPARK-24926][CORE] Ensure numCores is used consi...

2018-07-31 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21885#discussion_r206641823 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2668,17 +2668,24 @@ object SparkContext extends Logging { } /**

[GitHub] spark issue #21936: [SPARK-24981][Core] ShutdownHook timeout causes job to f...

2018-07-31 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21936 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21936: [SPARK-24981][Core] ShutdownHook timeout causes job to f...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21936 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21936: [SPARK-24981][Core] ShutdownHook timeout causes job to f...

2018-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21936 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   6   7   >