[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17848#discussion_r127635707
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
 ---
@@ -103,4 +110,19 @@ case class UserDefinedFunction protected[sql] (
   udf
 }
   }
+
+  /**
+   * Updates UserDefinedFunction to non-deterministic.
+   *
+   * @since 2.3.0
+   */
+  def nonDeterministic(): UserDefinedFunction = {
--- End diff --

shall we follow existing naming style and make it `def 
withDeterministic(deterministic: Boolean)`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18580: [SPARK-21354] [SQL] INPUT FILE related functions do not ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18580
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79651/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18580: [SPARK-21354] [SQL] INPUT FILE related functions do not ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18580
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18580: [SPARK-21354] [SQL] INPUT FILE related functions do not ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18580
  
**[Test build #79651 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79651/testReport)**
 for PR 18580 at commit 
[`c4de2b8`](https://github.com/apache/spark/commit/c4de2b8e2583c55f1b761569050d2c21506c2291).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18580: [SPARK-21354] [SQL] INPUT FILE related functions do not ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18580
  
LGTM, pending jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18555
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18555
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79650/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18555
  
**[Test build #79650 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79650/testReport)**
 for PR 18555 at commit 
[`722f26e`](https://github.com/apache/spark/commit/722f26e55e53dac263f3a0bce899d9881d9e01dd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF and Ja...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17848
  
**[Test build #79658 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79658/testReport)**
 for PR 17848 at commit 
[`a336ffc`](https://github.com/apache/spark/commit/a336ffc1bd70050d82eef79f6179e25231b23aa1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18649
  
**[Test build #79657 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79657/testReport)**
 for PR 18649 at commit 
[`b5c963f`](https://github.com/apache/spark/commit/b5c963fec11b87cc43f542483d71422b79867750).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18649
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18649
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79657/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18649
  
**[Test build #79657 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79657/testReport)**
 for PR 18649 at commit 
[`b5c963f`](https://github.com/apache/spark/commit/b5c963fec11b87cc43f542483d71422b79867750).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18649
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18628: [SPARK-18061][ThriftServer] Add spnego auth support for ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18628
  
looks good according to the existing similar code, but I'm not very 
familiar with this part code of code, will leave it for a few days if someone 
else has some comments.

Also cc @jiangxb1987 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18628: [SPARK-18061][ThriftServer] Add spnego auth suppo...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18628#discussion_r127631011
  
--- Diff: 
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala
 ---
@@ -57,6 +59,20 @@ private[hive] class SparkSQLCLIService(hiveServer: 
HiveServer2, sqlContext: SQLC
 case e @ (_: IOException | _: LoginException) =>
   throw new ServiceException("Unable to login to kerberos with 
given principal/keytab", e)
   }
+
+  // Try creating spnego UGI if it is configured.
+  val principal = 
hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_PRINCIPAL)
+  val keyTabFile = hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_KEYTAB)
+  if (principal.nonEmpty && keyTabFile.nonEmpty) {
+try {
+  httpUGI = 
HiveAuthFactory.loginFromSpnegoKeytabAndReturnUGI(hiveConf)
+  setSuperField(this, "httpUGI", httpUGI)
+} catch {
+  case e: IOException =>
+throw new ServiceException("Unable to login to spnego with 
given principal/keytab " +
+  s"$principal/$keyTabFile", e)
--- End diff --

shall we hide the principal/keytab?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18631: [SPARK-21410][CORE] Create less partitions for RangePart...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18631
  
**[Test build #79656 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79656/testReport)**
 for PR 18631 at commit 
[`7239734`](https://github.com/apache/spark/commit/7239734a74b5684e0d8c7f21502008133ca04a75).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18631: [SPARK-21410][CORE] Create less partitions for RangePart...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18631
  
cc @srowen is it ok to do this? e.g. `rdd.sortByKey(numPartitions = 10)`, 
do we have to return an RDD of 10 partitions even some of them are empty?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18631: [SPARK-21410][CORE] Create less partitions for RangePart...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18631
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18612: [SPARK-21388][ML][PySpark] GBTs inherit from HasStepSize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18612
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79652/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18612: [SPARK-21388][ML][PySpark] GBTs inherit from HasStepSize...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18612
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18612: [SPARK-21388][ML][PySpark] GBTs inherit from HasStepSize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18612
  
**[Test build #79652 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79652/testReport)**
 for PR 18612 at commit 
[`94f6ca5`](https://github.com/apache/spark/commit/94f6ca517717214fb95edfedc971fb3ae16ddd15).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18615
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18615
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79654/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18615
  
**[Test build #79654 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79654/testReport)**
 for PR 18615 at commit 
[`8167ea1`](https://github.com/apache/spark/commit/8167ea13ab49983228807a06e1de9150c1acec1e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18615
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79653/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18615
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18615
  
**[Test build #79653 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79653/testReport)**
 for PR 18615 at commit 
[`4950b8d`](https://github.com/apache/spark/commit/4950b8d9d856ee4645283bea7def8a55d0a1a22a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18624: [SPARK-21389][ML][MLLIB] Optimize ALS recommendForAll by...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18624
  
**[Test build #79655 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79655/testReport)**
 for PR 18624 at commit 
[`4a9b350`](https://github.com/apache/spark/commit/4a9b35031b006f5320fbfd6ed62a00e022e87b2e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18624: [SPARK-21389][ML][MLLIB] Optimize ALS recommendForAll by...

[mpjlu]

Github user mpjlu commented on the issue:

https://github.com/apache/spark/pull/18624
  
I have checked the results with the master method, the recommendation 
results are right. 
The master TestSuite is too simple, should be updated.  I will update it.
Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18615
  
**[Test build #79654 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79654/testReport)**
 for PR 18615 at commit 
[`8167ea1`](https://github.com/apache/spark/commit/8167ea13ab49983228807a06e1de9150c1acec1e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18615
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable obj...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18615
  
Thanks @viirya. 

@holdenk, I happened to find another issue and tried to fix both here 
together.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18615: [SPARK-21394][SPARK-21432][PYTHON] Reviving calla...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18615#discussion_r127627470
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -2087,10 +2087,22 @@ def _wrapped(self):
 """
 Wrap this udf with a function and attach docstring from func
 """
-@functools.wraps(self.func)
+
+# It is possible for a callable instance without __name__ 
attribute or/and
+# __module__ attribute to be wrapped here For example, 
functools.partial. In this case,
--- End diff --

here For example -> here. For example


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18615: [SPARK-21394][PYTHON] Reviving callable object support i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18615
  
**[Test build #79653 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79653/testReport)**
 for PR 18615 at commit 
[`4950b8d`](https://github.com/apache/spark/commit/4950b8d9d856ee4645283bea7def8a55d0a1a22a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18612: [SPARK-21388][ML][PySpark] GBTs inherit from HasStepSize...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18612
  
**[Test build #79652 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79652/testReport)**
 for PR 18612 at commit 
[`94f6ca5`](https://github.com/apache/spark/commit/94f6ca517717214fb95edfedc971fb3ae16ddd15).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18641#discussion_r127626187
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -273,12 +274,31 @@ case class CaseWhenCodegen(
 val cases = branches.map { case (condExpr, valueExpr) =>
   val cond = condExpr.genCode(ctx)
   val res = valueExpr.genCode(ctx)
+  val (condFunc, condIsNull, condValue) = if ((cond.code.length >= 
512) &&
--- End diff --

Is it enough to only consider the generated code of individual condition? 
If it is less than 512, e.g, 500, but the combination of all conditions can 
still be large to cause the same issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18641#discussion_r127626045
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -72,11 +72,11 @@ case class If(predicate: Expression, trueValue: 
Expression, falseValue: Expressi
   (ctx.INPUT_ROW != null && ctx.currentVars == null)) {
 
   val (condFuncName, condGlobalIsNull, condGlobalValue) =
-createAndAddFunction(ctx, condEval, predicate.dataType, 
"evalIfCondExpr")
+CondExpression.createAndAddFunction(ctx, condEval, 
predicate.dataType, "evalIfCondExpr")
--- End diff --

NVM. I saw there's the check already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18641#discussion_r127625999
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -72,11 +72,11 @@ case class If(predicate: Expression, trueValue: 
Expression, falseValue: Expressi
   (ctx.INPUT_ROW != null && ctx.currentVars == null)) {
 
   val (condFuncName, condGlobalIsNull, condGlobalValue) =
-createAndAddFunction(ctx, condEval, predicate.dataType, 
"evalIfCondExpr")
+CondExpression.createAndAddFunction(ctx, condEval, 
predicate.dataType, "evalIfCondExpr")
--- End diff --

Shall we check `INPUT_ROW` too for these three functions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18641#discussion_r127625968
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -273,12 +274,31 @@ case class CaseWhenCodegen(
 val cases = branches.map { case (condExpr, valueExpr) =>
   val cond = condExpr.genCode(ctx)
   val res = valueExpr.genCode(ctx)
+  val (condFunc, condIsNull, condValue) = if ((cond.code.length >= 
512) &&
+// Split these expressions only if they are created from a row 
object
+(ctx.INPUT_ROW != null && ctx.currentVars == null)) {
+val (funcName, globalIsNull, globalValue) =
+  CondExpression.createAndAddFunction(ctx, cond, 
condExpr.dataType, "caseWhenCondExpr")
+(s"$funcName(${ctx.INPUT_ROW});", globalIsNull, globalValue)
+  } else {
+(cond.code, cond.isNull, cond.value)
+  }
+  val (resFunc, resIsNull, resValue) = if ((res.code.length >= 512) &&
+// Split these expressions only if they are created from a row 
object
+(ctx.INPUT_ROW != null && ctx.currentVars == null)) {
--- End diff --

We can move this condition above  and reuse it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18580: [SPARK-21354] [SQL] INPUT FILE related functions ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18580#discussion_r127625865
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 
---
@@ -409,6 +409,42 @@ object HiveOnlyCheck extends (LogicalPlan => Unit) {
   }
 }
 
+
+/**
+ * A rule to do various checks before reading a table.
+ */
+object PreReadCheck extends (LogicalPlan => Unit) {
+  def apply(plan: LogicalPlan): Unit = {
+plan.foreach {
+  case operator: LogicalPlan =>
+operator transformExpressionsUp {
+  case e @ (_: InputFileName | _: InputFileBlockLength | _: 
InputFileBlockStart) =>
+checkNumInputFileBlockSources(e, operator)
+e
+}
+}
+  }
+
+  private def checkNumInputFileBlockSources(e: Expression, operator: 
LogicalPlan): Int = {
+operator match {
+  case _: CatalogRelation => 1
+  case _ @ LogicalRelation(_: HadoopFsRelation, _, _) => 1
+  case _: LeafNode => 0
+  // UNION ALL has multiple children, but these children do not 
concurrently use InputFileBlock.
+  case u: Union =>
+if (u.children.map(checkNumInputFileBlockSources(e, _)).sum >= 1) 
1 else 0
+  case o =>
+val numInputFileBlockSources = 
o.children.map(checkNumInputFileBlockSources(e, _)).sum
+if (numInputFileBlockSources > 1) {
+  e.failAnalysis(s"'${e.prettyName}' does not support more than 
one sources")
--- End diff --

Need to check it as early as possible; otherwise, `Union` might eat it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18580: [SPARK-21354] [SQL] INPUT FILE related functions do not ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18580
  
**[Test build #79651 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79651/testReport)**
 for PR 18580 at commit 
[`c4de2b8`](https://github.com/apache/spark/commit/c4de2b8e2583c55f1b761569050d2c21506c2291).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13599
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13599
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79649/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13599
  
**[Test build #79649 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79649/testReport)**
 for PR 13599 at commit 
[`abdf7b7`](https://github.com/apache/spark/commit/abdf7b7a8a75dfc7b8de597611bbfa0af126e24e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18555
  
**[Test build #79650 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79650/testReport)**
 for PR 18555 at commit 
[`722f26e`](https://github.com/apache/spark/commit/722f26e55e53dac263f3a0bce899d9881d9e01dd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18646: [SPARK-19810][MINOR][FOLLOW-UP] Follow-ups from S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18646


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18646: [SPARK-19810][MINOR][FOLLOW-UP] Follow-ups from SPARK-19...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18646
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18580: [SPARK-21354] [SQL] INPUT FILE related functions ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18580#discussion_r127621186
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 ---
@@ -74,6 +74,15 @@ trait CheckAnalysis extends PredicateHelper {
 }
   }
 
+  private def getNumInputFileBlockSources(operator: LogicalPlan): Int = {
+operator match {
+  case _: LeafNode => 1
--- End diff --

Unable to check it in `CheckAnalysis`. Both `HadoopRDD` and `FileScanRDD` 
have the same issues. To block both, we need to add the check as another rule. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13599
  
**[Test build #79649 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79649/testReport)**
 for PR 13599 at commit 
[`abdf7b7`](https://github.com/apache/spark/commit/abdf7b7a8a75dfc7b8de597611bbfa0af126e24e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[debugger87]

Github user debugger87 commented on the issue:

https://github.com/apache/spark/pull/18649
  
@cloud-fan Could you please help me to review this PR? Thanks a lot!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18649
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver do...

[debugger87]

GitHub user debugger87 opened a pull request:

https://github.com/apache/spark/pull/18649

[SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't register operation 
log before execute sql statement

## What changes were proposed in this pull request?

- Add a private method `registerCurrentOperationLog` in 
SparkExecuteStatementOperation
- Call `registerCurrentOperationLog` before `execute()` and unregister it 
after `execute()`  like implementation in 
[SQLOperation.java#L204](https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/SQLOperation.java#L204)

## How was this patch tested?
Add unittest in HiveThriftBinaryServerSuite

```
build/mvn  -Phive-thriftserver -Dtest=none 
-DwildcardSuites=org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite
 test
```
HiveThriftBinaryServerSuite tests passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/debugger87/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18649.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18649


commit b5c963fec11b87cc43f542483d71422b79867750
Author: debugger87 
Date:   2017-07-16T20:07:48Z

[SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't register operation 
log before execute sql statement




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127614336
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/MathUtils.scala 
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+
+object MathUtils {
+
+  /**
+   *  Returns the bucket number into which
+   *  the value of this expression would fall after being evaluated.
+   *
+   * @param expr is the expression for which the histogram is being created
+   * @param minValue is an expression that resolves
+   * to the minimum end point of the acceptable range for 
expr
+   * @param maxValue is an expression that resolves
+   * to the maximum end point of the acceptable range for 
expr
+   * @param numBucket is an expression that resolves to
+   *  a constant indicating the number of buckets
+   * @return Returns an long between 0 and numBucket+1 by mapping the expr 
into buckets defined by
+   * the range [minValue, maxValue].
+   */
+  def widthBucket(expr: Double, minValue: Double, maxValue: Double, 
numBucket: Long): Long = {
+
+if (numBucket <= 0) {
+  throw new AnalysisException(s"The num of bucket must be greater than 
0, but got ${numBucket}")
+}
+
+val lower: Double = Math.min(minValue, maxValue)
+val upper: Double = Math.max(minValue, maxValue)
+
+val result: Long = if (expr < lower) {
+  0
+} else if (expr >= upper) {
+  numBucket + 1L
--- End diff --

// an overflow bucket numbered num_buckets+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127614328
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/MathUtils.scala 
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+
+object MathUtils {
+
+  /**
+   *  Returns the bucket number into which
+   *  the value of this expression would fall after being evaluated.
+   *
+   * @param expr is the expression for which the histogram is being created
+   * @param minValue is an expression that resolves
+   * to the minimum end point of the acceptable range for 
expr
+   * @param maxValue is an expression that resolves
+   * to the maximum end point of the acceptable range for 
expr
+   * @param numBucket is an expression that resolves to
+   *  a constant indicating the number of buckets
+   * @return Returns an long between 0 and numBucket+1 by mapping the expr 
into buckets defined by
+   * the range [minValue, maxValue].
+   */
+  def widthBucket(expr: Double, minValue: Double, maxValue: Double, 
numBucket: Long): Long = {
+
+if (numBucket <= 0) {
+  throw new AnalysisException(s"The num of bucket must be greater than 
0, but got ${numBucket}")
+}
+
+val lower: Double = Math.min(minValue, maxValue)
+val upper: Double = Math.max(minValue, maxValue)
+
+val result: Long = if (expr < lower) {
--- End diff --

// Creates (when needed) an underflow bucket numbered 0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

[skonto]

Github user skonto commented on the issue:

https://github.com/apache/spark/pull/18630
  
@jerryshao 

> 

Are you trying to support --packages in standalone cluster?

Yes that is my intention as stated in the description. I will proceed with 
the changes thnx for the review.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/18630#discussion_r127614213
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala ---
@@ -66,4 +75,50 @@ object DriverWrapper {
 System.exit(-1)
 }
   }
+
+  // R or Python are not supported in cluster mode so just get jars and 
files for the driver
+  private def setupDependencies(loader: MutableURLClassLoader, userJar: 
String): Unit = {
+
+var packagesExclusions = sys.props.get("spark.jars.excludes").orNull
+var packages = sys.props.get("spark.jars.packages").orNull
+var repositories = sys.props.get("spark.jars.repositories").orNull
+val hadoopConf = new HadoopConfiguration()
+val childClasspath = new ArrayBuffer[String]()
+var jars = sys.props.get("spark.jars").orNull
+var files = sys.props.get("spark.files").orNull
+var ivyRepoPath = sys.props.get("spark.jars.ivy").orNull
+
+val exclusions: Seq[String] =
+  if (!StringUtils.isBlank(packagesExclusions)) {
+packagesExclusions.split(",")
+  } else {
+Nil
+  }
+
+// Create the IvySettings, either load from file or build defaults
+val ivySettings = sys.props.get("spark.jars.ivySettings").map { 
ivySettingsFile =>
+  SparkSubmitUtils.loadIvySettings(ivySettingsFile, 
Option(repositories),
+Option(ivyRepoPath))
+}.getOrElse {
+  SparkSubmitUtils.buildIvySettings(Option(repositories), 
Option(ivyRepoPath))
+}
+
+val resolvedMavenCoordinates = 
SparkSubmitUtils.resolveMavenCoordinates(packages,
+  ivySettings, exclusions = exclusions)
+
+if (!StringUtils.isBlank(resolvedMavenCoordinates)) {
+  jars = SparkSubmit.mergeFileLists(jars, resolvedMavenCoordinates)
+}
+
+// filter out the user jar
+jars = 
jars.split(",").filterNot(_.contains(userJar.split("/").last)).mkString(",")
+jars = Option(jars).map(SparkSubmit.downloadFileList(_, 
hadoopConf)).orNull
--- End diff --

Ok will fix that...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/18630#discussion_r127614182
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala ---
@@ -66,4 +75,50 @@ object DriverWrapper {
 System.exit(-1)
 }
   }
+
+  // R or Python are not supported in cluster mode so just get jars and 
files for the driver
+  private def setupDependencies(loader: MutableURLClassLoader, userJar: 
String): Unit = {
+
+var packagesExclusions = sys.props.get("spark.jars.excludes").orNull
+var packages = sys.props.get("spark.jars.packages").orNull
+var repositories = sys.props.get("spark.jars.repositories").orNull
+val hadoopConf = new HadoopConfiguration()
+val childClasspath = new ArrayBuffer[String]()
+var jars = sys.props.get("spark.jars").orNull
--- End diff --

Ivy does the resolution and downloads them locally at the file system's 
node where the driver is launched on. So then you use the hadoop utilities to 
fetch them. That's the idea about jars. I was only focusing on packages... but 
yes need to write some more code to handle files and remote jars already passed 
though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127614121
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -1186,3 +1186,51 @@ case class BRound(child: Expression, scale: 
Expression)
 with Serializable with ImplicitCastInputTypes {
   def this(child: Expression) = this(child, Literal(0))
 }
+
+/**
+ *  Returns the bucket number into which
+ *  the value of this expression would fall after being evaluated.
+ *
+ * @param expr is the expression for which the histogram is being created
+ * @param minValue is an expression that resolves
+ * to the minimum end point of the acceptable range for 
expr
+ * @param maxValue is an expression that resolves
+ * to the maximum end point of the acceptable range for 
expr
+ * @param numBucket is an expression that resolves to
+ *  a constant indicating the number of buckets
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(expr, min_value, max_value, num_bucket) - Returns an 
long between 0 and `num_buckets`+1 by mapping the `expr` into buckets defined 
by the range [`min_value`, `max_value`].",
+  extended = """
+Examples:
+  > SELECT _FUNC_(5.35, 0.024, 10.06, 5);
+   3
+  """)
+// scalastyle:on line.size.limit
+case class WidthBucket(
+  expr: Expression,
+  minValue: Expression,
+  maxValue: Expression,
+  numBucket: Expression) extends QuaternaryExpression with 
ImplicitCastInputTypes {
+
+  override def children: Seq[Expression] = Seq(expr, minValue, maxValue, 
numBucket)
+  override def inputTypes: Seq[AbstractDataType] = Seq(DoubleType, 
DoubleType, DoubleType, LongType)
+  override def dataType: DataType = LongType
+  override def nullable: Boolean = true
+
+  override def nullSafeEval(ex: Any, min: Any, max: Any, num: Any): Any = {
--- End diff --

We should not use nullSafeEval. See the answers I got from Oracle.
```
select width_bucket(col1, 0, 10, -9) from t;
ORA-30494: The argument [4] of WIDTH_BUCKET function is NULL or invalid.

select width_bucket(col1, 0, 10, null) from t;
ORA-30494: The argument [4] of WIDTH_BUCKET function is NULL or invalid.

select width_bucket(col1, null, 5, 9) from t;
ORA-30494: The argument [2] of WIDTH_BUCKET function is NULL or invalid.

select width_bucket(col1, 5, null, 9) from t;
ORA-30494: The argument [3] of WIDTH_BUCKET function is NULL or invalid.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/18630#discussion_r127613972
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -473,6 +474,12 @@ object SparkSubmit extends CommandLineUtils {
   OptionAssigner(args.driverExtraLibraryPath, ALL_CLUSTER_MGRS, 
ALL_DEPLOY_MODES,
 sysProp = "spark.driver.extraLibraryPath"),
 
+  // Standalone only - propagate attributes for dependency resolution 
at the driver side
+  OptionAssigner(args.packages, STANDALONE, CLUSTER, sysProp = 
"spark.jars.packages"),
--- End diff --

ok...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127613839
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathExpressionsSuite.scala
 ---
@@ -644,4 +645,37 @@ class MathExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
 checkEvaluation(BRound(-0.35, 1), -0.4)
 checkEvaluation(BRound(-35, -1), -40)
   }
+
+  test("width_bucket") {
+def test(
+  expr: Double,
+  minValue: Double,
+  maxValue: Double,
+  numBucket: Long,
+  expected: Long): Unit = {
+  checkEvaluation(WidthBucket(Literal.create(expr, DoubleType),
+Literal.create(minValue, DoubleType),
+Literal.create(maxValue, DoubleType),
+Literal.create(numBucket, LongType)),
+expected)
+}
+
+test(5.35, 0.024, 10.06, 5, 3)
+
+test(3.14, 0, 4, 3, 3)
+test(2, 0, 4, 3, 2)
+test(-1, 0, 3.2, 4, 0)
+
+test(3.14, 4, 0, 3, 1)
+test(2, 4, 0, 3, 2)
+test(-1, 3.2, 0, 4, 5)
+
+// numBucket <= 0
+intercept[AnalysisException]{
--- End diff --

Please capture the error message and verify it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127613829
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/MathUtils.scala 
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+
+object MathUtils {
+
+  /**
+   *  Returns the bucket number into which
+   *  the value of this expression would fall after being evaluated.
+   *
+   * @param expr is the expression for which the histogram is being created
+   * @param minValue is an expression that resolves
+   * to the minimum end point of the acceptable range for 
expr
+   * @param maxValue is an expression that resolves
+   * to the maximum end point of the acceptable range for 
expr
+   * @param numBucket is an expression that resolves to
+   *  a constant indicating the number of buckets
+   * @return Returns an long between 0 and numBucket+1 by mapping the expr 
into buckets defined by
+   * the range [minValue, maxValue].
+   */
+  def widthBucket(expr: Double, minValue: Double, maxValue: Double, 
numBucket: Long): Long = {
+
+if (numBucket <= 0) {
+  throw new AnalysisException(s"The num of bucket must be greater than 
0, but got ${numBucket}")
--- End diff --

This check needs to be moved to `case class WidthBucket`. We do not want to 
issue such an exception during the execution of the query. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127613753
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -1186,3 +1186,51 @@ case class BRound(child: Expression, scale: 
Expression)
 with Serializable with ImplicitCastInputTypes {
   def this(child: Expression) = this(child, Literal(0))
 }
+
+/**
+ *  Returns the bucket number into which
+ *  the value of this expression would fall after being evaluated.
+ *
+ * @param expr is the expression for which the histogram is being created
+ * @param minValue is an expression that resolves
+ * to the minimum end point of the acceptable range for 
expr
+ * @param maxValue is an expression that resolves
+ * to the maximum end point of the acceptable range for 
expr
+ * @param numBucket is an expression that resolves to
+ *  a constant indicating the number of buckets
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(expr, min_value, max_value, num_bucket) - Returns an 
long between 0 and `num_buckets`+1 by mapping the `expr` into buckets defined 
by the range [`min_value`, `max_value`].",
--- End diff --

> Return the `bucket` to which operand would be assigned in an equidepth 
histogram with `num_bucket` buckets, in the range `min_value` to `max_value`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18323: [SPARK-21117][SQL] Built-in SQL Function Support ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18323#discussion_r127613668
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
 ---
@@ -1186,3 +1186,51 @@ case class BRound(child: Expression, scale: 
Expression)
 with Serializable with ImplicitCastInputTypes {
   def this(child: Expression) = this(child, Literal(0))
 }
+
+/**
+ *  Returns the bucket number into which
+ *  the value of this expression would fall after being evaluated.
+ *
+ * @param expr is the expression for which the histogram is being created
+ * @param minValue is an expression that resolves
+ * to the minimum end point of the acceptable range for 
expr
+ * @param maxValue is an expression that resolves
+ * to the maximum end point of the acceptable range for 
expr
+ * @param numBucket is an expression that resolves to
+ *  a constant indicating the number of buckets
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(expr, min_value, max_value, num_bucket) - Returns an 
long between 0 and `num_buckets`+1 by mapping the `expr` into buckets defined 
by the range [`min_value`, `max_value`].",
--- End diff --

Nit: `an` -> `a`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18637: [SPARK-15526][ML][FOLLOWUP][test-maven] Make JPMML provi...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18637
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18637: [SPARK-15526][ML][FOLLOWUP][test-maven] Make JPMML provi...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18637
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79646/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18637: [SPARK-15526][ML][FOLLOWUP][test-maven] Make JPMML provi...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18637
  
**[Test build #79646 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79646/testReport)**
 for PR 18637 at commit 
[`0f7a248`](https://github.com/apache/spark/commit/0f7a248e2434741879a8f026debee889304e1767).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18555
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79648/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18555
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18555
  
**[Test build #79648 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79648/testReport)**
 for PR 18555 at commit 
[`8a6b817`](https://github.com/apache/spark/commit/8a6b817d62397b780eb5f0b3b3be7dd9d3a733f1).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18424: [SPARK-17091] Add rule to convert IN predicate to...

[a10y]

Github user a10y commented on a diff in the pull request:

https://github.com/apache/spark/pull/18424#discussion_r127611976
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
 ---
@@ -238,6 +238,14 @@ private[parquet] object ParquetFilters {
   case sources.Not(pred) =>
 createFilter(schema, pred).map(FilterApi.not)
 
+  case sources.In(name, values) if canMakeFilterOn(name) =>
+val conds = values.flatMap(v => 
makeEq.lift(nameToType(name)).map(_(name, v)))
+var filter = conds(0)
+conds.drop(1).foreach { v =>
--- End diff --

You can eliminate the var by using reduceLeft


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18646: [SPARK-19810][MINOR][FOLLOW-UP] Follow-ups from SPARK-19...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18646
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79644/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18646: [SPARK-19810][MINOR][FOLLOW-UP] Follow-ups from SPARK-19...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18646
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18646: [SPARK-19810][MINOR][FOLLOW-UP] Follow-ups from SPARK-19...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18646
  
**[Test build #79644 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79644/testReport)**
 for PR 18646 at commit 
[`ce35fa2`](https://github.com/apache/spark/commit/ce35fa2067b1134ea859b99141be99a792d41487).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611350
  
--- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala ---
@@ -89,6 +89,14 @@ trait FutureAction[T] extends Future[T] {
*/
   override def value: Option[Try[T]]
 
+  // These two methods must be implemented in Scala 2.12, but won't be 
used by Spark
+
+  def transform[S](f: (Try[T]) => Try[S])(implicit executor: 
ExecutionContext): Future[S] =
--- End diff --

These cause a MiMa warning, because they're new methods in a trait that 
might be extended by users. I'll have to go back and remember whether this is 
an actual problem, because the trait is providing a default implementation here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611390
  
--- Diff: sql/catalyst/pom.xml ---
@@ -39,6 +39,10 @@
   org.scala-lang
   scala-reflect
 
+
--- End diff --

Some stuff like this became necessary due to other dependency changes and 
changes in their transitive dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611412
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2537,8 +2537,9 @@ class Dataset[T] private[sql](
* @group action
* @since 1.6.0
*/
-  def foreachPartition(func: ForeachPartitionFunction[T]): Unit =
-foreachPartition(it => func.call(it.asJava))
+  def foreachPartition(func: ForeachPartitionFunction[T]): Unit = {
+foreachPartition((it: Iterator[T]) => func.call(it.asJava))
--- End diff --

In cases like this I resolved the ambiguity by casting the argument, 
instead of implementing an interface -- because in cases like this the 
interface version exists just for Java support.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611421
  
--- Diff: sql/hive/pom.xml ---
@@ -59,13 +59,6 @@
 
 
   org.apache.spark
-  spark-sql_${scala.binary.version}
--- End diff --

Not strictly related, just found this redundancy while editing. The exact 
same stanza appears below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611376
  
--- Diff: 
core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala ---
@@ -54,7 +54,10 @@ class TaskContextSuite extends SparkFunSuite with 
BeforeAndAfter with LocalSpark
 val rdd = new RDD[String](sc, List()) {
   override def getPartitions = Array[Partition](StubPartition(0))
   override def compute(split: Partition, context: TaskContext) = {
-context.addTaskCompletionListener(context => 
TaskContextSuite.completed = true)
+context.addTaskCompletionListener(new TaskCompletionListener {
--- End diff --

Changes like this resolve an ambiguity where two overloads of a method 
exist, one with a signature taking a lambda, and the other taking a 
trait/interface of one method, both of which could fit. In cases like this I 
resolved in favor of implementing a specific listener class. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18645: [SPARK-14280][BUILD][WIP] Update change-version.s...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18645#discussion_r127611395
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala
 ---
@@ -339,39 +335,4 @@ class ScalaReflectionSuite extends SparkFunSuite {
 assert(linkedHashMapDeserializer.dataType == 
ObjectType(classOf[LHMap[_, _]]))
   }
 
-  private val dataTypeForComplexData = dataTypeFor[ComplexData]
--- End diff --

Actually this could have been removed with Scala 2.10 and will be, 
separately, shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18643: [SPARK-21426] [2.0] [SQL] [TEST] Fix test failure...

[gatorsmile]

Github user gatorsmile closed the pull request at:

https://github.com/apache/spark/pull/18643


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18643: [SPARK-21426] [2.0] [SQL] [TEST] Fix test failure due to...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18643
  
Thanks! Merging to 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18648: [SPARK-21428] Set IsolatedClientLoader off while using b...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18648
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18648: [SPARK-21428] Set IsolatedClientLoader off while ...

[yaooqinn]

GitHub user yaooqinn opened a pull request:

https://github.com/apache/spark/pull/18648

[SPARK-21428] Set IsolatedClientLoader off while using builtin Hive jars 
for reusing CliSessionState

## What changes were proposed in this pull request?

Set isolated to false while using builtin hive jars

## How was this patch tested?

Manually verified: `hive.exec.strachdir` was only created once because of 
reusing cliSessionState 
```java
➜  spark git:(SPARK-21428) ✗ bin/spark-sql --conf 
spark.sql.hive.metastore.jars=builtin

log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
17/07/16 23:59:27 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
17/07/16 23:59:27 INFO HiveMetaStore: 0: Opening raw store with 
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
17/07/16 23:59:27 INFO ObjectStore: ObjectStore, initialize called
17/07/16 23:59:28 INFO Persistence: Property 
hive.metastore.integral.jdo.pushdown unknown - will be ignored
17/07/16 23:59:28 INFO Persistence: Property datanucleus.cache.level2 
unknown - will be ignored
17/07/16 23:59:29 INFO ObjectStore: Setting MetaStore object pin classes 
with 
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/07/16 23:59:30 INFO Datastore: The class 
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
"embedded-only" so does not have its own datastore table.
17/07/16 23:59:30 INFO Datastore: The class 
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so 
does not have its own datastore table.
17/07/16 23:59:31 INFO Datastore: The class 
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as 
"embedded-only" so does not have its own datastore table.
17/07/16 23:59:31 INFO Datastore: The class 
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so 
does not have its own datastore table.
17/07/16 23:59:31 INFO MetaStoreDirectSql: Using direct SQL, underlying DB 
is DERBY
17/07/16 23:59:31 INFO ObjectStore: Initialized ObjectStore
17/07/16 23:59:31 WARN ObjectStore: Version information not found in 
metastore. hive.metastore.schema.verification is not enabled so recording the 
schema version 1.2.0
17/07/16 23:59:31 WARN ObjectStore: Failed to get database default, 
returning NoSuchObjectException
17/07/16 23:59:32 INFO HiveMetaStore: Added admin role in metastore
17/07/16 23:59:32 INFO HiveMetaStore: Added public role in metastore
17/07/16 23:59:32 INFO HiveMetaStore: No user is added in admin role, since 
config is empty
17/07/16 23:59:32 INFO HiveMetaStore: 0: get_all_databases
17/07/16 23:59:32 INFO audit: ugi=Kent  ip=unknown-ip-addr  
cmd=get_all_databases
17/07/16 23:59:32 INFO HiveMetaStore: 0: get_functions: db=default pat=*
17/07/16 23:59:32 INFO audit: ugi=Kent  ip=unknown-ip-addr  
cmd=get_functions: db=default pat=*
17/07/16 23:59:32 INFO Datastore: The class 
"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as 
"embedded-only" so does not have its own datastore table.
17/07/16 23:59:32 INFO SessionState: Created local directory: 
/var/folders/k2/04p4k4ws73l6711h_mz2_tq0gn/T/beea7261-221a-4711-89e8-8b12a9d37370_resources
17/07/16 23:59:32 INFO SessionState: Created HDFS directory: 
/tmp/hive/Kent/beea7261-221a-4711-89e8-8b12a9d37370
17/07/16 23:59:32 INFO SessionState: Created local directory: 
/var/folders/k2/04p4k4ws73l6711h_mz2_tq0gn/T/Kent/beea7261-221a-4711-89e8-8b12a9d37370
17/07/16 23:59:32 INFO SessionState: Created HDFS directory: 
/tmp/hive/Kent/beea7261-221a-4711-89e8-8b12a9d37370/_tmp_space.db
17/07/16 23:59:32 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT
17/07/16 23:59:32 INFO SparkContext: Submitted application: 
SparkSQL::10.0.0.8
17/07/16 23:59:32 INFO SecurityManager: Changing view acls to: Kent
17/07/16 23:59:32 INFO SecurityManager: Changing modify acls to: Kent
17/07/16 23:59:32 INFO SecurityManager: Changing view acls groups to:
17/07/16 23:59:32 INFO SecurityManager: Changing modify acls groups to:
17/07/16 23:59:32 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users  with view permissions: Set(Kent); groups 
with view permissions: Set(); users  with modify permissions: Set(Kent); groups 
with modify permissions: Set()
17/07/16 23:59:33 INFO Utils: Successfully started service 'sparkDriver' on 
port 51889.
17/07/16 23:59:33 INFO SparkEnv: Registering MapOutputTracker
  

[GitHub] spark issue #18631: [SPARK-SPARK-21410][CORE] RangePartitioner.numPartitions...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18631
  
Please update the title to:
```
[SPARK-21410][CORE] Create less partitions for RangePartitioner if 
RDD.count() is less than `partitions`
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18631: [SPARK-SPARK-21410][CORE] RangePartitioner.numPartitions...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18631
  
Ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18555
  
**[Test build #79648 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79648/testReport)**
 for PR 18555 at commit 
[`8a6b817`](https://github.com/apache/spark/commit/8a6b817d62397b780eb5f0b3b3be7dd9d3a733f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18647: [MINOR][PYTHON] Remove obsolete codes for parsing abstra...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18647
  
cc @davies and @joshrosen, I was looking at the first place that introduced 
this - 
https://github.com/apache/spark/commit/880eabec37c69ce4e9594d7babfac291b0f93f50.
 Could you take a look and see if i missed something here when you have some 
time?

I was thinking we should explicitly expose this or remove them out rather 
than leaving them in the codes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18647: [MINOR][PYTHON] Remove obsolete codes for parsing abstra...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18647
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18647: [MINOR][PYTHON] Remove obsolete codes for parsing abstra...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18647
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79647/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18647: [MINOR][PYTHON] Remove obsolete codes for parsing abstra...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18647
  
**[Test build #79647 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79647/testReport)**
 for PR 18647 at commit 
[`83228cb`](https://github.com/apache/spark/commit/83228cbfd8e8e681bbbca2e970d55c917d4dbc89).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18644: case class should be independent

[piyushknoldus]

Github user piyushknoldus closed the pull request at:

https://github.com/apache/spark/pull/18644


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18647: [MINOR][PYTHON] Remove obsolete codes for parsing abstra...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18647
  
**[Test build #79647 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79647/testReport)**
 for PR 18647 at commit 
[`83228cb`](https://github.com/apache/spark/commit/83228cbfd8e8e681bbbca2e970d55c917d4dbc89).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18647: [MINOR][PYTHON] Remove obsolete codes for parsing...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/18647

[MINOR][PYTHON] Remove obsolete codes for parsing abstract schema strings

## What changes were proposed in this pull request?

This PR proposes to remove private functions that look not used in the main 
codes, `_split_schema_abstract`, `_parse_field_abstract`, 
`_parse_schema_abstract` and `_infer_schema_type`.

## How was this patch tested?

Existing tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark remove-abstract

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18647


commit 83228cbfd8e8e681bbbca2e970d55c917d4dbc89
Author: hyukjinkwon 
Date:   2017-07-16T14:41:59Z

Remove obsolete codes for parsing abstract schema strings




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18645
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18645
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79645/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18645
  
**[Test build #79645 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79645/testReport)**
 for PR 18645 at commit 
[`1204077`](https://github.com/apache/spark/commit/1204077c0cb67c17d49b3b490342c1d3aa2827ee).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18637: [SPARK-15526][ML][FOLLOWUP][test-maven] Make JPMML provi...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18637
  
**[Test build #79646 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79646/testReport)**
 for PR 18637 at commit 
[`0f7a248`](https://github.com/apache/spark/commit/0f7a248e2434741879a8f026debee889304e1767).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18637: [SPARK-15526][ML][FOLLOWUP][test-maven] Make JPMM...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18637#discussion_r127608414
  
--- Diff: pom.xml ---
@@ -2511,8 +2523,8 @@
   
 
 

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18645
  
**[Test build #79645 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79645/testReport)**
 for PR 18645 at commit 
[`1204077`](https://github.com/apache/spark/commit/1204077c0cb67c17d49b3b490342c1d3aa2827ee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org