[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214978891
  
**[Test build #57094 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57094/consoleFull)**
 for PR 12714 at commit 
[`3a6665c`](https://github.com/apache/spark/commit/3a6665c23131cdd0770621c21bddc5d76a130782).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/12714#discussion_r61207500
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1391,4 +1393,99 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   assert(message == "'path' is not specified")
 }
   }
+
+  test("derived from Hive query file: 
drop_database_removes_partition_dirs.q") {
--- End diff --

I feel this one and the next one are useful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214977910
  
(Please allow me just leave a short note for some files having unused 
imports in this PR)

```bash

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelation.scala

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214977713
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214977714
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57085/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214977634
  
**[Test build #57085 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57085/consoleFull)**
 for PR 12722 at commit 
[`df35a93`](https://github.com/apache/spark/commit/df35a9388860f468cdc6af9ba8d1af2c55e55ba8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214976458
  
**[Test build #57093 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57093/consoleFull)**
 for PR 12629 at commit 
[`f9837d8`](https://github.com/apache/spark/commit/f9837d83cf0e93c9809ff39a3500d373f8330354).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214976019
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214976015
  
**[Test build #57092 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57092/consoleFull)**
 for PR 12629 at commit 
[`0c32f4e`](https://github.com/apache/spark/commit/0c32f4ee6f3e5dd630ed663a1fc1e21d85d0541d).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214976020
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57092/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214975784
  
It looks Hadoop configuration can be set via `option()` now. It looks when 
it sets `null` for Hadoop configurations, it uses default values which is 
consistent with what this PR proposes. So, I think we don't need a special 
handling for Hadoop configuration.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-214975697
  
**[Test build #57092 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57092/consoleFull)**
 for PR 12629 at commit 
[`0c32f4e`](https://github.com/apache/spark/commit/0c32f4ee6f3e5dd630ed663a1fc1e21d85d0541d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214975016
  
**[Test build #2892 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2892/consoleFull)**
 for PR 12689 at commit 
[`086fa13`](https://github.com/apache/spark/commit/086fa138fb7fdf40f295c7d034c88fa9778bfea3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `abstract class HadoopFsRelationTest extends QueryTest with 
SQLTestUtils with TestHiveSingleton `
  * `class AlwaysFailOutputCommitter(`
  * `class AlwaysFailParquetOutputCommitter(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14942][SQL][Streaming] First construct ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12725#issuecomment-214973570
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57084/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14942][SQL][Streaming] First construct ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12725#issuecomment-214973566
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14427][SQL] Support persisting partitio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12204#issuecomment-214973567
  
**[Test build #57091 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57091/consoleFull)**
 for PR 12204 at commit 
[`22d6e35`](https://github.com/apache/spark/commit/22d6e3530a2889a5ea48c8f96cd80a80474c3b05).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14942][SQL][Streaming] First construct ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12725#issuecomment-214973085
  
**[Test build #57084 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57084/consoleFull)**
 for PR 12725 at commit 
[`8c8d73a`](https://github.com/apache/spark/commit/8c8d73a85550349faafef1cbde1c7094d384809e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL] Left Anti Join

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12214#issuecomment-214972960
  
@hvanhovell It sounds like something is wrong? 
```scala
  test("left anti join") {
val df = Seq(1, 3).toDF("int")
val df2 = Seq(1, 5).toDF("int2")

checkAnswer(
  df.as("a").join(df2.as("b"), $"a.int" === $"b.int2", "left_anti"),
  Row(3) :: Nil)
  }
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14940][SQL] Move ExternalCatalog to own...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12721#issuecomment-214972806
  
I guess ... conflict now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14427][SQL] Support persisting partitio...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12204#discussion_r61205281
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1346,6 +1346,96 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
 }
   }
 
+  test("writing partitioned DataFrame to Hive compatible format") {
+val schema = StructType(
+  Array(StructField("a", LongType, false),
+StructField("b", StringType, false),
+StructField("c", StringType, false),
+StructField("d", IntegerType, false)))
+
+val rdd = sparkContext.parallelize(Array(
+  Row(1L, "test", "NW", 1),
+  Row(2L, "test2", "SF", 1),
+  Row(3L, "test3", "NW", 1),
+  Row(1L, "test", "CA", 2)))
+
+val df = sqlContext.createDataFrame(rdd, schema)
+df.write.partitionBy("c", "d").saveAsTable("testDFTABLE1")
--- End diff --

SGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214972164
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214972166
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57082/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14944][SQL] Remove HiveConf from HiveTa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12727#issuecomment-214972091
  
**[Test build #57090 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57090/consoleFull)**
 for PR 12727 at commit 
[`df728f2`](https://github.com/apache/spark/commit/df728f2190be7a44bbf24858dc6c11313bcd2fcf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214972072
  
**[Test build #57082 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57082/consoleFull)**
 for PR 12715 at commit 
[`c9b5c3b`](https://github.com/apache/spark/commit/c9b5c3b42f32bcca9892fb4699afb53cde0f5e26).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14944][SQL] Remove HiveConf from HiveTa...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12727#issuecomment-214971725
  
Note that this one builds on https://github.com/apache/spark/pull/12726.

Let's see if tests can pass in one shot ...



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14944][SQL] Remove HiveConf from HiveTa...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/12727

[SPARK-14944][SQL] Remove HiveConf from HiveTableScanExec and 
HiveTableReader

## What changes were proposed in this pull request?
This patch removes HiveConf from HiveTableScanExec and HiveTableReader and 
instead just uses our own configuration system. I'm splitting the large change 
of removing HiveConf into multiple independent pull requests because it is very 
difficult to debug test failures when they are all combined in one giant one.

## How was this patch tested?
Should be covered by existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-14944

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12727.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12727


commit 869b9aab22a1660df88c3c6c5e21f14e59234c05
Author: Reynold Xin 
Date:   2016-04-27T05:05:52Z

[SPARK-14943][SQL] Remove HiveConf dependency from ScriptTransformation

commit df728f2190be7a44bbf24858dc6c11313bcd2fcf
Author: Reynold Xin 
Date:   2016-04-27T05:12:49Z

[SPARK-14944][SQL] Remove HiveConf from HiveTableScanExec and 
HiveTableReader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14427][SQL] Support persisting partitio...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12204#discussion_r61204556
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1346,6 +1346,96 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
 }
   }
 
+  test("writing partitioned DataFrame to Hive compatible format") {
+val schema = StructType(
+  Array(StructField("a", LongType, false),
+StructField("b", StringType, false),
+StructField("c", StringType, false),
+StructField("d", IntegerType, false)))
+
+val rdd = sparkContext.parallelize(Array(
+  Row(1L, "test", "NW", 1),
+  Row(2L, "test2", "SF", 1),
+  Row(3L, "test3", "NW", 1),
+  Row(1L, "test", "CA", 2)))
+
+val df = sqlContext.createDataFrame(rdd, schema)
+df.write.partitionBy("c", "d").saveAsTable("testDFTABLE1")
--- End diff --

Or just combine these two tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14943][SQL] Remove HiveConf dependency ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12726#issuecomment-214971072
  
**[Test build #57089 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57089/consoleFull)**
 for PR 12726 at commit 
[`869b9aa`](https://github.com/apache/spark/commit/869b9aab22a1660df88c3c6c5e21f14e59234c05).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14943][SQL] Remove HiveConf dependency ...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/12726

[SPARK-14943][SQL] Remove HiveConf dependency from ScriptTransformation

## What changes were proposed in this pull request?
This patch removes HiveConf from ScriptTransformation and instead just uses 
our own configuration system.

## How was this patch tested?
Updated tests to reflect this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-14943

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12726


commit 869b9aab22a1660df88c3c6c5e21f14e59234c05
Author: Reynold Xin 
Date:   2016-04-27T05:05:52Z

[SPARK-14943][SQL] Remove HiveConf dependency from ScriptTransformation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214970949
  
**[Test build #2891 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2891/consoleFull)**
 for PR 12689 at commit 
[`086fa13`](https://github.com/apache/spark/commit/086fa138fb7fdf40f295c7d034c88fa9778bfea3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `abstract class HadoopFsRelationTest extends QueryTest with 
SQLTestUtils with TestHiveSingleton `
  * `class AlwaysFailOutputCommitter(`
  * `class AlwaysFailParquetOutputCommitter(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12689


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14937][ML][Document]spark.ml LogisticRe...

[wangmiao1981]

Github user wangmiao1981 commented on the pull request:

https://github.com/apache/spark/pull/12717#issuecomment-214970534
  
@yanboliang  Can you take a look ? It is a simple fix. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214970558
  
Going to merge this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13289][MLLIB] Fix infinite distances be...

[flyjy]

Github user flyjy commented on the pull request:

https://github.com/apache/spark/pull/11812#issuecomment-214970336
  
Yes, I am working it. Will finish tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214970126
  
**[Test build #57088 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57088/consoleFull)**
 for PR 12724 at commit 
[`321ec4e`](https://github.com/apache/spark/commit/321ec4e999bfdef2f19c5b7888f8d8236d56c59c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12689#discussion_r61204047
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -237,14 +237,6 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   intercept[AnalysisException] {
 sql(s"""LOAD DATA LOCAL INPATH "$incorrectUri" INTO TABLE 
non_part_table""")
   }
-
-  // Unset default URI Scheme and Authority: throw exception
--- End diff --

  // Unset default URI Scheme and Authority: throw exception
  // HadoopConfiguration sets fs.default.name as alias of new YARN 
setting fs.defaultFS.
  // We only need to unset fs.default.name and reset it back.
  val originalFsName = 
hiveContext.sparkContext.hadoopConfiguration.get("fs.default.name")
  // SQLConf's settings will be added into sessionState.newHadoopConf,
  // so we need to unset it too. It has no alias, thus we need to unset 
both two settings.
  val oriDefaultFS =
hiveContext.sessionState.conf.getConfString("fs.defaultFS", null)
  val oriDefaultFSName =
hiveContext.sessionState.conf.getConfString("fs.default.name", null)

  hiveContext.sparkContext.hadoopConfiguration.unset("fs.default.name")
  hiveContext.sessionState.conf.unsetConf("fs.defaultFS")
  hiveContext.sessionState.conf.unsetConf("fs.default.name")
  intercept[AnalysisException] {
sql(s"""LOAD DATA INPATH "$testData" INTO TABLE non_part_table""")
  }
  hiveContext.sparkContext.hadoopConfiguration.set("fs.default.name", 
originalFsName)
  if (oriDefaultFS != null) {
hiveContext.sessionState.conf.setConfString("fs.defaultFS", 
oriDefaultFS)
  }
  if (oriDefaultFSName != null) {
hiveContext.sessionState.conf.setConfString("fs.default.name", 
oriDefaultFSName)
 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14427][SQL] Support persisting partitio...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12204#discussion_r61203984
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1346,6 +1346,96 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
 }
   }
 
+  test("writing partitioned DataFrame to Hive compatible format") {
+val schema = StructType(
+  Array(StructField("a", LongType, false),
+StructField("b", StringType, false),
+StructField("c", StringType, false),
+StructField("d", IntegerType, false)))
+
+val rdd = sparkContext.parallelize(Array(
+  Row(1L, "test", "NW", 1),
+  Row(2L, "test2", "SF", 1),
+  Row(3L, "test3", "NW", 1),
+  Row(1L, "test", "CA", 2)))
+
+val df = sqlContext.createDataFrame(rdd, schema)
+df.write.partitionBy("c", "d").saveAsTable("testDFTABLE1")
--- End diff --

simply drop this table?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12919][SPARKR] Implement dapply() on Da...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/12493#issuecomment-214970063
  
@sun-rui Do the tests pass locally for you ? Because the error in Jenkins 
doesn't seem like a flaky test but from the `dapply` test (error pasted below).

Also I still see that the PR doesn't merge cleanly. Can you bring it up to 
date with master ?


```
1. Error: dapply() on a DataFrame 
--
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 
in stage 1154.0 failed 1 times, most recent failure: Lost task 0.0 in stage 
1154.0 (TID 9753, localhost): org.apache.spark.SparkException: R computation 
failed with
 [1] 2
[1] 3
[1] 1
[1] 2
[1] 1
[1] 3
[1] 2
[1] 2
[1] 2
[1] 2
[1] 2
[1] 2

  Unsupported type for serialization factor
Calls: source ... serializeRow -> writeList -> writeObject -> writeType
Execution halted
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214969720
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12689#discussion_r61203819
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -237,14 +237,6 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   intercept[AnalysisException] {
 sql(s"""LOAD DATA LOCAL INPATH "$incorrectUri" INTO TABLE 
non_part_table""")
   }
-
-  // Unset default URI Scheme and Authority: throw exception
--- End diff --

If you post the code here I can incorporate it directly. Alternatively you 
can also submit a pull request against my branch on github.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14729][Scheduler] Refactored YARN sched...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12641#issuecomment-214969466
  
**[Test build #57087 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57087/consoleFull)**
 for PR 12641 at commit 
[`8117184`](https://github.com/apache/spark/commit/811718492e7b93af3b1c857829a687a7ba2cae39).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12689#discussion_r61203719
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala
 ---
@@ -237,14 +237,6 @@ class HiveCommandSuite extends QueryTest with 
SQLTestUtils with TestHiveSingleto
   intercept[AnalysisException] {
 sql(s"""LOAD DATA LOCAL INPATH "$incorrectUri" INTO TABLE 
non_part_table""")
   }
-
-  // Unset default URI Scheme and Authority: throw exception
--- End diff --

I do fix this. Should I add it back when this is merged or just post codes 
here?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14729][Scheduler] Refactored YARN sched...

[hbhanawat]

Github user hbhanawat commented on the pull request:

https://github.com/apache/spark/pull/12641#issuecomment-214969001
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214967977
  
**[Test build #2893 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2893/consoleFull)**
 for PR 12689 at commit 
[`55a3676`](https://github.com/apache/spark/commit/55a3676e7f10997af97168d714750c3f104a090f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214967911
  
**[Test build #57085 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57085/consoleFull)**
 for PR 12722 at commit 
[`df35a93`](https://github.com/apache/spark/commit/df35a9388860f468cdc6af9ba8d1af2c55e55ba8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214967908
  
**[Test build #57086 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57086/consoleFull)**
 for PR 12689 at commit 
[`55a3676`](https://github.com/apache/spark/commit/55a3676e7f10997af97168d714750c3f104a090f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214967827
  
OK I pushed a fix. Let's see if that works ...



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12713


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214966991
  
Merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965587
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965590
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57079/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965489
  
**[Test build #57079 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57079/consoleFull)**
 for PR 12713 at commit 
[`2e0be54`](https://github.com/apache/spark/commit/2e0be54b949125edbe201665c1d4f624607551f9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965335
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965336
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57080/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965242
  
**[Test build #57080 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57080/consoleFull)**
 for PR 12713 at commit 
[`32d596c`](https://github.com/apache/spark/commit/32d596c4e281d3c36e1210d2119e5c5bae4cc344).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214965127
  
**[Test build #2888 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2888/consoleFull)**
 for PR 12713 at commit 
[`32d596c`](https://github.com/apache/spark/commit/32d596c4e281d3c36e1210d2119e5c5bae4cc344).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214964912
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57078/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214964911
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214964764
  
**[Test build #57078 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57078/consoleFull)**
 for PR 12714 at commit 
[`5b013e9`](https://github.com/apache/spark/commit/5b013e9f07f5c5f96d7e440f906020a1df188f5e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214964709
  
**[Test build #2890 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2890/consoleFull)**
 for PR 12713 at commit 
[`32d596c`](https://github.com/apache/spark/commit/32d596c4e281d3c36e1210d2119e5c5bae4cc344).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214964626
  
**[Test build #2889 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2889/consoleFull)**
 for PR 12713 at commit 
[`32d596c`](https://github.com/apache/spark/commit/32d596c4e281d3c36e1210d2119e5c5bae4cc344).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12714#discussion_r61201204
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -1391,4 +1393,136 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
   assert(message == "'path' is not specified")
 }
   }
+
+  test("derived from Hive query file: date_3.q") {
--- End diff --

none of these seem that useful to have?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14942][SQL][Streaming] First construct ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12725#issuecomment-214964435
  
**[Test build #57084 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57084/consoleFull)**
 for PR 12725 at commit 
[`8c8d73a`](https://github.com/apache/spark/commit/8c8d73a85550349faafef1cbde1c7094d384809e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214964419
  
I see. : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214964247
  
One thing is that I'm trying to get rid of the HiveConf, so I can't just 
apply your change directly.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14942][SQL][Streaming] First construct ...

[lw-lin]

GitHub user lw-lin opened a pull request:

https://github.com/apache/spark/pull/12725

[SPARK-14942][SQL][Streaming] First construct a batch then run the batch 
for continuous queries

## Problem

Currently in `StreamExecution`, we first run the batch, then construct the 
next:
```scala
if (dataAvailable) runBatch()
constructNextBatch()
```

This is good if we run batches ASAP, where data would get processed in the 
**very next batch**:


![1](https://cloud.githubusercontent.com/assets/15843379/14779964/2786e698-0b0d-11e6-9d2c-bb41513488b2.png)

However, if we run batches at trigger like `ProcessTime("1 minute")`, data 
- such as _y_ below - may not get processed in the very next batch i.e. _batch 
1_, but in _batch 2_:


![2](https://cloud.githubusercontent.com/assets/15843379/14779818/6f3bb064-0b0c-11e6-9f16-c1ce4897186b.png)

## What changes were proposed in this pull request?

This patch reverse the order of `constructNextBatch()` and `runBatch()`. 
After this patch, data would get processed in the **very next batch**, i.e. 
_batch 1_:


![3](https://cloud.githubusercontent.com/assets/15843379/14779816/6f36ee62-0b0c-11e6-9e53-bc8397fade18.png)

In addition, this patch alters when we do `currentBatchId += 1`: let's do 
that when the processing of the current batch of data is complete, so we won't 
bother passing `currentBatchId + 1` or  `currentBatchId - 1` to states or sinks.


## How was this patch tested?

This should be covered by existing test suits including stress tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lw-lin/spark construct-before-run-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12725


commit 8c8d73a85550349faafef1cbde1c7094d384809e
Author: Liwei Lin 
Date:   2016-04-27T02:19:14Z

constructNextBatch() before runBatch()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214963780
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57077/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214963778
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13477][SQL] Expose new user-facing Cata...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12713#issuecomment-214963666
  
**[Test build #57077 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57077/consoleFull)**
 for PR 12713 at commit 
[`91e0d5a`](https://github.com/apache/spark/commit/91e0d5af6addf4cc47aca035f210f376fffa6ddf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214963609
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57076/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214963608
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14130] [SQL] Throw exceptions for ALTER...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12714#issuecomment-214963507
  
**[Test build #57076 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57076/consoleFull)**
 for PR 12714 at commit 
[`5a3cd57`](https://github.com/apache/spark/commit/5a3cd575f516fcd76abe2c79536d421e3064afa7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214963442
  
Thanks, @gatorsmile!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214963288
  
**[Test build #2892 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2892/consoleFull)**
 for PR 12689 at commit 
[`086fa13`](https://github.com/apache/spark/commit/086fa138fb7fdf40f295c7d034c88fa9778bfea3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14911][Core] Fix a potential data race ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12681#issuecomment-214962374
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14911][Core] Fix a potential data race ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12681#issuecomment-214962376
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57072/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14911][Core] Fix a potential data race ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12681#issuecomment-214962280
  
**[Test build #57072 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57072/consoleFull)**
 for PR 12681 at commit 
[`6b72b96`](https://github.com/apache/spark/commit/6b72b963d54855771dcabc1fca8ed963be28303c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4105][CORE] regenerate the shuffle file...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/12700#issuecomment-214962220
  
Since I don't meet this problem recently, so I cannot exactly tell what 
actually cause it, maybe race condition, maybe flush problem. 

Since you already have the reproducible case, why not dig into more details.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12689#issuecomment-214962084
  
**[Test build #2891 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2891/consoleFull)**
 for PR 12689 at commit 
[`086fa13`](https://github.com/apache/spark/commit/086fa138fb7fdf40f295c7d034c88fa9778bfea3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12689#discussion_r61199823
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala ---
@@ -48,8 +48,21 @@ private[sql] class SessionState(sparkSession: 
SparkSession) {
* SQL-specific key-value configurations.
*/
   lazy val conf: SQLConf = new SQLConf
-  lazy val hadoopConf: Configuration = {
-new Configuration(sparkSession.sparkContext.hadoopConfiguration)
+
+  def newHadoopConf(): Configuration = {
+val hadoopConf = new 
Configuration(sparkSession.sparkContext.hadoopConfiguration)
+conf.getAllConfs.foreach { case (k, v) => if (v ne null) 
hadoopConf.set(k, v) }
--- End diff --

we definitely want override here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14941][SQL] Remove runtime HiveConf - W...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12722#issuecomment-214960786
  
To save your time, I hit the same errors. Want to share what I found. One 
test case called `SQLConf.clear()` and cleaned all the configurations. When we 
reloading the conf, we did not load the Hive-related configurations.

It might not be the best way to fix the issue. Below is what I changed. 
Hopefully, it helps you in this PR. Thanks!


https://github.com/apache/spark/pull/12081/commits/1da261a395bd799b6942e04fa142372a5bee3a8c


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214960674
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214960670
  
**[Test build #57083 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57083/consoleFull)**
 for PR 12724 at commit 
[`321ec4e`](https://github.com/apache/spark/commit/321ec4e999bfdef2f19c5b7888f8d8236d56c59c).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214960676
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57083/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214960489
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57070/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214960488
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214960402
  
**[Test build #57070 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57070/consoleFull)**
 for PR 12715 at commit 
[`c9b5c3b`](https://github.com/apache/spark/commit/c9b5c3b42f32bcca9892fb4699afb53cde0f5e26).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12919][SPARKR] Implement dapply() on Da...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12493#issuecomment-214960152
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12919][SPARKR] Implement dapply() on Da...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12493#issuecomment-214960153
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57066/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12919][SPARKR] Implement dapply() on Da...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12493#issuecomment-214960044
  
**[Test build #57066 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57066/consoleFull)**
 for PR 12493 at commit 
[`a88c1db`](https://github.com/apache/spark/commit/a88c1dbda92791aec49e3fd122a9f7137939ae96).
 * This patch **fails SparkR unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12724#issuecomment-214959745
  
**[Test build #57083 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57083/consoleFull)**
 for PR 12724 at commit 
[`321ec4e`](https://github.com/apache/spark/commit/321ec4e999bfdef2f19c5b7888f8d8236d56c59c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12638#issuecomment-214959647
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57073/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12638#issuecomment-214959646
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14874][SQL][Streaming] Remove the obsol...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12638#issuecomment-214959537
  
**[Test build #57073 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57073/consoleFull)**
 for PR 12638 at commit 
[`653fa52`](https://github.com/apache/spark/commit/653fa52c35a668896c33d6868637a9011fd9ad1e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14913][SQL] Simplify configuration API

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/12689#discussion_r61198809
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala ---
@@ -48,8 +48,21 @@ private[sql] class SessionState(sparkSession: 
SparkSession) {
* SQL-specific key-value configurations.
*/
   lazy val conf: SQLConf = new SQLConf
-  lazy val hadoopConf: Configuration = {
-new Configuration(sparkSession.sparkContext.hadoopConfiguration)
+
+  def newHadoopConf(): Configuration = {
+val hadoopConf = new 
Configuration(sparkSession.sparkContext.hadoopConfiguration)
+conf.getAllConfs.foreach { case (k, v) => if (v ne null) 
hadoopConf.set(k, v) }
--- End diff --

oh, I thought we do not want to override. Then even we directly set 
sparkContext.hadoopConfiguration, it is still possibly overridden by conf.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11940][PYSPARK][ML] Python API for ml.c...

[zjffdu]

Github user zjffdu closed the pull request at:

https://github.com/apache/spark/pull/10242


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14783] [SPARK-14786] [BRANCH-1.6] Prese...

[yhuai]

GitHub user yhuai opened a pull request:

https://github.com/apache/spark/pull/12724

[SPARK-14783] [SPARK-14786] [BRANCH-1.6] Preserve full exception stacktrace 
in IsolatedClientLoader and Remove hive-cli dependency from hive subproject

This PR is the branch-1.6 version of the following commits from @JoshRosen. 
We probably cannot change the dependency of sql/hive again. But, if anyone 
needs this, they can use this PR.
 
**[SPARK-14783] Preserve full exception stacktrace in IsolatedClientLoader**
In IsolatedClientLoader, we have a`catch` block which throws an exception 
without wrapping the original exception, causing the full exception stacktrace 
and any nested exceptions to be lost. This patch fixes this, improving the 
usefulness of classloading error messages.

**[SPARK-14786] Remove hive-cli dependency from hive subproject**
The `hive` subproject currently depends on `hive-cli` in order to perform a 
check to see whether a `SessionState` is an instance of 
`org.apache.hadoop.hive.cli.CliSessionState` (see #9589). The introduction of 
this `hive-cli` dependency has caused problems for users whose Hive metastore 
JAR classpaths don't include the `hive-cli` classes (such as in #11495).

This patch removes this dependency on `hive-cli` and replaces the 
`isInstanceOf` check by reflection. I added a Maven Enforcer rule to ban 
`hive-cli` from the `hive` subproject in order to make sure that this 
dependency is not accidentally reintroduced.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yhuai/spark hivecli

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12724.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12724


commit 0d5607676f5d8f95dd87ba43fe0f93e2d546b84e
Author: Josh Rosen 
Date:   2016-04-21T17:43:22Z

[SPARK-14783] Preserve full exception stacktrace in IsolatedClientLoader

In IsolatedClientLoader, we have a`catch` block which throws an exception 
without wrapping the original exception, causing the full exception stacktrace 
and any nested exceptions to be lost. This patch fixes this, improving the 
usefulness of classloading error messages.

Author: Josh Rosen 

Closes #12548 from JoshRosen/improve-logging-for-hive-classloader-issues.

commit 321ec4e999bfdef2f19c5b7888f8d8236d56c59c
Author: Yin Huai 
Date:   2016-04-27T03:12:11Z

[SPARK-14786] Remove hive-cli dependency from hive subproject




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214959194
  
**[Test build #57082 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57082/consoleFull)**
 for PR 12715 at commit 
[`c9b5c3b`](https://github.com/apache/spark/commit/c9b5c3b42f32bcca9892fb4699afb53cde0f5e26).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214959134
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12715#issuecomment-214958903
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org