[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-218543181
  
Saw a discussion about the path limit. 
https://www.quora.com/Why-does-UNIX-system-have-maximum-path-of-108-bytes 

Not sure if we need to add extra check here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12871


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217932629
  
LGTM merging into master 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217805312
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58124/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217805309
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217805072
  
**[Test build #58124 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58124/consoleFull)**
 for PR 12871 at commit 
[`487f020`](https://github.com/apache/spark/commit/487f0208cf5269d8e80756589eb45f5aa0483327).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class InMemoryCatalog(hadoopConfig: Configuration = new Configuration) 
extends ExternalCatalog `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217804368
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217804371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58123/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217804149
  
**[Test build #58123 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58123/consoleFull)**
 for PR 12871 at commit 
[`b4f4926`](https://github.com/apache/spark/commit/b4f49263e8a1ba1019daf828f1a3d18ae251ca54).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class InMemoryCatalog(hadoopConfig: Configuration = new Configuration) 
extends ExternalCatalog `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217790134
  
**[Test build #58124 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58124/consoleFull)**
 for PR 12871 at commit 
[`487f020`](https://github.com/apache/spark/commit/487f0208cf5269d8e80756589eb45f5aa0483327).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217789379
  
**[Test build #58123 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58123/consoleFull)**
 for PR 12871 at commit 
[`b4f4926`](https://github.com/apache/spark/commit/b4f49263e8a1ba1019daf828f1a3d18ae251ca54).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217788838
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217788305
  
Hive is not case-preserving, and so does `HiveExternalCatalog`. When we 
save table `myTable` to hive, a dir named `mytable` is created, instead of 
`myTable` as I expected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217787634
  
The PR looks good. Let's resolve the conflicts and get it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217787572
  
btw, what's the reason of having 
https://github.com/apache/spark/pull/12871/commits/aefade3924b52ab05f26d9a8af4f63555e243b24?
 (where do we turn the string to its lower case form?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62454941
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -73,6 +79,8 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
 
+  private val fs = FileSystem.get(new Configuration)
--- End diff --

I think it is better to use `sparkContext.hadoopConfiguration` (we can 
access sparkContext in SharedState). We can use `new Configuration` as a 
default value of the hadoop conf to avoid of changing those tests. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217608493
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58055/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217608492
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217608468
  
**[Test build #58055 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58055/consoleFull)**
 for PR 12871 at commit 
[`4d400ca`](https://github.com/apache/spark/commit/4d400ca5e5d0b184540f8e91188daeaa1303ff5e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217604019
  
**[Test build #58055 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58055/consoleFull)**
 for PR 12871 at commit 
[`4d400ca`](https://github.com/apache/spark/commit/4d400ca5e5d0b184540f8e91188daeaa1303ff5e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217485501
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217485502
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57999/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217485371
  
**[Test build #57999 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57999/consoleFull)**
 for PR 12871 at commit 
[`0261d25`](https://github.com/apache/spark/commit/0261d252f8baa1e823a97261e111a3e93019a0dc).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217465398
  
**[Test build #57999 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57999/consoleFull)**
 for PR 12871 at commit 
[`0261d25`](https://github.com/apache/spark/commit/0261d252f8baa1e823a97261e111a3e93019a0dc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217401682
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57972/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217401679
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217401550
  
**[Test build #57972 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57972/consoleFull)**
 for PR 12871 at commit 
[`4d8c366`](https://github.com/apache/spark/commit/4d8c366f45173da6ae06db20d6117cc23cd8ed00).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217399145
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57971/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217399142
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217399031
  
**[Test build #57971 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57971/consoleFull)**
 for PR 12871 at commit 
[`aefade3`](https://github.com/apache/spark/commit/aefade3924b52ab05f26d9a8af4f63555e243b24).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217388375
  
**[Test build #57972 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57972/consoleFull)**
 for PR 12871 at commit 
[`4d8c366`](https://github.com/apache/spark/commit/4d8c366f45173da6ae06db20d6117cc23cd8ed00).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217383775
  
**[Test build #57971 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57971/consoleFull)**
 for PR 12871 at commit 
[`aefade3`](https://github.com/apache/spark/commit/aefade3924b52ab05f26d9a8af4f63555e243b24).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217370473
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57967/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217370472
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217370390
  
**[Test build #57967 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57967/consoleFull)**
 for PR 12871 at commit 
[`e975047`](https://github.com/apache/spark/commit/e97504759ad6ba640e858555fc6dbf7e9604ec1a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217369151
  
looks pretty good. Left a few comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62296164
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
 ---
@@ -488,6 +491,79 @@ abstract class ExternalCatalogSuite extends 
SparkFunSuite with BeforeAndAfterEac
 assert(catalog.listFunctions("db2", "func*").toSet == Set("func1", 
"func2"))
   }
 
+  // 
--
+  // File System operations
+  // 
--
+
+  private def exists(uri: String, children: String*): Boolean = {
+val base = new File(new URI(uri))
+
logWarning(base.listFiles().toSeq.map(_.getAbsolutePath).mkString("\n###"))
+val f = children.foldLeft(base) {
+  case (parent, child) => new File(parent, child)
+}
+logWarning("exists? " + f.getAbsolutePath)
+f.exists()
+  }
+
+  test("create/drop database should create/delete the directory") {
+val catalog = newBasicCatalog()
+val db = newDb("mydb")
+catalog.createDatabase(db, ignoreIfExists = false)
+assert(exists(db.locationUri))
+
+catalog.dropDatabase("mydb", ignoreIfNotExists = false, cascade = 
false)
+assert(!exists(db.locationUri))
+  }
+
+  test("create/drop/rename table should create/delete/rename the 
directory") {
+val catalog = newBasicCatalog()
+val db = catalog.getDatabase("db1")
+val table = CatalogTable(
+  identifier = TableIdentifier("myTable", Some("db1")),
+  tableType = CatalogTableType.MANAGED,
+  storage = CatalogStorageFormat(None, None, None, None, false, 
Map.empty),
+  schema = Seq(CatalogColumn("a", "int"), CatalogColumn("b", "string"))
+)
+
+catalog.createTable("db1", table, ignoreIfExists = false)
+assert(exists(db.locationUri, "myTable"))
+
+catalog.renameTable("db1", "myTable", "yourTable")
+assert(!exists(db.locationUri, "myTable"))
+assert(exists(db.locationUri, "yourTable"))
+
+catalog.dropTable("db1", "yourTable", ignoreIfNotExists = false)
+assert(!exists(db.locationUri, "yourTable"))
+  }
+
+  test("create/drop/rename partitions should create/delete/rename the 
directory") {
+val catalog = newBasicCatalog()
+val databaseDir = catalog.getDatabase("db1").locationUri
+val table = CatalogTable(
+  identifier = TableIdentifier("myTable", Some("db1")),
+  tableType = CatalogTableType.MANAGED,
+  storage = CatalogStorageFormat(None, None, None, None, false, 
Map.empty),
+  schema = Seq(
+CatalogColumn("col1", "int"),
+CatalogColumn("col2", "string"),
+CatalogColumn("a", "int"),
+CatalogColumn("b", "string")),
+  partitionColumnNames = Seq("a", "b")
+)
+catalog.createTable("db1", table, ignoreIfExists = false)
+
+catalog.createPartitions("db1", "myTable", Seq(part1, part2), 
ignoreIfExists = false)
+assert(exists(databaseDir, "myTable", "a=1", "b=2"))
+assert(exists(databaseDir, "myTable", "a=3", "b=4"))
+
+catalog.renamePartitions("db1", "myTable", Seq(part1.spec), 
Seq(part3.spec))
+assert(!exists(databaseDir, "myTable", "a=1", "b=2"))
+assert(exists(databaseDir, "myTable", "a=5", "b=6"))
+
+catalog.dropPartitions("db1", "myTable", Seq(part2.spec, part3.spec), 
ignoreIfNotExists = false)
+assert(!exists(databaseDir, "myTable", "a=3", "b=4"))
+assert(!exists(databaseDir, "myTable", "a=5", "b=6"))
+  }
--- End diff --

Also test the case when partition has a user-specified location?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62296175
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
 ---
@@ -488,6 +491,79 @@ abstract class ExternalCatalogSuite extends 
SparkFunSuite with BeforeAndAfterEac
 assert(catalog.listFunctions("db2", "func*").toSet == Set("func1", 
"func2"))
   }
 
+  // 
--
+  // File System operations
+  // 
--
+
+  private def exists(uri: String, children: String*): Boolean = {
+val base = new File(new URI(uri))
+
logWarning(base.listFiles().toSeq.map(_.getAbsolutePath).mkString("\n###"))
+val f = children.foldLeft(base) {
+  case (parent, child) => new File(parent, child)
+}
+logWarning("exists? " + f.getAbsolutePath)
+f.exists()
+  }
+
+  test("create/drop database should create/delete the directory") {
+val catalog = newBasicCatalog()
+val db = newDb("mydb")
+catalog.createDatabase(db, ignoreIfExists = false)
+assert(exists(db.locationUri))
+
+catalog.dropDatabase("mydb", ignoreIfNotExists = false, cascade = 
false)
+assert(!exists(db.locationUri))
+  }
+
+  test("create/drop/rename table should create/delete/rename the 
directory") {
+val catalog = newBasicCatalog()
+val db = catalog.getDatabase("db1")
+val table = CatalogTable(
+  identifier = TableIdentifier("myTable", Some("db1")),
+  tableType = CatalogTableType.MANAGED,
+  storage = CatalogStorageFormat(None, None, None, None, false, 
Map.empty),
+  schema = Seq(CatalogColumn("a", "int"), CatalogColumn("b", "string"))
+)
+
+catalog.createTable("db1", table, ignoreIfExists = false)
+assert(exists(db.locationUri, "myTable"))
+
+catalog.renameTable("db1", "myTable", "yourTable")
+assert(!exists(db.locationUri, "myTable"))
+assert(exists(db.locationUri, "yourTable"))
+
+catalog.dropTable("db1", "yourTable", ignoreIfNotExists = false)
+assert(!exists(db.locationUri, "yourTable"))
+  }
--- End diff --

Also test external table?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62295951
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -272,9 +362,29 @@ class InMemoryCatalog extends ExternalCatalog {
   specs: Seq[TablePartitionSpec],
   newSpecs: Seq[TablePartitionSpec]): Unit = synchronized {
 require(specs.size == newSpecs.size, "number of old and new partition 
specs differ")
+
+val tableDir = new Path(catalog(db).db.locationUri, table)
+val partitionColumnNames = getTable(db, table).partitionColumnNames
+// TODO: we should follow hive to roll back if one partition path 
failed to rename.
 specs.zip(newSpecs).foreach { case (oldSpec, newSpec) =>
   val newPart = getPartition(db, table, oldSpec).copy(spec = newSpec)
   val existingParts = catalog(db).tables(table).partitions
+
+  if (newPart.storage.locationUri.isEmpty) {
--- End diff --

Let's add a comment at here to explain the meaning of this condition. (if 
location is not set, the partition is using the default partition location, 
right?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217358886
  
**[Test build #57967 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57967/consoleFull)**
 for PR 12871 at commit 
[`e975047`](https://github.com/apache/spark/commit/e97504759ad6ba640e858555fc6dbf7e9604ec1a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217348613
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57957/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217348611
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217348567
  
**[Test build #57957 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57957/consoleFull)**
 for PR 12871 at commit 
[`8ff5d9c`](https://github.com/apache/spark/commit/8ff5d9c27f6d1fcb86de4b4b9c06511250c7e7af).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217343030
  
**[Test build #57957 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57957/consoleFull)**
 for PR 12871 at commit 
[`8ff5d9c`](https://github.com/apache/spark/commit/8ff5d9c27f6d1fcb86de4b4b9c06511250c7e7af).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217332358
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57942/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217332357
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217332280
  
**[Test build #57942 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57942/consoleFull)**
 for PR 12871 at commit 
[`53aa782`](https://github.com/apache/spark/commit/53aa782433ee0ee3e7555ddd25e7d4e3ef864eba).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217324219
  
**[Test build #57942 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57942/consoleFull)**
 for PR 12871 at commit 
[`53aa782`](https://github.com/apache/spark/commit/53aa782433ee0ee3e7555ddd25e7d4e3ef864eba).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217322740
  
**[Test build #57940 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57940/consoleFull)**
 for PR 12871 at commit 
[`a239860`](https://github.com/apache/spark/commit/a239860694d94c8508d75dc3db1bd899c6cd7d58).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217322743
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217322745
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57940/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217322585
  
**[Test build #57940 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57940/consoleFull)**
 for PR 12871 at commit 
[`a239860`](https://github.com/apache/spark/commit/a239860694d94c8508d75dc3db1bd899c6cd7d58).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217321466
  
**[Test build #57931 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57931/consoleFull)**
 for PR 12871 at commit 
[`002fdb3`](https://github.com/apache/spark/commit/002fdb330468ac0169281efa60a675af70e50383).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217321534
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217321536
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57931/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217314164
  
**[Test build #57931 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57931/consoleFull)**
 for PR 12871 at commit 
[`002fdb3`](https://github.com/apache/spark/commit/002fdb330468ac0169281efa60a675af70e50383).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r6518
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -104,6 +119,14 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
   // Remove the database.
+  val dbDefinition = catalog(db).db
+  try {
+fs.delete(new Path(dbDefinition.locationUri), true)
--- End diff --

we don't support purge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217217119
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217217121
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57897/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217216871
  
**[Test build #57897 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57897/consoleFull)**
 for PR 12871 at commit 
[`ed86c0e`](https://github.com/apache/spark/commit/ed86c0ef244ad691aee03aa1d759e89739fe5197).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217201923
  
This is weird, I can pass these tests locally. Maybe it's related to path 
format, added some log to investigate it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217201731
  
**[Test build #57897 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57897/consoleFull)**
 for PR 12871 at commit 
[`ed86c0e`](https://github.com/apache/spark/commit/ed86c0ef244ad691aee03aa1d759e89739fe5197).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217111974
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217111975
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57869/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217111876
  
**[Test build #57869 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57869/consoleFull)**
 for PR 12871 at commit 
[`6fe20cb`](https://github.com/apache/spark/commit/6fe20cbf3f95929147788fa1a072d8918e4d010a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217103030
  
**[Test build #57869 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57869/consoleFull)**
 for PR 12871 at commit 
[`6fe20cb`](https://github.com/apache/spark/commit/6fe20cbf3f95929147788fa1a072d8918e4d010a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217101578
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57863/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217101573
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217101528
  
**[Test build #57863 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57863/consoleFull)**
 for PR 12871 at commit 
[`eeccf69`](https://github.com/apache/spark/commit/eeccf694d1ba000324c22ed641712e2f8360404d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217100208
  
**[Test build #57863 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57863/consoleFull)**
 for PR 12871 at commit 
[`eeccf69`](https://github.com/apache/spark/commit/eeccf694d1ba000324c22ed641712e2f8360404d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217086705
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57851/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217086702
  
**[Test build #57851 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57851/consoleFull)**
 for PR 12871 at commit 
[`6bc6c7a`](https://github.com/apache/spark/commit/6bc6c7a9d93b10777356f6dea92d0849b816c4a1).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217086704
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217085785
  
**[Test build #57851 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57851/consoleFull)**
 for PR 12871 at commit 
[`6bc6c7a`](https://github.com/apache/spark/commit/6bc6c7a9d93b10777356f6dea92d0849b816c4a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62152383
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -245,7 +301,21 @@ class InMemoryCatalog extends ExternalCatalog {
   s"'$db' table '$table':\n$dupSpecsStr")
   }
 }
-parts.foreach { p => existingParts.put(p.spec, p) }
+
+val tableDir = new Path(catalog(db).db.locationUri, table)
+// TODO: we should follow hive to roll back if one partition path 
failed to create.
+parts.foreach { p =>
+  if (p.storage.locationUri.isEmpty) {
+val partitionPath = p.spec.map { case (k, v) => s"$k=$v" 
}.mkString("/")
--- End diff --

good catch!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62149583
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -245,7 +301,21 @@ class InMemoryCatalog extends ExternalCatalog {
   s"'$db' table '$table':\n$dupSpecsStr")
   }
 }
-parts.foreach { p => existingParts.put(p.spec, p) }
+
+val tableDir = new Path(catalog(db).db.locationUri, table)
+// TODO: we should follow hive to roll back if one partition path 
failed to create.
+parts.foreach { p =>
+  if (p.storage.locationUri.isEmpty) {
+val partitionPath = p.spec.map { case (k, v) => s"$k=$v" 
}.mkString("/")
--- End diff --

Yeah, you are right! @dilipbiswal We need to resolve it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62149522
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -245,7 +301,21 @@ class InMemoryCatalog extends ExternalCatalog {
   s"'$db' table '$table':\n$dupSpecsStr")
   }
 }
-parts.foreach { p => existingParts.put(p.spec, p) }
+
+val tableDir = new Path(catalog(db).db.locationUri, table)
+// TODO: we should follow hive to roll back if one partition path 
failed to create.
+parts.foreach { p =>
+  if (p.storage.locationUri.isEmpty) {
+val partitionPath = p.spec.map { case (k, v) => s"$k=$v" 
}.mkString("/")
--- End diff --

One potential issue is that the partition spec is a map and is un-ordered. 
In one of my PR, we had to first compute the order based on partition column 
names. Lian had found this issue while reviewing my PR :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62149040
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -245,7 +301,21 @@ class InMemoryCatalog extends ExternalCatalog {
   s"'$db' table '$table':\n$dupSpecsStr")
   }
 }
-parts.foreach { p => existingParts.put(p.spec, p) }
+
+val tableDir = new Path(catalog(db).db.locationUri, table)
+// TODO: we should follow hive to roll back if one partition path 
failed to create.
+parts.foreach { p =>
+  if (p.storage.locationUri.isEmpty) {
+val partitionPath = p.spec.map { case (k, v) => s"$k=$v" 
}.mkString("/")
--- End diff --

I am not 100% sure if this always works. Here, do we support all the 
possible inputs for `key` and `value`? Any limit when we create a directory 
name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62148788
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -104,6 +119,14 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
   // Remove the database.
+  val dbDefinition = catalog(db).db
+  try {
+fs.delete(new Path(dbDefinition.locationUri), true)
--- End diff --

So far, in `HiveExternalCatalog`, I saw this issue. Hive metastore has the 
same behavior. 

Regarding `Purge`, DROP DATABASE does not have such an option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62148671
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -85,6 +93,13 @@ class InMemoryCatalog extends ExternalCatalog {
 throw new AnalysisException(s"Database '${dbDefinition.name}' 
already exists.")
   }
 } else {
+  try {
+fs.mkdirs(new Path(dbDefinition.locationUri))
--- End diff --

Just made a try. If the directory already exists, it does not issue any 
exception. You know, when users provide a location, it could have been created. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62143683
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -104,6 +119,14 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
   // Remove the database.
+  val dbDefinition = catalog(db).db
+  try {
+fs.delete(new Path(dbDefinition.locationUri), true)
--- End diff --

Does DROP DATABASE has PURGE option? what does hive do for this case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62143626
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -85,6 +93,13 @@ class InMemoryCatalog extends ExternalCatalog {
 throw new AnalysisException(s"Database '${dbDefinition.name}' 
already exists.")
   }
 } else {
+  try {
+fs.mkdirs(new Path(dbDefinition.locationUri))
--- End diff --

We already checked the metadata, I don't think we need to check it again 
here. BTW, I guess it's OK if the dir already exists and is empty?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62141623
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -104,6 +119,14 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
   // Remove the database.
+  val dbDefinition = catalog(db).db
+  try {
+fs.delete(new Path(dbDefinition.locationUri), true)
--- End diff --

Just one possible issue. If we create a database with an non-existent 
directory, it could create `.../x/y/z/`, `.../x/y/` and `.../x/`. When we drop 
a database, we will just drop the leaf directory `.../x/y/z/`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62141537
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -85,6 +93,13 @@ class InMemoryCatalog extends ExternalCatalog {
 throw new AnalysisException(s"Database '${dbDefinition.name}' 
already exists.")
   }
 } else {
+  try {
+fs.mkdirs(new Path(dbDefinition.locationUri))
--- End diff --

How about checking if it already exists?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-217006919
  
This looks pretty good. It's failing tests though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216907386
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216907389
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57765/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216907174
  
**[Test build #57765 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57765/consoleFull)**
 for PR 12871 at commit 
[`a744099`](https://github.com/apache/spark/commit/a74409937bb7ec7c23f8bc563dd32ed5de9fcdf7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216891935
  
@cloud-fan uh, I see. Then, there is a lot of things to do for building our 
own metastore. : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12871#discussion_r62052925
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -73,6 +79,8 @@ class InMemoryCatalog extends ExternalCatalog {
 }
   }
 
+  private val fs = FileSystem.get(new Configuration)
--- End diff --

The passed in database location URI is already qualified, so I think it's 
safe to use a empty configuration to get the file system. Please correct me if 
I was wrong. cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-21614
  
@gatorsmile , There is no use case for it currently. This is an attempt to 
make `InMemoryCatalog` more consistent with hive catalog, and also a step 
forward to make our own metastore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216888219
  
**[Test build #57765 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57765/consoleFull)**
 for PR 12871 at commit 
[`a744099`](https://github.com/apache/spark/commit/a74409937bb7ec7c23f8bc563dd32ed5de9fcdf7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216596397
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57643/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216596396
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216596200
  
**[Test build #57643 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57643/consoleFull)**
 for PR 12871 at commit 
[`667f089`](https://github.com/apache/spark/commit/667f089a1ab4b3b733a2af7907e3aa00224d9ecc).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216586798
  
Just a question. The directory for InMemoryCatalog is not being used. Do 
you have any use case for it? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216575435
  
**[Test build #57643 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57643/consoleFull)**
 for PR 12871 at commit 
[`667f089`](https://github.com/apache/spark/commit/667f089a1ab4b3b733a2af7907e3aa00224d9ecc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/12871#issuecomment-216575162
  
cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15093][SQL] create/delete/rename direct...

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/12871

[SPARK-15093][SQL] create/delete/rename directory for InMemoryCatalog 
operations if needed

## What changes were proposed in this pull request?

following operations have file system operation now:

1. CREATE DATABASE: create a dir
2. DROP DATABASE: delete the dir
3. ALTER DATABASE: rename the dir
4. CREATE TABLE: create a dir
5. DROP TABLE: delete the dir
6. RENAME TABLE: rename the dir

TODO:
we should also add file system operation for partition related operations, 
like CREATE PARTITION. But one problem is, should we do transaction for it? 
e.g. if we are trying to create 4 partitions, the first 3 success but the last 
one failed because of some IO problems, should we roll back the first 3 
partition directory creation?

## How was this patch tested?

todo


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark catalog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12871


commit 667f089a1ab4b3b733a2af7907e3aa00224d9ecc
Author: Wenchen Fan 
Date:   2016-05-03T14:25:44Z

create/delete/rename directory for InMemoryCatalog operations if needed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org