Github user vanzin commented on a diff in the pull request:
https://github.com/apache/incubator-livy/pull/121#discussion_r231341354
--- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
@@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
@Test
public void testSparkSQLJob() throws Exception {
- runTest(true, new TestFunction() {
+ runTest(true, false, new TestFunction() {
--- End diff --
I think this deserves some more investigation. It seems to maybe be an
issue in Spark, even if it might not affect Livy outside of tests.
If I re-enable the old test, and comment out these lines, things pass:
```
File metaStore = new File("metastore_db");
if (metaStore.exists()) {
FileUtils.deleteDirectory(metaStore);
}
```
This suite runs multiple different SparkContext / SparkSession instances in
the same JVM. Each one is supposed to be independent of one another, so the
lines above shouldn't cause problems.
Since they seem to be causing problems, either Livy is not shutting down
sessions appropriately, or Spark is keeping state across different SparkSession
initializations. Neither is very good...
---
