ABHISHEK KUMAR GUPTA created SPARK-24170:
--------------------------------------------
Summary: [Spark SQL] json file format is not dropped after
dropping table
Key: SPARK-24170
URL: https://issues.apache.org/jira/browse/SPARK-24170
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Environment: OS: SUSE 11
Spark Version: 2.3
Reporter: ABHISHEK KUMAR GUPTA
Steps:
# Launch spark-sql --master yarn
# create table json(name STRING, age int, gender string, id INT) using
org.apache.spark.sql.json options(path "hdfs:///user/testdemo/");
# Execute the below SQL queries
INSERT into json
SELECT 'Shaan',21,'Male',1
UNION ALL
SELECT 'Xing',20,'Female',11
UNION ALL
SELECT 'Mile',4,'Female',20
UNION ALL
SELECT 'Malan',10,'Male',9;
Below 4 json file format created
BLR1000023111:/opt/Antsecure/install/hadoop/namenode/bin # ./hdfs dfs -ls
/user/testdemo
Found 14 items
-rw-r--r-- 3 spark hadoop 0 2018-04-26 17:44 /user/testdemo/_SUCCESS
-rw-r--r-- 3 spark hadoop 4802 2018-04-24 18:20 /user/testdemo/customer1.csv
-rw-r--r-- 3 spark hadoop 92 2018-04-26 17:02 /user/testdemo/json1.txt
-rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32
/user/testdemo/part-00000-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
-rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44
/user/testdemo/part-00000-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
-rw-r--r-- 3 spark hadoop 51 2018-04-26 17:32
/user/testdemo/part-00001-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
-rw-r--r-- 3 spark hadoop 51 2018-04-26 17:44
/user/testdemo/part-00001-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
-rw-r--r-- 3 spark hadoop 50 2018-04-26 17:32
/user/testdemo/part-00002-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
-rw-r--r-- 3 spark hadoop 50 2018-04-26 17:44
/user/testdemo/part-00002-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
-rw-r--r-- 3 spark hadoop 49 2018-04-26 17:32
/user/testdemo/part-00003-4311f66b-ba1b-4a4d-a289-1a211f27f653-c000.json
-rw-r--r-- 3 spark hadoop 49 2018-04-26 17:44
/user/testdemo/part-00003-b8a8e16a-91a8-48ec-9998-2d741c52cf5a-c000.json
Issue is:
Now executed below drop command
spark-sql> drop table json;
Table dropped successfully but json file still present in the path
/user/testdemo
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
