[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766407#comment-15766407 ]
luat edited comment on SPARK-18941 at 12/21/16 8:06 AM: -------------------------------------------------------- Hi [~dongjoon], Thank for your reply. I download spark 2.0.2 from link http://spark.apache.org/downloads.html Then, I unzip and start spark thrift server and test as below steps: [root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./sbin/start-thriftserver.sh \ > --hiveconf hive.server2.thrift.port=10005 \ > --hiveconf hive.server2.thrift.bind.host=10.30.132.76 \ > --master yarn [root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./bin/beeline Beeline version 1.2.1.spark2 by Apache Hive beeline> !connect jdbc:hive2://10.30.132.76:10005 Connecting to jdbc:hive2://10.30.132.76:10005 Enter username for jdbc:hive2://10.30.132.76:10005: hdfs Enter password for jdbc:hive2://10.30.132.76:10005: 16/12/21 14:36:54 INFO jdbc.Utils: Supplied authorities: 10.30.132.76:10005 16/12/21 14:36:54 INFO jdbc.Utils: Resolved authority: 10.30.132.76:10005 16/12/21 14:36:54 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://10.30.132.76:10005 Connected to: Spark SQL (version 2.0.2) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://10.30.132.76:10005> 0: jdbc:hive2://10.30.132.76:10005> create table test(a String) stored as orc location '/test'; +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.11 seconds) 0: jdbc:hive2://10.30.132.76:10005> insert into test values("abc"); +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.294 seconds) 0: jdbc:hive2://10.30.132.76:10005> select * from test; +------+--+ | a | +------+--+ | abc | +------+--+ 1 row selected (0.158 seconds) Checking result on the hdfs file system: [hdfs@namenode76 ~]$ hdfs dfs -ls /test Found 3 items drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4 -rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000 -rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001 Then, I try to drop table: 0: jdbc:hive2://10.30.132.76:10005> drop table test; +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.112 seconds) 0: jdbc:hive2://10.30.132.76:10005> select * from test; Error: org.apache.spark.sql.AnalysisException: Table or view not found: test; line 1 pos 14 (state=,code=0) 0: jdbc:hive2://10.30.132.76:10005> Checking again result on the hdfs file system: [hdfs@namenode76 ~]$ hdfs dfs -ls /test Found 3 items drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4 -rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000 -rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001 I'm running on HDP 2.5 cluster (Hadoop 2.7.3). What is wrong with my testing? was (Author: luatnc): Hi [~dongjoon], Thank for your reply. I download spark 2.0.2 from link http://spark.apache.org/downloads.html Then, I unzip and start spark thrift server and testing as below steps: [root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./sbin/start-thriftserver.sh \ > --hiveconf hive.server2.thrift.port=10005 \ > --hiveconf hive.server2.thrift.bind.host=10.30.132.76 \ > --master yarn [root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./bin/beeline Beeline version 1.2.1.spark2 by Apache Hive beeline> !connect jdbc:hive2://10.30.132.76:10005 Connecting to jdbc:hive2://10.30.132.76:10005 Enter username for jdbc:hive2://10.30.132.76:10005: hdfs Enter password for jdbc:hive2://10.30.132.76:10005: 16/12/21 14:36:54 INFO jdbc.Utils: Supplied authorities: 10.30.132.76:10005 16/12/21 14:36:54 INFO jdbc.Utils: Resolved authority: 10.30.132.76:10005 16/12/21 14:36:54 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://10.30.132.76:10005 Connected to: Spark SQL (version 2.0.2) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://10.30.132.76:10005> 0: jdbc:hive2://10.30.132.76:10005> create table test(a String) stored as orc location '/test'; +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.11 seconds) 0: jdbc:hive2://10.30.132.76:10005> insert into test values("abc"); +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.294 seconds) 0: jdbc:hive2://10.30.132.76:10005> select * from test; +------+--+ | a | +------+--+ | abc | +------+--+ 1 row selected (0.158 seconds) Checking result on the hdfs file system: [hdfs@namenode76 ~]$ hdfs dfs -ls /test Found 3 items drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4 -rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000 -rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001 Then, I try to drop table: 0: jdbc:hive2://10.30.132.76:10005> drop table test; +---------+--+ | Result | +---------+--+ +---------+--+ No rows selected (0.112 seconds) 0: jdbc:hive2://10.30.132.76:10005> select * from test; Error: org.apache.spark.sql.AnalysisException: Table or view not found: test; line 1 pos 14 (state=,code=0) 0: jdbc:hive2://10.30.132.76:10005> Checking again result on the hdfs file system: [hdfs@namenode76 ~]$ hdfs dfs -ls /test Found 3 items drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4 -rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000 -rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001 I'm running on HDP 2.5 cluster (Hadoop 2.7.3). What is wrong with my testing? > Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the > directory associated with the Hive table (not EXTERNAL table) from the HDFS > file system > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-18941 > URL: https://issues.apache.org/jira/browse/SPARK-18941 > Project: Spark > Issue Type: Bug > Components: Java API > Affects Versions: 2.0.2 > Reporter: luat > > Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the > directory associated with the Hive table (not EXTERNAL table) from the HDFS > file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org