Weizhong created SPARK-15335: -------------------------------- Summary: In Spark 2.0 TRUNCATE TABLE is unsupported Key: SPARK-15335 URL: https://issues.apache.org/jira/browse/SPARK-15335 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.0.0 Reporter: Weizhong Priority: Minor
Spark version based on commit b3930f74a0929b2cdcbbe5cbe34f0b1d35eb01cc, test result is like below: {noformat} spark-sql> create table truncateTT(c string); 16/05/16 10:23:15 INFO execution.SparkSqlParser: Parsing command: create table truncateTT(c string) 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: get_database: default 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_database: default 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: get_database: default 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_database: default 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: create_table: Table(tableName:truncatett, dbName:default, owner:root, createTime:1463365395, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:c, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null)) 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=create_table: Table(tableName:truncatett, dbName:default, owner:root, createTime:1463365395, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:c, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null)) 16/05/16 10:23:15 INFO common.FileUtils: Creating directory if it doesn't exist: hdfs://vm001:9000/opt/apache/spark/spark-warehouse/truncatett 16/05/16 10:23:16 INFO spark.SparkContext: Starting job: processCmd at CliDriver.java:376 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Got job 1 (processCmd at CliDriver.java:376) with 1 output partitions 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (processCmd at CliDriver.java:376) 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Parents of final stage: List() 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Missing parents: List() 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at processCmd at CliDriver.java:376), which has no missing parents 16/05/16 10:23:16 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.2 KB, free 1823.2 MB) 16/05/16 10:23:16 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1965.0 B, free 1823.2 MB) 16/05/16 10:23:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.151.146:47228 (size: 1965.0 B, free: 1823.2 MB) 16/05/16 10:23:16 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1012 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at processCmd at CliDriver.java:376) 16/05/16 10:23:16 INFO cluster.YarnScheduler: Adding task set 1.0 with 1 tasks 16/05/16 10:23:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, vm001, partition 0, PROCESS_LOCAL, 5387 bytes) 16/05/16 10:23:16 INFO cluster.YarnClientSchedulerBackend: Launching task 1 on executor id: 1 hostname: vm001. 16/05/16 10:23:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on vm001:35665 (size: 1965.0 B, free: 4.4 GB) 16/05/16 10:23:18 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2105 ms on vm001 (1/1) 16/05/16 10:23:18 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 16/05/16 10:23:18 INFO scheduler.DAGScheduler: ResultStage 1 (processCmd at CliDriver.java:376) finished in 2.105 s 16/05/16 10:23:18 INFO scheduler.DAGScheduler: Job 1 finished: processCmd at CliDriver.java:376, took 2.121866 s Time taken: 2.691 seconds 16/05/16 10:23:18 INFO CliDriver: Time taken: 2.691 seconds spark-sql> truncate table truncateTT; 16/05/16 10:23:32 INFO execution.SparkSqlParser: Parsing command: truncate table truncateTT Error in query: Unsupported SQL statement == SQL == truncate table truncateTT spark-sql> {noformat} Before 2.0 we will run 'TRUNCATE TABLE ...' as a Hive native command, so it can work. In Spark 2.0 we also need to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org