[
https://issues.apache.org/jira/browse/SPARK-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin resolved SPARK-15335.
---------------------------------
Resolution: Fixed
Assignee: Lianhui Wang
Fix Version/s: 2.0.0
> Implement TRUNCATE TABLE Command
> --------------------------------
>
> Key: SPARK-15335
> URL: https://issues.apache.org/jira/browse/SPARK-15335
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Weizhong
> Assignee: Lianhui Wang
> Priority: Minor
> Fix For: 2.0.0
>
>
> Spark version based on commit b3930f74a0929b2cdcbbe5cbe34f0b1d35eb01cc, test
> result is like below:
> {noformat}
> spark-sql> create table truncateTT(c string);
> 16/05/16 10:23:15 INFO execution.SparkSqlParser: Parsing command: create
> table truncateTT(c string)
> 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: get_database: default
> 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr
> cmd=get_database: default
> 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: get_database: default
> 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr
> cmd=get_database: default
> 16/05/16 10:23:15 INFO metastore.HiveMetaStore: 0: create_table:
> Table(tableName:truncatett, dbName:default, owner:root,
> createTime:1463365395, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:c, type:string, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{},
> viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
> privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null,
> rolePrivileges:null))
> 16/05/16 10:23:15 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr
> cmd=create_table: Table(tableName:truncatett, dbName:default, owner:root,
> createTime:1463365395, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:c, type:string, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{},
> viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
> privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null,
> rolePrivileges:null))
> 16/05/16 10:23:15 INFO common.FileUtils: Creating directory if it doesn't
> exist: hdfs://vm001:9000/opt/apache/spark/spark-warehouse/truncatett
> 16/05/16 10:23:16 INFO spark.SparkContext: Starting job: processCmd at
> CliDriver.java:376
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Got job 1 (processCmd at
> CliDriver.java:376) with 1 output partitions
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 1
> (processCmd at CliDriver.java:376)
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Parents of final stage: List()
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Missing parents: List()
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Submitting ResultStage 1
> (MapPartitionsRDD[5] at processCmd at CliDriver.java:376), which has no
> missing parents
> 16/05/16 10:23:16 INFO memory.MemoryStore: Block broadcast_1 stored as values
> in memory (estimated size 3.2 KB, free 1823.2 MB)
> 16/05/16 10:23:16 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 1965.0 B, free 1823.2 MB)
> 16/05/16 10:23:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
> memory on 192.168.151.146:47228 (size: 1965.0 B, free: 1823.2 MB)
> 16/05/16 10:23:16 INFO spark.SparkContext: Created broadcast 1 from broadcast
> at DAGScheduler.scala:1012
> 16/05/16 10:23:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
> from ResultStage 1 (MapPartitionsRDD[5] at processCmd at CliDriver.java:376)
> 16/05/16 10:23:16 INFO cluster.YarnScheduler: Adding task set 1.0 with 1 tasks
> 16/05/16 10:23:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
> 1.0 (TID 1, vm001, partition 0, PROCESS_LOCAL, 5387 bytes)
> 16/05/16 10:23:16 INFO cluster.YarnClientSchedulerBackend: Launching task 1
> on executor id: 1 hostname: vm001.
> 16/05/16 10:23:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in
> memory on vm001:35665 (size: 1965.0 B, free: 4.4 GB)
> 16/05/16 10:23:18 INFO scheduler.TaskSetManager: Finished task 0.0 in stage
> 1.0 (TID 1) in 2105 ms on vm001 (1/1)
> 16/05/16 10:23:18 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose
> tasks have all completed, from pool
> 16/05/16 10:23:18 INFO scheduler.DAGScheduler: ResultStage 1 (processCmd at
> CliDriver.java:376) finished in 2.105 s
> 16/05/16 10:23:18 INFO scheduler.DAGScheduler: Job 1 finished: processCmd at
> CliDriver.java:376, took 2.121866 s
> Time taken: 2.691 seconds
> 16/05/16 10:23:18 INFO CliDriver: Time taken: 2.691 seconds
> spark-sql> truncate table truncateTT;
> 16/05/16 10:23:32 INFO execution.SparkSqlParser: Parsing command: truncate
> table truncateTT
> Error in query:
> Unsupported SQL statement
> == SQL ==
> truncate table truncateTT
> spark-sql>
> {noformat}
> Before 2.0 we will run 'TRUNCATE TABLE ...' as a Hive native command, so it
> can work. In Spark 2.0 we also need to support it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
