Thanks, Ryan OK, I will create one for this and you guys can delete if duplicated. I wanted to run spark-sql-perf to verify basic functionality and performance and found that It can't convert from spark/hive table to iceberg table.
On Wed, Jul 3, 2019 at 1:07 AM Ryan Blue <rb...@netflix.com.invalid> wrote: > > Right now, we don't have a utility to convert tables automatically. I don't > think there is an open issue for it (though Anton may have opened one). > > On Tue, Jul 2, 2019 at 9:43 AM 俊杰陈 <cjjnj...@gmail.com> wrote: >> >> Thanks for the explanation. >> >> So to migrate a spark table to iceberg table, I can append the >> SparkDataFiles from listParquetPartition to an iceberg table, right? I >> don't see an existing open issue for this, is this still wokring in progress? >> >> On Mon, Jul 1, 2019 at 7:34 PM Anton Okolnychyi >> <aokolnyc...@apple.com.invalid> wrote: >>> >>> Hey, >>> >>> The issue you pointed out is about tracking Iceberg tables in HMS and >>> levering HMS locks to commit metadata instead of relying on renames. This >>> allows Iceberg to reliably manage metadata when it is persisted in object >>> stores. >>> >>> At the same time, it is possible to migrate Spark tables to Iceberg without >>> ETL jobs. You take a look at org.apache.iceberg.spark.SparkTableUtil. There >>> is a plan to provide a proper API for migrating tables but it is not done >>> yet. >>> >>> Thanks, >>> Anton >>> >>> On 25 Jun 2019, at 16:35, 俊杰陈 <cjjnj...@gmail.com> wrote: >>> >>> Hi >>> >>> Do we support to read a hive table and convert it to iceberg table? Is this >>> issue for tracking the feature? >>> >>> -- >>> Thanks & Best Regards >>> >>> >> >> >> -- >> Thanks & Best Regards > > > > -- > Ryan Blue > Software Engineer > Netflix -- Thanks & Best Regards
