Re: How to use iceberg in spark?

2019-05-14 Thread 俊杰陈
Try to send this again since I can't see this on dev mail list. On Tue, May 14, 2019 at 1:24 PM 俊杰陈 wrote: > Hi > > Do we have wiki or doc for spark usage? I tried to put jar in spark jars > dir, but when importing org.apache.iceberg. in spark-shell, it shows > iceberg is not a member of pa

Re: Extend SparkTableUtil to Handle Tables Not Tracked in Hive Metastore

2019-05-14 Thread Anton Okolnychyi
I would like to resume this topic. How do we see the proper API for migration? I have a couple of questions in mind: - Now, it is based on a Spark job. Do we want to keep it that way because the number of files might be huge? Will it be too much for the driver? - What should be the scope? Do we w