Google meet link for sync

Hey everyone, looks like the google meet link for our sync broke so I created a new one: https://meet.google.com/jpj-qevt-xko Please join that instead. Ryan -- Ryan Blue Tabular

Re: Writing iceberg table to S3

`SET iceberg.mr.catalog=hive` works!!! Thanks Ryan, you rock!!! You may consider adding the below into iceberg document to help other newcomers. Add `SET iceberg.mr.catalog=hive` to https://iceberg.apache.org/hive/. Add `.tableProperty("location", filePath)` to https://iceberg.apache.org/spark-wri

Re: Writing iceberg table to S3

Looks like the table is set up correctly. I think the problem might be how Hive is configured. I think by default it will try to load tables by location in 0.11.1. You need to tell it to load tables as metastore tables, not HDFS tables by running `SET iceberg.mr.catalog=hive`. On Wed, Aug 11, 2021

Re: Writing iceberg table to S3

hive> describe formatted mytable3; OK # col_name data_type comment value int # Detailed Table Information Database: mydb OwnerType: USER Owner: root CreateTime: Wed Aug 11 20:02:14 UTC 2021 LastAcc

Re: Writing iceberg table to S3

Can you run `DESCRIBE FORMATTED` for the table? Then we can see if there is a storage handler set up for it. On Wed, Aug 11, 2021 at 1:46 PM Lian Jiang wrote: > Thanks guys. tableProperty("location", ...) works. > > I have trouble making hive query an iceberg table by following > https://iceberg

Re: Writing iceberg table to S3

Thanks guys. tableProperty("location", ...) works. I have trouble making hive query an iceberg table by following https://iceberg.apache.org/hive/. I have done: * in Hive shell, do `add jar /path/to/iceberg-hive-runtime.jar;` * in hive-site.xml, add hive.vectorized.execution.enabled=false and ic

Re: Writing iceberg table to S3

The problem for #3 is how Spark handles the options. The option method sets write options, not table properties. The write options aren’t passed when creating the table. Instead, you should use tableProperty("location", ...). Ryan On Wed, Aug 11, 2021 at 9:17 AM Russell Spitzer wrote: > 2) Hive

Re: Writing iceberg table to S3

2) Hive cannot read Iceberg tables without configuring the MR Hive integration from iceberg. So you shouldn't see it in hive unless you have configured that, see https://iceberg.apache.org/hive/. 3) https://github.com/apache/iceberg/blob/master/spark3/src/main/java/org/apache/iceberg/spark/Sp

Re: Writing iceberg table to S3

Any help is highly appreciated! On Tue, Aug 10, 2021 at 11:06 AM Lian Jiang wrote: > Thanks Russell. > > I tried: > > /spark/bin/spark-shell --packages > org.apache.iceberg:iceberg-hive-runtime:0.11.1,org.apache.iceberg:iceberg-spark3-runtime:0.11.1 > --conf spark.sql.catalog.hive_test=org.apach

Re: Subject: [VOTE] Release Apache Iceberg 0.12.0 RC3

+1 (non-binding) - Checked signature, checksum, and license. - Ran build and test (failures in iceberg-mr and iceberg-hive3) On Tue, Aug 10, 2021 at 10:12 PM Szehon Ho wrote: > +1 (non binding) > > * Checked Signature Keys > * Verified Checksum > * Rat checks > * Build and run tests, most funct