Behavior of dropping table with HadoopCatalog

2023-08-29 Thread Manu Zhang
Hi all, The current behavior of dropping a table with HadoopCatalog looks inconsistent to me. When stored at the default location under table path, metadata and data will be deleted regardless of the "purge" flag. When stored elsewhere, they will be left there if not "purge". Is this by design? @

Table owned locations

2023-08-29 Thread Szehon Ho
Hi all, As you know, there is a recurring Iceberg issue where delete orphan file operations may inadvertently delete other table's data, if they are misconfigured to have the same location. A while back, Anton had a proposal for 'owned.locations' in: https://github.com/apache/iceberg/issues/4159

Re: two proposed spec changes

2023-08-29 Thread Jacob Marble
Please define "derived field"? We don't allow empty string as a tag value, so that sentinel value is available. However, there are some second-order effects that need to be considered. Just thinking out loud, I haven't explored using default values for tags in our Iceberg export code; certainly n

Re: two proposed spec changes

2023-08-29 Thread Ryan Blue
Jacob, could you model this with a derived field? Or could you require the tags and use a "unknown" value? On Mon, Aug 28, 2023 at 11:18 AM Jacob Marble wrote: > On Fri, Aug 25, 2023 at 3:23 PM Ryan Blue wrote: > >> I don't think that we should introduce nanosecond precision types without >> at