paul-rogers commented on a change in pull request #1953: Add docs for Drill Metastore URL: https://github.com/apache/drill/pull/1953#discussion_r374452765
########## File path: _docs/performance-tuning/drill-metastore/030-drill-iceberg-metastore.md ########## @@ -0,0 +1,69 @@ +--- +title: "Drill Iceberg Metastore" +parent: "Drill Metastore" +date: 2020-01-31 +--- + +Drill uses Iceberg Metastore implementation based on [Iceberg tables](http://iceberg.incubator.apache.org). For Drill 1.17, + this is default Drill Metastore implementation. For details on how to configure Iceberg Metastore implementation and + its option descriptions, please refer to [Iceberg Metastore docs](https://github.com/apache/drill/blob/master/metastore/iceberg-metastore/README.md). + +{% include startnote.html %} +Iceberg table supports concurrent writes and transactions but they are only effective on file systems that support + atomic rename. +If the file system does not support atomic rename, it could lead to inconsistencies during concurrent writes. +{% include endnote.html %} + +### Iceberg Tables Location + +Iceberg tables will reside on the file system in the location based on +Iceberg Metastore base location `drill.metastore.iceberg.location.base_path` and component specific location. +If Iceberg Metastore base location is `/drill/metastore/iceberg` +and tables component location is `tables`. Iceberg table for tables component +will be located in `/drill/metastore/iceberg/tables` folder. + +Metastore metadata will be stored inside Iceberg table location provided +in the configuration file. Drill table metadata location will be constructed +based on specific component storage keys. For example, for `tables` component, +storage keys are storage plugin, workspace and table name: unique table identifier in Drill. + +Assume Iceberg table location is `/drill/metastore/iceberg/tables`, metadata for the table +`dfs.tmp.nation` will be stored in the `/drill/metastore/iceberg/tables/dfs/tmp/nation` folder. Review comment: This would be clearer with a concrete example. If you ran through the tutorial, Metastore files are stored in (where.) If you inspect this directory, you will see the following directories "dfs/tmp/lineitem". Note: if this example is accurate, we've got a problem. "dfs" is the plugin name. I can have multiple such plugins, each of which may have a "tmp" workspace. Metastore can't tell them apart. If this is true, let's file a bug and get this fixed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
