Re: Schema store for Parquet

2020-03-09 Thread Ruijing Li
Thanks Magnus, I’ll explore Atlas and see what I can find. On Wed, Mar 4, 2020 at 11:10 AM Magnus Nilsson wrote: > Apache Atlas is the apache data catalog. Maybe want to look into that. It > depends on what your use case is. > > On Wed, Mar 4, 2020 at 8:01 PM Ruijing Li wrote: > >> Thanks

Re: Schema store for Parquet

2020-03-04 Thread Magnus Nilsson
Apache Atlas is the apache data catalog. Maybe want to look into that. It depends on what your use case is. On Wed, Mar 4, 2020 at 8:01 PM Ruijing Li wrote: > Thanks Lucas and Magnus, > > Would there be any open source solutions other than Apache Hive metastore, > if we don’t wish to use Apache

Re: Schema store for Parquet

2020-03-04 Thread Ruijing Li
Thanks Lucas and Magnus, Would there be any open source solutions other than Apache Hive metastore, if we don’t wish to use Apache Hive and spark? Thanks. On Wed, Mar 4, 2020 at 10:40 AM lucas.g...@gmail.com wrote: > Or AWS glue catalog if you're in AWS > > On Wed, 4 Mar 2020 at 10:35, Magnus

Re: Schema store for Parquet

2020-03-04 Thread lucas.g...@gmail.com
Or AWS glue catalog if you're in AWS On Wed, 4 Mar 2020 at 10:35, Magnus Nilsson wrote: > Google hive metastore. > > On Wed, Mar 4, 2020 at 7:29 PM Ruijing Li wrote: > >> Hi all, >> >> Has anyone explored efforts to have a centralized storage of schemas of >> different parquet files? I know

Re: Schema store for Parquet

2020-03-04 Thread Magnus Nilsson
Google hive metastore. On Wed, Mar 4, 2020 at 7:29 PM Ruijing Li wrote: > Hi all, > > Has anyone explored efforts to have a centralized storage of schemas of > different parquet files? I know there is schema management for Avro, but > couldn’t find solutions for parquet schema management.