parquet-tools can be used only for inspecting parquet files, not for creating new parquet files. Yes, you can use CTAS to do this. You have to manually remove the old files and move the new files. It does impact the metadata caching mechanism. You need to regenerate metadata cache.
Thanks Padma > On Oct 31, 2017, at 5:08 AM, Rahul Raj <rahul....@option3consulting.com> > wrote: > > Hi, > > I have few questions on modeling a time series use case with parquet and > drill. I have seen the topic discussed at > https://issues.apache.org/jira/browse/DRILL-3534. > > My requirements are: > > * Keep the parquet files partitioned by year and month > * For the current month, the data needs to be further partitioned by Week > and Day > * End of the running week, 7 daily parquets will be merged to a single > weekly file > * Similarly, weekly files will to be merged to form a monthly file during > month end > > I will have a web application to generate the daily data and to ensure the > batch runs/ atomic writes/locking etc. > > What are the possible ways to merge parquet files? Another CTAS? > > Is it possible to use parquet-tools(part of Parquet-MR) to merge multiple > parquets(java jar ./parquet-tools-<VERSION>.jar <command> <input-directory> > <output-file>) and then let drill query the results?. Will it impact the > drill meta data caching mechanism? > > Regards, > Rahul > > -- > **** This email and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom it is > addressed. If you are not the named addressee then you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately and delete this e-mail from your system.****
