Frank, You can play with a couple of configs to keep X number of older file versions. Take a look at these configs : https://hudi.apache.org/configurations.html#withCompactionConfig. Specifically, you can choose the number of commits you want to keep, here commits = versions. Depending on how long your query runs, you might want to keep the older data file for a configured amount of time after which it will be cleaned.
Thanks, Nishith On Mon, Mar 11, 2019 at 7:42 PM kaka chen <[email protected]> wrote: > Hi Vinoth, > > To use this feature, I find the new file will write a new file with old > inserted records. > But how to cleanup the old files when use cow tables? > > Thanks, > Frank > > Vinoth Chandar <[email protected]> 于2019年2月28日周四 上午3:24写道： > > > Similarly, please try the 0.4.5 release. This has small file handling > > turned on by default.. > > > > Also please use the insert api/operation, (not bulk_insert) if you want > > this behavior. > > > > Let us know if you still run into issues.. > > > > On Tue, Feb 26, 2019 at 11:09 PM kaka chen <[email protected]> > wrote: > > > > > Thanks! > > > > > > nishith agarwal <[email protected]> 于2019年2月27日周三 下午2:56写道： > > > > > > > Hi Kaka, > > > > > > > > Hudi automatically does file sizing for you. As you ingest more > inserts > > > the > > > > existing file will be automatically sized. You can play with a few > > > configs > > > > : > > > > > > > > https://hudi.apache.org/configurations.html#withStorageConfig -> > This > > > > config allows you to set a max size for your output file. > > > > https://hudi.apache.org/configurations.html#compactionSmallFileSize > -> > > > > This > > > > config allows you to set a minimum file size that will be > automatically > > > > sized. > > > > > > > > As you can guess, the limitFileSize >= compactionFileSize. > > > > Hope this helps. > > > > > > > > Thanks, > > > > Nishith > > > > > > > > On Tue, Feb 26, 2019 at 6:52 PM kaka chen <[email protected]> > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I found Insert will generate at least one file each time when each > > > spark > > > > or > > > > > spark streaming batch. > > > > > Is it expected result? If it is, how to control these small files, > is > > > > hudi > > > > > provide some tools to compact it? > > > > > > > > > > Thanks, > > > > > Frank > > > > > > > > > > > > > > >
