With a large number of table you also run into the following potential issues: 1. Consistency: There is no single timeline so different tables (per partition) expose data from different times of ingestion. If the data within partitions is inter-dependent then the queries may see inconsistent results.
2. Complicated error handling / debugging: If some of the pipelines fail then data in some partitions may not have been updated for some time. This may lead to data consistency issues on the query side. Debugging any issue when 1000 separate datasets are involved is much more complicated then a single dataset (e.g. hudi-cli connects to one dataset at a time). 3. (Possibly minor) Excess load on the infra: With several parallel operations, the worst case load on the Namenode may go up N times (N=number of parallel pipelines). Under provisioned NameNode may lead to out of resource errors. 4. Adding new partitions would be complicated: Assuming you would want a new partition in future, the steps would be more involved. If only a few partitions are having the load issues, you can also look into the partitioning scheme. 1. Maybe invent a new column in the schema which is more uniformly distributed 2. Maybe split the loaded partitions into two partitions (range based or something like that) 3. If possible (depending on the ingestion source), prioritize ingestion for particular partitions (partition priority queue) 4. Limit the number of records ingested at a time to limit maximum job time Thanks Prashant On Thu, Jul 9, 2020 at 12:00 AM Shayan Hati <shayanh...@gmail.com> wrote: > Thanks for your response. > > @Mario: So the metastore can be something like a Glue/Hive metastore which > basically has the metadata about different partitions in a single table. > One challenge is per partition Hudi table can be queried using Hudi library > bundle, but across partitions it has to be queried based on the metastore > itself. > > @Vinoth: The use-case is we have different partitions and the data as well > as the load is skewed on them. So one partition has to ingest much more > data than another. Basically, one large partition delta affects the > ingestion time of a smaller size partition as well. Also failure/corrupt > data of one partition delta affects others if we have single write. So we > wanted these writes to be independent per partition. > > Also any timeline when 0.6.0 will be released? > > Thanks, > Shayan > > > On Thu, Jul 9, 2020 at 9:22 AM Vinoth Chandar <vin...@apache.org> wrote: > > > We are looking into adding support for parallel writers in 0.6.0. So that > > should help. > > > > I am curious to understand though why you prefer to have 1000 different > > writer jobs, as opposed to having just one writer. Typical use cases for > > parallel writing I have seen are related to backfills and such. > > > > +1 to Mario’s comment. Can’t think of anything else if your users are > happy > > querying 1000 tables. > > > > On Wed, Jul 8, 2020 at 7:28 AM Mario de Sá Vera <desav...@gmail.com> > > wrote: > > > > > hey Shayan, > > > > > > that seems actually a very good approach ... just curious with the glue > > > metastore you mentioned. Would it be an external metastore for spark to > > > query over ??? external in terms of not managed by Hudi ??? > > > > > > that would be my only concern ... how to maintain the sync between all > > > metadata partitions but , again, a very promising approach ! > > > > > > regards, > > > > > > Mario. > > > > > > Em qua., 8 de jul. de 2020 às 15:20, Shayan Hati <shayanh...@gmail.com > > > > > escreveu: > > > > > > > Hi folks, > > > > > > > > We have a use-case where we want to ingest data concurrently for > > > different > > > > partitions. Currently Hudi doesn't support concurrent writes on the > > same > > > > Hudi table. > > > > > > > > One of the approaches we were thinking was to use one hudi table per > > > > partition of data. So let us say we have 1000 partitions, we will > have > > > 1000 > > > > Hudi tables which will enable us to write concurrently on each > > partition. > > > > And the metadata for each partition will be synced to a single > > metastore > > > > table (Assumption here is schema is same for all partitions). So this > > > > single metastore table can be used for all the spark, hive queries > when > > > > querying data. Basically this metastore glues all the different hudi > > > table > > > > data together in a single table. > > > > > > > > We already tested this approach and its working fine and each > partition > > > > will have its own timeline and hudi table. > > > > > > > > We wanted to know if there are some gotchas or any other issues with > > this > > > > approach to enable concurrent writes? Or if there are any other > > > approaches > > > > we can take? > > > > > > > > Thanks, > > > > Shayan > > > > > > > > > > > > -- > Shayan Hati >
