+1 on that answer. It's pretty spot on. Even as random prefix helps with HBase balancing, the issue then becomes that you lose all the key ordering inside the Hudi table, which can be a nice thing if you even want range pruning/indexing to be effective.
To paint a picture of all the work being done around this area. This work, driven by uber engineers https://github.com/apache/hudi/pull/3508 could technically solve the issue by directly reading HFiles for the indexing, avoiding going to HBase servers. But obviously, it could be less performant for small upsert batches than HBase (given the region servers will cache etc). If your backing storage is a cloud/object storage, which again throttles by prefixes etc, then we could run into the same hotspotting problem again. Otherwise, for larger batches, this would be far more scalable. On Mon, Oct 4, 2021 at 7:06 PM 管梓越 <guanziyue....@bytedance.com> wrote: > Hi jianfeng > As far as I know, there may not be a solution in hudi side yet. > However, I have met this problem before so hope my experience could help. > Just like other usages of hbase, adding a random prefix to rowkey may be > the most universal solution to this problem. > We may change the primary key for hudi by adding such prefix before the > data is ingested into hudi. A new column could be added to save original > primary key for query and hide the pk of hudi. > Also, we may have a small modification to hbase index. Copy the code of > hbase index, add the prefix on the aspect of query and update hbase. By > this way, the pk in hbase will be different with the one in hudi but such > logic will be transparent to business logic. I have adopted this method in > prod environment. Using withIndexClass config in IndexConfig could specify > custom index which allows the change of index without re compilation of the > whole hudi project. > > On Mon, Oct 4, 2021, 11:29 PM <jian.f...@shopee.com> wrote: > when I bootstrape a huge hbase index table, I found all keys have a prefix > 'itemid:', then it caused data skew, there are 100 region servers in hbase > but only one was handle datas Is there any way to avoid this issue on the > Hudi side ? -- *Jian Feng,冯健* Shopee | Engineer | Data Infrastructure >
