jerqi opened a new issue, #134:
URL: https://github.com/apache/incubator-uniffle/issues/134
The current blockId is designed as following:
```
// BlockId is long and composed by partitionId, executorId and AtomicInteger
// AtomicInteger is first 19 bit, max value is 2^19 - 1
// partitionId is next 24 bit, max value is 2^24 - 1
// taskAttemptId is rest of 20 bit, max value is 2^20 - 1
```
**Why we need blockId?**
It's designed for data check, filter, memory data read, etc.
**Why blockId is designed as above?**
BlockId will be stored in Shuffle server, to reduce memory cost.
Roaringbitmap is used to cache it.
According to implementation of Roaringbitmap, the design of BlockId is
target to use `BitmapContainer` instead of `ArrayContainer` for memory saving.
**What's the problem of blockId?**
It can't support taskId which is greater than 2^20 - 1
**Proposal**
I think the first 19 bit is too much for atomic int, and we can leverage
some of them for taskId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
