Thanks, Haibo I can help change the release notes accordingly. Also, the issue is not about backward incompatibility. The format change is backward compatible. Let me explain
I had done two format changes for raw index: https://github.com/apache/incubator-pinot/pull/5256 To handle very large STRING/BYTES column values, derive number of docs per chunk using metadata. Before this change, the docs per chunk was hardcoded to 1000. https://github.com/apache/incubator-pinot/pull/5285 Follow up to previous one -- if the raw index is greater than 2GB, the offset of each chunk has to be 8-byte (LONG) in the file header instead of 4-byte (INT) Both the above changes were backward compatible. So the new code can read old segments (both with 1000 docs per chunk and 4-byte chunk offsets). The first change caused performance issues for high QPS use cases since based on the access pattern, you are going to uncompress a bigger chunk to fish out the rows from forward index. For example, if we now pack 5000 rows per chunk, and the docIds to select are 0, 5, 1000, 1500, 2000. In the old code, we will uncompress 2 chunks of 1000 rows each. In the new code, we will uncompress a single chunk of 5000 rows. Due to this, the first change is made configurable. So the release notes have to changed. Thanks, Sidd On Mon, Jun 1, 2020 at 9:02 AM H <[email protected]> wrote: > Please ignore this version. > > A backward incompatible issue was reported. I will cherry-pick 5470 > <https://github.com/apache/incubator-pinot/pull/5470> and 5472 > <https://github.com/apache/incubator-pinot/pull/5472> and send a new > release candidate. Please let me know if there are other changes that need > to be included. > > Thanks > > On Mon, Jun 1, 2020 at 8:39 AM H <[email protected]> wrote: > > > Hi Pinot Community, > > > > This is a call for vote to the release Apache Pinot (incubating) version > > 0.4.0. > > > > The release candidate: > > > > > https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc1 > > > > Git tag for this release: > > https://github.com/apache/incubator-pinot/tree/release-0.4.0-rc1 > > > > Git hash for this release: > > fcfd016ae9c4f922c31b46bd8dc3fd624e1a297a > > > > The artifacts have been signed with key: 6CC169A6FC19C470, which can be > > found in the following KEYS file. > > https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS > > > > Release notes: > > GitHub: > > https://github.com/apache/incubator-pinot/releases/tag/release-0.4.0-rc1 > > > > Staging repository: > > https://repository.apache.org/content/repositories/orgapachepinot-1012 > > > > Documentation on verifying a release candidate: > > > > > https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate > > > > > > The vote will be open for at least 72 hours or until necessary number of > > votes are reached. > > > > Please vote accordingly, > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Thanks, > > Apache Pinot (incubating) team > > > -- *Best Regards,* *SIDDHARTH TEOTIA* *2008C6PS540G* *BITS PILANI- GOA CAMPUS* *+91 87911 75932*
