>From Ritik Raj <[email protected]>:
Attention is currently required from: Ian Maxon, Murtadha Hubail, Ritik Raj.
Hello Anon. E. Moose #1000171, Jenkins, Murtadha Hubail,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21372?usp=email
to look at the new patch set (#4).
The following approvals got outdated and were removed:
Code-Review+1 by Ritik Raj, Integration-Tests+1 by Jenkins, Verified+1 by
Jenkins, Verified-1 by Anon. E. Moose #1000171
Change subject: [ASTERIXDB-3788][STO] flush .dic LAF pages in ascending page-id
order
......................................................................
[ASTERIXDB-3788][STO] flush .dic LAF pages in ascending page-id order
On cloud-backed deployments the columnar .dic (Look-Aside File) is
written through an append-only cloud writer whose ensurePosition
invariant requires pages to arrive in strictly ascending offset
(page-id) order. LAFWriter did not honor that:
- writeFullPage() flushed a LAF frame as soon as it filled, i.e. in
completion order, not page-id order; and
- endWriting() flushed the remaining frames in HashMap iteration
order.
During a large MERGE the bulkloader writes a leaf's column pages first
and its page-zero last, and multi-page column values add extra-page
entries that straddle LAF-page boundaries. So a higher LAF page could
fill and flush while a lower one was still incomplete. Harmless on the
random-access local file, but fatal on the append-only cloud copy:
AbstractCloudIOManager.ensurePosition throws "Misaligned positions" and
the merge thread halts the JVM, which on the affected collection turned
into a crash/rollback/re-backfill loop.
Fix: hold a LAF frame back until every lower-id page has been flushed.
A nextPageToFlush cursor releases only the contiguous ascending prefix
of completed frames; endWriting flushes the remainder in ascending
page-id order. The local path is random-access and unaffected; only the
append-only cloud writer needs the ordering guarantee, and
ensurePosition stays intact as a corruption guard.
Adds a dedicated regression test, CloudStorageMergeTest, that runs only
cloud_storage/dic-merge-misalign: a COLUMN dataset with incompressible
columns inserted in batches, then COMPACT, forces a merged component
whose .dic spans multiple LAF pages. Without this change it halts the
JVM during COMPACT (mock S3, append-only cloud writer); with it the
merge completes and the count is correct.
To surface the bug with little data, the test uses its own config
(cc-cloud-storage-merge.conf.ftl: buffercache.pagesize=2KB,
memorycomponent.globalbudget=8MB, column.max.tuple.count=200) so a LAF
page holds few entries and inserts flush into many small components.
Kept as a separate test class + config so these aggressive storage
knobs do not perturb the shared CloudStorageTest suite.
ext-ref: MB-72604
Change-Id: I2a8a83cd1b57d3ceb1d9be381edd6589a0e138b9
---
A
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/cloud_storage/CloudStorageMergeTest.java
A asterixdb/asterix-app/src/test/resources/cc-cloud-storage-merge.conf.ftl
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.000.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.001.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.002.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.003.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.004.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.005.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.999.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/results/cloud_storage/dic-merge-misalign/mis.005.adm
A
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage_merge.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage_merge_only.xml
M
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
13 files changed, 539 insertions(+), 14 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/72/21372/4
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21372?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: lumina
Gerrit-Change-Id: I2a8a83cd1b57d3ceb1d9be381edd6589a0e138b9
Gerrit-Change-Number: 21372
Gerrit-PatchSet: 4
Gerrit-Owner: Ritik Raj <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Reviewer: Murtadha Hubail <[email protected]>
Gerrit-Reviewer: Ritik Raj <[email protected]>
Gerrit-CC: Ian Maxon <[email protected]>
Gerrit-Attention: Murtadha Hubail <[email protected]>
Gerrit-Attention: Ian Maxon <[email protected]>
Gerrit-Attention: Ritik Raj <[email protected]>
