>From Murtadha Hubail <[email protected]>: Attention is currently required from: Ritik Raj. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 )
Change subject: [ASTERIXDB-3636][STO] Introducing column buffer pool ...................................................................... Patch Set 4: (23 comments) Commit Message: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/0b798fc2_9e602e37 PS4, Line 9: - user model changes: no : - storage format changes: no : - interface changes: no update and add details File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/73f7132f_a2b2c6ae PS4, Line 333: , hence choosing ; using File asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/6a24a52a_ece3d029 PS4, Line 460: //Dereference multiPageOp move this down File asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/merge/MergeColumnTupleWriter.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/e9b617fa_ce55fb0f PS4, Line 248: onHouse remove onHouse File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/StorageProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/882ac3cf_232443f9 PS4, Line 75: STORAGE_COLUMN_BUFFER_GRANULE_SIZE STORAGE_COLUMN_BUFFER_SIZE https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/d064ad7d_cc1e1cc2 PS4, Line 76: STORAGE_COLUMN_BUFFER_POOL_CAPPED_SIZE_PERCENTAGE STORAGE_COLUMN_BUFFERS_MAX_SIZE_PERCENTAGE https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/cd2f98f0_47150ee8 PS4, Line 77: STORAGE_COLUMN_BUFFER_POOL_SIZE STORAGE_COLUMN_BUFFER_POOL_MAX_SIZE https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/06102280_9079c0d2 PS4, Line 169: case STORAGE_COLUMN_BUFFER_GRANULE_SIZE: : return "The size of the granule for the column buffer pool. This is the size of each buffer in the pool."; : case STORAGE_COLUMN_BUFFER_POOL_CAPPED_SIZE_PERCENTAGE: : return "The percentage of the total memory allocated to the column buffer pool. This is used to " : + "control the size of the column buffer pool relative to the total runtime available memory."; : case STORAGE_COLUMN_BUFFER_POOL_SIZE: : return "The count of buffers in the column buffer pool."; update accordingly File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnWriteMultiPageOp.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/99788fcf_706a8948 PS4, Line 61: confiscateTemporaryOnHouseBuffer remove OnHouse https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/e99c7339_c294f36d PS4, Line 67: releaseOnHouseBuffer remove OnHouse File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/ColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/b1233ea3_f64da92d PS4, Line 72: // Stats dumper thread : statsDumpScheduler = Executors.newSingleThreadScheduledExecutor(r -> { : Thread t = new Thread(r, "ColumnBufferPoolStatsDumper"); : t.setDaemon(true); : return t; : }); : startStatsDumpThread(); If you need those stats, make this a TimerTask https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/e57ee8a2_7600d9ec PS4, Line 155: if (totalAllocatedMemoryInBytes.get() + (long) poolSize * columnBufferGranuleInBytes > cappedMemoryInBytes) { ensure.... https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/ea997bf7_4061a927 PS4, Line 156: throw new OutOfMemoryError("Cannot allocate more buffers, memory limit reached."); log error with all details (stats). Also, tell the logs reader how to change the config to fix this https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/4945d2bd_14338c1c PS4, Line 160: // grab a new buffer Remember, a comment should tell you 'why', not 'what' https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/db21caa6_9bd7511c PS4, Line 171: if (buffer == null) { : // unreachable : return; : } if you are 100% sure, this isn't reachable, it should be illegal state exception to catch issues File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/IColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/92c9dabb_35256cdc PS4, Line 23: IColumnBufferPool Try to simplify this https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/3a78a632_2b826ab2 PS4, Line 34: reserveCredits reserve https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/3f06b40d_6b80827c PS4, Line 41: releaseCredits release https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/5971a54e_ba27f49c PS4, Line 49: reserveExtraCredits reserveExtraBuffers https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/fe4e536d_026ff056 PS4, Line 58: tryAcquiringExtraCredits tryReserveExtraBuffers https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/c91eab84_d3ffac28 PS4, Line 66: acquireBuffer get https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/3e30b32e_b64890ca PS4, Line 72: releaseBuffer recycle File hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-common-test/src/test/java/org/apache/hyracks/storage/common/ColumnBufferPoolTest.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/1140937c_ba68d876 PS4, Line 21: .* replace this -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: ionic Gerrit-Change-Id: I62437839d89b11d950e6715a00e844aedd0dab8e Gerrit-Change-Number: 20249 Gerrit-PatchSet: 4 Gerrit-Owner: Ritik Raj <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-CC: Murtadha Hubail <[email protected]> Gerrit-Attention: Ritik Raj <[email protected]> Gerrit-Comment-Date: Mon, 25 Aug 2025 13:00:03 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
