[ https://issues.apache.org/jira/browse/CASSANDRA-18424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726639#comment-17726639 ]
Jacek Lewandowski edited comment on CASSANDRA-18424 at 5/26/23 2:19 PM: ------------------------------------------------------------------------ [~jmckenzie] I haven't delved into that problem yet, but if I guess correctly, you would like something like internal paging, just like something used for aggregation, is that right? btw. I've pushed my branch on CASSANDRA-11745, you may have a look, it is different to what we have in datastax/cassandra was (Author: jlewandowski): [~jmckenzie] I haven't delved into that problem yet, but if I guess correctly, you would like something like internal paging, just like something used for aggregation, is that right? > Implement graceful paging across tombstones with short-circuit on paging > rather than throwing TombstoneOverwhelmingExceptions > ----------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-18424 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18424 > Project: Cassandra > Issue Type: New Feature > Components: Messaging/Client, Messaging/Internode > Reporter: Josh McKenzie > Assignee: Josh McKenzie > Priority: Normal > > We implemented the hard stop with a {{TombstoneOverwhelmingException}} almost > a decade ago since paging across many tombstones was the most common way for > nodes to OOM as they iterated across all this data during queries and paging. > With our current implementations and architecture / codebase, we should be > able to combine the {{StoppingTransformation}} and existing {{clustering}} > blob we pass back to clients to allow clients to optionally page across > tombstones when using the async api via the driver and short-circuit a page > when they hit the tombstone failure threshold rather than throwing a > {{{}TombstoneOverwhelmingException{}}}. This would allow for more flexible > data modeling on users' side as well as removing one of the fairly rough > edges of our API's we're currently constrained by. > Making sure this is correct will require extensive fuzz-testing of > pagination; this should likely happen in the Harry project but we could also > have a bespoke model / implementation in the C* codebase we rely on in the > interim. > Client warnings at the current default levels would remain; the gap between > warn and "short-circuit pages" (100x ratio currently, 1000 vs. 100000) should > be sufficient for clients to take action on their data models well before > they hit this limit. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org