Hi Zakelly, I now have access, thank you!
I have filed two tickets, and they are dependent: https://issues.apache.org/jira/browse/FLINK-39455 - enhancement to the FrocksDB https://issues.apache.org/jira/browse/FLINK-39456 - leverage the enhancement in the FrocksDB to add this feature in Flink I am starting to prepare the FLIP in wiki for "FLINK-39455 - enhancement to the FrocksDB". I would wait for approval of this one, then create the FLIP for FLINK-39456, what do you suggest? Best, -Soumitra. On Sat, Apr 25, 2026 at 8:43 PM Zakelly Lan <[email protected]> wrote: > Hi Soumitra, > > We welcome AI, however there are currently no guidelines for using AI in > FLIP or design. I think AIs often introduce unnecessary code details or > repeated words, which makes the article lose its focus, leaving many people > confused and reluctant to participate in the discussion. We need to > minimize the reader's cognitive overhead. I suggest not diving into > excessive detail, as many code details evolve flexibly during development. > It would be best if we could clearly define the flink's public API and the > use patterns, as well as present the overall technical architecture and > benchmark results. > > I have processed your application and you might have access to the wiki > now. Please check. > > > Best, > Zakelly > > On Thu, Apr 23, 2026 at 7:41 PM Soumitra Kumar <[email protected]> > wrote: > > > Hi Zakelly, > > > > I applied for wiki access. I will wait for a few days to get the > approval. > > If there is a delay, I will use google doc for the next email thread. > > I used AI, but not to generate it. I will browse around for the AI use > > policy and abide by that. If there is a doc, please send it. Since I have > > the prototype, I could provide the details. In the next FLIP, I will be > > concise and expand as reviewers ask. > > > > Looking forward, best, > > -Soumitra. > > > > On Wed, Apr 22, 2026 at 11:13 PM Zakelly Lan <[email protected]> > > wrote: > > > > > Hi Soumitra, > > > > > > Good to know. It's better to sign up for an account to access the wiki, > > but > > > google doc is also fine, up to you. And I suggest starting a separate > > > discussion thread for this FLIP to attract more attention, since you > > need 3 > > > binding votes to get it passed. > > > > > > I still need some time to fully read your FLIP. Also, may I ask if any > of > > > this content was AI-generated? Could you provide a more concise > overview > > of > > > the architecture and user examples, rather than focusing so heavily on > > > minute code details? > > > > > > > > > Best, > > > Zakelly > > > > > > On Wed, Apr 22, 2026 at 12:33 PM Soumitra Kumar < > > [email protected]> > > > wrote: > > > > > > > Hi Zakelly, > > > > > > > > Since I don't have access to the wiki, > > > > > > > > > > > > > > https://docs.google.com/document/d/1bBbqBC50zIHVRd3JAR4-bSrvmlOLkDGI7A60b2VZRkQ/edit?usp=sharing > > > > is the FLIP as a Google doc. It is ready for your review. > > > > > > > > I am working on getting the JNI overhead and other plots you asked > for. > > > > > > > > Best, > > > > -Soumitra. > > > > > > > > On Thu, Apr 16, 2026 at 3:33 AM Zakelly Lan <[email protected]> > > > wrote: > > > > > > > > > Hi Soumitra, > > > > > > > > > > It's a good improvement, In general I'm +1 on this. Several > > questions / > > > > > suggestions: > > > > > > > > > > 1. Could you please share the benchmark results in brief? I have > seen > > > > your > > > > > perf's README but I could only find a throughput of 1,000 rec/s > v.s. > > > 500 > > > > > rec/s comparison. I would like to know the state size and the usage > > of > > > > cpu > > > > > or I/O across different setups. It would be even better if we could > > > > > determine the overhead of the JNI. I'm also thinking that we may > > > provide > > > > > some built-in merge operators for common primitives like sum of > long > > > > > written in C on frocksdb side, to save the JNI overhead during > > > > compaction. > > > > > > > > > > 2. I noticed that you are using the DataStream API for testing. So > > the > > > > > built-in windows cannot benefit from this improvement due to the > > > > > retractions or late messages? > > > > > > > > > > 3. I can see there are some changes to the flink's public API in > your > > > > > branch, right? I thought that no public API changes would be > > required. > > > We > > > > > need to maintain API signature compatibility. If the changes are > > truly > > > > > necessary, it is required to file a FLIP under [1] and collect > > further > > > > > comments from the mailing list. Once the FLIP has been approved > > > through a > > > > > vote, we can proceed. > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > > > > Best, > > > > > Zakelly > > > > > > > > > > On Thu, Apr 16, 2026 at 12:38 PM Soumitra Kumar < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hello Community, > > > > > > > > > > > > I wanted to share some work I have been doing on the frocksdb and > > > Flink > > > > > > that I think is useful for the Flink community. > > > > > > > > > > > > I have implemented support for Java base associative merge > > operators > > > in > > > > > > https://github.com/ververica/frocksdb and use that to support > > > > additional > > > > > > reducing and aggregating state variables in Flink. I have used > this > > > do > > > > > > event reordering in a flink app. All the code is in my github > repo > > ( > > > > > > https://github.com/soumitrak) and I will be more than happy to > > work > > > > with > > > > > > the members to contribute the code back to frocksdb and Flink. > > > > > > > > > > > > I have filed a followup task ( > > > > > > https://issues.apache.org/jira/browse/FLINK-39456) to leverage > the > > > > > support > > > > > > in frocksdb to expose the state variables in the Flink. > > > > > > > > > > > > Code in my forked repos: > > > > > > > https://github.com/soumitrak/frocksdb/commits/FRocksDB-8.10.0-SK/ > > - > > > > > > Created > > > > > > a branch off FRocksDB-8.10.0 and committed the changes > > > > > > > https://github.com/soumitrak/flink/commits/rocksdb-merge-operator/ > > - > > > > > > Created a branch off master and committed the changes > > > > > > https://github.com/soumitrak/flink_streaming_event_reordering - > > Test > > > > > > project used to build, test, perf test, and compare the > performance > > > of > > > > > > heap-based, ValueState, and new MergeState (using associative > merge > > > > > > operator). > > > > > > > > > > > > I should have started two email threads, but they are related, so > > > added > > > > > the > > > > > > details in one. > > > > > > > > > > > > Looking forward to guidance on how to go about this. > > > > > > Best, -Soumitra. > > > > > > > > > > > > > > > > > > > > >
