Hi Zakelly, I applied for wiki access. I will wait for a few days to get the approval. If there is a delay, I will use google doc for the next email thread. I used AI, but not to generate it. I will browse around for the AI use policy and abide by that. If there is a doc, please send it. Since I have the prototype, I could provide the details. In the next FLIP, I will be concise and expand as reviewers ask.
Looking forward, best, -Soumitra. On Wed, Apr 22, 2026 at 11:13 PM Zakelly Lan <[email protected]> wrote: > Hi Soumitra, > > Good to know. It's better to sign up for an account to access the wiki, but > google doc is also fine, up to you. And I suggest starting a separate > discussion thread for this FLIP to attract more attention, since you need 3 > binding votes to get it passed. > > I still need some time to fully read your FLIP. Also, may I ask if any of > this content was AI-generated? Could you provide a more concise overview of > the architecture and user examples, rather than focusing so heavily on > minute code details? > > > Best, > Zakelly > > On Wed, Apr 22, 2026 at 12:33 PM Soumitra Kumar <[email protected]> > wrote: > > > Hi Zakelly, > > > > Since I don't have access to the wiki, > > > > > https://docs.google.com/document/d/1bBbqBC50zIHVRd3JAR4-bSrvmlOLkDGI7A60b2VZRkQ/edit?usp=sharing > > is the FLIP as a Google doc. It is ready for your review. > > > > I am working on getting the JNI overhead and other plots you asked for. > > > > Best, > > -Soumitra. > > > > On Thu, Apr 16, 2026 at 3:33 AM Zakelly Lan <[email protected]> > wrote: > > > > > Hi Soumitra, > > > > > > It's a good improvement, In general I'm +1 on this. Several questions / > > > suggestions: > > > > > > 1. Could you please share the benchmark results in brief? I have seen > > your > > > perf's README but I could only find a throughput of 1,000 rec/s v.s. > 500 > > > rec/s comparison. I would like to know the state size and the usage of > > cpu > > > or I/O across different setups. It would be even better if we could > > > determine the overhead of the JNI. I'm also thinking that we may > provide > > > some built-in merge operators for common primitives like sum of long > > > written in C on frocksdb side, to save the JNI overhead during > > compaction. > > > > > > 2. I noticed that you are using the DataStream API for testing. So the > > > built-in windows cannot benefit from this improvement due to the > > > retractions or late messages? > > > > > > 3. I can see there are some changes to the flink's public API in your > > > branch, right? I thought that no public API changes would be required. > We > > > need to maintain API signature compatibility. If the changes are truly > > > necessary, it is required to file a FLIP under [1] and collect further > > > comments from the mailing list. Once the FLIP has been approved > through a > > > vote, we can proceed. > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > Best, > > > Zakelly > > > > > > On Thu, Apr 16, 2026 at 12:38 PM Soumitra Kumar < > > [email protected]> > > > wrote: > > > > > > > Hello Community, > > > > > > > > I wanted to share some work I have been doing on the frocksdb and > Flink > > > > that I think is useful for the Flink community. > > > > > > > > I have implemented support for Java base associative merge operators > in > > > > https://github.com/ververica/frocksdb and use that to support > > additional > > > > reducing and aggregating state variables in Flink. I have used this > do > > > > event reordering in a flink app. All the code is in my github repo ( > > > > https://github.com/soumitrak) and I will be more than happy to work > > with > > > > the members to contribute the code back to frocksdb and Flink. > > > > > > > > I have filed a followup task ( > > > > https://issues.apache.org/jira/browse/FLINK-39456) to leverage the > > > support > > > > in frocksdb to expose the state variables in the Flink. > > > > > > > > Code in my forked repos: > > > > https://github.com/soumitrak/frocksdb/commits/FRocksDB-8.10.0-SK/ - > > > > Created > > > > a branch off FRocksDB-8.10.0 and committed the changes > > > > https://github.com/soumitrak/flink/commits/rocksdb-merge-operator/ - > > > > Created a branch off master and committed the changes > > > > https://github.com/soumitrak/flink_streaming_event_reordering - Test > > > > project used to build, test, perf test, and compare the performance > of > > > > heap-based, ValueState, and new MergeState (using associative merge > > > > operator). > > > > > > > > I should have started two email threads, but they are related, so > added > > > the > > > > details in one. > > > > > > > > Looking forward to guidance on how to go about this. > > > > Best, -Soumitra. > > > > > > > > > >
