Hi Zakelly, I am almost ready with the document, but then I noticed that I don't have write access to the wiki. Shall you grant me write access? I can start with a Google doc as well.
Best, -Soumitra. On Mon, Apr 27, 2026 at 10:34 PM Soumitra Kumar <[email protected]> wrote: > Hi Zakelly, > > That's a great idea, let me create one FLIP and send it out in a few days. > > Best, > -Soumitra. > > On Sun, Apr 26, 2026, 11:49 PM Zakelly Lan <[email protected]> wrote: > >> Hi Soumitra, >> >> I think there is only one story from Flink's perspective. So maybe you can >> prepare one single FLIP for all those? >> >> >> Best, >> Zakelly >> >> On Sun, Apr 26, 2026 at 10:34 PM Soumitra Kumar <[email protected] >> > >> wrote: >> >> > Hi Zakelly, >> > >> > I now have access, thank you! >> > >> > I have filed two tickets, and they are dependent: >> > https://issues.apache.org/jira/browse/FLINK-39455 - enhancement to the >> > FrocksDB >> > https://issues.apache.org/jira/browse/FLINK-39456 - leverage the >> > enhancement in the FrocksDB to add this feature in Flink >> > >> > I am starting to prepare the FLIP in wiki for "FLINK-39455 - >> enhancement to >> > the FrocksDB". I would wait for approval of this one, then create the >> FLIP >> > for FLINK-39456, what do you suggest? >> > >> > Best, >> > -Soumitra. >> > >> > >> > On Sat, Apr 25, 2026 at 8:43 PM Zakelly Lan <[email protected]> >> wrote: >> > >> > > Hi Soumitra, >> > > >> > > We welcome AI, however there are currently no guidelines for using AI >> in >> > > FLIP or design. I think AIs often introduce unnecessary code details >> or >> > > repeated words, which makes the article lose its focus, leaving many >> > people >> > > confused and reluctant to participate in the discussion. We need to >> > > minimize the reader's cognitive overhead. I suggest not diving into >> > > excessive detail, as many code details evolve flexibly during >> > development. >> > > It would be best if we could clearly define the flink's public API and >> > the >> > > use patterns, as well as present the overall technical architecture >> and >> > > benchmark results. >> > > >> > > I have processed your application and you might have access to the >> wiki >> > > now. Please check. >> > > >> > > >> > > Best, >> > > Zakelly >> > > >> > > On Thu, Apr 23, 2026 at 7:41 PM Soumitra Kumar < >> [email protected] >> > > >> > > wrote: >> > > >> > > > Hi Zakelly, >> > > > >> > > > I applied for wiki access. I will wait for a few days to get the >> > > approval. >> > > > If there is a delay, I will use google doc for the next email >> thread. >> > > > I used AI, but not to generate it. I will browse around for the AI >> use >> > > > policy and abide by that. If there is a doc, please send it. Since I >> > have >> > > > the prototype, I could provide the details. In the next FLIP, I >> will be >> > > > concise and expand as reviewers ask. >> > > > >> > > > Looking forward, best, >> > > > -Soumitra. >> > > > >> > > > On Wed, Apr 22, 2026 at 11:13 PM Zakelly Lan <[email protected] >> > >> > > > wrote: >> > > > >> > > > > Hi Soumitra, >> > > > > >> > > > > Good to know. It's better to sign up for an account to access the >> > wiki, >> > > > but >> > > > > google doc is also fine, up to you. And I suggest starting a >> separate >> > > > > discussion thread for this FLIP to attract more attention, since >> you >> > > > need 3 >> > > > > binding votes to get it passed. >> > > > > >> > > > > I still need some time to fully read your FLIP. Also, may I ask if >> > any >> > > of >> > > > > this content was AI-generated? Could you provide a more concise >> > > overview >> > > > of >> > > > > the architecture and user examples, rather than focusing so >> heavily >> > on >> > > > > minute code details? >> > > > > >> > > > > >> > > > > Best, >> > > > > Zakelly >> > > > > >> > > > > On Wed, Apr 22, 2026 at 12:33 PM Soumitra Kumar < >> > > > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Hi Zakelly, >> > > > > > >> > > > > > Since I don't have access to the wiki, >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1bBbqBC50zIHVRd3JAR4-bSrvmlOLkDGI7A60b2VZRkQ/edit?usp=sharing >> > > > > > is the FLIP as a Google doc. It is ready for your review. >> > > > > > >> > > > > > I am working on getting the JNI overhead and other plots you >> asked >> > > for. >> > > > > > >> > > > > > Best, >> > > > > > -Soumitra. >> > > > > > >> > > > > > On Thu, Apr 16, 2026 at 3:33 AM Zakelly Lan < >> [email protected] >> > > >> > > > > wrote: >> > > > > > >> > > > > > > Hi Soumitra, >> > > > > > > >> > > > > > > It's a good improvement, In general I'm +1 on this. Several >> > > > questions / >> > > > > > > suggestions: >> > > > > > > >> > > > > > > 1. Could you please share the benchmark results in brief? I >> have >> > > seen >> > > > > > your >> > > > > > > perf's README but I could only find a throughput of 1,000 >> rec/s >> > > v.s. >> > > > > 500 >> > > > > > > rec/s comparison. I would like to know the state size and the >> > usage >> > > > of >> > > > > > cpu >> > > > > > > or I/O across different setups. It would be even better if we >> > could >> > > > > > > determine the overhead of the JNI. I'm also thinking that we >> may >> > > > > provide >> > > > > > > some built-in merge operators for common primitives like sum >> of >> > > long >> > > > > > > written in C on frocksdb side, to save the JNI overhead during >> > > > > > compaction. >> > > > > > > >> > > > > > > 2. I noticed that you are using the DataStream API for >> testing. >> > So >> > > > the >> > > > > > > built-in windows cannot benefit from this improvement due to >> the >> > > > > > > retractions or late messages? >> > > > > > > >> > > > > > > 3. I can see there are some changes to the flink's public API >> in >> > > your >> > > > > > > branch, right? I thought that no public API changes would be >> > > > required. >> > > > > We >> > > > > > > need to maintain API signature compatibility. If the changes >> are >> > > > truly >> > > > > > > necessary, it is required to file a FLIP under [1] and collect >> > > > further >> > > > > > > comments from the mailing list. Once the FLIP has been >> approved >> > > > > through a >> > > > > > > vote, we can proceed. >> > > > > > > >> > > > > > > >> > > > > > > [1] >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals >> > > > > > > >> > > > > > > Best, >> > > > > > > Zakelly >> > > > > > > >> > > > > > > On Thu, Apr 16, 2026 at 12:38 PM Soumitra Kumar < >> > > > > > [email protected]> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hello Community, >> > > > > > > > >> > > > > > > > I wanted to share some work I have been doing on the >> frocksdb >> > and >> > > > > Flink >> > > > > > > > that I think is useful for the Flink community. >> > > > > > > > >> > > > > > > > I have implemented support for Java base associative merge >> > > > operators >> > > > > in >> > > > > > > > https://github.com/ververica/frocksdb and use that to >> support >> > > > > > additional >> > > > > > > > reducing and aggregating state variables in Flink. I have >> used >> > > this >> > > > > do >> > > > > > > > event reordering in a flink app. All the code is in my >> github >> > > repo >> > > > ( >> > > > > > > > https://github.com/soumitrak) and I will be more than >> happy to >> > > > work >> > > > > > with >> > > > > > > > the members to contribute the code back to frocksdb and >> Flink. >> > > > > > > > >> > > > > > > > I have filed a followup task ( >> > > > > > > > https://issues.apache.org/jira/browse/FLINK-39456) to >> leverage >> > > the >> > > > > > > support >> > > > > > > > in frocksdb to expose the state variables in the Flink. >> > > > > > > > >> > > > > > > > Code in my forked repos: >> > > > > > > > >> > > https://github.com/soumitrak/frocksdb/commits/FRocksDB-8.10.0-SK/ >> > > > - >> > > > > > > > Created >> > > > > > > > a branch off FRocksDB-8.10.0 and committed the changes >> > > > > > > > >> > > https://github.com/soumitrak/flink/commits/rocksdb-merge-operator/ >> > > > - >> > > > > > > > Created a branch off master and committed the changes >> > > > > > > > >> https://github.com/soumitrak/flink_streaming_event_reordering >> > - >> > > > Test >> > > > > > > > project used to build, test, perf test, and compare the >> > > performance >> > > > > of >> > > > > > > > heap-based, ValueState, and new MergeState (using >> associative >> > > merge >> > > > > > > > operator). >> > > > > > > > >> > > > > > > > I should have started two email threads, but they are >> related, >> > so >> > > > > added >> > > > > > > the >> > > > > > > > details in one. >> > > > > > > > >> > > > > > > > Looking forward to guidance on how to go about this. >> > > > > > > > Best, -Soumitra. >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
