Hi Yuan Thanks for the proposal. The 512 MiB target is reasonable, but I’m concerned that the 1 MiB sort/modification buffer may be too restrictive in practice. Even moderate ORDER BY queries could spill to disk frequently, hurting usability.
Could we consider making this buffer tunable per session or allow it to grow automatically when memory is available? This would keep the default safe while avoiding frustration for users who occasionally need larger sorts. Otherwise, the JVM options and package naming look good. I support the initiative with this adjustment. Best regards, Xuan Wang Yuan Tian <[email protected]> 于2026年8月27日周四 09:15写道： > Hi Chris, > > Thanks for offering to join. I agree that maintaining this in the > upstream project would be much better than keeping a separate fork. > > My current rough plan is to proceed in three steps: > > 1. First, adjust only the JVM options and the default configuration > parameters. We will run a series of tests under a hard 512 MiB > combined memory limit for the ConfigNode and DataNode, in order to > determine the best configuration for this target and characterize the > read and write workloads it can support. > > 2. Next, try to run the DataNode and ConfigNode in a single JVM > process. Since the ConfigNode module already depends on the DataNode > module, this may be relatively straightforward. For example, the > ConfigNode main entry point could also invoke the DataNode startup > logic. At this stage, we would still keep the existing RPC > communication path between the ConfigNode and DataNode to minimize > code changes. > > 3. Finally, abstract the communication layer behind an interface, with > one RPC-based implementation and one in-VM function-call > implementation. To keep the code simple and maintainable, the > interface may continue to use many of the existing Thrift request and > response types. Therefore, the in-VM implementation may still depend > on the generated Thrift classes for parameter definitions, while > bypassing the network call itself. > > The first two steps are mainly configuration changes and should > require relatively little code modification. I plan to submit PRs for > them, and it would be great if you could help review them or share > parameter recommendations based on your experience. > > For the third step, I would be very happy to work on it together. > Could you share the current status of your fork, the repository link, > and the approach you have taken so far? We can study the design > together and decide how to move it upstream. > > Best regards, > Yuan >
