Yes, it is somewhat of an unorthodox approach and makes sense if we want to make this flag configuration absolutely noticeable to users during an upgrade.
Abhishek's idea of pushing an alert to WebUI/ksck seems reasonable. Certainly, more noticeable to users than other approaches. On a side note, we could also think of enabling memory budgeting logic (1556a353e6) driven by flags (i.e. rowset_compaction_memory_estimate_enabled, rowset_compaction_delta_memory_factor, etc). This may require some additional tuning based on the cluster node configuration. Also, enabling this flag means rowset compaction will not happen for rowsets that have memory requirements going beyond a certain limit thereby avoiding OOM condition but potentially hitting scan performance. Regards, Ashwani On Thu, Feb 13, 2025 at 3:20 PM Attila Bukor <[email protected]> wrote: > > I am thinking if it is possible to have this flag as a mandatory one. > > I guess we could do this, but I'm not a fan of this approach. We should > try to > limit the required flags to those that can't have a reasonable default > like the > paths and addresses. > > > Alternatively, we could think of reducing the value to 24 hours since > that > > seems to be a general backup schedule. > > This would still be a breaking change, even if it would break fewer > use-cases. > > > Make the change now which could break existing workflows of backups OR > Do not > > make the change now leading to more and more users experiencing servers > > crashing due to OOM (apart from performance overhead) under certain > workloads > > Exactly. > > > perhaps we could have some message displayed in WebUI/logs/ksck by > monitoring > > diff scans. > > I like this approach. > > > We could take a step ahead and since this is marked as a runtime flag we > might > > as well lower it down to 15 minutes(or the longest time period seen in > those > > past 7 days) and log the configuration change. > > This is interesting, and probably a good idea to think about it, but I'm > not > convinced yet. > > I can think of some edge cases that this would break. For example, if > there is > an application that runs once a month, but requires historical data going > back a > day, this would break it. > > -- Attila > >
