I think I can finish the FATE refactor PR [1] for 2.1. I had been keeping it up to date with the latest in main but stopped because it was too much work. I was waiting until the ZK property changes are completed before resolving the latest conflicts. I don't think it is much of a risk. It is mostly cleanup and refactoring to remove generics from the serialization code. It will be some work to revisit but I think the risk is pretty low. It would allow changing the serialization, which we may be able to get into 2.1 as well.
[1] https://github.com/apache/accumulo/pull/2475 On Mon, Apr 4, 2022 at 11:50 AM Keith Turner <ke...@deenlo.com> wrote: > On Mon, Apr 4, 2022 at 11:17 AM Christopher <ctubb...@apache.org> wrote: > > > > I haven't seen the metrics test fail very often lately. If it's stable, I > > don't mind removing the blocker on that issue, but I'd be reluctant to > > close it entirely just yet, until we can verify it doesn't happen > anymore. > > > > As for the original list of potential issues to include, I'm in favor of > > trying to get #2197 in. It was started awhile ago, is relatively simple > and > > well understood by several of us already... it just needs a bit of > > attention to finalize reviews so it can be merged. > > > > However, I'm reluctant to include #2422, because I don't think it's near > > ready enough, and by the time it is, it will be very last minute, and I > > don't want to delay 2.1 further for it. Even if it's included as an > > experimental feature, I think it has huge potential to be disruptive, or > to > > have a lot of churn by the time people actually have a chance to review > it > > thoroughly. Furthermore, I think there are possible alternatives (like a > > fully client-side implementation, based on offline scanners) that would > > avoid the tight coupling of a new service to Accumulo's core code. This > > There are some advantages to scan servers over direct file access to > consider. One is scalability of computation, if a web server is > serving N client queries with scan servers those can potentially go to > different scan servers. With direct file access, all N queries and > their iterator stacks would have to run in the web server. Another is > scalability of caching/memory. When web servers send queries to scan > servers using a sticky algorithm for assigning tablets to groups of > scan servers, it could lead to good cache utilization and sharing that > may not be possible when running scans directly in the web server. So > scan servers allow scaling cache and computations for queries > independently of web servers in way that may not be possible with > direct file access. > > Another advantage to consider is isolation. With direct file access > and queries running directly in a web server, a bad query could bring > down a web server and lots of unrelated queries. Having a bad query > bring down a scan server may be less disruptive. > > > thread isn't for discussing this in depth, so we can have that discussion > > in a separate thread, but I'm generally opposed to including it this late > > in 2.1's development, given the timing, size and scope, tight coupling, > and > > current state. > > > > I don't know enough about #2475 to have a strong opinion, but it looks > big, > > and possibly high-risk, given the critical code it touches. It currently > > has a substantial number of conflicts with the main branch. However, I > was > > thinking that *some* minimal refactoring (like low-risk automatic > > refactoring, like moving packages) could be done. So, if that's all this > > does, it might be okay. Otherwise, maybe it can be simplified? At the > very > > least, I was thinking it would be a good opportunity to move the > > `org.apache.accumulo.fate` packages into an appropriate > > `org.apache.accumulo.core` parent package (some would go to > o.a.a.core.fate > > and others might go to o.a.a.core.util or similar) to keep the package > > namespaces standardized, which is helpful to avoid naming collisions and > > jar sealing issues, as well as for less complicated jigsaw module > > definitions in future. Since 2.1 FaTE is already incompatible with prior > > versions, a rename at this time would be less disruptive. > > > > Another task I had wanted to be done for 2.1, before I got distracted > > fixing test failures during and after Christmas and trying to work > through > > the singleton manager zookeeper stuff to see what we could simplify. > What I > > had wanted done was to standardize the way we pass table identifiers > (name, > > IDs) across the RPC layer, since we currently do that inconsistently. I > > don't remember if there's an existing ticket open for it, but I have a > > working branch I had started working out of for it before Christmas. It's > > relatively simple work, and would set us up for some much better APIs > going > > forward, as well as help with logging information about table actions. If > > necessary, it could be bumped to a future version, but then we'd have > more > > churn in the thrift layer. So, I'd prefer to get it for 2.1 to avoid > that. > > > > As for planning, I was thinking early May for a code freeze (except bug > > fixes and small improvements found during testing), so we can try to > > release towards the end of May/early June. If we go with that timeline, > > that's not a lot of time to wrap up features and have time for > > review/testing, so we may need to be selective about what we hold off > until > > the next version, unless we want to further delay 2.1. > > > > > > On Mon, Apr 4, 2022 at 9:13 AM Dave Marion <dmario...@gmail.com> wrote: > > > > > I think [3] is OBE and can be closed. > > > > > > On Mon, Apr 4, 2022 at 9:11 AM Mike Miller <mmil...@apache.org> wrote: > > > > > > > Yes I agree, that was the goal of this email thread. I found a few > more > > > > tickets that should be addressed for the next release. > > > > > > > > Ivan - There was some work done on this PR but it has been some > time. Do > > > > you want to take a look at it? Implement a Thread limit. [1] > > > > Keith T - I think we should get this one merged to fix that > consistency > > > > check bug I found. It looks like it is finished. [2] > > > > Dave & Dom - Were you guys able to figure out a fix for the new > external > > > > compaction metrics test? [3] > > > > > > > > FYI we have 6 blockers for 2.1: > > > > https://github.com/apache/accumulo/labels/blocker > > > > > > > > This is almost definitely going into 2.1 [4]. Thanks Jeff! > > > > > > > > [1] https://github.com/apache/accumulo/pull/1487 > > > > [2] https://github.com/apache/accumulo/pull/2574 > > > > [3] https://github.com/apache/accumulo/issues/2406 > > > > [4] https://github.com/apache/accumulo/pull/2215 > > > > > > > > On Fri, Apr 1, 2022 at 2:21 PM Dave Marion <dmario...@gmail.com> > wrote: > > > > > > > > > I think it would be useful to do some release planning so that we > know > > > > what > > > > > features we are working towards and in which release they will be > in. > > > > This > > > > > would be helpful for determining what existing PRs need to make it > into > > > > > 2.1.0. 2.1.0 is the LTM release, so patches for existing features > will > > > be > > > > > backported (2.1.1, 2.1.2, 2.1.3, etc.) However, as defined in [1], > > > > features > > > > > that don't make it into 2.1.0 will go into the next non-LTM release > > > > (2.2.0) > > > > > and any patches to bugs in those features will go into the next > non-LTM > > > > > release after that (2.3.0). > > > > > > > > > > I'm not trying to hold up the 2.1.0 release by suggesting that we > > > perform > > > > > this activity. I'm just asking what the future holds, even if it's > just > > > > one > > > > > feature in the next non-LTM release. My concern is that the next > > > release > > > > > will be open-ended and anything not included in 2.1.0 might not > get put > > > > > into a release for a very long time. > > > > > > > > > > [1] https://accumulo.apache.org/contributor/versioning.html#LTM > > > > > > > > > > > > > > > On Thu, Mar 31, 2022 at 11:43 AM Mike Miller <mmil...@apache.org> > > > wrote: > > > > > > > > > > > Starting an email chain of things that folks want to finish for > 2.1. > > > > Here > > > > > > is what we currently have in the works that are most likely going > > > into > > > > > 2.1: > > > > > > https://github.com/apache/accumulo/pull/2569 > > > > > > https://github.com/apache/accumulo/pull/2600 > > > > > > https://github.com/apache/accumulo/pull/2293 > > > > > > > > > > > > Some things that may go into 2.1: > > > > > > https://github.com/apache/accumulo/pull/2422 > > > > > > https://github.com/apache/accumulo/pull/2475 > > > > > > https://github.com/apache/accumulo/pull/2197 > > > > > > > > > > > > I created a Project for follow on work to the ZK property > change. I > > > was > > > > > > planning on putting tasks in there that we want to complete for > 2.1. > > > > But > > > > > we > > > > > > could also use it for post 2.1 work. > > > > > > https://github.com/apache/accumulo/projects/24 > > > > > > https://github.com/apache/accumulo/issues/2469 > > > > > > > > > > > > FYI a draft copy of the release notes has already been on the > > > website: > > > > > > https://accumulo.apache.org/release/accumulo-2.1.0/ > > > > > > > > > > > > This may be a good thread to discuss whether or not a task needs > to > > > go > > > > > into > > > > > > 2.1 or should wait for the next version. We currently have 32 > open > > > pull > > > > > > requests so please email me if there is one that you would like > > > > > prioritized > > > > > > for 2.1. > > > > > > > > > > > > > > > > > > >
