On Mon, Apr 4, 2022 at 11:17 AM Christopher <ctubb...@apache.org> wrote: > > I haven't seen the metrics test fail very often lately. If it's stable, I > don't mind removing the blocker on that issue, but I'd be reluctant to > close it entirely just yet, until we can verify it doesn't happen anymore. > > As for the original list of potential issues to include, I'm in favor of > trying to get #2197 in. It was started awhile ago, is relatively simple and > well understood by several of us already... it just needs a bit of > attention to finalize reviews so it can be merged. > > However, I'm reluctant to include #2422, because I don't think it's near > ready enough, and by the time it is, it will be very last minute, and I > don't want to delay 2.1 further for it. Even if it's included as an > experimental feature, I think it has huge potential to be disruptive, or to > have a lot of churn by the time people actually have a chance to review it > thoroughly. Furthermore, I think there are possible alternatives (like a > fully client-side implementation, based on offline scanners) that would > avoid the tight coupling of a new service to Accumulo's core code. This
There are some advantages to scan servers over direct file access to consider. One is scalability of computation, if a web server is serving N client queries with scan servers those can potentially go to different scan servers. With direct file access, all N queries and their iterator stacks would have to run in the web server. Another is scalability of caching/memory. When web servers send queries to scan servers using a sticky algorithm for assigning tablets to groups of scan servers, it could lead to good cache utilization and sharing that may not be possible when running scans directly in the web server. So scan servers allow scaling cache and computations for queries independently of web servers in way that may not be possible with direct file access. Another advantage to consider is isolation. With direct file access and queries running directly in a web server, a bad query could bring down a web server and lots of unrelated queries. Having a bad query bring down a scan server may be less disruptive. > thread isn't for discussing this in depth, so we can have that discussion > in a separate thread, but I'm generally opposed to including it this late > in 2.1's development, given the timing, size and scope, tight coupling, and > current state. > > I don't know enough about #2475 to have a strong opinion, but it looks big, > and possibly high-risk, given the critical code it touches. It currently > has a substantial number of conflicts with the main branch. However, I was > thinking that *some* minimal refactoring (like low-risk automatic > refactoring, like moving packages) could be done. So, if that's all this > does, it might be okay. Otherwise, maybe it can be simplified? At the very > least, I was thinking it would be a good opportunity to move the > `org.apache.accumulo.fate` packages into an appropriate > `org.apache.accumulo.core` parent package (some would go to o.a.a.core.fate > and others might go to o.a.a.core.util or similar) to keep the package > namespaces standardized, which is helpful to avoid naming collisions and > jar sealing issues, as well as for less complicated jigsaw module > definitions in future. Since 2.1 FaTE is already incompatible with prior > versions, a rename at this time would be less disruptive. > > Another task I had wanted to be done for 2.1, before I got distracted > fixing test failures during and after Christmas and trying to work through > the singleton manager zookeeper stuff to see what we could simplify. What I > had wanted done was to standardize the way we pass table identifiers (name, > IDs) across the RPC layer, since we currently do that inconsistently. I > don't remember if there's an existing ticket open for it, but I have a > working branch I had started working out of for it before Christmas. It's > relatively simple work, and would set us up for some much better APIs going > forward, as well as help with logging information about table actions. If > necessary, it could be bumped to a future version, but then we'd have more > churn in the thrift layer. So, I'd prefer to get it for 2.1 to avoid that. > > As for planning, I was thinking early May for a code freeze (except bug > fixes and small improvements found during testing), so we can try to > release towards the end of May/early June. If we go with that timeline, > that's not a lot of time to wrap up features and have time for > review/testing, so we may need to be selective about what we hold off until > the next version, unless we want to further delay 2.1. > > > On Mon, Apr 4, 2022 at 9:13 AM Dave Marion <dmario...@gmail.com> wrote: > > > I think [3] is OBE and can be closed. > > > > On Mon, Apr 4, 2022 at 9:11 AM Mike Miller <mmil...@apache.org> wrote: > > > > > Yes I agree, that was the goal of this email thread. I found a few more > > > tickets that should be addressed for the next release. > > > > > > Ivan - There was some work done on this PR but it has been some time. Do > > > you want to take a look at it? Implement a Thread limit. [1] > > > Keith T - I think we should get this one merged to fix that consistency > > > check bug I found. It looks like it is finished. [2] > > > Dave & Dom - Were you guys able to figure out a fix for the new external > > > compaction metrics test? [3] > > > > > > FYI we have 6 blockers for 2.1: > > > https://github.com/apache/accumulo/labels/blocker > > > > > > This is almost definitely going into 2.1 [4]. Thanks Jeff! > > > > > > [1] https://github.com/apache/accumulo/pull/1487 > > > [2] https://github.com/apache/accumulo/pull/2574 > > > [3] https://github.com/apache/accumulo/issues/2406 > > > [4] https://github.com/apache/accumulo/pull/2215 > > > > > > On Fri, Apr 1, 2022 at 2:21 PM Dave Marion <dmario...@gmail.com> wrote: > > > > > > > I think it would be useful to do some release planning so that we know > > > what > > > > features we are working towards and in which release they will be in. > > > This > > > > would be helpful for determining what existing PRs need to make it into > > > > 2.1.0. 2.1.0 is the LTM release, so patches for existing features will > > be > > > > backported (2.1.1, 2.1.2, 2.1.3, etc.) However, as defined in [1], > > > features > > > > that don't make it into 2.1.0 will go into the next non-LTM release > > > (2.2.0) > > > > and any patches to bugs in those features will go into the next non-LTM > > > > release after that (2.3.0). > > > > > > > > I'm not trying to hold up the 2.1.0 release by suggesting that we > > perform > > > > this activity. I'm just asking what the future holds, even if it's just > > > one > > > > feature in the next non-LTM release. My concern is that the next > > release > > > > will be open-ended and anything not included in 2.1.0 might not get put > > > > into a release for a very long time. > > > > > > > > [1] https://accumulo.apache.org/contributor/versioning.html#LTM > > > > > > > > > > > > On Thu, Mar 31, 2022 at 11:43 AM Mike Miller <mmil...@apache.org> > > wrote: > > > > > > > > > Starting an email chain of things that folks want to finish for 2.1. > > > Here > > > > > is what we currently have in the works that are most likely going > > into > > > > 2.1: > > > > > https://github.com/apache/accumulo/pull/2569 > > > > > https://github.com/apache/accumulo/pull/2600 > > > > > https://github.com/apache/accumulo/pull/2293 > > > > > > > > > > Some things that may go into 2.1: > > > > > https://github.com/apache/accumulo/pull/2422 > > > > > https://github.com/apache/accumulo/pull/2475 > > > > > https://github.com/apache/accumulo/pull/2197 > > > > > > > > > > I created a Project for follow on work to the ZK property change. I > > was > > > > > planning on putting tasks in there that we want to complete for 2.1. > > > But > > > > we > > > > > could also use it for post 2.1 work. > > > > > https://github.com/apache/accumulo/projects/24 > > > > > https://github.com/apache/accumulo/issues/2469 > > > > > > > > > > FYI a draft copy of the release notes has already been on the > > website: > > > > > https://accumulo.apache.org/release/accumulo-2.1.0/ > > > > > > > > > > This may be a good thread to discuss whether or not a task needs to > > go > > > > into > > > > > 2.1 or should wait for the next version. We currently have 32 open > > pull > > > > > requests so please email me if there is one that you would like > > > > prioritized > > > > > for 2.1. > > > > > > > > > > > > > >
