Thanks all, We have a test system that we could try rolling back. If something does break, will it be obvious?
Dave, the ingest rates are slightly more spikey, but I think it's mostly because tservers are bouncing and the cluster is working to catch up. Nothing major jumps out as an increase in throughput (i.e. ingest rate in terms of operations per second seem to be roughly equivalent. The same is true for the ingest rate in MB/s.) On Fri, Nov 4, 2022 at 4:21 PM Christopher <ctubb...@apache.org> wrote: > I don't think it has any changes that would prevent rollback, but it's not > a scenario that has been tested to my knowledge. > > On Fri, Nov 4, 2022, 16:15 Dave Marion <dmario...@gmail.com> wrote: > > > It's going to take some time to review the changes[1], but I don't see > > changes in the default JVM sizes. I was wondering if maybe the issue is > > that it's running faster. You are loading the same amount of data, but is > > it going faster by chance? If so, you could be creating more garbage per > > unit time putting more pressure on the GC. Just a thought. > > > > [1] https://github.com/apache/accumulo/compare/rel/1.9.3..rel/1.10.2 > > > > On Fri, Nov 4, 2022 at 4:02 PM Logan Jones <lo...@codescratch.com> > wrote: > > > > > Yeah, our memory usage is drastically different since the upgrade. > > > > > > We are seeing spikes in heap utilization on tablet servers that weren't > > > happening before the upgrade despite our ingest load being roughly the > > > same. This increase in heap utilization seems to be causing long GC > > times. > > > Those GC times are long enough that the tablet servers lose their locks > > and > > > then die. > > > > > > Looking into the JVM options, we don't see anything obvious that > changed > > > around the garbage collector, and looking at the Accumulo release notes > > > didn't leave us any indication that something like this should have > > > changed, but nevertheless we are seeing crashes of tservers. I'm mostly > > > trying to identify whether or not rollback is even an option. > > > > > > - Logan > > > > > > On Fri, Nov 4, 2022 at 3:49 PM Dave Marion <dmario...@gmail.com> > wrote: > > > > > > > Are you running into an error or some other issue that is making > you > > > > think that you have to rollback? I don't know that rolling back has > > been > > > > tested. > > > > > > > > On Fri, Nov 4, 2022 at 3:40 PM Logan Jones <lo...@codescratch.com> > > > wrote: > > > > > > > > > Hello: > > > > > > > > > > We recently upgraded from Accumulo 1.9.3 to 1.10.2. Is it safe to > > roll > > > > back > > > > > to Accumulo 1.9.3? > > > > > > > > > > Thanks in advance, > > > > > > > > > > - Logan > > > > > > > > > > > > > > >
