I am not sure what's up with the VFS error. I generally recommend against using commons-vfs, because of the lack of any known good versions, and the fact that most of what it is used for can be done better in other ways. We have taken steps in recent versions to make sure that the classloader stuff is more pluggable, so people do not have to rely on commons-vfs, if they don't want to use it. One thing that's concerning is that VFS is using some temp file in /tmp, which could be wiped at any time by the OS, so if you do have need of VFS, you may want to consider customizing it to use a different directory, that isn't going to be wiped periodically by the OS. I believe the relevant Accumulo property is: general.vfs.cache.dir
The manager is down message is expected on initial startup if the monitor starts faster than the manager, but refreshing the page should make it work eventually if the manager is actually running. If that doesn't happen eventually, I'm not sure what could be happening without further troubleshooting. On Thu, Oct 12, 2023 at 12:36 PM Vincent Russell <[email protected]> wrote: > > Christopher, > > I have run through an upgrade with a bit more data than I did previously > and monitors with high availability just don't seem as consistent as they > were with 2.0.1. > > After upgrading both accumulo monitors are reporting "Manager is Down" and > I was only able to achieve that after restarting both of the monitors and > managers multiple times. > > The only ERROR log that I see in the manager logs are (at startup): > > java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname > .... > at > org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) > > I see this on both managers. > > Any ideas? > > Thanks, > > On Mon, Sep 25, 2023 at 12:21 PM Christopher <[email protected]> wrote: > > > I'm not sure about the monitor behavior. I've never had a reason to > > run more than one, and I don't think we have any well-defined behavior > > for trying to run more than one (I could see an argument for both). It > > would be weird if one incorrectly reported "Master[now Manager] is > > down" because another monitor was running, though. I would expect the > > 2.1 behavior you described over that 2.0 behavior. > > > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > > <[email protected]> wrote: > > > > > > Thank you Christopher, > > > > > > I did neglect to update the log4j2 files. > > > > > > One thing that is weird is that the monitors appear to work differently > > > with high availability. > > > > > > With 2.0.1 the accumulo monitors both work and you can hit them both (one > > > just says 'Master is down'; however, with 2.1.1, it appears that only one > > > works. Is this expected? > > > > > > Thank you, > > > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher <[email protected]> wrote: > > > > > > > I do not know about the error message you saw. StatusConsoleListener > > > > is not an Accumulo class, but it looks like it's a log4j one. My best > > > > guess is that during your upgrade, you did not migrate your log4j > > > > config files over to be based on the log4j2 ones we have as an example > > > > in the 2.1 tarball. It's possible that there's some log4j > > > > configuration issue that's preventing you from seeing the reason why > > > > the Monitor failed to start. > > > > > > > > To troubleshoot, I would compare your log4j config with what we ship > > > > in the 2.1 tarball and also check to see if you've done any log4j > > > > config overrides in the system properties or environment variables > > > > that control log4j. > > > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > > <[email protected]> wrote: > > > > > > > > > > Thank you Christopher, > > > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a test > > > > > cluster. :) > > > > > > > > > > I just upgraded a test cluster and everything appeared to go smoothly > > > > > except for one thing. > > > > > > > > > > I am able to login to the accumulo shell and I can see the table > > that I > > > > > created before the upgrade exists and I can scan that table. > > > > > > > > > > Before the upgrade the monitor was running on port 9995, but now > > nothing > > > > > appears to be running on that port. Nothing in the monitor log > > > > exception: > > > > > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender > > [MonitorLog] > > > > > IllegalARgumentException: invalid URI schema <<hostname>> > > > > > > > > > > > > > > > This seems not great, but I am not sure if it's related. > > > > > > > > > > It doesn't look like the monitor is running on another port. Any > > > > > suggestions for debugging this? > > > > > > > > > > Thanks, > > > > > Vincent > > > > > > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher <[email protected]> > > wrote: > > > > > > > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > > > > would consult the ZooKeeper community for advice on that, since > > they > > > > > > are the experts. For Accumulo's part, I believe 2.0 worked fine > > with > > > > > > newer versions of ZK (3.5+), so you should be able to update ZK > > first > > > > > > if you didn't want to do them at the same time. There may be some > > > > > > things you might need to do to ensure your classpath is set up > > > > > > correctly for Accumulo if using it with 2.0, since 2.0 might > > assume ZK > > > > > > 3.4 jar locations. I don't recall the specifics, but I remember it > > > > > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > > > > > while. The necessary config changes to support that might even > > still > > > > > > be scripted in its repo. > > > > > > > > > > > > I'm really not familiar with Hadoop upgrades, but as far as > > Accumulo > > > > > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > > > > > essentially equivalent. I don't think anything substantially > > changed > > > > > > that would cause Accumulo to even notice a difference. Seems like > > it > > > > > > would just be a quick shutdown, update the classpath, restart, for > > > > > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > > > > > with the Hadoop community first. You could probably upgrade Hadoop > > > > > > before or after you upgrade Accumulo for this one. I don't think > > it's > > > > > > going to matter much. > > > > > > > > > > > > If you're concerned about problems you might encounter, I would run > > > > > > through the upgrades on a small test instance first, just to > > practice. > > > > > > If you run into a specific problem doing that, we could probably > > offer > > > > > > more specific insight, having probably seen it before. > > > > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Thank you Christopher. > > > > > > > > > > > > > > Does there exist some documentation for upgrading zookeeper from > > > > 3.4.14 > > > > > > to > > > > > > > 3.8.1? Is there a preferred upgrade path? > > > > > > > > > > > > > > Also...Is there any documentation on how this affects hadoop when > > > > using > > > > > > > high availability? We are using hadoop 3.3.1. I want to > > upgrade to > > > > > > > hadoop 3.3.5, but I didn't know if that should be done as part of > > > > > > upgrading > > > > > > > accumulo or later. > > > > > > > > > > > > > > Any experience folks have would be greatly appreciated. > > > > > > > > > > > > > > Thank you, > > > > > > > Vincent > > > > > > > > > > > > > > On Thu, Aug 24, 2023 at 12:58 PM Christopher < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > The correct link seems to be > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > > > > > > > > > > > > > It looks like both `bin/accumulo dump-zoo` and > > `bin/accumulo-util > > > > > > > > dump-zoo` will do the same thing. It was originally a > > disconnected > > > > > > > > utility that existed for convenience, but was incorporated > > into our > > > > > > > > regular tooling recently as part of an effort to make all these > > > > > > > > utilities more easily discoverable and executable. I don't > > think we > > > > > > > > realized that it already had a convenient and documented entry > > > > point. > > > > > > > > We were just looking for utilities with a "main" method. Either > > > > should > > > > > > > > work fine. They call the same code. > > > > > > > > > > > > > > > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > Also the link to: > > > > > > > > > > > > > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > > > > > > > > > > > > > > > is not resolving. > > > > > > > > > > > > > > > > > > What needs to be done at this step? > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Vincent > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell < > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > I'm practicing going to the upgrade instructions from > > 2.0.1 to > > > > > > 2.1.1 > > > > > > > > and I > > > > > > > > > > wanted to confirm that the zoo-dump command is run via the > > > > > > > > accumulo-util > > > > > > > > > > command and not via accumulo. > > > > > > > > > > > > > > > > > > > > The instructions say: > > > > > > > > > > > > > > > > > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root > > /accumulo | > > > > tee > > > > > > > > > > PATH_TO_SNAPSHOT > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > Vincent > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
