On Mon, Jul 2, 2018 at 2:15 PM, Victor Pickard <vpick...@redhat.com> wrote:
> Hi all, > > I'm looking at clustering stability. One of the jobs I've been looking at is > controller clustering. This is a good CSIT, in that it stops and starts ODL > several times during the run. > > In one of failed test runs (sandbox, logs wiped from last week, but I do have > this particular karaf log archived locally), ODL is started, and rest calls > fail during the test. Looking at the logs, I can see why. Karaf failed to > start, or better yet, took a really long time to start. From the snipped > below, you can see about 7 mins between when Karaf launched, and did > something?, maybe restarted again. But the main thing is that karaf failed to > start in a timely manner, taking over 7 minutes to begin to start up > blueprints, etc. > > > I ran a job that had karaf debug logging enabled with this setting: > > log4j.rootLogger=DEBUG > > > This did not go very well. This generates way too much debug info, and was > causing timeouts and other various errors in the CSIT run. > > > So, my questions are: > > 1. Has anyone see this issue where karaf seems to hang on startup (after a > kill -9 on karaf pid)? If so, is this a known issue? > > 2. What debug would be needed to figure out why karaf was hanging? Note the > above generated a log file of ~768 MB in a very short timespan. > > > Vic - does this happen if you gracefully shut it down? In years past with karaf I recall corruption could occur in the bundle cache under data if the karaf process was killed. I don't know if that potential issue is still present with karaf 4. Does it clean the data dir before restarting? If not, it would be good to do so to be safe. Other than that, we probably need to get a thread dump. > Thanks, > > Vic > > > > > Jun 29, 2018 3:43:47 PM org.apache.karaf.main.Main launch > INFO: Installing and starting initial bundles > Jun 29, 2018 3:43:47 PM org.apache.karaf.main.Main launch > INFO: All initial bundles installed and set to start > Jun 29, 2018 3:43:47 PM org.apache.karaf.main.lock.SimpleFileLock lock > INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock > Jun 29, 2018 3:43:47 PM org.apache.karaf.main.lock.SimpleFileLock lock > INFO: Lock acquiredJun 29, 2018 3:43:47 PM > org.apache.karaf.main.Main$KarafLockCallback lockAquired > INFO: Lock acquired. Setting startlevel to 100 > Jun 29, 2018 3:50:48 PM org.apache.karaf.main.Main launch > INFO: Installing and starting initial bundles > Jun 29, 2018 3:50:49 PM org.apache.karaf.main.Main launch > INFO: All initial bundles installed and set to start > Jun 29, 2018 3:50:49 PM org.apache.karaf.main.lock.SimpleFileLock lock > INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock > Jun 29, 2018 3:50:49 PM org.apache.karaf.main.lock.SimpleFileLock lock > INFO: Lock acquired > Jun 29, 2018 3:50:49 PM org.apache.karaf.main.Main$KarafLockCallback > lockAquired > INFO: Lock acquired. Setting startlevel to 100 > > > > _______________________________________________ > controller-dev mailing list > controller-dev@lists.opendaylight.org > https://lists.opendaylight.org/mailman/listinfo/controller-dev > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
