How do I determine if the rest of Accumulo is up and running since I can't use the monitor?
Inside the monitor logs I see messages that it connected to Hadoop and Zookeeper, but then later on I see this message stating there are no tablet servers: 2013-09-11 11:39:29,199 [server.Accumulo] WARN : System swappiness setting is greater than ten (60) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement. 2013-09-11 11:39:32,506 [impl.ServerClient] WARN : tracer:localhost.localdomain There are no tablet servers: check that zookeeper and accumulo are running. And when I log in to the accumulo shell I get the warning: There are no tablet servers: check that zookeeper and accumulo are running. Could the system swappiness setting of 60 be causing my issue? Regards, Pete On Wed, Sep 11, 2013 at 1:13 PM, John Vines <[email protected]> wrote: > Is the rest of Accumulo up and running? And is that the only type of error > in the Monitor logs? > > > On Wed, Sep 11, 2013 at 1:04 PM, Pete Carlson <[email protected]> wrote: > >> Ok, I deleted /accumulo out of hdfs, initialized accumulo, and re-started >> Accumulo. I still can't see the monitor IN my VM browser. I still get the >> same Monitor info message in the logs. >> >> Pete >> >> On Wed, Sep 11, 2013 at 11:55 AM, John Vines <[email protected]> wrote: >> >>> You need to delete the /accumulo directory out of hdfs >>> >>> >>> On Wed, Sep 11, 2013 at 11:54 AM, Pete Carlson <[email protected]>wrote: >>> >>>> I removed the monitor certificates as suggested, and I no longer get >>>> any messages in the monitor_localhost.err file. >>>> >>>> When I look in the monitor_localhost.localdomain.log file and saw the >>>> following INFO/Exception message: >>>> >>>> 2013-09-11 11:42:50,012 [monitor.Monitor] INFO : Failed to obtain >>>> problem reports >>>> java.lang.RuntimeException: >>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>> at >>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174) >>>> at >>>> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:241) >>>> at >>>> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:299) >>>> at >>>> org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:392) >>>> at >>>> org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:504) >>>> at >>>> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) >>>> at java.lang.Thread.run(Thread.java:724) >>>> Caused by: >>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>> at >>>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212) >>>> at >>>> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) >>>> at >>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) >>>> ... 6 more >>>> >>>> I was thinking this was related to the fact I didn't yet have Thrift >>>> 0.9.0 installed (due to dependency issues). I now have Thrift 0.9.0 >>>> installed. I saved off my config files, blew away the accumulo directory >>>> and extracted it again from the tar.gz and copied over my conf files. >>>> >>>> However, when I run the accumulo-1.5.0/bin/accumulo init command again >>>> it gives me the following error: >>>> >>>> [util.Initialize] FATAL: It appears this location was previously >>>> initialized, exiting ... >>>> >>>> I can still start Accumulo like before with the bin/start-all.sh >>>> command but the Accumulo monitor still doesn't work IN my VM. >>>> >>>> Any other ideas to try? >>>> >>>> Pete >>>> >>>> On Wed, Sep 11, 2013 at 10:57 AM, John Vines <[email protected]> wrote: >>>> >>>>> Either try rerunning the monitor certificates or remove the portion >>>>> from your accumulo-site that defines use of the certificate. I suggest the >>>>> latter simply to remove variables at hand. >>>>> >>>>> >>>>> On Wed, Sep 11, 2013 at 10:54 AM, Pete Carlson <[email protected]>wrote: >>>>> >>>>>> John, >>>>>> >>>>>> What is the monitor process called so I can verify its running? I >>>>>> did look at the thread monitor logs, and under monitor_localhost.err it >>>>>> gives a FileNotFoundException because it can't find the keystore.jks >>>>>> under >>>>>> accumulo-1.5.0/conf/ >>>>>> >>>>>> I would have thought when I ran ./generate_monitor_certificate.sh it >>>>>> would have created this. >>>>>> >>>>>> I was trying to run the monitor in a browser IN the VM. Thats how I >>>>>> found this problem. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Pete >>>>>> >>>>>> >>>>>> On Wed, Sep 11, 2013 at 10:41 AM, John Vines <[email protected]>wrote: >>>>>> >>>>>>> Again, please check to make sure the monitor process is running and >>>>>>> it's not reporting any errors in it's logs. >>>>>>> >>>>>>> If you can verify that (and please verify that first), try accessing >>>>>>> the monitor in a browser IN the VM. >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 11, 2013 at 10:36 AM, Pete Carlson >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> All, >>>>>>>> >>>>>>>> Just to clarify I have a Centos VM running on my Windows 7 box, and >>>>>>>> I have Accumulo, Hadoop, and Zookeeper running on it standalone. I was >>>>>>>> trying to open the Accumulo monitor within that VM and it failed. >>>>>>>> However >>>>>>>> I can open the Hadoop Namenode and Jobtracker web interfaces in my >>>>>>>> browser >>>>>>>> just fine. >>>>>>>> >>>>>>>> Brendan, I was able to connect to my VM using netcat on port 50095. >>>>>>>> >>>>>>>> However, when I try to telnet to that port on my Win7 with telnet >>>>>>>> using my VM's ip address and port i.e., >>>>>>>> >>>>>>>> telnet <IP address of vm> 50095 >>>>>>>> >>>>>>>> I get a message: ....Could not open connection to the host, on >>>>>>>> port 50095: Connect failed. >>>>>>>> >>>>>>>> Pete >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 11, 2013 at 9:15 AM, Brendan Heussler < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Can you try to connect to your VM using netcat on port 50095? >>>>>>>>> >>>>>>>>> On VM: >>>>>>>>> nc -l 50095 >>>>>>>>> >>>>>>>>> On Client: >>>>>>>>> nc <ip-address of VM> 50095 >>>>>>>>> >>>>>>>>> Then type "test" or something on your Client machine. The >>>>>>>>> message should be sent to your VM. >>>>>>>>> >>>>>>>>> This will help to test the connectivity of the two machines over >>>>>>>>> port 50095 >>>>>>>>> >>>>>>>>> >>>>>>>>> Brendan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 10, 2013 at 5:22 PM, John Vines <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> Are you certain the monitor is running and was able to bind to >>>>>>>>>> the port? Please check the monitor logs for any errors. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 10, 2013 at 5:18 PM, Pete Carlson < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> John, >>>>>>>>>>> >>>>>>>>>>> I don't see port 50095 in use in the VM. When I become root and >>>>>>>>>>> type lsof -i -P | grep :50095 I don't get anything back. >>>>>>>>>>> >>>>>>>>>>> Any suggestions on how to get the port running? I tried editing >>>>>>>>>>> the iptables with the following command: >>>>>>>>>>> >>>>>>>>>>> iptables -I INPUT -p tcp --dport 50095 -j ACCEPT >>>>>>>>>>> >>>>>>>>>>> and then I typed >>>>>>>>>>> >>>>>>>>>>> service iptables save >>>>>>>>>>> >>>>>>>>>>> to save my iptable changes, but when I restarted my VM I didn't >>>>>>>>>>> see any difference. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Pete >>>>>>>>>>> >>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>> >>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 10, 2013 at 4:56 PM, John Vines <[email protected]>wrote: >>>>>>>>>>> >>>>>>>>>>>> You don't see 50095 in use locally on the VM or in your box? >>>>>>>>>>>> You should only see it in use in the VM. If not, make sure it's >>>>>>>>>>>> running. If >>>>>>>>>>>> it is, check the monitor logs and see what's up. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:51 PM, Pete Carlson < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi John, >>>>>>>>>>>>> >>>>>>>>>>>>> No, it looks like port 50095 wasn't open on my VM. I tried >>>>>>>>>>>>> modifying the IP tables to make it opened, and restarted my VM >>>>>>>>>>>>> but when I >>>>>>>>>>>>> run the lsof -i command I don't see port 50095 listed. >>>>>>>>>>>>> >>>>>>>>>>>>> Pete >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Pete >>>>>>>>>>>>> >>>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>>> >>>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:56 PM, John Vines >>>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Do you have the port open to your VM? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:54 PM, Pete Carlson < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I just set up a free standing Accumulo 1.5 instance on my >>>>>>>>>>>>>>> linux VM, and when I run bin/start-all.sh it starts up >>>>>>>>>>>>>>> accumulo, and also >>>>>>>>>>>>>>> connects to my free standing Zookeeper and HDFS instances . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However, when I try to pull up the Accumulo master in my >>>>>>>>>>>>>>> FireFox browser (Firefox ESR 10.0.12) i.e., >>>>>>>>>>>>>>> http://accumulomaster:50095/status for health and >>>>>>>>>>>>>>> monitoring, I get an unable to connect error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I also tried running ./generate_monitor_certificate.sh and >>>>>>>>>>>>>>> copied those properties in my accumulo-site.xml but I still get >>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>> result. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is there something else I need to configure? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
