Yes, fix your swappiness setting. Also, put the external address of your VM into conf/monitor
-Eric On Wed, Sep 11, 2013 at 2:10 PM, John Vines <[email protected]> wrote: > Check the tserver logs. > > > On Wed, Sep 11, 2013 at 2:08 PM, Pete Carlson <[email protected]> wrote: > >> How do I determine if the rest of Accumulo is up and running since I >> can't use the monitor? >> >> Inside the monitor logs I see messages that it connected to Hadoop and >> Zookeeper, but then later on I see this message stating there are no tablet >> servers: >> >> 2013-09-11 11:39:29,199 [server.Accumulo] WARN : System swappiness >> setting is greater than ten (60) which can cause time-sensitive operations >> to be delayed. Accumulo is time sensitive because it needs to maintain >> distributed lock agreement. >> 2013-09-11 11:39:32,506 [impl.ServerClient] WARN : >> tracer:localhost.localdomain There are no tablet servers: check that >> zookeeper and accumulo are running. >> >> And when I log in to the accumulo shell I get the warning: There are no >> tablet servers: check that zookeeper and accumulo are running. >> >> Could the system swappiness setting of 60 be causing my issue? >> >> Regards, >> >> Pete >> >> >> On Wed, Sep 11, 2013 at 1:13 PM, John Vines <[email protected]> wrote: >> >>> Is the rest of Accumulo up and running? And is that the only type of >>> error in the Monitor logs? >>> >>> >>> On Wed, Sep 11, 2013 at 1:04 PM, Pete Carlson <[email protected]>wrote: >>> >>>> Ok, I deleted /accumulo out of hdfs, initialized accumulo, and >>>> re-started Accumulo. I still can't see the monitor IN my VM browser. I >>>> still get the same Monitor info message in the logs. >>>> >>>> Pete >>>> >>>> On Wed, Sep 11, 2013 at 11:55 AM, John Vines <[email protected]> wrote: >>>> >>>>> You need to delete the /accumulo directory out of hdfs >>>>> >>>>> >>>>> On Wed, Sep 11, 2013 at 11:54 AM, Pete Carlson <[email protected]>wrote: >>>>> >>>>>> I removed the monitor certificates as suggested, and I no longer get >>>>>> any messages in the monitor_localhost.err file. >>>>>> >>>>>> When I look in the monitor_localhost.localdomain.log file and saw the >>>>>> following INFO/Exception message: >>>>>> >>>>>> 2013-09-11 11:42:50,012 [monitor.Monitor] INFO : Failed to obtain >>>>>> problem reports >>>>>> java.lang.RuntimeException: >>>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>>> at >>>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174) >>>>>> at >>>>>> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:241) >>>>>> at >>>>>> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:299) >>>>>> at >>>>>> org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:392) >>>>>> at >>>>>> org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:504) >>>>>> at >>>>>> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) >>>>>> at java.lang.Thread.run(Thread.java:724) >>>>>> Caused by: >>>>>> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException >>>>>> at >>>>>> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212) >>>>>> at >>>>>> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) >>>>>> at >>>>>> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) >>>>>> ... 6 more >>>>>> >>>>>> I was thinking this was related to the fact I didn't yet have Thrift >>>>>> 0.9.0 installed (due to dependency issues). I now have Thrift 0.9.0 >>>>>> installed. I saved off my config files, blew away the accumulo directory >>>>>> and extracted it again from the tar.gz and copied over my conf files. >>>>>> >>>>>> However, when I run the accumulo-1.5.0/bin/accumulo init command >>>>>> again it gives me the following error: >>>>>> >>>>>> [util.Initialize] FATAL: It appears this location was previously >>>>>> initialized, exiting ... >>>>>> >>>>>> I can still start Accumulo like before with the bin/start-all.sh >>>>>> command but the Accumulo monitor still doesn't work IN my VM. >>>>>> >>>>>> Any other ideas to try? >>>>>> >>>>>> Pete >>>>>> >>>>>> On Wed, Sep 11, 2013 at 10:57 AM, John Vines <[email protected]>wrote: >>>>>> >>>>>>> Either try rerunning the monitor certificates or remove the portion >>>>>>> from your accumulo-site that defines use of the certificate. I suggest >>>>>>> the >>>>>>> latter simply to remove variables at hand. >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 11, 2013 at 10:54 AM, Pete Carlson >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> John, >>>>>>>> >>>>>>>> What is the monitor process called so I can verify its running? I >>>>>>>> did look at the thread monitor logs, and under monitor_localhost.err it >>>>>>>> gives a FileNotFoundException because it can't find the keystore.jks >>>>>>>> under >>>>>>>> accumulo-1.5.0/conf/ >>>>>>>> >>>>>>>> I would have thought when I ran ./generate_monitor_certificate.sh >>>>>>>> it would have created this. >>>>>>>> >>>>>>>> I was trying to run the monitor in a browser IN the VM. Thats how >>>>>>>> I found this problem. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Pete >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 11, 2013 at 10:41 AM, John Vines <[email protected]>wrote: >>>>>>>> >>>>>>>>> Again, please check to make sure the monitor process is running >>>>>>>>> and it's not reporting any errors in it's logs. >>>>>>>>> >>>>>>>>> If you can verify that (and please verify that first), try >>>>>>>>> accessing the monitor in a browser IN the VM. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Sep 11, 2013 at 10:36 AM, Pete Carlson < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> All, >>>>>>>>>> >>>>>>>>>> Just to clarify I have a Centos VM running on my Windows 7 box, >>>>>>>>>> and I have Accumulo, Hadoop, and Zookeeper running on it standalone. >>>>>>>>>> I was >>>>>>>>>> trying to open the Accumulo monitor within that VM and it failed. >>>>>>>>>> However >>>>>>>>>> I can open the Hadoop Namenode and Jobtracker web interfaces in my >>>>>>>>>> browser >>>>>>>>>> just fine. >>>>>>>>>> >>>>>>>>>> Brendan, I was able to connect to my VM using netcat on port >>>>>>>>>> 50095. >>>>>>>>>> >>>>>>>>>> However, when I try to telnet to that port on my Win7 with telnet >>>>>>>>>> using my VM's ip address and port i.e., >>>>>>>>>> >>>>>>>>>> telnet <IP address of vm> 50095 >>>>>>>>>> >>>>>>>>>> I get a message: ....Could not open connection to the host, on >>>>>>>>>> port 50095: Connect failed. >>>>>>>>>> >>>>>>>>>> Pete >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Sep 11, 2013 at 9:15 AM, Brendan Heussler < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Can you try to connect to your VM using netcat on port 50095? >>>>>>>>>>> >>>>>>>>>>> On VM: >>>>>>>>>>> nc -l 50095 >>>>>>>>>>> >>>>>>>>>>> On Client: >>>>>>>>>>> nc <ip-address of VM> 50095 >>>>>>>>>>> >>>>>>>>>>> Then type "test" or something on your Client machine. The >>>>>>>>>>> message should be sent to your VM. >>>>>>>>>>> >>>>>>>>>>> This will help to test the connectivity of the two machines over >>>>>>>>>>> port 50095 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Brendan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 10, 2013 at 5:22 PM, John Vines <[email protected]>wrote: >>>>>>>>>>> >>>>>>>>>>>> Are you certain the monitor is running and was able to bind to >>>>>>>>>>>> the port? Please check the monitor logs for any errors. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 10, 2013 at 5:18 PM, Pete Carlson < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> John, >>>>>>>>>>>>> >>>>>>>>>>>>> I don't see port 50095 in use in the VM. When I become root >>>>>>>>>>>>> and type lsof -i -P | grep :50095 I don't get anything back. >>>>>>>>>>>>> >>>>>>>>>>>>> Any suggestions on how to get the port running? I tried >>>>>>>>>>>>> editing the iptables with the following command: >>>>>>>>>>>>> >>>>>>>>>>>>> iptables -I INPUT -p tcp --dport 50095 -j ACCEPT >>>>>>>>>>>>> >>>>>>>>>>>>> and then I typed >>>>>>>>>>>>> >>>>>>>>>>>>> service iptables save >>>>>>>>>>>>> >>>>>>>>>>>>> to save my iptable changes, but when I restarted my VM I >>>>>>>>>>>>> didn't see any difference. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Pete >>>>>>>>>>>>> >>>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>>> >>>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:56 PM, John Vines >>>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> You don't see 50095 in use locally on the VM or in your box? >>>>>>>>>>>>>> You should only see it in use in the VM. If not, make sure it's >>>>>>>>>>>>>> running. If >>>>>>>>>>>>>> it is, check the monitor logs and see what's up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 4:51 PM, Pete Carlson < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi John, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No, it looks like port 50095 wasn't open on my VM. I tried >>>>>>>>>>>>>>> modifying the IP tables to make it opened, and restarted my VM >>>>>>>>>>>>>>> but when I >>>>>>>>>>>>>>> run the lsof -i command I don't see port 50095 listed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Looking for a unique woodcraft? Check out my Facebook page: >>>>>>>>>>>>>>> www.facebook.com/oqwcrafts >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also on Etsy: https://www.etsy.com/shop/OlneyQualityWoodCrft >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:56 PM, John Vines < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you have the port open to your VM? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Sep 10, 2013 at 3:54 PM, Pete Carlson < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I just set up a free standing Accumulo 1.5 instance on my >>>>>>>>>>>>>>>>> linux VM, and when I run bin/start-all.sh it starts up >>>>>>>>>>>>>>>>> accumulo, and also >>>>>>>>>>>>>>>>> connects to my free standing Zookeeper and HDFS instances . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, when I try to pull up the Accumulo master in my >>>>>>>>>>>>>>>>> FireFox browser (Firefox ESR 10.0.12) i.e., >>>>>>>>>>>>>>>>> http://accumulomaster:50095/status for health and >>>>>>>>>>>>>>>>> monitoring, I get an unable to connect error. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I also tried running ./generate_monitor_certificate.sh and >>>>>>>>>>>>>>>>> copied those properties in my accumulo-site.xml but I still >>>>>>>>>>>>>>>>> get the same >>>>>>>>>>>>>>>>> result. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is there something else I need to configure? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Pete >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
