Thanks for all the responses. I do not see cloud.log and cloud.out logs are zipped in /var/log and /var/log/cloud respectively. Only file that was zipped was cron.log. The two largest files are: cloud.out.2 with following: — + keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 console=hvc0 template=domP type=secstorage host=10.84.58.252 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource instance=SecStorage sslcopy=true role=templateProcessor mtu=1500' + for i in '$CMDLINE' ++ cut -s -d= -f1 ++ echo eth2ip=10.84.59.176 + KEY=eth2ip ++ cut -s -d= -f2 ++ echo eth2ip=10.84.59.176 + VALUE=10.84.59.176 + '[' eth2ip == '' ']' + case $KEY in + keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 console=hvc0 template=domP type=secstorage host=10.84.58.252 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM resource=org.apache.cloudstack.sJava HotSpot(TM) Client VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_root/8004 Try using the -Djava.io.tmpdir= option to select an alternate temp location.
log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No such property [maxFileSize] in org.apache.log4j.rolling.RollingFileAppender. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.rolling.RollingFileAppender. log4j:WARN Please set a rolling policy for the RollingFileAppender named 'FILE3' 08:31:55,320 INFO AgentShell:318 - Agent started log4j:ERROR Failed to flush writer, java.io.IOException: No space left on device — and, repetition of following in /var/log/cloud.log.1 — 2014-03-01 07:05:08,607 DEBUG [cloud.utils.ProcessUtil] (main:null) PID TTY TIME CMD 3938 ? 00:00:36 java 2014-03-01 07:05:08,607 ERROR [cloud.agent.AgentShell] (main:null) Unable to start agent: Java process is being started twice. If this is not true, remove /var/run/agent.SecStorage.pid 2014-03-01 07:05:19,028 INFO [cloud.agent.AgentShell] (main:null) Agent started 2014-03-01 07:05:19,030 INFO [cloud.agent.AgentShell] (main:null) Implementation Version is 4.3.0-SNAPSHOT 2014-03-01 07:05:19,030 INFO [cloud.agent.AgentShell] (main:null) agent.properties found at /usr/local/cloud/systemvm/conf/agent.properties 2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found property: instance 2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found property: resource 2014-03-01 07:05:19,038 INFO [cloud.agent.AgentShell] (main:null) Defaulting to using properties file for storage 2014-03-01 07:05:19,039 INFO [cloud.agent.AgentShell] (main:null) Defaulting to the constant time backoff algorithm 2014-03-01 07:05:19,048 INFO [cloud.utils.LogUtils] (main:null) log4j configuration found at /usr/local/cloud/systemvm/conf/log4j-cloud.xml 2014-03-01 07:05:19,062 DEBUG [cloud.agent.AgentShell] (main:null) Checking to see if agent.SecStorage.pid exists. 2014-03-01 07:05:19,064 DEBUG [cloud.utils.ProcessUtil] (main:null) environment.properties could not be opened 2014-03-01 07:05:19,071 DEBUG [cloud.utils.ProcessUtil] (main:null) Executing: bash -c ps -p 3938 2014-03-01 07:05:19,077 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution is successful. eventually the log gets full with, 2014-03-10 09:54:09,420 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.84.58.252:8250 2014-03-10 09:54:36,916 WARN [utils.nio.NioConnection] (Agent-Selector:null) Unable to connect to remote: is there a server running on port 8250 — Looks like the agent on s-1-vm could not connect to the management server at some point of time after the system vm startup and the log gets filled with above ‘Unable to connect’ messages. Anirban On Mar 7, 2014, at 11:37 PM, Rajesh Battala <rajesh.batt...@citrix.com> wrote: > AFAIK, log roation is enabled in the systemvm. > Can you check whether the logs are getting zipped .? > > -----Original Message----- > From: Anirban Chakraborty [mailto:abc...@juniper.net] > Sent: Saturday, March 8, 2014 12:46 PM > To: dev@cloudstack.apache.org > Subject: system vm disk space issue in ACS 4.3 > > Hi All, > > I am seeing system vm disk has no space left after running for few days. > Cloudstack UI shows the agent in v-2-VM in alert state, while agent state of > s-1-VM shows blank (hyphen in the UI). > Both the system vms are running and ssh-able from the host. The log in s-1-Vm > shows following errors: > > root@s-1-VM:~# grep 'Exception' /var/log/cloud/*.* > /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device > /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device > > whereas logs in v-1-VM shows > /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device > /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device > /var/log/cloud/cloud.out.3:07:18:00,547 INFO CSExceptionErrorCode:87 - Could > not find exception: com.cloud.exception.AgentControlChannelException in error > code list for exceptions > /var/log/cloud/cloud.out.3:com.cloud.exception.AgentControlChannelException: > Unable to post agent control request as link is not available > > Looks like cloud agent is filling up the log, which is leading to the disk > full state. > > Is this a known issue? Thanks. > > Anirban > >
