Ok, my Cloudstack installation is now so broken that I think it's probably
best to backup all my instances and templates, wipe the databases, and
start from scratch. However, I can't take snapshots or download volumes
anymore. What's causing these errors?
2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands
841744457 to Host 1 timed out after 21600
2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands
841744457 to Host 1 timed out after 21600
2014-04-29 17:40:51,269 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object
(VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7bbbd901), no
need to delete from object in store ref table
2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
(Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing
org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to copy the volume
from the source primary storage pool to secondary storage.
2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to copy the volume from the source primary storage pool to secondary
storage."}
On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <iyo...@ratespecial.com> wrote:
> I downgraded to 4.2.1 again but cloudstack-management won't start because
> the database is version 4.3. Is it safe to restore the database backup I
> made prior to this whole process? In the meantime I have destroyed and
> created system VMs, so I'm not sure it's a good idea.
> On Apr 29, 2014 3:09 PM, "Ian Young" <iyo...@ratespecial.com> wrote:
>
>> @stevenliang: I take it back--you can't set the VM size when you register
>> the template.
>>
>>
>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <motty.c...@gmail.com> wrote:
>>
>>> yes, you would have to shutdown the router, then click on "Change Service
>>> Offering"
>>> restart the VR.
>>>
>>> To Ian,
>>>
>>> I suspect you forgot the last step: " cloudstack-setup-management"
>>>
>>> that would fix your issue, I think,
>>>
>>> Thanks,
>>> ---
>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>> cloudstack-management service can't start because it can't connect to the
>>> database.
>>>
>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null) Unable
>>> to
>>> get a new db connection
>>> Caused by: java.sql.SQLException: Access denied for user 'cloud'@
>>> 'localhost'
>>> (using password: YES)
>>>
>>> Where are the credentials stored?
>>>
>>>
>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <stevenli...@yesup.com>
>>> wrote:
>>>
>>> > oh, then change service offering for vr?
>>> >
>>> >
>>> > On 29/04/14 05:53 PM, motty cruz wrote:
>>> >
>>> >> for my VR, I created a new
>>> >>
>>> >> "System Offering For Software Router"
>>> >> CPU in (MHz) 1.00GHz
>>> >> Memory (in MB) 1.00GB
>>> >>
>>> >> this are my current offerings, I'm sure the more RAM and CPU better
>>> >> performance.
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <stevenli...@yesup.com>
>>> >> wrote:
>>> >>
>>> >> Thank you again, motty.
>>> >>> I didn't notice this earlier.
>>> >>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>> >>>
>>> >>>
>>> >>>
>>> >>> On 29/04/14 05:33 PM, motty cruz wrote:
>>> >>>
>>> >>> Stevellang,
>>> >>>> I not sure if you saw this in the forums earlier :
>>> >>>>
>>> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%
>>> >>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>> c2rfedroy...@mail.gmail.com%3E
>>> >>>>
>>> >>>> I don't know if the bug was fixed yet,
>>> >>>>
>>> >>>> I will try upgrade in the next couple of days on a testing cluster,
>>> will
>>> >>>> report back if the bug was fixed.
>>> >>>>
>>> >>>> Thanks,
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <stevenli...@yesup.com
>>> >
>>> >>>> wrote:
>>> >>>>
>>> >>>> Thank you, motty.
>>> >>>>
>>> >>>>> I am also running kvm. Since that time I failed upgrade, I am still
>>> >>>>> using
>>> >>>>> 4.2.1. I'll try as your advice.
>>> >>>>>
>>> >>>>>
>>> >>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>> >>>>>
>>> >>>>> Stevenllang,
>>> >>>>>
>>> >>>>>> I had the similar issue with VR, I notice it was because I leave
>>> the
>>> >>>>>> default system specs on the VR, for instance by default 500MHz on
>>> CPU
>>> >>>>>> and
>>> >>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and 512MB of
>>> RAM
>>> >>>>>> your
>>> >>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>> >>>>>>
>>> >>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs were not
>>> >>>>>> able
>>> >>>>>> to
>>> >>>>>> access outside world, even if I created a new router.
>>> >>>>>>
>>> >>>>>> wish you the best,
>>> >>>>>> -motty
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>> stevenli...@yesup.com>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>> Yes, I had two zones(one is basic, another is advanced mode).
>>> >>>>>>
>>> >>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>> >>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>> >>>>>>>
>>> >>>>>>> Did rolling back to 4.2 fix the problem?
>>> >>>>>>>
>>> >>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>> stevenli...@yesup.com
>>> >>>>>>>> >
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> I met your situation before. Finally I rolled back to 4.2
>>> >>>>>>>>
>>> >>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>> >>>>>>>>
>>> >>>>>>>>> I destroyed the old virtual router and was able to create
>>> a new
>>> >>>>>>>>> one
>>> >>>>>>>>> by
>>> >>>>>>>>>
>>> >>>>>>>>> adding a new instance. However, this new router also failed
>>> to
>>> >>>>>>>>>
>>> >>>>>>>>>> start,
>>> >>>>>>>>>> citing the same error. After that, the expungement delay
>>> elapsed
>>> >>>>>>>>>> and
>>> >>>>>>>>>> the
>>> >>>>>>>>>> virtual router was expunged, so now I have none.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>> >>>>>>>>>> iyo...@ratespecial.com>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight, following the
>>> >>>>>>>>>> instructions
>>> >>>>>>>>>> here:
>>> >>>>>>>>>>
>>> >>>>>>>>>> http://docs.cloudstack.apache.org/projects/cloudstack-
>>> >>>>>>>>>>
>>> >>>>>>>>>> release-notes/en/latest/rnotes.html#upgrade-from-4-2-x-to-4-3
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> At the last step, I tried to restart the system VMs. The
>>> virtual
>>> >>>>>>>>>>> router
>>> >>>>>>>>>>> failed to start. Here is the message that was displayed in
>>> the
>>> >>>>>>>>>>> web
>>> >>>>>>>>>>> UI:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to start
>>> >>>>>>>>>>> instance
>>> >>>>>>>>>>> due
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to error in
>>> >>>>>>>>>>> finalizeStart,
>>> >>>>>>>>>>> not
>>> >>>>>>>>>>> retrying
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I tried running the script to restart the VMs but this time
>>> it
>>> >>>>>>>>>>> failed
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> start the console proxy:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d 192.168.100.6 -u
>>> >>>>>>>>>>> cloud
>>> >>>>>>>>>>> -p
>>> >>>>>>>>>>> -a
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>>> >>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>> >>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Is there a way to wipe the system VMs out and start over?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >
>>>
>>
>>
