Well, Im glad that you managed to fix it Alejandro. Thought, as Nicolas pointed out, there seems to be some limitations in place (that I also wasn't aware of).
Best Andrija On Sun, Jun 16, 2019, 10:07 Nicolas Vazquez <nicolas.vazq...@shapeblue.com> wrote: > Hi Alejandro, > > AFAIK it is not possible to add data disks unless you add a Ceph storage > pool. > > References: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/LXC+Enhancements > https://issues.apache.org/jira/browse/CLOUDSTACK-8574 > > Regards, > Nicolas Vazquez > > De: Alejandro Ruiz Bermejo > Enviado: sábado, 15 de junio 16:15 > Asunto: Re: launch instance error > Para: users@cloudstack.apache.org > > > Hi, > I made a fresh install of the CoudStack environment with 2 nodes: > management and server. The primary and secondary storage inside the > management server, both using NFS. > > 1 Zone > 1 Pod > 1 Cluster > 1 Host (LXC) > > The secondary and console proxy VMs where created successfully and i > created an LXC template according to > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/LXC+Template+creation#LXCTemplatecreation-CreatingtemplateonUbuntu > > > When i try to launc the new instance i get again errors thi is the log > <https://pastebin.com/rdCYMYDZ>. As i see now the manager allocate CPU, > RAM > and Root disk in a storage pool but can't find a storage pool for Data > disk. > > I can create now an instance without assign a Data Disk. How can i correct > this? > > On Fri, Jun 14, 2019 at 5:54 PM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > I would go with 2nd aproach - I don't expect "same" issues actually - you > > can either add a new pod/cluster or just new cluster in same pod. > > > > On Fri, 14 Jun 2019 at 21:25, Alejandro Ruiz Bermejo < > > arbermejo0...@gmail.com> wrote: > > > > > I created a new zone, pod and cluster and added the LXC host to the new > > > cluster. CloudStack did everything for me. Since i am in a test > > environment > > > i used the same subnet for both zones. > > > > > > I can do 2 things: > > > 1. I will wipe the host and do a fresh install(only on the compute > host), > > > but i will probably have the same issues. I will add it to the same > zone > > in > > > the same pod as the KVM cluster and in an LXC cluster as i previously > > did. > > > > > > 2. I also can do fresh a CloudStack install (management server > included) > > > and use only LXC as hypervisor technology in order to find where the > > > troubles are. After i have LXC working can later add KVM > > > > > > Wich approach would be more helpful to find where the trouble is. > > > > > > Regards, > > > Alejandro > > > > > > > > > On Fri, Jun 14, 2019 at 2:59 PM Andrija Panic <andrija.pa...@gmail.com > > > > > wrote: > > > > > > > I'm not sure how you moved the host to another Zone ? (Please > describe) > > > > > > > > But anyway, I would wipe that host (perhaps some settings are kept > > > locally, > > > > etc), and add it as a fresh host to a new cluster (could be in same > Pod > > > as > > > > well, ot a new one).i.e. start from scratch please - since your setup > > > seems > > > > problematic at this moment. > > > > > > > > I would be happy to learn if you managed to have it working - so > please > > > let > > > > me know. > > > > > > > > Best > > > > Andrija > > > > > > > > On Fri, Jun 14, 2019, 20:03 Alejandro Ruiz Bermejo < > > > > arbermejo0...@gmail.com> > > > > wrote: > > > > > > > > > Yes trying to find solutions i did create a new zone pod and > cluster > > > and > > > > > moved the lxc host to it, but i had the same errors. So i moved it > > back > > > > to > > > > > my original LXC cluster into my original zone. I guess thats why it > > > shows > > > > > those records. > > > > > > > > > > I made all the movements using the UI, it seems like the values on > > the > > > DB > > > > > didn't updated, how do i change that? > > > > > > > > > > On Fri, Jun 14, 2019 at 1:32 PM Andrija Panic < > > andrija.pa...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > So, there are a couple of problems I see, based on your storage > > pool > > > DB > > > > > > records: > > > > > > > > > > > > your storage (for LXC) is CLUSTER wide, so back to first SQL I > > share, > > > > as > > > > > > following: > > > > > > SELECT storage_pool.id, storage_pool.name, storage_pool.uuid, > > > > > > storage_pool.pool_type, storage_pool.created, > storage_pool.removed, > > > > > > storage_pool.update_time, storage_pool.data_center_id, > > > > > storage_pool.pod_id, > > > > > > storage_pool.used_bytes, storage_pool.capacity_bytes, > > > > > storage_pool.status, > > > > > > storage_pool.storage_provider_name, storage_pool.host_address, > > > > > > storage_pool.path, storage_pool.port, storage_pool.user_info, > > > > > > storage_pool.cluster_id, storage_pool.scope, > storage_pool.managed, > > > > > > storage_pool.capacity_iops, storage_pool.hypervisor FROM > > storage_pool > > > > > > WHERE storage_pool.data_center_id = 1 > > > > > > AND storage_pool.status = 'Up' > > > > > > AND storage_pool.scope = 'CLUSTER' > > > > > > AND ( storage_pool.pod_id IS NULL OR storage_pool.pod_id = 1 ) > > > > > > AND ( storage_pool.cluster_id IS NULL OR > storage_pool.cluster_id > > = > > > > 2 ) > > > > > > AND storage_pool.removed IS NULL > > > > > > > > > > > > If you compare WHERE clause i.e. conditions, you will see your > > > > > > data_center=2 instead of 1, you pod_id=2 instead of 1, your > > > > cluster_id=3 > > > > > > instead of 2. > > > > > > So you probably have been adding back and forth primary storage > > pool > > > > for > > > > > > LXC or something fishy there... > > > > > > > > > > > > Please investigate - based on your records, it seems you have a > > zone > > > > for > > > > > > KVM, another zone for LXC, and then there are more clusters/pods > > > than 1 > > > > > in > > > > > > each of them... > > > > > > We could easily change DB records, but you need to understand why > > > these > > > > > > records are "shifted" by one - i.e. wrong zone, wrong pod, wrong > > > > cluster > > > > > > expected by CloudStack versus the actual records in storage_pool > > > table. > > > > > > > > > > > > Best > > > > > > Andrija > > > > > > > > > > > > > > > > > > On Fri, 14 Jun 2019 at 18:53, Alejandro Ruiz Bermejo < > > > > > > arbermejo0...@gmail.com> wrote: > > > > > > > > > > > > > this was the ouputs > > > > > > > > > > > > > > mysql> SELECT storage_pool.id, storage_pool.name, > > > storage_pool.uuid, > > > > > > > -> storage_pool.pool_type, storage_pool.created, > > > > > > storage_pool.removed, > > > > > > > -> storage_pool.update_time, storage_pool.data_center_id, > > > > > > > storage_pool.pod_id, > > > > > > > -> storage_pool.used_bytes, storage_pool.capacity_bytes, > > > > > > > storage_pool.status, > > > > > > > -> storage_pool.storage_provider_name, > > > storage_pool.host_address, > > > > > > > -> storage_pool.path, storage_pool.port, > > > storage_pool.user_info, > > > > > > > -> storage_pool.cluster_id, storage_pool.scope, > > > > > storage_pool.managed, > > > > > > > -> storage_pool.capacity_iops, storage_pool.hypervisor FROM > > > > > > > storage_pool WHERE > > > > > > > -> storage_pool.data_center_id = 1 AND > storage_pool.status = > > > > 'Up' > > > > > > AND > > > > > > > -> storage_pool.scope = 'ZONE' AND > storage_pool.hypervisor = > > > > 'LXC' > > > > > > > AND > > > > > > > -> storage_pool.removed IS NULL; > > > > > > > Empty set (0.00 sec) > > > > > > > > > > > > > > mysql> select * from storage_pool; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > > > > > > > | id | name | uuid | > pool_type > > > > > > | > > > > > > > port | data_center_id | pod_id | cluster_id | used_bytes | > > > > > > capacity_bytes | > > > > > > > host_address | user_info | path | created > > | > > > > > > removed > > > > > > > | update_time | status | storage_provider_name | scope | > > > > hypervisor | > > > > > > > managed | capacity_iops | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > > > > > > > | 1 | primary | 672631ab-5c14-3f60-97c6-1d633e60f7bd | > > > > > > NetworkFilesystem | > > > > > > > 2049 | 1 | NULL | NULL | 3413114880 | > > > > > > 984374837248 | > > > > > > > 10.8.9.235 | NULL | /export/primary | 2019-06-06 > 20:05:26 > > | > > > > NULL > > > > > > > | NULL | Up | DefaultPrimary | ZONE | KVM > > > > > | > > > > > > > 0 | NULL | > > > > > > > | 3 | lxc | 3b062c9d-0333-41c4-9ba8-f71f75a81d1f | > > > > > > SharedMountPoint | > > > > > > > 0 | 2 | 2 | 3 | 3413114880 | > > > > > > 984374837248 | > > > > > > > localhost | NULL | /mnt/primary | 2019-06-13 > 18:32:26 > > | > > > > NULL > > > > > > > | NULL | Up | DefaultPrimary | CLUSTER | > NULL > > > > > | > > > > > > > 0 | NULL | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+------+----------------+--------+------------+------------+----------------+--------------+-----------+-----------------+---------------------+---------+-------------+--------+-----------------------+---------+------------+---------+---------------+ > > > > > > > 2 rows in set (0.00 sec) > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 12:51 PM Andrija Panic < > > > > > andrija.pa...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > right... based on logs, it used different sql to search for > > > > ZONE-wide > > > > > > > > storage, execute this one please: > > > > > > > > > > > > > > > > SELECT storage_pool.id, storage_pool.name, > storage_pool.uuid, > > > > > > > > storage_pool.pool_type, storage_pool.created, > > > storage_pool.removed, > > > > > > > > storage_pool.update_time, storage_pool.data_center_id, > > > > > > > storage_pool.pod_id, > > > > > > > > storage_pool.used_bytes, storage_pool.capacity_bytes, > > > > > > > storage_pool.status, > > > > > > > > storage_pool.storage_provider_name, > storage_pool.host_address, > > > > > > > > storage_pool.path, storage_pool.port, storage_pool.user_info, > > > > > > > > storage_pool.cluster_id, storage_pool.scope, > > > storage_pool.managed, > > > > > > > > storage_pool.capacity_iops, storage_pool.hypervisor FROM > > > > storage_pool > > > > > > > WHERE > > > > > > > > storage_pool.data_center_id = 1 AND storage_pool.status = > 'Up' > > > > AND > > > > > > > > storage_pool.scope = 'ZONE' AND storage_pool.hypervisor = > > 'LXC' > > > > AND > > > > > > > > storage_pool.removed IS NULL > > > > > > > > > > > > > > > > This SHOULD return 1 pool, unless there are some funny > > issues... > > > > > > > > I expect the storage_pool.hypervisor might be NULL instead of > > > "LXC" > > > > > > > > > > > > > > > > Anyway, execute the following: "select * from > > > storage_pool;" > > > > > > > > to see what you have in DB... > > > > > > > > > > > > > > > > Best > > > > > > > > Andrija > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 14 Jun 2019 at 18:13, Alejandro Ruiz Bermejo < > > > > > > > > arbermejo0...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Hi and thanks again for your help, > > > > > > > > > > > > > > > > > > So i run the query with both level scopes of storage > > > > > > > > > > > > > > > > > > with storage_pool.scope = 'CLUSTER' > > > > > > > > > > > > > > > > > > output: > > > > > > > > > Empty set (0.00 sec) > > > > > > > > > > > > > > > > > > with storage_pool.scope = 'ZONE' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+---------------------+---------+-------------+----------------+--------+------------+----------------+--------+-----------------------+--------------+-----------------+------+-----------+------------+-------+---------+---------------+------------+ > > > > > > > > > | id | name | uuid | > > > pool_type > > > > > > > > | > > > > > > > > > created | removed | update_time | > data_center_id > > | > > > > > > pod_id | > > > > > > > > > used_bytes | capacity_bytes | status | > storage_provider_name > > | > > > > > > > > host_address > > > > > > > > > | path | port | user_info | cluster_id | scope | > > > > > managed | > > > > > > > > > capacity_iops | hypervisor | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+---------------------+---------+-------------+----------------+--------+------------+----------------+--------+-----------------------+--------------+-----------------+------+-----------+------------+-------+---------+---------------+------------+ > > > > > > > > > | 1 | primary | 672631ab-5c14-3f60-97c6-1d633e60f7bd | > > > > > > > > NetworkFilesystem | > > > > > > > > > 2019-06-06 20:05:26 | NULL | NULL | > 1 > > | > > > > > > NULL | > > > > > > > > > 3413114880 | 984374837248 | Up | DefaultPrimary > > | > > > > > > > 10.8.9.235 > > > > > > > > > | /export/primary | 2049 | NULL | NULL | ZONE | > > > > > 0 | > > > > > > > > > NULL | KVM | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +----+---------+--------------------------------------+-------------------+---------------------+---------+-------------+----------------+--------+------------+----------------+--------+-----------------------+--------------+-----------------+------+-----------+------------+-------+---------+---------------+------------+ > > > > > > > > > 1 row in set (0.00 sec) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 11:57 AM Andrija Panic < > > > > > > > andrija.pa...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Execute this one (from logs) against your DB - does it > > > > > > return/finds a > > > > > > > > > > storage pool ? > > > > > > > > > > > > > > > > > > > > SELECT storage_pool.id, storage_pool.name, > > > storage_pool.uuid, > > > > > > > > > > storage_pool.pool_type, storage_pool.created, > > > > > storage_pool.removed, > > > > > > > > > > storage_pool.update_time, storage_pool.data_center_id, > > > > > > > > > storage_pool.pod_id, > > > > > > > > > > storage_pool.used_bytes, storage_pool.capacity_bytes, > > > > > > > > > storage_pool.status, > > > > > > > > > > storage_pool.storage_provider_name, > > > storage_pool.host_address, > > > > > > > > > > storage_pool.path, storage_pool.port, > > storage_pool.user_info, > > > > > > > > > > storage_pool.cluster_id, storage_pool.scope, > > > > > storage_pool.managed, > > > > > > > > > > storage_pool.capacity_iops, storage_pool.hypervisor FROM > > > > > > storage_pool > > > > > > > > > WHERE > > > > > > > > > > storage_pool.data_center_id = 1 AND storage_pool.status > = > > > 'Up' > > > > > > AND > > > > > > > > > > storage_pool.scope = 'CLUSTER' AND ( > storage_pool.pod_id > > IS > > > > > NULL > > > > > > > OR > > > > > > > > > > storage_pool.pod_id = 1 ) AND ( > storage_pool.cluster_id > > IS > > > > > NULL > > > > > > > OR > > > > > > > > > > storage_pool.cluster_id = 2 ) AND storage_pool.removed > IS > > > > NULL > > > > > > > > > > > > > > > > > > > > tip from thre query - it seems to be looking for CLUSTER > > > level > > > > > not > > > > > > > ZONE > > > > > > > > > > level scope of storage. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 13 Jun 2019 at 16:35, Alejandro Ruiz Bermejo < > > > > > > > > > > arbermejo0...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I really need to get LXC working, I'm very interested > on > > > > using > > > > > CS > > > > > > > > and i > > > > > > > > > > > need to move LXC instances to the new cloud. Thats's > why > > > i'm > > > > > > > > insisting > > > > > > > > > on > > > > > > > > > > > it. So please if you see any errors or can help me i > will > > > > > > > appreciate > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > 1. I haven't used any Tag till now (should i?) > > > > > > > > > > > 2. Already set the TRACE logging level here is the > > content > > > > for > > > > > > an > > > > > > > > > > instance > > > > > > > > > > > createion (https://pastebin.com/Yd9as8uF) > > > > > > > > > > > > > > > > > > > > > > Thanks for the help > > > > > > > > > > > Alejandro > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 12, 2019 at 5:06 PM Andrija Panic < > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Well, this one still doesn't provide any usefull > info. > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you confirm if you have tags set on LXC > Compute > > > > > > Offerings > > > > > > > or > > > > > > > > > any > > > > > > > > > > > > storage used by LXC cluster? > > > > > > > > > > > > 2. Consider change logging level for > > > management-server.log > > > > to > > > > > > > TRACE > > > > > > > > > > mode, > > > > > > > > > > > > which will show also all SQL queries executed, so > > perhaps > > > > we > > > > > > can > > > > > > > > see > > > > > > > > > > why > > > > > > > > > > > it > > > > > > > > > > > > craps out. > > > > > > > > > > > > 3. Not meant to discourage you, but I'm not sure > > weather > > > > LXC > > > > > > code > > > > > > > > > still > > > > > > > > > > > > works with any more recent ACS release - i.e. I'm not > > > aware > > > > > of > > > > > > > any > > > > > > > > > > users, > > > > > > > > > > > > nor do I (personally) remember another user > mentioning > > > LXC > > > > > here > > > > > > > any > > > > > > > > > > time > > > > > > > > > > > > recently. > > > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > Andrija > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 12 Jun 2019 at 15:13, Alejandro Ruiz Bermejo > < > > > > > > > > > > > > arbermejo0...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi this is the full log for one instance cretion > with > > > the > > > > > > same > > > > > > > > > error > > > > > > > > > > ( > > > > > > > > > > > > > https://pastebin.com/HHKnHi36) > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 11, 2019 at 5:10 PM Andrija Panic < > > > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Alejandro, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please send the whole log on pastebin or similar, > > > this > > > > > one > > > > > > > > > doesn't > > > > > > > > > > > > > provide > > > > > > > > > > > > > > enough information... (beside the " Found pools > > > > matching > > > > > > > tags: > > > > > > > > [] > > > > > > > > > > ") > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 11 Jun 2019 at 17:57, Alejandro Ruiz > > Bermejo > > > < > > > > > > > > > > > > > > arbermejo0...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ok, so i registered a new template and unset-ed > > the > > > > HVM > > > > > > > > option > > > > > > > > > > and > > > > > > > > > > > > > > > launched a new instance using this themplate. > > > > > > > > > > > > > > > Now the planner can find the host inside the > LXC > > > > > cluster > > > > > > > but > > > > > > > > it > > > > > > > > > > > seems > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > it can't find now a propper storage pool. I > > already > > > > > have > > > > > > > > > > everything > > > > > > > > > > > > > > > configured as it appears on the installation > > guide > > > > and > > > > > > > > > configured > > > > > > > > > > > the > > > > > > > > > > > > > > > secondary and primary storage on the lxc host. > I > > > > attach > > > > > > and > > > > > > > > > paste > > > > > > > > > > > > here > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > log of the VM creation process > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > root@cloudstack-manager:/home/team# grep -i > > > "unable > > > > > to" > > > > > > > > > > > > > > > > > > /var/log/cloudstack/management/management-server.log > > > > > > > > > > > > > > > 2019-06-11 11:12:59,917 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-19:ctx-23ea94e8 job-51 > > > > ctx-06951cdb) > > > > > > > > > > > > (logid:0afee059) > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > > create a > > > > > > > > > > > > > > > deployment for > VM[User|i-2-11-VM]Scope=interface > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:12:59,917 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-19:ctx-23ea94e8 job-51 > > > > ctx-06951cdb) > > > > > > > > > > > > (logid:0afee059) > > > > > > > > > > > > > > > Unable to create a deployment for > > > VM[User|i-2-11-VM] > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > create > > > > > > > > > > > > > > > a deployment for > > VM[User|i-2-11-VM]Scope=interface > > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:12:59,918 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-19:ctx-23ea94e8 job-51) > > > > > > (logid:0afee059) > > > > > > > > > > Complete > > > > > > > > > > > > > async > > > > > > > > > > > > > > > job-51, jobStatus: FAILED, resultCode: 530, > > result: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable > > > > > > > > > > > > > > > to create a deployment for VM[User|i-2-11-VM]"} > > > > > > > > > > > > > > > 2019-06-11 11:17:33,162 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-22:ctx-e044255f job-57 > > > > ctx-87af2b3f) > > > > > > > > > > > > (logid:0e16600e) > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > > create a > > > > > > > > > > > > > > > deployment for > VM[User|i-2-12-VM]Scope=interface > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:17:33,162 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-22:ctx-e044255f job-57 > > > > ctx-87af2b3f) > > > > > > > > > > > > (logid:0e16600e) > > > > > > > > > > > > > > > Unable to create a deployment for > > > VM[User|i-2-12-VM] > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > create > > > > > > > > > > > > > > > a deployment for > > VM[User|i-2-12-VM]Scope=interface > > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:17:33,162 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-22:ctx-e044255f job-57) > > > > > > (logid:0e16600e) > > > > > > > > > > Complete > > > > > > > > > > > > > async > > > > > > > > > > > > > > > job-57, jobStatus: FAILED, resultCode: 530, > > result: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable > > > > > > > > > > > > > > > to create a deployment for VM[User|i-2-12-VM]"} > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > > create a > > > > > > > > > > > > > > > deployment for > VM[User|i-2-13-VM]Scope=interface > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Unable to create a deployment for > > > VM[User|i-2-13-VM] > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > create > > > > > > > > > > > > > > > a deployment for > > VM[User|i-2-13-VM]Scope=interface > > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62) > > > > > > (logid:5b3ad768) > > > > > > > > > > Complete > > > > > > > > > > > > > async > > > > > > > > > > > > > > > job-62, jobStatus: FAILED, resultCode: 530, > > result: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable > > > > > > > > > > > > > > > to create a deployment for VM[User|i-2-13-VM]"} > > > > > > > > > > > > > > > root@cloudstack-manager:/home/team# grep -i > > > "job-62" > > > > > > > > > > > > > > > > > > /var/log/cloudstack/management/management-server.log > > > > > > > > > > > > > > > 2019-06-11 11:37:03,453 INFO > > > > > > [o.a.c.f.j.i.AsyncJobMonitor] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62) > > > > > > (logid:fb7a7cb5) > > > > > > > > Add > > > > > > > > > > > job-62 > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > job monitoring > > > > > > > > > > > > > > > 2019-06-11 11:37:03,459 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (qtp895947612-374:ctx-4c42085e ctx-d0095860) > > > > > > > (logid:baf4b65e) > > > > > > > > > > > submit > > > > > > > > > > > > > > async > > > > > > > > > > > > > > > job-62, details: AsyncJobVO {id:62, userId: 2, > > > > > accountId: > > > > > > > 2, > > > > > > > > > > > > > > instanceType: > > > > > > > > > > > > > > > VirtualMachine, instanceId: 13, cmd: > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, > > > > > > > > > > > > cmdInfo: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"httpmethod":"GET","templateid":"f2f98443-3793-452d-8e35-aaf631428c50","ctxAccountId":"2","uuid":"56f3829d-722b-45a1-9084-2576eae34068","cmdEventType":"VM.CREATE","diskofferingid":"3dd93d79-62b0-427c-8c15-1050305eb90e","serviceofferingid":"57675b96-98fa-49a3-9437-24f9dbf3fd90","response":"json","ctxUserId":"2","hypervisor":"LXC","zoneid":"5b03beea-ffdd-45ea-84eb-64110f3ff0d0","ctxStartEventId":"147","id":"13","ctxDetails":"{\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.vm.VirtualMachine\":\"56f3829d-722b-45a1-9084-2576eae34068\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.offering.ServiceOffering\":\"57675b96-98fa-49a3-9437-24f9dbf3fd90\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.dc.DataCenter\":\"5b03beea-ffdd-45ea-84eb-64110f3ff0d0\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.offering.DiskOffering\":\"3dd93d79-62b0-427c-8c15-1050305eb90e\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.template.VirtualMachineTemplate\":\"f2f98443-3793-452d-8e35-aaf631428c50\"}","_":"1560267550351"}, > > > > > > > > > > > > > > > cmdVersion: 0, status: IN_PROGRESS, > > processStatus: > > > 0, > > > > > > > > > resultCode: > > > > > > > > > > > 0, > > > > > > > > > > > > > > > result: null, initMsid: 207380201932, > > completeMsid: > > > > > null, > > > > > > > > > > > > lastUpdated: > > > > > > > > > > > > > > > null, lastPolled: null, created: null} > > > > > > > > > > > > > > > 2019-06-11 11:37:03,460 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62) > > > > > > (logid:5b3ad768) > > > > > > > > > > > Executing > > > > > > > > > > > > > > > AsyncJobVO {id:62, userId: 2, accountId: 2, > > > > > instanceType: > > > > > > > > > > > > > VirtualMachine, > > > > > > > > > > > > > > > instanceId: 13, cmd: > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, > > > > > > > > > > > > cmdInfo: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"httpmethod":"GET","templateid":"f2f98443-3793-452d-8e35-aaf631428c50","ctxAccountId":"2","uuid":"56f3829d-722b-45a1-9084-2576eae34068","cmdEventType":"VM.CREATE","diskofferingid":"3dd93d79-62b0-427c-8c15-1050305eb90e","serviceofferingid":"57675b96-98fa-49a3-9437-24f9dbf3fd90","response":"json","ctxUserId":"2","hypervisor":"LXC","zoneid":"5b03beea-ffdd-45ea-84eb-64110f3ff0d0","ctxStartEventId":"147","id":"13","ctxDetails":"{\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.vm.VirtualMachine\":\"56f3829d-722b-45a1-9084-2576eae34068\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.offering.ServiceOffering\":\"57675b96-98fa-49a3-9437-24f9dbf3fd90\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.dc.DataCenter\":\"5b03beea-ffdd-45ea-84eb-64110f3ff0d0\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.offering.DiskOffering\":\"3dd93d79-62b0-427c-8c15-1050305eb90e\",\"interface > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.template.VirtualMachineTemplate\":\"f2f98443-3793-452d-8e35-aaf631428c50\"}","_":"1560267550351"}, > > > > > > > > > > > > > > > cmdVersion: 0, status: IN_PROGRESS, > > processStatus: > > > 0, > > > > > > > > > resultCode: > > > > > > > > > > > 0, > > > > > > > > > > > > > > > result: null, initMsid: 207380201932, > > completeMsid: > > > > > null, > > > > > > > > > > > > lastUpdated: > > > > > > > > > > > > > > > null, lastPolled: null, created: null} > > > > > > > > > > > > > > > 2019-06-11 11:37:03,561 DEBUG > > > > [c.c.n.NetworkModelImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Service SecurityGroup is not supported in the > > > network > > > > > > > id=204 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,562 DEBUG > > > > [c.c.n.NetworkModelImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Service SecurityGroup is not supported in the > > > network > > > > > > > id=204 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,569 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > DeploymentPlanner allocation algorithm: null > > > > > > > > > > > > > > > 2019-06-11 11:37:03,569 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Trying to allocate a host and storage pools > from > > > > dc:1, > > > > > > > > > > > > > > > pod:null,cluster:null, requested cpu: 1000, > > > requested > > > > > > ram: > > > > > > > > > > > 1073741824 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,569 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > Is > > > > > > > > > > > > > > > ROOT volume READY (pool already allocated)?: No > > > > > > > > > > > > > > > 2019-06-11 11:37:03,573 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Deploy avoids pods: [], clusters: [], hosts: [] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,573 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Searching all possible resources under this > > Zone: 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,573 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Listing clusters in order of aggregate > capacity, > > > that > > > > > > have > > > > > > > > > > (atleast > > > > > > > > > > > > one > > > > > > > > > > > > > > > host with) enough CPU and RAM capacity under > this > > > > > Zone: 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,575 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Removing from the clusterId list these clusters > > > from > > > > > > avoid > > > > > > > > set: > > > > > > > > > > [] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,580 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Checking resources in Cluster: 2 under Pod: 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,581 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > > Looking > > > > for > > > > > > > hosts > > > > > > > > in > > > > > > > > > > > dc: 1 > > > > > > > > > > > > > > > pod:1 cluster:2 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,582 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > > > > > > > FirstFitAllocator > > > > > > > > > has > > > > > > > > > > 1 > > > > > > > > > > > > > hosts > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > check for allocation: [Host[-4-Routing]] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,585 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > Found > > 1 > > > > > hosts > > > > > > > for > > > > > > > > > > > > allocation > > > > > > > > > > > > > > > after prioritization: [Host[-4-Routing]] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,585 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > > Looking > > > > for > > > > > > > > > > > speed=1000Mhz, > > > > > > > > > > > > > > > Ram=1024 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,588 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > Host: > > 4 > > > > has > > > > > > cpu > > > > > > > > > > > capability > > > > > > > > > > > > > > > (cpu:8, speed:2666) to support requested CPU: 1 > > and > > > > > > > requested > > > > > > > > > > > speed: > > > > > > > > > > > > > 1000 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,588 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > > Checking > > > > if > > > > > > > host: > > > > > > > > 4 > > > > > > > > > > has > > > > > > > > > > > > > enough > > > > > > > > > > > > > > > capacity for requested CPU: 1000 and requested > > RAM: > > > > > > > > 1073741824 > > > > > > > > > , > > > > > > > > > > > > > > > cpuOverprovisioningFactor: 1.0 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,589 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > > Hosts's > > > > > actual > > > > > > > > total > > > > > > > > > > > CPU: > > > > > > > > > > > > > > 21328 > > > > > > > > > > > > > > > and CPU after applying overprovisioning: 21328 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,589 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) Free > > > CPU: > > > > > > 21328 > > > > > > > , > > > > > > > > > > > > Requested > > > > > > > > > > > > > > CPU: > > > > > > > > > > > > > > > 1000 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) Free > > > RAM: > > > > > > > > > 15751778304 > > > > > > > > > > , > > > > > > > > > > > > > > > Requested RAM: 1073741824 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) Host > > has > > > > > > enough > > > > > > > > CPU > > > > > > > > > > and > > > > > > > > > > > > RAM > > > > > > > > > > > > > > > available > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > STATS: > > > Can > > > > > > alloc > > > > > > > > CPU > > > > > > > > > > > from > > > > > > > > > > > > > > host: > > > > > > > > > > > > > > > 4, used: 0, reserved: 0, actual total: 21328, > > total > > > > > with > > > > > > > > > > > > > > overprovisioning: > > > > > > > > > > > > > > > 21328; requested > > > cpu:1000,alloc_from_last_host?:false > > > > > > > > > > > > > > > ,considerReservedCapacity?: true > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > STATS: > > > Can > > > > > > alloc > > > > > > > > MEM > > > > > > > > > > > from > > > > > > > > > > > > > > host: > > > > > > > > > > > > > > > 4, used: 0, reserved: 0, total: 15751778304; > > > > requested > > > > > > mem: > > > > > > > > > > > > > > > 1073741824,alloc_from_last_host?:false > > > > > > > > > > ,considerReservedCapacity?: > > > > > > > > > > > > true > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) > Found > > a > > > > > > suitable > > > > > > > > > host, > > > > > > > > > > > > > adding > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > list: 4 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,590 DEBUG > > > > > > > [c.c.a.m.a.i.FirstFitAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > ctx-7fcf55ab > > > > > > > > > > > > > > > FirstFitRoutingAllocator) (logid:5b3ad768) Host > > > > > Allocator > > > > > > > > > > > returning 1 > > > > > > > > > > > > > > > suitable hosts > > > > > > > > > > > > > > > 2019-06-11 11:37:03,591 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Checking suitable pools for volume (Id, Type): > > > > > (22,ROOT) > > > > > > > > > > > > > > > 2019-06-11 11:37:03,591 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > need to allocate new storagepool for this > volume > > > > > > > > > > > > > > > 2019-06-11 11:37:03,592 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Calling StoragePoolAllocators to find suitable > > > pools > > > > > > > > > > > > > > > 2019-06-11 11:37:03,592 DEBUG > > > > > > > > > > [o.a.c.s.a.LocalStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > LocalStoragePoolAllocator trying to find > storage > > > pool > > > > > to > > > > > > > fit > > > > > > > > > the > > > > > > > > > > vm > > > > > > > > > > > > > > > 2019-06-11 11:37:03,592 DEBUG > > > > > > > > > > > > > > [o.a.c.s.a.ClusterScopeStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > ClusterScopeStoragePoolAllocator looking for > > > storage > > > > > pool > > > > > > > > > > > > > > > 2019-06-11 11:37:03,592 DEBUG > > > > > > > > > > > > > > [o.a.c.s.a.ClusterScopeStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Looking for pools in dc: 1 pod:1 cluster:2. > > > > Disabled > > > > > > > pools > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > > > > ignored. > > > > > > > > > > > > > > > 2019-06-11 11:37:03,593 DEBUG > > > > > > > > > > > > > > [o.a.c.s.a.ClusterScopeStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Found pools matching tags: [] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,594 DEBUG > > > > > > > > > > > > > > [o.a.c.s.a.ClusterScopeStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > No > > > > > > > > > > > > > > > storage pools available for shared volume > > > allocation, > > > > > > > > returning > > > > > > > > > > > > > > > 2019-06-11 11:37:03,594 DEBUG > > > > > > > > > > > > [o.a.c.s.a.ZoneWideStoragePoolAllocator] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > ZoneWideStoragePoolAllocator to find storage > pool > > > > > > > > > > > > > > > 2019-06-11 11:37:03,595 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > No > > > > > > > > > > > > > > > suitable pools found for volume: > > Vol[22|vm=13|ROOT] > > > > > under > > > > > > > > > > cluster: > > > > > > > > > > > 2 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,595 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > No > > > > > > > > > > > > > > > suitable pools found > > > > > > > > > > > > > > > 2019-06-11 11:37:03,595 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > No > > > > > > > > > > > > > > > suitable storagePools found under this > Cluster: 2 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,598 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Cluster: 1 has HyperVisorType that does not > match > > > the > > > > > VM, > > > > > > > > > > skipping > > > > > > > > > > > > this > > > > > > > > > > > > > > > cluster > > > > > > > > > > > > > > > 2019-06-11 11:37:03,598 DEBUG > > > > > > > > > > [c.c.d.DeploymentPlanningManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Could not find suitable Deployment Destination > > for > > > > this > > > > > > VM > > > > > > > > > under > > > > > > > > > > > any > > > > > > > > > > > > > > > clusters, returning. > > > > > > > > > > > > > > > 2019-06-11 11:37:03,598 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Searching all possible resources under this > > Zone: 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,598 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Listing clusters in order of aggregate > capacity, > > > that > > > > > > have > > > > > > > > > > (atleast > > > > > > > > > > > > one > > > > > > > > > > > > > > > host with) enough CPU and RAM capacity under > this > > > > > Zone: 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:03,600 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Removing from the clusterId list these clusters > > > from > > > > > > avoid > > > > > > > > set: > > > > > > > > > > [1, > > > > > > > > > > > > 2] > > > > > > > > > > > > > > > 2019-06-11 11:37:03,601 DEBUG > > > [c.c.d.FirstFitPlanner] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > No > > > > > > > > > > > > > > > clusters found after removing disabled clusters > > and > > > > > > > clusters > > > > > > > > in > > > > > > > > > > > avoid > > > > > > > > > > > > > > list, > > > > > > > > > > > > > > > returning. > > > > > > > > > > > > > > > 2019-06-11 11:37:03,603 DEBUG > > > > [c.c.v.UserVmManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Destroying vm VM[User|i-2-13-VM] as it failed > to > > > > create > > > > > > on > > > > > > > > Host > > > > > > > > > > > with > > > > > > > > > > > > > > Id:null > > > > > > > > > > > > > > > 2019-06-11 11:37:03,769 DEBUG > > > > > [c.c.c.CapacityManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > VM > > > > > > > > > > > > > > > state transitted from :Stopped to Error with > > event: > > > > > > > > > > > > > > > OperationFailedToErrorvm's original host id: > null > > > new > > > > > > host > > > > > > > > id: > > > > > > > > > > null > > > > > > > > > > > > > host > > > > > > > > > > > > > > id > > > > > > > > > > > > > > > before state transition: null > > > > > > > > > > > > > > > 2019-06-11 11:37:04,152 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = volume count for > Account > > > = 2 > > > > > > > > > Operation = > > > > > > > > > > > > > > > decreasing Amount = 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:04,235 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = primary_storage count > > for > > > > > > Account > > > > > > > = > > > > > > > > 2 > > > > > > > > > > > > > Operation > > > > > > > > > > > > > > = > > > > > > > > > > > > > > > decreasing Amount = 358010880 > > > > > > > > > > > > > > > 2019-06-11 11:37:04,585 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = volume count for > Account > > > = 2 > > > > > > > > > Operation = > > > > > > > > > > > > > > > decreasing Amount = 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:04,710 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = primary_storage count > > for > > > > > > Account > > > > > > > = > > > > > > > > 2 > > > > > > > > > > > > > Operation > > > > > > > > > > > > > > = > > > > > > > > > > > > > > > decreasing Amount = 5368709120 > > > > > > > > > > > > > > > 2019-06-11 11:37:04,894 WARN [o.a.c.alerts] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > AlertType:: 8 | dataCenterId:: 1 | podId:: > null | > > > > > > > clusterId:: > > > > > > > > > > null > > > > > > > > > > > | > > > > > > > > > > > > > > > message:: Failed to deploy Vm with Id: 13, on > > Host > > > > with > > > > > > Id: > > > > > > > > > null > > > > > > > > > > > > > > > 2019-06-11 11:37:04,969 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = user_vm count for > > Account > > > = > > > > 2 > > > > > > > > > Operation > > > > > > > > > > = > > > > > > > > > > > > > > > decreasing Amount = 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,051 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = cpu count for Account > = > > 2 > > > > > > > Operation > > > > > > > > = > > > > > > > > > > > > > decreasing > > > > > > > > > > > > > > > Amount = 1 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,135 DEBUG > > > > > > > > [c.c.r.ResourceLimitManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Updating resource Type = memory count for > Account > > > = 2 > > > > > > > > > Operation = > > > > > > > > > > > > > > > decreasing Amount = 1024 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > > > > > > com.cloud.exception.InsufficientServerCapacityException: > > > > > > > > Unable > > > > > > > > > > to > > > > > > > > > > > > > > create a > > > > > > > > > > > > > > > deployment for > VM[User|i-2-13-VM]Scope=interface > > > > > > > > > > > > > com.cloud.dc.DataCenter; > > > > > > > > > > > > > > > id=1 > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 INFO > > > > > > > > > [o.a.c.a.c.a.v.DeployVMCmdByAdmin] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34beda4d job-62 > > > > ctx-7fcf55ab) > > > > > > > > > > > > (logid:5b3ad768) > > > > > > > > > > > > > > > Unable to create a deployment for > > > VM[User|i-2-13-VM] > > > > > > > > > > > > > > > 2019-06-11 11:37:05,302 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > (API-Job-Executor-24:ctx-34bed
