Sergey, I tried what you said in your last mail with a fake and a real ESXi.
We built a new 6.5 ESXi host and added it successfully to the cluster: 2019-04-18 16:54:06,969 INFO [c.c.r.ResourceManagerImpl] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Trying to add a new host at http://vcenter/datacenter/cluster1/esx3 in data center 1 2019-04-18 16:54:06,969 INFO [c.c.h.v.VmwareServerDiscoverer] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Discover host. dc: 1, pod: 1, cluster: 16, uri host: vcenter 2019-04-18 16:54:07,073 DEBUG [c.c.h.v.r.VmwareContextFactory] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) initialize VmwareContext. url: https://vcenter/sdk/vimService, username: csadmin@vsphere.local, password: o******* 2019-04-18 16:54:07,440 INFO [c.c.h.v.u.VmwareContext] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) New VmwareContext object, current outstanding count: 1 2019-04-18 16:54:07,602 INFO [c.c.h.v.m.VmwareManagerImpl] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Preparing network on host com.cloud.hypervisor.vmware.util.VmwareContext@506842c3 for vSwitch0 2019-04-18 16:54:07,678 DEBUG [c.c.h.v.m.HostMO] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Port group cloud.private.untagged.0.1-vSwitch0 doesn't exist in vSwitch vSwitch0. Attempting to create port group in this vSwitch. 2019-04-18 16:54:07,765 INFO [c.c.h.v.m.HostMO] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Waiting for network cloud.private.untagged.0.1-vSwitch0 to be ready [...] 2019-04-18 16:54:12,233 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-1:ctx-67c624e1 esx3, cmd: ReadyCommand) (logid:1144ccef) Executing resource ReadyCommand: {"dcId":1,"hostId":69,"wait":0} 2019-04-18 16:54:12,233 TRACE [c.c.h.v.u.VmwareContextPool] (DirectAgent-1:ctx-67c624e1 esx3, cmd: ReadyCommand) (logid:1144ccef) Return a VmwareContext from the idle pool: csadmin@vsphere.local@vcenter. current pool size: 0, outstanding count: 1 2019-04-18 16:54:12,255 TRACE [c.c.h.v.r.VmwareResource] (DirectAgent-1:ctx-67c624e1 esx3, cmd: ReadyCommand) (logid:1144ccef) Reset threadlocal context to null 2019-04-18 16:54:12,255 TRACE [c.c.h.v.r.VmwareResource] (DirectAgent-1:ctx-67c624e1 esx3, cmd: ReadyCommand) (logid:1144ccef) Recycling threadlocal context to pool 2019-04-18 16:54:12,255 TRACE [c.c.h.v.u.VmwareContextPool] (DirectAgent-1:ctx-67c624e1 esx3, cmd: ReadyCommand) (logid:1144ccef) Recycle VmwareContext into idle pool: csadmin@vsphere.local@vcenter, current idle pool size: 1, outstanding count: 1 2019-04-18 16:54:12,255 TRACE [c.c.h.v.r.VmwareResource] (DirectAgent-1:ctx-67c624e1) (logid:1144ccef) End executeRequest(), cmd: ReadyCommand 2019-04-18 16:54:12,255 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-1:ctx-67c624e1) (logid:1144ccef) Seq 69-6034823500676464645: Response Received: 2019-04-18 16:54:12,255 DEBUG [c.c.a.t.Request] (DirectAgent-1:ctx-67c624e1) (logid:1144ccef) Seq 69-6034823500676464645: Processing: { Ans: , MgmtId: 345052098654, via: 69(esx3), Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] } 2019-04-18 16:54:12,255 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-1:ctx-67c624e1) (logid:1144ccef) Seq 69-6034823500676464645: No more commands found 2019-04-18 16:54:12,255 DEBUG [c.c.a.t.Request] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Seq 69-6034823500676464645: Received: { Ans: , MgmtId: 345052098654, via: 69(esx3), Ver: v1, Flags: 110, { ReadyAnswer } } 2019-04-18 16:54:12,255 DEBUG [c.c.h.Status] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) Transition:[Resource state = Enabled, Agent event = Ready, Host id = 69, name = esx3 2019-04-18 16:54:12,263 INFO [c.c.r.ResourceManagerImpl] (qtp510113906-283:ctx-908e5b6a ctx-4eb36509) (logid:1144ccef) server resources successfully discovered by VMware Discover It works but if you restart the management server same behavior as before : no more traffic between the CS controller and the vCenter. I tried to add a new fake ESXi, I see VMware context init and net packets. vCenter fails to add the host because it can't find it and then nothing more happens, no more packets between CS and the vCenter... The only errors I can find in the logs looks like this : INFO [c.c.u.e.CSExceptionErrorCode] (StatsCollector-4:ctx-b2532a68) (logid:98b87cb6) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions Florent ----- Mail original ----- > De: "Sergey Levitskiy" <serg...@hotmail.com> > À: "users" <users@cloudstack.apache.org> > Envoyé: Mercredi 17 Avril 2019 22:45:23 > Objet: Re: Problem with ACS 4.11.2 and vCenter server > It doesn’t look VMware context is getting initialized. This something > fundamental e.g. API version mismatch or missing vmware API Jar file. > You can try re-adding exising vmware DC and cluster using fake cluster name to > force connectivity probing during the addition. If you do please post the > log. > >On 4/17/19, 7:48 AM, "Florent Paillot" <florent.pail...@inria.fr> wrote: > > The only lines I can find in the log are linked to "*VmwareContext*". > > I tried to activate the TRACE level log for com.cloud.hypervisor.vmware > but it > didn't help. > Maybe my conf in log4j is wrong : > > <category name="com.cloud.hypervisor.vmware"> > <priority value="TRACE"/> > </category> > > > Thanks for your help. > > Florent > > ----- Mail original ----- > > De: "Suresh Kumar Anaparti" <sureshkumar.anapa...@gmail.com> > > À: "users" <users@cloudstack.apache.org> > > Envoyé: Mardi 16 Avril 2019 19:25:50 > > Objet: Re: Problem with ACS 4.11.2 and vCenter server > > > Hi Florent, > > > > As soon as CS is started, it will try to connect to the ESXi host with > the > > vCenter details in the zone (Check details in tables vmware_data_center & > > host_details). Grep the below text in the log to see the connection > > progress or errors: > > - *Loading directly connected host* > > - *initialize VmwareContext* > > - *Unable to connect to vSphere server* > > - *can not load directly connected host* > > > > CS keeps trying to connect to the host, by default every 90 secs (agent > > scan interval). > > > > Regards, > > Suresh > > > > On Tue, Apr 16, 2019 at 8:57 PM Sergey Levitskiy <serg...@hotmail.com> > > wrote: > > > >> Please post a full management-server.log without parsing. > >> > >> On 4/16/19, 7:49 AM, "Florent Paillot" <florent.pail...@inria.fr> wrote: > >> > >> I did this on purpose to limit the size of the log. > >> > >> I let the controller run for more than 30mins. Can't find any > revelant > >> error... :( > >> I put some grep results here : > >> https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1 > >> > >> On the production controller (4.9.3), if we have connectivity > problem > >> with the vCenter, I see log with the "com.cloud.hypervisor.vmware.*" > >> context. > >> Is there something to change in the log4j config to see this logs in > >> 4.11 ? > >> > >> The vim25.jar is the right one, after unziping > >> "/usr/share/cloudstack-management/lib/cloudstack-4.11.2.0.jar" in > >> com/vmware/vim25/vim.wsdl, I see " <soap:operation > >> soapAction="urn:vim25/6.7" style="document" />". > >> > >> > >> Florent > >> > >> ----- Mail original ----- > >> > De: "Suresh Kumar Anaparti" <sureshkumar.anapa...@gmail.com> > >> > À: "users" <users@cloudstack.apache.org> > >> > Envoyé: Mardi 16 Avril 2019 14:22:28 > >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server > >> > >> > Florent, The log doesn't have any CS to vCenter connectivity > >> details. The > >> > CS management server was started and stopped in 2 mins. It needs > >> some time > >> > to load modules and connect to the hosts and storages. Wait for > some > >> more > >> > time for the connectivity details in the log. > >> > > >> > You can install vim25.jar for 6.7 version and give a try. > >> > > >> > - Suresh > >> > > >> > On Tue, Apr 16, 2019 at 5:34 PM Florent Paillot < > >> florent.pail...@inria.fr> > >> > wrote: > >> > > >> >> Here is the log : https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1 > >> >> It's a complete 3 minutes start/run/stop. > >> >> > >> >> Thanks for the info about "cs.vmware.api.version*". For 4.11.2, > it > >> seems > >> >> that vSphere 6.7 is the target : > >> >> pom.xml: <cs.vmware.api.version>6.7</cs.vmware.api.version> > >> >> > >> >> > >> >> Florent > >> >> > >> >> ----- Mail original ----- > >> >> > De: "Suresh Kumar Anaparti" <sureshkumar.anapa...@gmail.com> > >> >> > À: "users" <users@cloudstack.apache.org> > >> >> > Envoyé: Mardi 16 Avril 2019 13:05:20 > >> >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server > >> >> > >> >> > Hi Florent, > >> >> > > >> >> > Log is missing. I think attachments are ignored. Either > provide a > >> link to > >> >> > access the log or paste the logs in email (few logs of the > issue). > >> >> > > >> >> > Irrespective of the vCenter version (hard coded in the build > param > >> >> > *cs.vmware.api.version*), CS would try to connect to the > vCenter > >> server. > >> >> > > >> >> > - Suresh > >> >> > > >> >> > > >> >> > On Tue, Apr 16, 2019 at 2:44 PM Florent Paillot < > >> >> florent.pail...@inria.fr> > >> >> > wrote: > >> >> > > >> >> >> Thanks for the answers, > >> >> >> > >> >> >> Paul : I know that the vCenter 6.7 is not officialy supported > >> but my > >> >> >> colleague reported that it worked with 4.9.3 . > >> >> >> When you say "cloudstack won't recognise it", is it hard > coded ? > >> If the > >> >> >> the vCenter auth API doesn't change between 6.5 and 6.7 it > >> should work > >> >> or > >> >> >> at least try to connect to the vCenter. > >> >> >> > >> >> >> I'm agree with you Suresh, even if CS can't communicate with > the > >> vCenter > >> >> >> we should see something in the logs or in the packet capture. > >> >> >> > >> >> >> I am able to ping and telnet from the CS controller to the > >> vCenter on > >> >> port > >> >> >> 443 (and I see the packet with tcpdump). > >> >> >> > >> >> >> The log is in attachement. > >> >> >> > >> >> >> Florent > >> >> >> > >> >> >> > >> >> >> ----- Mail original ----- > >> >> >> > De: "Suresh Kumar Anaparti" <sureshkumar.anapa...@gmail.com> > >> >> >> > À: "users" <users@cloudstack.apache.org> > >> >> >> > Envoyé: Lundi 15 Avril 2019 17:54:00 > >> >> >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server > >> >> >> > >> >> >> > Hi Florent, What was the CS behavior & vCenter/ESXi hosts > >> connectivity > >> >> >> > status before upgrade to 4.11.2 from 4.9.3? Have you > upgraded > >> vCenter > >> >> as > >> >> >> > well? Are you able to ping vCenter from CS management > server? > >> >> >> > > >> >> >> > Even vCenter 6.7 is not recognised, CS would try to connect > >> with the > >> >> >> > details specified, which is logged. Have you enabled debug > >> logs? > >> >> >> > > >> >> >> > - Suresh > >> >> >> > > >> >> >> > On Mon, Apr 15, 2019 at 7:38 PM Paul Angus < > >> paul.an...@shapeblue.com> > >> >> >> wrote: > >> >> >> > > >> >> >> >> Ah. vCenter 6.7 isn't supported yet, cloudstack won't > >> recognise it. > >> >> >> >> > >> >> >> >> paul.an...@shapeblue.com > >> >> >> >> www.shapeblue.com > >> >> >> >> Amadeus House, Floral Street, London WC2E 9DPUK > >> >> >> >> @shapeblue > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> -----Original Message----- > >> >> >> >> From: Florent Paillot <florent.pail...@inria.fr> > >> >> >> >> Sent: 15 April 2019 14:26 > >> >> >> >> To: users <users@cloudstack.apache.org> > >> >> >> >> Subject: Re: Problem with ACS 4.11.2 and vCenter server > >> >> >> >> > >> >> >> >> Hi Suresh, > >> >> >> >> The vCenter is a 6.7u2 (last build). > >> >> >> >> > >> >> >> >> There is no issue log, that the problem... I don't see any > >> error > >> >> message > >> >> >> >> concerning a failed connection to the vCenter and CS does > not > >> seems > >> >> to > >> >> >> send > >> >> >> >> any packet to the vCenter. > >> >> >> >> > >> >> >> >> Florent > >> >> >> >> > >> >> >> >> > >> >> >> >> ----- Mail original ----- > >> >> >> >> > De: "Suresh Kumar Anaparti" > <sureshkumar.anapa...@gmail.com > >> > > >> >> >> >> > À: "users" <users@cloudstack.apache.org> > >> >> >> >> > Envoyé: Lundi 15 Avril 2019 15:10:44 > >> >> >> >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server > >> >> >> >> > >> >> >> >> > Hi Florent, > >> >> >> >> > > >> >> >> >> > Can you share the vCenter version and build details. > Also, > >> the > >> >> >> >> > cloudstack management server log with the issue logs > while > >> >> connecting > >> >> >> to > >> >> >> >> vCenter. > >> >> >> >> > > >> >> >> >> > - Suresh > >> >> >> >> > > >> >> >> >> > On Mon, Apr 15, 2019 at 6:24 PM Paul Angus < > >> >> paul.an...@shapeblue.com> > >> >> >> >> wrote: > >> >> >> >> > > >> >> >> >> >> Assuming that you have updated the core code to > recognise > >> and use > >> >> >> >> 'newer' > >> >> >> >> >> libraries/SDKs than the ones currently in use (and > >> thoroughly > >> >> tested > >> >> >> >> >> it) then, please do create a pull request. Otherwise, I > >> >> personally > >> >> >> >> >> would advise that you leave things 'as-is' to get the > most > >> stable > >> >> >> >> experience. > >> >> >> >> >> > >> >> >> >> >> I don't *think* that you need to run > >> >> >> >> >> > >> >> >> >> >> mvn clean > >> >> >> >> >> mvn install -Dnoredist > >> >> >> >> >> > >> >> >> >> >> *After* installing the non-oss components. > >> >> >> >> >> > >> >> >> >> >> You might want to do a mvn clean before you start. > >> >> >> >> >> > >> >> >> >> >> It's been a while since I've done it manually now. > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> paul.an...@shapeblue.com > >> >> >> >> >> www.shapeblue.com > >> >> >> >> >> Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> -----Original Message----- > >> >> >> >> >> From: Florent Paillot <florent.pail...@inria.fr> > >> >> >> >> >> Sent: 15 April 2019 13:17 > >> >> >> >> >> To: users <users@cloudstack.apache.org> > >> >> >> >> >> Subject: Re: Problem with ACS 4.11.2 and vCenter server > >> >> >> >> >> > >> >> >> >> >> Yes, this is the list of the files in deps/ : > >> >> >> >> >> cloud-iControl.jar cloud-netscaler-sdx.jar > >> iControl-12.1.0.jar > >> >> >> >> >> install-non-oss.sh manageontap.jar vim25_51.jar > >> vim25_55.jar > >> >> >> >> >> vim25_60.jar vim25_65.jar vim25_67.jar > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> I had to add the following lines to "install-non-oss.sh" > >> to get > >> >> the > >> >> >> >> build > >> >> >> >> >> pass : > >> >> >> >> >> > >> >> >> >> >> mvn install:install-file -Dfile=vim25_67.jar > >> >> >> >> >> -DgroupId=com.cloud.com.vmware -DartifactId=vmware-vim25 > >> >> >> >> -Dversion=6.7 > >> >> >> >> >> -Dpackaging=jar > >> >> >> >> >> mvn install:install-file -Dfile=iControl-12.1.0.jar > >> >> >> >> >> -DgroupId=com.cloud.com.f5 -DartifactId=icontrol > >> >> >> >> >> -Dversion=12.1 -Dpackaging=jar > >> >> >> >> >> > >> >> >> >> >> I can open a PR or an issue if it makes sense. > >> >> >> >> >> > >> >> >> >> >> Florent > >> >> >> >> >> > >> >> >> >> >> -- > >> >> >> >> >> Florent PAILLOT > >> >> >> >> >> SESI - Cellule architecture > >> >> >> >> >> > >> >> >> >> >> INRIA Bordeaux -Sud-Ouest > >> >> >> >> >> 05.24.57.41.25 > >> >> >> >> >> > >> >> >> >> >> ----- Mail original ----- > >> >> >> >> >> > De: "Paul Angus" <paul.an...@shapeblue.com> > >> >> >> >> >> > À: "users" <users@cloudstack.apache.org> > >> >> >> >> >> > Envoyé: Lundi 15 Avril 2019 14:10:21 > >> >> >> >> >> > Objet: RE: Problem with ACS 4.11.2 and vCenter server > >> >> >> >> >> > >> >> >> >> >> > When packaging, did you install all of the nonoss > >> libraries > >> >> into > >> >> >> the > >> >> >> >> >> > build before packaging? including > vmware-vim25-6.5.jar > >> >> >> >> >> > > >> >> >> >> >> > paul.an...@shapeblue.com > >> >> >> >> >> > www.shapeblue.com > >> >> >> >> >> > Amadeus House, Floral Street, London WC2E 9DPUK > >> @shapeblue > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > -----Original Message----- > >> >> >> >> >> > From: Dag Sonstebo <dag.sonst...@shapeblue.com> > >> >> >> >> >> > Sent: 15 April 2019 12:43 > >> >> >> >> >> > To: users@cloudstack.apache.org > >> >> >> >> >> > Subject: Re: Problem with ACS 4.11.2 and vCenter > server > >> >> >> >> >> > > >> >> >> >> >> > Florent, > >> >> >> >> >> > > >> >> >> >> >> > Not aware of any issues but I'm no packaging expert so > >> will let > >> >> >> others > >> >> >> >> >> > respond to this. > >> >> >> >> >> > > >> >> >> >> >> > However - any reason you're not just using the RPM > >> repositories > >> >> >> rather > >> >> >> >> >> > than packaging yourself? > >> >> >> >> >> > > >> >> >> >> >> > Regards, > >> >> >> >> >> > Dag Sonstebo > >> >> >> >> >> > Cloud Architect > >> >> >> >> >> > ShapeBlue > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> >On 15/04/2019, 11:46, "Florent Paillot" < > >> >> florent.pail...@inria.fr> > >> >> >> >> >> wrote: > >> >> >> >> >> > > >> >> >> >> >> > Hi everyone, > >> >> >> >> >> > We have a problem with our test environment running > >> ACS > >> >> 4.11.2 > >> >> >> on > >> >> >> >> >> CentOS 7, not > >> >> >> >> >> > communicating with a vCenter (upgraded from 4.9.3 > to > >> 4.11.2 > >> >> and > >> >> >> >> >> CentOS 6 o > >> >> >> >> >> > CentOS 7). We have no problems with our KVM host. > >> >> >> >> >> > Our production is using 4.9.3 o, Centos 6 and > >> everything is > >> >> OK. > >> >> >> >> >> > > >> >> >> >> >> > I used tcpdump and see nothing going out from the > >> management > >> >> >> server > >> >> >> >> >> to the > >> >> >> >> >> > vCenter or the ESXi host. > >> >> >> >> >> > > >> >> >> >> >> > Could it be a package building problem or a vmware > >> module > >> >> >> loading? > >> >> >> >> >> > > >> >> >> >> >> > I used the following commands to build the > packages : > >> >> >> >> >> > install-non-oss.sh > >> >> >> >> >> > mvn clean > >> >> >> >> >> > mvn install -Dnoredist > >> >> >> >> >> > package.sh -p noredist -d centos7 > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > On the production server I see a : > >> >> >> >> >> > > >> >> >> >> >> > >> >> >> >> > >> >> >> > >> >> > >> > /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/vmware-vim25-6.0.jar > >> >> >> >> >> > file. > >> >> >> >> >> > I can't find it on the test environnement, is it > >> linked to > >> >> >> jetty ? > >> >> >> >> >> > > >> >> >> >> >> > Any help is welcome :) > >> >> >> >> >> > Regards, > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > dag.sonst...@shapeblue.com > >> >> >> >> >> > www.shapeblue.com > >> >> >> >> >> > Amadeus House, Floral Street, London WC2E 9DPUK > >> @shapeblue > >>
