I ran into this issue a couple of weeks ago and with the assistance of Andy (NCSU) was able to determine the issue is related to one of our blades not being accessible via SSH (management node uses SSH to "check" on the reservation).
Looking through the error log I see where it mentions vclesxi26 isn't accessible via SSH. > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|OS.pm:is_ssh_responding(682)|vclesxi26 > is NOT responding to SSH, SSH command failed, port 22: open, port 24: > closed Hope this helps... Thanks, David Creech ECU - ITCS Academic Technologies Sent from my iPhone On Mar 2, 2013, at 1:46 PM, "Alexander Patterson" <[email protected]> wrote: > VCL::inuse object could not be created and initialized > ------------------------------------------------------------------------ > time: 2013-03-02 10:45:17 > caller: vcld:make_new_child(574) > ( 0) vcld, make_new_child (line: 574) > (-1) vcld, main (line: 350) > ------------------------------------------------------------------------ > management node: vclnfs01.calstate.edu > reservation PID: 4983 > 2013-03-02 10:45:02|4983|108039:101671|inuse|vcld:make_new_child(558)|vcld > environment variable set to 0 for this process > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(643)|image > ID argument was specified: noimage, DataStructure object will contain > image information for the production imagerevision of this image > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(674)|retrieved > data for imagerevision ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(690)|retrieved > data for image ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_mn_os_object(376)|VCL::Module::OS::Linux::ManagementNode > module loaded > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:new(196)|VCL::Module::OS::Linux::ManagementNode > object created for image noimage, address: 18a0b840 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_automethod(847)|data > structure updated, hash path: > $self->request_data->{reservation}{0}{computer}{hostname}, data > identifier: computer_hostname, data: > |4983|108039:101671|inuse| : "vclnfs01.calstate.edu" > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_automethod(847)|data > structure updated, hash path: > $self->request_data->{reservation}{0}{computer}{NODENAME}, data > identifier: computer_node_name, data: > |4983|108039:101671|inuse| : "vclnfs01" > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_automethod(847)|data > structure updated, hash path: > $self->request_data->{reservation}{0}{computer}{SHORTNAME}, data > identifier: computer_short_name, data: > |4983|108039:101671|inuse| : "vclnfs01" > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_automethod(847)|data > structure updated, hash path: > $self->request_data->{reservation}{0}{computer}{IPaddress}, data > identifier: computer_ip_address, data: > |4983|108039:101671|inuse| : "137.145.56.10" > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_mn_os_object(381)|VCL::Module::OS::Linux::ManagementNode > OS object created, address: 18a0b840 > 2013-03-02 10:45:02|4983|108039:101671|inuse|Module.pm:new(192)|VCL::inuse > object created for state inuse, address: 183826f0 > 2013-03-02 10:45:02|4983|108039:101671|inuse|State.pm:initialize(88)|obtained > a database handle for this state process, stored as $ENV{dbh} > 2013-03-02 10:45:02|4983|108039:101671|inuse|State.pm:check_image_os(771)|no > corrections need to be made, not an imaging request, returning 1 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:is_parent_reservation(1004)|returning > true: parent reservation ID for this request: 101671 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|utils.pm:rename_vcld_process(6752)|reservation > count: 1 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|utils.pm:rename_vcld_process(6771)|PARENTIMAGE: > 1 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|utils.pm:rename_vcld_process(6772)|SUBIMAGE: > 0 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|utils.pm:rename_vcld_process(6790)|renamed > process to 'vcld 108039:101671 inuse vmguest-196>vclesxi26 > vmwarewinxp-CSUEBSPSS18withOffice2007124-v9 iy8492' > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:is_parent_reservation(1004)|returning > true: parent reservation ID for this request: 101671 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:is_parent_reservation(1004)|returning > true: parent reservation ID for this request: 101671 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_os_object(308)|VCL::Module::OS::Windows::Version_5::XP > module loaded > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:new(196)|VCL::Module::OS::Windows::Version_5::XP > object created for image vmwarewinxp-CSUEBSPSS18withOffice2007124-v9, > address: 18a253e0 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Windows.pm:initialize(214)|beginning > Windows module initialization > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Windows.pm:initialize(224)|Windows > module initialization complete > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_os_object(314)|VCL::Module::OS::Windows::Version_5::XP > OS object created, address: 18a253e0 > 2013-03-02 10:45:02|4983|108039:101671|inuse|State.pm:initialize(125)|computer > is a VM, attempting to create VM host OS object > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(619)|computer > ID argument was specified, retrieving data for computer ID: 127 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(643)|image > ID argument was specified: 4, DataStructure object will contain image > information for the production imagerevision of this image > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(674)|retrieved > data for imagerevision ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(690)|retrieved > data for image ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_vmhost_os_object(454)|attempting > to load VM host OS module: VCL::Module::OS::Linux::UnixLab (image: 4) > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_vmhost_os_object(460)|VM > host OS module loaded: VCL::Module::OS::Linux::UnixLab > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:new(196)|VCL::Module::OS::Linux::UnixLab > object created for image noimage, address: 18a38ab0 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_vmhost_os_object(465)|VCL::Module::OS::Linux::UnixLab > OS object created, address: 18a38ab0 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:create_provisioning_object(519)|VCL::Module::Provisioning::VMware::VMware > module loaded > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:new(200)|VCL::Module::Provisioning::VMware::VMware > object created for computer vmguest-196, address: 19065120 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:initialize(267)|initializing > VCL::Module::Provisioning::VMware::VMware object > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(619)|computer > ID argument was specified, retrieving data for computer ID: 127 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(643)|image > ID argument was specified: 4, DataStructure object will contain image > information for the production imagerevision of this image > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(674)|retrieved > data for imagerevision ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(690)|retrieved > data for image ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:get_vmhost_datastructure(1397)|created > DataStructure object for VM host: vclesxi26 > 2013-03-02 10:45:02|4983|108039:101671|inuse|VMware.pm:initialize(287)|VM > profile assigned to vclesxi26: VMware ESXI Network Storage > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(619)|computer > ID argument was specified, retrieving data for computer ID: 127 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(643)|image > ID argument was specified: 4, DataStructure object will contain image > information for the production imagerevision of this image > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(674)|retrieved > data for imagerevision ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|DataStructure.pm:_initialize(690)|retrieved > data for image ID: 4 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:get_vmhost_datastructure(1397)|created > DataStructure object for VM host: vclesxi26 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:get_vmhost_api_object(1500)|attempting > to load VMware control module: > VCL::Module::Provisioning::VMware::vSphere_SDK > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:get_vmhost_api_object(1506)|loaded > VMware control module: VCL::Module::Provisioning::VMware::vSphere_SDK > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|Module.pm:new(200)|VCL::Module::Provisioning::VMware::vSphere_SDK > object created for computer vmguest-196, address: 195e4eb0 > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|vSphere_SDK.pm:initialize(98)|vSphere > SDK for Perl does not appear to be installed on this managment node, > unable to load VMware vSphere SDK Perl modules, error: > |4983|108039:101671|inuse| Can't locate VMware/VIRuntime.pm in @INC > (@INC contains: /usr/local/vcl/bin/../../../.. > /usr/local/vcl/bin/../.. /usr/local/vcl/bin/../../.. > /usr/local/vcl/bin/../../../../.. /usr/local/vcl/bin/.. > /usr/local/vcl/bin/../lib > /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl > /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl > /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 > .) at (eval 320510) line 1. > |4983|108039:101671|inuse| BEGIN failed--compilation aborted at (eval > 320510) line 1. > 2013-03-02 > 10:45:02|4983|108039:101671|inuse|VMware.pm:get_vmhost_api_object(1520)|API > object could not be created: > VCL::Module::Provisioning::VMware::vSphere_SDK > D > 2013-03-02 10:45:17|4983|108039:101671|inuse|VMware.pm:initialize(307)|unable > to control OS of VM host vclesxi26 using > VCL::Module::OS::Linux::UnixLab OS object because VM host is not > responding to SSH > |4983|108039:101671|inuse| ---- WARNING ---- > |4983|108039:101671|inuse| 2013-03-02 > 10:45:17|4983|108039:101671|inuse|Module.pm:create_provisioning_object(530)|provisioning > object could not be created, returning 0 > |4983|108039:101671|inuse| ( 0) Module.pm, create_provisioning_object > (line: 530) > |4983|108039:101671|inuse| (-1) State.pm, initialize (line: 136) > |4983|108039:101671|inuse| (-2) Module.pm, new (line: 207) > |4983|108039:101671|inuse| (-3) vcld, make_new_child (line: 567) > |4983|108039:101671|inuse| (-4) vcld, main (line: 350) > |4983|108039:101671|inuse| ---- WARNING ---- > |4983|108039:101671|inuse| 2013-03-02 > 10:45:17|4983|108039:101671|inuse|State.pm:initialize(145)|failed to > create provisioning object > |4983|108039:101671|inuse| ( 0) State.pm, initialize (line: 145) > |4983|108039:101671|inuse| (-1) Module.pm, new (line: 207) > |4983|108039:101671|inuse| (-2) vcld, make_new_child (line: 567) > |4983|108039:101671|inuse| (-3) vcld, main (line: 350) > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|utils.pm:delete_computerloadlog_reservation(6396)|removing > computerloadlog entries matching loadstate = begin > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|utils.pm:delete_computerloadlog_reservation(6443)|deleted > rows from computerloadlog for reservation id=101671 > 2013-03-02 10:45:17|4983|108039:101671|inuse|State.pm:DESTROY(929)|VCL::inuse > process duration: 15 seconds > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|utils.pm:get_management_node_info(5456)|retrieving > current management node info for 'vclnfs01.calstate.edu' from > database, cached data is stale: 63 seconds old > 2013-03-02 10:45:17|4983|108039:101671|inuse|utils.pm:getnewdbh(2474)|unable > to use database handle stored in $ENV{dbh} > 2013-03-02 10:45:17|4983|108039:101671|inuse|utils.pm:getnewdbh(2531)|database > handle stored in $ENV{dbh} > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|utils.pm:get_management_node_info(5603)|retrieved > management node info: 'vclnfs01.calstate.edu' (vclnfs01) > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|DataStructure.pm:get_computer_private_ip_address(1641)|attempting > to retrieve private IP address for computer: vmguest-196 > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|OS.pm:get_file_contents(1897)|retrieved > 531 lines from file on vclnfs01: '/etc/hosts' > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|DataStructure.pm:get_computer_private_ip_address(1697)|returning > IP address from /etc/hosts file: 10.10.11.24 > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|utils.pm:is_inblockrequest(5760)|zero > rows were returned from database select > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|DataStructure.pm:get_image_affiliation_name(2121)|image > owner id: 10 > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|DataStructure.pm:retrieve_user_data(1412)|attempting > to retrieve and store data for user: user.id = '10' > 2013-03-02 > 10:45:17|4983|108039:101671|inuse|DataStructure.pm:retrieve_user_data(1475)|data > has been retrieved for user: wv4973 (id: 10) > > -- > Thanks, > Alex Patterson > User Support Services > Operating System Analyst > California State University, East Bay
