Hey Everyone, I am trying to run the testsuite against a three-node DRBD/KVM Debian Buster cluster setup and currently the TestInstanceDeviceNames test fails with the following error:
Command: ssh gnt-test01 'gnt-instance modify --disk=test_device:remove kvm-test-instance01' Stderr was: Failure: prerequisites not met for this operation: error type: wrong_state, error details: Instance 'kvm-test-instance01' is marked to be up, can't remove volume from a running instance without using hotplug ---- FAILED <function TestInstanceDeviceNames at 0x7f789cb8b2a8>: Command 'gnt-instance modify --disk=test_device:remove kvm-test-instance01' on node gnt-test01 failed, exit code 1 >>>> 2019-07-17 08:52:13.428087 time=0:00:02.451124 <function TestInstanceDeviceNames at 0x7f789cb8b2a8> ---- BLOCK FAILED [RunInstanceTests] Create and exercise instances: Command 'gnt-instance modify --disk=test_device:remove kvm-test-instance01' on node gnt-test01 failed, exit code 1 ^^^^ BLOCK 2019-07-17 08:52:13.428177 time=0:05:30.631610 [RunInstanceTests] Create and exercise instances Adding of the same disk succeeds (that happens right before the step above): <<<< 2019-07-17 08:52:10.976963 start <function TestInstanceDeviceNames at 0x7f789cb8b2a8> Command: ssh gnt-test01 'gnt-instance modify --disk=-1:add,name=test_device,size=512M kvm-test-instance01' Wed Jul 17 08:52:11 2019 * disk 2, size 512M Wed Jul 17 08:52:12 2019 - INFO: Waiting for instance kvm-test-instance01 to sync disks Wed Jul 17 08:52:12 2019 - INFO: Instance kvm-test-instance01's disks are in sync Modified instance kvm-test-instance01 - disk/2 -> add:size=512,mode=rw Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough. Just for the fun of it I modified the code in qa_instance.py to use the --hotplug parameter. Now the command seems to fail but in a way where it does not affect the test evaluation: Command: ssh gnt-test01 'gnt-instance modify --disk=test_device:remove --hotplug kvm-test-instance01' Wed Jul 17 07:53:39 2019 - INFO: Trying to hotplug device... Wed Jul 17 07:53:45 2019 - WARNING: Could not hotplug device: Error while executing backend function: Device disk-a47d61ab-6fb3-4178 should have been removed but is still there Wed Jul 17 07:53:45 2019 - INFO: Continuing execution.. Modified instance kvm-test-instance01 - disk/2 -> remove - disk/2 -> hotplug:failed Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough. Attached to this mail you'll find the json config for the QA configuration and also the current cluster configuration. Packages/Versions used are 2.16.0 from Debian Buster. Only two things were changed: * I manually added the file colors.py to /usr/share/ganeti/testsuite/qa from the sources as its missing in the ganeti-testsuite package (PR has been opened for that) * I changed the sources to disable strict ssh host key checking (yeah, blame me later for that. im currently focused on the tests ;-) ) Maybe someone with some QA experience can help me out with this one :-) On a side node - the cluster is itself virtualised using nested KVM. Cheers, Rudi -- You received this message because you are subscribed to the Google Groups "ganeti-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti-devel/a6115738-12cc-4b47-aa07-0f9ec770252a%40googlegroups.com.
kvm-drbd.json
Description: application/json
Cluster name: staging-cluster.ganeti.org
Cluster UUID: ea5f7b37-2724-45bf-aa14-539a4e2f0218
Creation time: 2019-07-17 08:40:26
Modification time: 2019-07-17 08:46:41
Master node: gnt-test01
Architecture (this node): 64bits (x86_64)
Tags: (none)
Default hypervisor: kvm
Enabled hypervisors: kvm
Hypervisor parameters:
kvm:
acpi: True
boot_order: disk
cdrom2_image_path:
cdrom_disk_type:
cdrom_image_path:
cpu_cores: 0
cpu_mask: all
cpu_sockets: 0
cpu_threads: 0
cpu_type:
disk_aio: threads
disk_cache: default
disk_discard: default
disk_type: paravirtual
floppy_image_path:
initrd_path:
kernel_args: ro
kernel_path:
keymap:
kvm_extra:
kvm_flag:
kvm_path: /usr/bin/kvm
kvm_pci_reservations: 12
machine_version:
mem_path:
migration_bandwidth: 32
migration_caps:
migration_downtime: 30
migration_mode: live
migration_port: 8102
nic_type: paravirtual
reboot_behavior: reboot
root_path: /dev/vda1
scsi_controller_type: lsi
security_domain:
security_model: none
serial_console: True
serial_speed: 38400
soundhw:
spice_bind: 0.0.0.0
spice_image_compression:
spice_ip_version: 0
spice_jpeg_wan_compression:
spice_password_file:
spice_playback_compression: True
spice_streaming_video:
spice_tls_ciphers: HIGH:-DES:-3DES:-EXPORT:-DH
spice_use_tls: True
spice_use_vdagent: True
spice_zlib_glz_wan_compression:
usb_devices:
usb_mouse:
use_chroot: False
use_guest_agent: False
use_localtime: False
user_shutdown: False
vga:
vhost_net: False
virtio_net_queues: 1
vnc_bind_address:
vnc_password_file:
vnc_tls: False
vnc_x509_path:
vnc_x509_verify: False
vnet_hdr: True
OS-specific hypervisor parameters:
OS parameters:
Hidden OSes:
Blacklisted OSes:
Cluster parameters:
candidate pool size: 10
maximal number of jobs running simultaneously: 20
maximal number of jobs simultaneously tracked by the scheduler: 25
mac prefix: aa:00:00
master netdev: eth0
master netmask: 24
use external master IP address setup script: False
lvm volume group: gnt
lvm reserved volumes: (none)
drbd usermode helper: /bin/true
file storage path: /srv/ganeti/file-storage
shared file storage path: /srv/ganeti/shared-file-storage
gluster storage path: /var/run/ganeti/gluster
maintenance of node health: False
uid pool:
default instance allocator: hail
default instance allocator parameters:
primary ip version: 4
preallocation wipe disks: False
OS search path: /srv/ganeti/os, /usr/local/lib/ganeti/os, /usr/lib/ganeti/os,
/usr/share/ganeti/os
ExtStorage Providers search path: /srv/ganeti/extstorage,
/usr/local/lib/ganeti/extstorage, /usr/lib/ganeti/extstorage,
/usr/share/ganeti/extstorage
enabled disk templates: drbd, plain
install image:
instance communication network:
zeroing image:
compression tools:
- gzip
- gzip-fast
- gzip-slow
enabled user shutdown: False
modify ssh setup: True
ssh_key_type: rsa
ssh_key_bits: 2048
Default node parameters:
cpu_speed: 1
exclusive_storage: False
oob_program:
ovs: False
ovs_link:
ovs_name: switch1
spindle_count: 1
ssh_port: 22
Default instance parameters:
default:
always_failover: False
auto_balance: True
maxmem: 128
minmem: 128
spindle_use: 1
vcpus: 1
Default nic parameters:
default:
link: virt-bridge
mode: bridged
vlan:
Default disk parameters:
blockdev:
diskless:
drbd:
c-delay-target: 1
c-fill-target: 0
c-max-rate: 61440
c-min-rate: 4096
c-plan-ahead: 20
data-stripes: 1
disk-barriers: n
disk-custom:
dynamic-resync: False
meta-barriers: False
meta-stripes: 1
metavg: gnt
net-custom:
protocol: C
resync-rate: 61440
ext:
access: kernelspace
file:
gluster:
access: kernelspace
host: 127.0.0.1
port: 24007
volume: gv0
plain:
stripes: 1
rbd:
access: kernelspace
pool: rbd
sharedfile:
Instance policy - limits for instances:
bounds specs:
- max/0:
cpu-count: 48
disk-count: 8
disk-size: 1048576
memory-size: 1048576
nic-count: 8
spindle-use: 8
min/0:
cpu-count: 1
disk-count: 0
disk-size: 512
memory-size: 128
nic-count: 0
spindle-use: 0
std:
cpu-count: 1
disk-count: 1
disk-size: 1024
memory-size: 128
nic-count: 1
spindle-use: 1
allowed disk templates: drbd, plain
vcpu-ratio: 4
spindle-ratio: 32
Data collectors:
cpu-avg-load:
active: True
interval: 5.000s
diskstats:
active: True
interval: 5.000s
drbd:
active: True
interval: 5.000s
inst-status-xen:
active: True
interval: 5.000s
lv:
active: True
interval: 5.000s
xen-cpu-avg-load:
active: True
interval: 5.000s
