[Yahoo-eng-team] [Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
This is an old bug in GRUB. This commit fixes it: http://git.savannah.gnu.org/cgit/grub.git/commit/?id=e20aa39ea4298011ba716087713cff26c6c52006 To test it, apply it to a GRUB source tree, compile it, install it, and then reinstall the bootloader with "sudo grub-install /dev/sda". ** Also affects: grub Importance: Undecided Status: New ** Changed in: linux-gcp (Ubuntu) Status: New => Won't Fix ** Also affects: grub (Ubuntu) Importance: Undecided Status: New ** No longer affects: grub ** Also affects: grub (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: linux-gcp (Ubuntu Xenial) Importance: Undecided Status: New ** No longer affects: linux-gcp (Ubuntu) ** No longer affects: linux-gcp (Ubuntu Xenial) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT Status in cloud-init: Won't Fix Status in grub package in Ubuntu: New Status in grub source package in Xenial: New Bug description: CPC team has recently converted Xenial images to use GPT instead of MBR. However, after booting an instance that has a disk size of 2049 GB or higher, we hang on the next subsequent boot (Logs indicate it hanging on "Booting Hard Disk 0". This works on Bionic, but what makes it strange is that they have the same kernel revision - 4.15.0-1-37. patrick_viafore@patviafore-test-3072-xenial:~$ lsb_release -rd Description:Ubuntu 16.04.6 LTS Release:16.04 patrick_viafore@patviafore-test-3072-xenial:~$ sudo dpkg -l | grep linux-gcp ii linux-gcp4.15.0.1037.51 amd64Complete Google Cloud Platform (GCP) Linux kernel and headers ii linux-gcp-headers-4.15.0-10374.15.0-1037.39~16.04.1 amd64Header files related to Linux kernel version 4.15.0 To reproduce: 1) Create an image with a disk size of 3072 using a serial that has GPT gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 Reboot the instance 2) It will hang on reboot and you cannot connect 3) Please note that later serials have the GPT change reverted. You can replace xenial with bionic in the above commands to get a bionic instance instead. To test this out in a more slower fashion: 1) Create an image with a disk size of 2048 using a serial that has GPT gcloud compute instances create test-2048-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 2048 2) Resize the disk to 3072 3) Issue growpart /dev/sda 1 4) Issue resize2fs /dev/sda1 5) Issue rsize2fs /dev/sda1 again On the second resize2fs, it tries to resize again, but on a working instance, it says there's nothing to resize. I've tried starting from a Xenial instance and doing a do-release- upgrade to get to bionic and then doing the growpart/resize2fs, but the issue still shows up. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1846511] Re: cloud-init does not run after upgrade due to bad 90_dpkg.cfg
This was an release-branch only issue; it did not affect upstream cloud- init. ** Also affects: cloud-init (Ubuntu) Importance: Undecided Status: New ** Also affects: cloud-init (Ubuntu Eoan) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu Eoan) Status: New => Fix Released ** Changed in: cloud-init (Ubuntu Eoan) Importance: Undecided => Critical ** Changed in: cloud-init Status: Fix Committed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1846511 Title: cloud-init does not run after upgrade due to bad 90_dpkg.cfg Status in cloud-init: Invalid Status in cloud-init package in Ubuntu: Fix Released Status in cloud-init source package in Eoan: Fix Released Bug description: Integration tests on Eoan are failing on the lxd and nocloud kvm platforms. In both cases the test system is not reachable via SSH. LXD log: https://paste.ubuntu.com/p/r2hwqZQfm2/ KVM log: https://paste.ubuntu.com/p/W3N22yBxKN/ This failure mode began with the test run of September 26. On the same system the Disco test run succeeds. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1846511/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1847604] [NEW] Can't reset password for VM instance
Public bug reported: We use Apache Cloud Stack. Since we move to Ubuntu 18.04 it's not possible to change default password of the user. The only issue I can see in the logs is that /sbin/restorecon can't be found, indeed there is no such binary, even if the package "restorecond" is installed. Attachment: relevant logs. ** Affects: cloud-init Importance: Undecided Status: New ** Attachment added: "cloud-init.log" https://bugs.launchpad.net/bugs/1847604/+attachment/5296215/+files/cloud-init.log -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1847604 Title: Can't reset password for VM instance Status in cloud-init: New Bug description: We use Apache Cloud Stack. Since we move to Ubuntu 18.04 it's not possible to change default password of the user. The only issue I can see in the logs is that /sbin/restorecon can't be found, indeed there is no such binary, even if the package "restorecond" is installed. Attachment: relevant logs. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1847604/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1847589] [NEW] Nova compute service restarts with break of rabbitmq server connection
Public bug reported: The nova-compute services restarts every time when the rabbitmq connection is lost and established back again also we can see lost of error logs for timeout to message reply There is no lot of connections observed at rabbitmq end LOgs 2019-10-10 10:34:31,073.073 40972 INFO nova.compute.resource_tracker [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c2 96a2bee1 - default default] Final resource view: name=cmp023.jpe2a.jiocloud.com phys_ram=161156MB used_ram=160844MB phys_disk=886759GB used_disk=0GB total_vcpus=32 used _vcpus=58 pci_stats=[] 2019-10-10 10:35:10,488.488 40972 WARNING oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd 3828eb8c296a2bee1 - default default] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 110] Connection timed out 2019-10-10 10:35:14,253.253 40972 ERROR oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd38 28eb8c296a2bee1 - default default] [833f2e5e-13a2-40d0-96ed-f73be982853d] AMQP server on 10.157.218.30:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker for ced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connectio n closure with reason 'shutdown' 2019-10-10 10:35:15,260.260 40972 ERROR oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd38 28eb8c296a2bee1 - default default] [833f2e5e-13a2-40d0-96ed-f73be982853d] AMQP server on 10.157.218.30:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED 2019-10-10 10:35:16,979.979 40972 INFO oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c296a2bee1 - default default] [833f2e5e-13a2-40d0-96ed-f73be982853d] Reconnected to AMQP server on 10.157.218.32:5672 via [amqp] client with port 56002. 2019-10-10 10:35:39,068.068 40972 ERROR oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c296a2bee1 - default default] [e28c93ab-7cf1-47f9-b37f-933c23881d0b] AMQP server 10.157.218.30:5672 closed the connection. Check login credentials: Socket closed: IOError: Socket closed 2019-10-10 10:35:40,079.079 40972 INFO oslo.messaging._drivers.impl_rabbit [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c296a2bee1 - default default] [e28c93ab-7cf1-47f9-b37f-933c23881d0b] Reconnected to AMQP server on 10.157.218.30:5672 via [amqp] client with port 39778. 2019-10-10 10:35:40,084.084 40972 WARNING oslo.service.loopingcall [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c296a2bee1 - default default] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 1.02 sec 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task [req-0ea34030-6eac-46aa-aac0-aec64d1f72a6 b04e76da460b4d4c900905a361981083 1f9ae90366ce4cd3828eb8c296a2bee1 - default default] Error during ComputeManager._run_image_cache_manager_pass: MessagingTimeout: Timed out waiting for a reply to message ID 7d25d4ef1e4e4417a2c57c2418d5579f 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task Traceback (most recent call last): 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task task(self, context) 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7137, in _run_image_cache_manager_pass 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task filters, expected_attrs=[], use_slave=True) 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task args, kwargs) 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task args=args, kwargs=kwargs) 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2019-10-10 11:35:32,542.542 40972 ERROR oslo_service.periodic_task retry=self.retry)
[Yahoo-eng-team] [Bug 1847586] [NEW] Loading neutron-lib internationalized file
Public bug reported: neutron-lib needs to load internationalized file in /user/share/locale at initialization， The translation of zh_CN has been completed. Refer to https://review.opendev.org/#/c/536011/ ** Affects: neutron Importance: Undecided Assignee: Frank Wang (wangpeihui) Status: In Progress ** Changed in: neutron Assignee: (unassigned) => Frank Wang (wangpeihui) ** Changed in: neutron Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1847586 Title: Loading neutron-lib internationalized file Status in neutron: In Progress Bug description: neutron-lib needs to load internationalized file in /user/share/locale at initialization， The translation of zh_CN has been completed. Refer to https://review.opendev.org/#/c/536011/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1847586/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1847583] [NEW] netplan configuration not re-generated
Public bug reported:
via https://bugs.launchpad.net/cloud-init/+bug/1846535/comments/40
steps to reproduce: (ec2/gcp/azure all have the same issue)
* start a ubuntu 18.04 instance
* update cloud-init to `19.2-36-g059d049c-0ubuntu2~18.04.1`
cloud-init-output.log
```
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'init-local' at Thu,
10 Oct 2019 08:33:00 +. Up 11.53 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'init' at Thu, 10 Oct
2019 08:33:03 +. Up 14.77 seconds.
ci-info: ++Net device
info++
ci-info:
++--++---++---+
ci-info: | Device | Up | Address | Mask | Scope
| Hw-Address|
ci-info:
++--++---++---+
ci-info: | ens5 | True | 172.31.18.122| 255.255.240.0 | global
| 02:a8:af:c9:34:90 |
ci-info: | ens5 | True | fe80::a8:afff:fec9:3490/64 | . | link
| 02:a8:af:c9:34:90 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host
| . |
ci-info: | lo | True | ::1/128 | . | host
| . |
ci-info:
++--++---++---+
ci-info: ++Route IPv4
info++
ci-info:
+---+-+-+-+---+---+
ci-info: | Route | Destination | Gateway | Genmask | Interface |
Flags |
ci-info:
+---+-+-+-+---+---+
ci-info: | 0 | 0.0.0.0 | 172.31.16.1 | 0.0.0.0 |ens5 |
UG |
ci-info: | 1 | 172.31.16.0 | 0.0.0.0 | 255.255.240.0 |ens5 |
U |
ci-info: | 2 | 172.31.16.1 | 0.0.0.0 | 255.255.255.255 |ens5 |
UH |
ci-info:
+---+-+-+-+---+---+
ci-info: +++Route IPv6 info+++
ci-info: +---+-+-+---+---+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +---+-+-+---+---+
ci-info: | 1 | fe80::/64 |:: |ens5 | U |
ci-info: | 3 |local|:: |ens5 | U |
ci-info: | 4 | ff00::/8 |:: |ens5 | U |
ci-info: +---+-+-+---+---+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
root@ip-172-31-18-122:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 ens5 ether routableconfigured
2 links listed.
root@ip-172-31-18-122:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
ethernets:
ens5:
dhcp4: true
dhcp6: true
match:
macaddress: 02:a8:af:c9:34:90
set-name: ens5
```
* update packages and install kernel 5.3.5
* create a image of the now running instance
* start a instance from the image
* unable to connect
* stop the instance
* mount the disk to another running instance
mounting the disk to another instance, one can see that the netplan
configuration was not updated
```
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'modules:config' at
Thu, 10 Oct 2019 08:53:35 +. Up 24.40 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'modules:final' at
Thu, 10 Oct 2019 08:53:36 +. Up 25.32 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 finished at Thu, 10 Oct 2019
08:53:36 +. Datasource DataSourceEc2Local. Up 25.43 seconds
Cloud-init v. 19.2-36-g059d049c-0ubuntu2~18.04.1 running 'init-local' at Thu,
10 Oct 2019 09:15:51 +. Up 16.36 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu2~18.04.1 running 'init' at Thu, 10 Oct
2019 09:15:52 +. Up 17.47 seconds.
ci-info: +++Net device info
ci-info: ++---+---+---+---+---+
ci-info: | Device | Up | Address |Mask | Scope | Hw-Address|
ci-info: ++---+---+---+---+---+
ci-info: | ens5 | False | . | . | . | 06:c1:88:5c:97:58 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo |
