[Group.of.nepali.translators] [Bug 1616116] Re: Unrecoverable resyncs if DB is restored from backup
** Changed in: landscape-client
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1616116
Title:
Unrecoverable resyncs if DB is restored from backup
Status in Landscape Client:
Fix Released
Status in landscape-client package in Ubuntu:
Fix Released
Status in landscape-client source package in Trusty:
Fix Released
Status in landscape-client source package in Xenial:
Fix Released
Status in landscape-client source package in Bionic:
Fix Released
Status in landscape-client source package in Cosmic:
Fix Released
Bug description:
[Impact]
* In some cases generally involving backups/restore, client would get
inconsistent package data and keep that data upon resync, thus getting
stuck in a resync loop. This usually gets noticed through the stress
it adds on the server and though logs which grow abnormally.
[Test Case]
* deploy landscape-server quickstart from ppa:landscape/18.03
* register client against server. wait for package info
* pg_dumpall
* add a repo and wait for new package to show on in landscape.
* restore the postgres backup.
* run ./scripts/hash_id_databases.sh from the server to complete
the restore.
* trigger a package install from the new repo to create some package
info to update
* client should resync once then will re-fetch hash on the next run.
[Regression Potential]
* Modified code is used only during resync operations and removes
cached data when the client state is deemed inconsistent.
* In the unlikely event the code is called outside of the expected
cases, the end result would be limited to the package-monitor
having to re-download the hash-id databases, which shouldn't
cause issues as that is the behaviour at client registration.
[Other Info]
* Other cases than server restores have been noticed to generate the
bug but they are far less common.
[Original description]
Landscape with live clients cannot handle a DB restore to a point in
the past.
The scenario is Landscape running as usual, with live clients,
restoring to a DB backup taken in the past. After the service ir
brought up again with this data, clients will start resyncing and
becoming wedged with all sorts of tracebacks on the message server.
I left such a scenario running overnight, hoping that eventually the
resyncs would settle down and everything recover, but that didn't
happen. The resyncs continued, in the packages scope.
An interesting one in particular was this:
Aug 22 21:46:26 message-server-2 ERR Error handling message
'operation-result' for computer 104: {'status': 6, 'timestamp': 1471901963,
'result-text': u'Mon Aug 22 21:39:23 UTC 2016\n', 'api': '3.3', 'operation-id':
533, 'type': 'operation-result'}#012Traceback (most recent call last):#012
File "/opt/canonical/landscape/canonical/landscape/message/apis.py", line 358,
in _process_messages#012self.handle(message["type"], message)#012 File
"/opt/canonical/landscape/canonical/message/api.py", line 66, in handle#012
return handler(type, body)#012 File
"/opt/canonical/landscape/canonical/message/handler.py", line 30, in
__call__#012return function(self.message_api, type, body)#012 File
"/opt/canonical/landscape/canonical/lib/arguments.py", line 79, in
replacement#012return original(*new_args, **new_kwargs)#012 File
"/opt/canonical/landscape/canonical/landscape/message/handlers/activity.py",
line 32, in handle_activity_result#012activity.succeed(code=result_code,
text=result_text)#012AttributeError: 'NoneType' object has no attribute
'succeed'
That was about an activity that had been delivered already, but did
not exist in the restored DB.
To manage notifications about this bug go to:
https://bugs.launchpad.net/landscape-client/+bug/1616116/+subscriptions
___
Mailing list: https://launchpad.net/~group.of.nepali.translators
Post to : group.of.nepali.translators@lists.launchpad.net
Unsubscribe : https://launchpad.net/~group.of.nepali.translators
More help : https://help.launchpad.net/ListHelp
[Group.of.nepali.translators] [Bug 1583276] Re: glance restarted during image upload, image stuck in "saving" state
** Changed in: landscape
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1583276
Title:
glance restarted during image upload, image stuck in "saving" state
Status in Glance - Simplestreams Sync Charm:
Invalid
Status in Landscape Server:
Fix Released
Status in Landscape Server 16.06 series:
Fix Released
Status in simplestreams:
Fix Released
Status in simplestreams package in Ubuntu:
Fix Released
Status in simplestreams source package in Xenial:
Won't Fix
Status in simplestreams source package in Bionic:
Fix Released
Status in simplestreams source package in Disco:
Fix Released
Bug description:
[Impact]
CI run: https://ci.lscape.net/job/landscape-system-tests/2451/
gs3 log attached. Below some excerpts:
It looks like the problem started when it uploaded the first image (14.04) to
glance. That failed:
WARNING * 05-18 13:36:48 [PID:10259] * root * No rabbitmq connection
available for msg{'status': 'Syncing', 'message':
'ubuntu-trusty-14.04-amd64-server-20160516 99%\n
DEBUG * 05-18 13:36:50 [PID:10259] * glanceclient.common.http * curl -i
-X POST -H 'x-image-meta-property-source_content_id:
com.ubuntu.cloud:released:download' -H 'INFO * 05-18 13:37:01 [PID:10364]
* root * glance-simplestreams-sync started.
INFO * 05-18 13:37:01 [PID:10364] * root *
/var/run/glance-simplestreams-sync.pid is locked, exiting
ERROR * 05-18 13:37:07 [PID:10259] * root * Glance Client exception
during do_sync:
Error communicating with http://10.96.10.146:9292 [Errno 32] Broken pipe
Will continue polling.
Traceback (most recent call last):
File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py",
line 444, in main
do_sync(charm_conf, status_exchange)
File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py",
line 215, in do_sync
tmirror.sync(smirror, path=initial_path)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py",
line 91, in sync
return self.sync_index(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py",
line 254, in sync_index
self.sync(reader, path=epath)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py",
line 89, in sync
return self.sync_products(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py",
line 341, in sync_products
self.insert_item(item, src, target, pgree, ipath_cs)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py",
line 303, in insert_item
ret = self.gclient.images.create(**create_kwargs)
File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line
253, in create
'POST', '/v1/images', headers=hdrs, body=image_data)
File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line
289, in raw_request
return self._http_request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line
235, in _http_request
raise exc.CommunicationError(message=message)
CommunicationError: Error communicating with http://10.96.10.146:9292 [Errno
32] Broken pipe
INFO * 05-18 13:37:07 [PID:10259] * root * sync done.
INFO * 05-18 13:38:01 [PID:10372] * root * glance-simplestreams-sync
started.
[Test Case]
Found evidence in the juju logs for glance/0 that it restarted right at
13:37:07:
2016-05-18 13:37:07 INFO ceph-relation-changed ^MReading state information...
0%^MReading state information... 0%^MReading state information... Done
2016-05-18 13:37:07 INFO juju-log ceph:50: Loaded template from
templates/ceph.conf
2016-05-18 13:37:07 INFO juju-log ceph:50: Rendering from template: ceph.conf
2016-05-18 13:37:07 INFO juju-log ceph:50: Wrote template
/var/lib/charm/glance/ceph.conf.
2016-05-18 13:37:07 INFO ceph-relation-changed glance-api stop/waiting
2016-05-18 13:37:07 INFO ceph-relation-changed glance-api start/running,
process 62839
glance/1 had its last restart later:
2016-05-18 13:32:01 INFO ceph-relation-changed glance-api start/running,
process 31045
glance/2 at that time too:
2016-05-18 13:32:00 INFO ceph-relation-changed glance-api start/running,
process 30584
In gs3, a few log entries later, we can see that 14.04 is in state "saving"
in glance:
(...)
{"images": [{"status": "saving", "deleted_at": null, "name":
"auto-sync/ubuntu-trusty-14.04-amd64-server-20160516-disk1.img", "deleted":
false, "container_format": "bare
(...)
It remains in this state throughout the logs.
gs3 then proceeds to download 16.04, upload it to glance, and then
finally publishes the streams. It's unknown if 14.04 is part of this
publication or not.
[Group.of.nepali.translators] [Bug 1886491] Re: Applying package profile on Xenial doesn't work
Subsequent releases already had this corrected. I have updated the task status. ** Changed in: landscape-client (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1886491 Title: Applying package profile on Xenial doesn't work Status in landscape-client package in Ubuntu: Fix Released Status in landscape-client source package in Xenial: Fix Committed Bug description: [Impact] * Users trying to apply a landscape package profile with a recent version of python-apt on xenial will get a stuck activity and an incomplete sources.list.d entry. * The change adds a [trusted=yes] option to temporary local sources created by landscape. Global sources and apt options [Test Case] * install landscape-client and register it. * create and associate a package profile. * the packages are installed and /var/log/landscape/package- changer.log has no errors [Regression Potential] * This is a backport of fixes already released since yakkety. The change has been released for a long time. * Change is agnostic of whether the issue is present or not and should work no matter the apt configuration. * The affected code path is limited to profiles, but is also used a lot in unit tests fixtures, thus making it reasonably exercised. [Other Info] * This change was not previously required on xenial, as it was not required. Acquire::AllowInsecureRepositories was set to be backward compatible and was accepted by python-apt. Changes in python-apt now looks for APT::Get::AllowUnauthenticated. * We fix landcape-client instead of raising the python-apt regression because the new behaviour is consistent with later releases, which we already handle. Also, it decouples landscape from requiring specific configuration. [Original Description] * install landscape-server-quickstart and landscape-api * Create a package profile "minimum packages" and add basic packages (byobu, vim, ubuntu-cloud-keyring). Enable for all computers * create a xenial container or machine and install/configure landscape-client Note that the apply package profile gets stuck and doesn't progress. On the container that was created, running apt update shows: sudo apt update Get:1 file:/var/lib/landscape/client/package/binaries ./ InRelease Ign:1 file:/var/lib/landscape/client/package/binaries ./ InRelease Get:2 file:/var/lib/landscape/client/package/binaries ./ Release Err:2 file:/var/lib/landscape/client/package/binaries ./ Release File not found - /var/lib/landscape/client/package/binaries/./Release (2: No such file or directory) Get:3 http://mirror/ubuntu xenial InRelease [247 kB] Hit:4 http://mirror/ubuntu xenial-updates InRelease Hit:5 http://mirror/ubuntu xenial-backports InRelease Hit:6 http://mirror/ubuntu xenial-security InRelease Reading package lists... Done E: The repository 'file:/var/lib/landscape/client/package/binaries ./ Release' does not have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. apt-cache policy landscape-client landscape-client: Installed: 16.03-0ubuntu2.16.04.7 Candidate: 16.03-0ubuntu2.16.04.7 Version table: *** 16.03-0ubuntu2.16.04.7 500 500 http://mirror/ubuntu xenial-updates/main amd64 Packages 100 /var/lib/dpkg/status 16.03-0ubuntu2 500 500 http://mirror/ubuntu xenial/main amd64 Packages This can be easily worked around by: * Remove the file /etc/apt/sources.list.d/_landscape-internal-facade.list * add-apt-repository ppa:landscape/19.01 * apt install landscape-client apt-cache policy landscape-client landscape-client: Installed: 18.03~1070~git.8e52e1d~ubuntu16.04.1 Candidate: 18.03~1070~git.8e52e1d~ubuntu16.04.1 Version table: *** 18.03~1070~git.8e52e1d~ubuntu16.04.1 500 500 http://ppa.launchpad.net/landscape/19.01/ubuntu xenial/main amd64 Packages 100 /var/lib/dpkg/status 16.03-0ubuntu2.16.04.7 500 500 http://mirror.glacier.dev/ubuntu xenial-updates/main amd64 Packages 16.03-0ubuntu2 500 500 http://mirror.glacier.dev/ubuntu xenial/main amd64 Packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/landscape-client/+bug/1886491/+subscriptions ___ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launchpad.net/ListHelp
[Group.of.nepali.translators] [Bug 1886491] Re: Applying package profile on Xenial doesn't work
It looks like python-apt (1.1.0~beta1ubuntu0.16.04.7) created a regression and elects to ignore Acquire::AllowInsecureRepositories in favor of APT::Get::AllowUnauthenticated. The simpler approach would likely be to backport the [trusted=true] fix we've got working with later releases, instead of trying anything related to the global apt config. ** Also affects: landscape-client (Ubuntu) Importance: Undecided Status: New ** Also affects: landscape-client (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: landscape-client (Ubuntu Xenial) Status: New => Confirmed ** No longer affects: landscape-client -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1886491 Title: Applying package profile on Xenial doesn't work Status in landscape-client package in Ubuntu: New Status in landscape-client source package in Xenial: Confirmed Bug description: [Impact] * Users trying to apply changes to package profiles through landscape will get an error in the activity. This will result in incomplete apt source list on those clients. [Test Case] * install landscape-server-quickstart and landscape-api * Create a package profile "minimum packages" and add basic packages (byobu, vim, ubuntu-cloud-keyring). Enable for all computers * create a xenial container or machine and install/configure landscape-client Note that the apply package profile gets stuck and doesn't progress. On the container that was created, running apt update shows: sudo apt update Get:1 file:/var/lib/landscape/client/package/binaries ./ InRelease Ign:1 file:/var/lib/landscape/client/package/binaries ./ InRelease Get:2 file:/var/lib/landscape/client/package/binaries ./ Release Err:2 file:/var/lib/landscape/client/package/binaries ./ Release File not found - /var/lib/landscape/client/package/binaries/./Release (2: No such file or directory) Get:3 http://mirror/ubuntu xenial InRelease [247 kB] Hit:4 http://mirror/ubuntu xenial-updates InRelease Hit:5 http://mirror/ubuntu xenial-backports InRelease Hit:6 http://mirror/ubuntu xenial-security InRelease Reading package lists... Done E: The repository 'file:/var/lib/landscape/client/package/binaries ./ Release' does not have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. apt-cache policy landscape-client landscape-client: Installed: 16.03-0ubuntu2.16.04.7 Candidate: 16.03-0ubuntu2.16.04.7 Version table: *** 16.03-0ubuntu2.16.04.7 500 500 http://mirror/ubuntu xenial-updates/main amd64 Packages 100 /var/lib/dpkg/status 16.03-0ubuntu2 500 500 http://mirror/ubuntu xenial/main amd64 Packages This can be easily worked around by: * Remove the file /etc/apt/sources.list.d/_landscape-internal-facade.list * add-apt-repository ppa:landscape/19.01 * apt install landscape-client apt-cache policy landscape-client landscape-client: Installed: 18.03~1070~git.8e52e1d~ubuntu16.04.1 Candidate: 18.03~1070~git.8e52e1d~ubuntu16.04.1 Version table: *** 18.03~1070~git.8e52e1d~ubuntu16.04.1 500 500 http://ppa.launchpad.net/landscape/19.01/ubuntu xenial/main amd64 Packages 100 /var/lib/dpkg/status 16.03-0ubuntu2.16.04.7 500 500 http://mirror.glacier.dev/ubuntu xenial-updates/main amd64 Packages 16.03-0ubuntu2 500 500 http://mirror.glacier.dev/ubuntu xenial/main amd64 Packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/landscape-client/+bug/1886491/+subscriptions ___ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launchpad.net/ListHelp
[Group.of.nepali.translators] [Bug 1685885] Re: Extreme RAM and SWAP usage
** No longer affects: landscape-client (Ubuntu Trusty) -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1685885 Title: Extreme RAM and SWAP usage Status in Landscape Client: Fix Committed Status in landscape-client package in Ubuntu: In Progress Status in landscape-client source package in Xenial: In Progress Status in landscape-client source package in Bionic: In Progress Status in landscape-client source package in Cosmic: In Progress Bug description: This has been noticed on several systems of mine, all 16.04. Landscape Client takes up, over time, an *extreme* amount of system resources. On a 1024MB RAM system, within 3 days to a month, Landscape Client and it's python scripts (possibly cached data?) are consuming massive quantities of memory - in the realm of 512MB RAM and over 400MB of the Swap on the system. This is repeatably shown in other systems of mine as well. Is there a memory leak or something in landscape-client that would contribute to this issue? ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: landscape-client 16.03-0ubuntu2.16.04.1 ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49 Uname: Linux 4.4.0-72-generic x86_64 NonfreeKernelModules: talpa_vfshook talpa_pedconnector talpa_vcdevice talpa_core talpa_linux talpa_syscallhook talpa_pedevice ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 Date: Mon Apr 24 14:42:37 2017 InstallationDate: Installed on 2017-04-02 (21 days ago) InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3) ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: landscape-client UpgradeStatus: No upgrade log present (probably fresh install) --- ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 DistroRelease: Ubuntu 16.04 InstallationDate: Installed on 2017-04-02 (21 days ago) InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3) Package: landscape-client 16.03-0ubuntu2.16.04.1 PackageArchitecture: amd64 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49 Tags: third-party-packages xenial Uname: Linux 4.4.0-72-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/landscape-client/+bug/1685885/+subscriptions ___ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launchpad.net/ListHelp
