will also add since it could help resolve this, there is no
"mgr/cephadm/registry_json" config option. The whole reason for moving from
the previous 3 options to the new json object was actually to move it from
config options that can get spit out in logs to the config-key store where
it's a bit more secure, given we're working with a password. If you want to
see what it's currently set to you would do a "ceph config-key get
mgr/cephadm/registry_credentials" and then for setting it roughly the same
but swap get for set and provide the json string as you had been trying to
do before.
On Sat, Nov 19, 2022 at 8:05 AM Adam King <adk...@redhat.com> wrote:
> I don't know for sure if it will fix the issue, but the migrations happen
> based on a config option "mgr/cephadm/migration_current". You could try
> setting that back to 0 and it would at least trigger the migrations to
> happen again after restarting/failing over the mgr. They're meant to be
> idempotent so in the worst case it just won't accomplish anything. Also,
> you're correct about it not being in the docs. The migrations were intended
> to be internal and never require user actions but it appears something has
> gone wrong in this case.
>
> On Fri, Nov 18, 2022 at 3:06 PM Seth T Graham <set...@fnal.gov> wrote:
>
>> We have a cluster running Octopus (15.2.17) that I need to get updated
>> and am getting cephadm failures when updating the managers, and have tried
>> both Pacific and Quincy with the same results. The cluster was deployed
>> with cephadm on centos stream 8 using podman and due to network isolation
>> of the cluster the images are being pulled from a private registry. When I
>> issue the 'ceph orch upgrade' command it starts out well by updating two of
>> the three managers. When it gets to the point of transitioning to one of
>> the upgraded managers the process stops with an error, with 'ceph status'
>> reporting that the cephadm module has failed.
>>
>> Digging through the logs, I find a python stack trace that reads:
>>
>> File "/usr/share/ceph/mgr/cephadm/module.py", line 587, in serve
>> serve.serve()
>> File "/usr/share/ceph/mgr/cephadm/serve.py", line 67, in serve
>> self.convert_tags_to_repo_digest()
>> File "/usr/share/ceph/mgr/cephadm/serve.py", line 974, in
>> convert_tags_to_repo_digest
>> self._get_container_image_info(container_image_ref))
>> File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async
>> return self.event_loop.get_result(coro)
>> File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result
>> return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
>> File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
>> return self.__get_result()
>> File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in
>> __get_result
>> raise self._exception
>> File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in
>> _get_container_image_info
>> await self._registry_login(host,
>> json.loads(str(self.mgr.get_store('registry_credentials'))))
>> File "/lib64/python3.6/json/__init__.py", line 354, in loads
>> return _default_decoder.decode(s)
>> File "/lib64/python3.6/json/decoder.py", line 339, in decode
>> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>> File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
>> raise JSONDecodeError("Expecting value", s, err.value) from None
>> json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>
>>
>> Looking through the ceph config there is indeed no setting for the
>> 'registry_credentials' value. Instead I have the registry_password,
>> registry_url and registry_username values that were set when the cluster
>> was provisioned.
>>
>> I do find mention of this key in the migrations.py script (lives in
>> /usr/share/ceph/mgr/cephadm), under the function 'migrate_4_5' which reads
>> to me like the old keys have been retired in favor of a unified key
>> containing a json object. So I attempted to recreate what that function is
>> doing by setting that key manually but unfortunately this didn't help.
>>
>> (eg, 'ceph config set mgr mgr/cephadm/registry_credentials '{ "url":
>> "XXX", "username": "XXX", "password": "XXX" }'')
>>
>> I'm not sure where to go from here. Is there a 'migrate' option I can
>> specify somewhere to properly upgrade this cluster, and perhaps run the
>> code found in migrations.py? I don't see any mention of this in the
>> documentation, but there's a lot of documentation so it's possible I missed
>> it.
>>
>> Failing that, are there any suggestions for a workaround so I can get
>> this upgrade completed?
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
