Hi, sorry for not replying inline, but I thought I'd just share my general opinion on this.
The biggest issue in maintaining ceph is to make it build on 32 bit architectures. This seems not to be supported at all by upstream anymore. Between 14.2.7 and 14.2.9 I had a longer look into the issue and started to fix some issues, for example the parsing of config options does pretty broken things if the default for the option does not fit into a 32bit integer. Fixing this properly brought me to various other places where size_t is being used in the code, but actually an (at least) uint64_t is being required. Fedora already removed ceph for all 32bit architectures with a "not supported by upstream anymore", but I was not able to find an official statement from ceph upstream. Also unfortunately I did not yet find the time to collect my findings and send them to the ceph devel mailinglist, but I'd assume that they just don't want to support 32bit anymore, otherwise they'd test it properly. As the work to fix this is properly seems to be a rather long task, I definitely won't do this. But I also don't want to upload maybe-working binaries to Debian anymore. So unless somebody fixes and tests ceph for 32bit (or does this for Debian, also fine for me - running the regression test suite is possible with enough resources and some hardware), I will remove all 32bit architectures with the next upload. I guess those are not the news you wanted to hear, but so fard thats the situation. Bernd On 5/27/20 10:54 AM, Ard van Breemen wrote: > Hi, > > On Tue, May 26, 2020 at 06:35:20PM +0200, Val Lorentz wrote: >> Thanks for the tip. >> >> I just tried downgrading an OSD (armhf) and a monitor (amd64) to >> 14.2.7-1~bpo10+1 using http://snapshot.debian.org/ ; but they are still >> unable to communicate ("failed decoding of frame header: >> buffer::bad_alloc"). >> >> So this might be a different issue, although related. > > Well, 14.2.7-~bpo something did work on my armhf osd cluster, > with 2 mons running on armhf, and one on proxmox pve 6 running > ceph 14.2.8 . > What Already did not work was OSD's on AMD64 working together > with a 2xarmhf and 1xamd64 mon setup. > I had a lot of problems getting it to work at all, but I thought > it was just my lack of knowledge at that time. 99% of the > problems is with setting up the correct secrets, or in other > words, the handling of the "keyrings". Even between amd64 and > amd64 this has been buggy if I look at the release notes. > Specifically 14.2.6 to 14.2.7 I think. > I assume bugs are in authentication, because as long as I did not > reboot the amd64 it works. > The daemons authenticate using the secrets, and the secret gives > an authentication ticket. > > Anyway: the most simple test is to install a system, rsync > /etc/ceph and type in ceph status. It either works (on 32 bits, > fix the timeout in the python script, because if you don't it > won't work at all) or it doesn't return at all. > > I will test if it's also the case with armhf ceph cli client to a > amd64 cluster. I only have one working amd64 cluster though, and > it has 2 fake OSD's, because amd64 clusters are too expensive to > experiment with. > I have to do some networking hacks though to connect the systems. > > Anyway: the kernel has no problem talking to either OSD types, so > the kernel's protocol handling is implemented correctly, and > cephx works between an rbd amd64 or armhf kernel client and armhf > userspace. > The rbd amd64 userspace utility however does not work at all. As > far as I can see it can't get past authentication, but without > any logs I am a bit riddled. > > By the way: the mgr dashboard modules is about 99% correct. The > disk space is obviously calculated incorrectly. > > Regards, > Ard > -- Bernd Zeimetz Debian GNU/Linux Developer http://bzed.de http://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
