Re: [ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
On Fri, 2020-01-17 at 17:10 +0100, Ilya Dryomov wrote: > On Fri, Jan 17, 2020 at 2:21 AM Aaron wrote: > > No worries, can definitely do that. > > > > Cheers > > Aaron > > > > On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote: > > > On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote: > > > > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote: > > > > > Seeing a weird mount issue. Some info: > > > > > > > > > > No LSB modules are available. > > > > > Distributor ID: Ubuntu > > > > > Description: Ubuntu 18.04.3 LTS > > > > > Release: 18.04 > > > > > Codename: bionic > > > > > > > > > > Ubuntu 18.04.3 with kerne 4.15.0-74-generic > > > > > Ceph 14.2.5 & 14.2.6 > > > > > > > > > > With ceph-common, ceph-base, etc installed: > > > > > > > > > > ceph/stable,now 14.2.6-1bionic amd64 [installed] > > > > > ceph-base/stable,now 14.2.6-1bionic amd64 [installed] > > > > > ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] > > > > > ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] > > > > > ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] > > > > > ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] > > > > > libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > python-ceph-argparse/stable,stable,now 14.2.6-1bionic all > > > > > [installed,automatic] > > > > > python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > > > > > I create a user via get-or-create cmd, and I have a users/secret now. > > > > > When I try to mount on these Ubuntu nodes, > > > > > > > > > > The mount cmd I run for testing is: > > > > > sudo mount -t ceph -o > > > > > name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== > > > > > 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 /tmp/test > > > > > > > > > > I get the error: > > > > > couldn't finalize options: -34 > > > > > > > > > > From some tracking down, it's part of the get_secret_option() in > > > > > common/secrets.c and the Linux System Error: > > > > > > > > > > #define ERANGE 34 /* Math result not representable */ > > > > > > > > > > Now the weird part...when I remove all the above libs above, the mount > > > > > command works. I know that there are ceph.ko modules in the Ubuntu > > > > > filesystems DIR, and that Ubuntu comes with some understanding of how > > > > > to mount a cephfs system. So, that explains how it can mount > > > > > cephfs...but, what I don't understand is why I'm getting that -34 > > > > > error with the 14.2.5 and 14.2.6 libs installed. I didn't have this > > > > > issue with 14.2.3 or 14.2.4. > > > > > > > > This sounds like a regression in mount.ceph, probably due to something > > > > that went in for v14.2.5. I can reproduce the problem on Fedora, and I > > > > think it has something to do with the very long username you're using. > > > > > > > > I'll take a closer look and let you know. Stay tuned. > > > > > > > > > > I think I see the issue. The SECRET_OPTION_BUFSIZE is just too small for > > > your use case. We need to make that a little larger than the largest > > > name= parameter can be. Prior to v14.2.5, it was ~1000 bytes, but I made > > > it smaller in that set thinking that was too large. Mea culpa. > > > > > > The problem is determining how big that size can be. AFAICT EntityName > > > is basically a std::string in the ceph code, which can be an arbitrary > > > size (up to 4g or so). > > It's just that you made SECRET_OPTION_BUFSIZE account precisely for > "secret=", but it can also be "key=". > > I don't think there is much of a problem. Defining it back to ~1000 is > guaranteed to work. Or we could remove it and just compute the size of > secret_option exactly the same way as get_secret_option() does it: > > strlen(cmi->cmi_secret) + strlen(cmi->cmi_name) + 7 + 1 > Yeah, it's not hard to do a simple fix like that, but I opted to rework the code to just safe_cat the secret option string(s) directly into the options buffer. That eliminates some extra copies of this info and the need for an arbitrary limit altogether. It also removes a chunk of code that doesn't really need to be in the common lib. See: https://github.com/ceph/ceph/pull/32706 Aaron, if you have a way to build and test this, it'd be good if you could confirm that it fixes the problem for you. -- Jeff Layton ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
On Fri, Jan 17, 2020 at 2:21 AM Aaron wrote: > > No worries, can definitely do that. > > Cheers > Aaron > > On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote: >> >> On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote: >> > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote: >> > > Seeing a weird mount issue. Some info: >> > > >> > > No LSB modules are available. >> > > Distributor ID: Ubuntu >> > > Description: Ubuntu 18.04.3 LTS >> > > Release: 18.04 >> > > Codename: bionic >> > > >> > > Ubuntu 18.04.3 with kerne 4.15.0-74-generic >> > > Ceph 14.2.5 & 14.2.6 >> > > >> > > With ceph-common, ceph-base, etc installed: >> > > >> > > ceph/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-base/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] >> > > ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] >> > > libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > python-ceph-argparse/stable,stable,now 14.2.6-1bionic all >> > > [installed,automatic] >> > > python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > >> > > I create a user via get-or-create cmd, and I have a users/secret now. >> > > When I try to mount on these Ubuntu nodes, >> > > >> > > The mount cmd I run for testing is: >> > > sudo mount -t ceph -o >> > > name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== >> > > 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 /tmp/test >> > > >> > > I get the error: >> > > couldn't finalize options: -34 >> > > >> > > From some tracking down, it's part of the get_secret_option() in >> > > common/secrets.c and the Linux System Error: >> > > >> > > #define ERANGE 34 /* Math result not representable */ >> > > >> > > Now the weird part...when I remove all the above libs above, the mount >> > > command works. I know that there are ceph.ko modules in the Ubuntu >> > > filesystems DIR, and that Ubuntu comes with some understanding of how >> > > to mount a cephfs system. So, that explains how it can mount >> > > cephfs...but, what I don't understand is why I'm getting that -34 >> > > error with the 14.2.5 and 14.2.6 libs installed. I didn't have this >> > > issue with 14.2.3 or 14.2.4. >> > >> > This sounds like a regression in mount.ceph, probably due to something >> > that went in for v14.2.5. I can reproduce the problem on Fedora, and I >> > think it has something to do with the very long username you're using. >> > >> > I'll take a closer look and let you know. Stay tuned. >> > >> >> I think I see the issue. The SECRET_OPTION_BUFSIZE is just too small for >> your use case. We need to make that a little larger than the largest >> name= parameter can be. Prior to v14.2.5, it was ~1000 bytes, but I made >> it smaller in that set thinking that was too large. Mea culpa. >> >> The problem is determining how big that size can be. AFAICT EntityName >> is basically a std::string in the ceph code, which can be an arbitrary >> size (up to 4g or so). It's just that you made SECRET_OPTION_BUFSIZE account precisely for "secret=", but it can also be "key=". I don't think there is much of a problem. Defining it back to ~1000 is guaranteed to work. Or we could remove it and just compute the size of secret_option exactly the same way as get_secret_option() does it: strlen(cmi->cmi_secret) + strlen(cmi->cmi_name) + 7 + 1 Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
Actually, scratch that. I went ahead and opened this: https://tracker.ceph.com/issues/43649 Feel free to watch that one for updates. On Fri, 2020-01-17 at 07:43 -0500, Jeff Layton wrote: > No problem. Can you let me know the tracker bug number once you've > opened it? > > Thanks, > Jeff > > On Thu, 2020-01-16 at 20:24 -0500, Aaron wrote: > > This debugging started because the ceph-provisioner from k8s was making > > those users...but what we found was doing something similar by hand caused > > the same issue. Just surprised no one else using k8s and ceph backed > > PVC/PVs ran into this issue. > > > > Thanks again for all your help! > > > > Cheers > > Aaron > > > > On Thu, Jan 16, 2020 at 8:21 PM Aaron wrote: > > > No worries, can definitely do that. > > > > > > Cheers > > > Aaron > > > > > > On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote: > > > > On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote: > > > > > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote: > > > > > > Seeing a weird mount issue. Some info: > > > > > > > > > > > > No LSB modules are available. > > > > > > Distributor ID: Ubuntu > > > > > > Description: Ubuntu 18.04.3 LTS > > > > > > Release: 18.04 > > > > > > Codename: bionic > > > > > > > > > > > > Ubuntu 18.04.3 with kerne 4.15.0-74-generic > > > > > > Ceph 14.2.5 & 14.2.6 > > > > > > > > > > > > With ceph-common, ceph-base, etc installed: > > > > > > > > > > > > ceph/stable,now 14.2.6-1bionic amd64 [installed] > > > > > > ceph-base/stable,now 14.2.6-1bionic amd64 [installed] > > > > > > ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] > > > > > > ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] > > > > > > ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] > > > > > > ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] > > > > > > libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > python-ceph-argparse/stable,stable,now 14.2.6-1bionic all > > > > > > [installed,automatic] > > > > > > python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > > > > > > > I create a user via get-or-create cmd, and I have a users/secret > > > > > > now. > > > > > > When I try to mount on these Ubuntu nodes, > > > > > > > > > > > > The mount cmd I run for testing is: > > > > > > sudo mount -t ceph -o > > > > > > name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== > > > > > > 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 > > > > > > /tmp/test > > > > > > > > > > > > I get the error: > > > > > > couldn't finalize options: -34 > > > > > > > > > > > > From some tracking down, it's part of the get_secret_option() in > > > > > > common/secrets.c and the Linux System Error: > > > > > > > > > > > > #define ERANGE 34 /* Math result not representable */ > > > > > > > > > > > > Now the weird part...when I remove all the above libs above, the > > > > > > mount > > > > > > command works. I know that there are ceph.ko modules in the Ubuntu > > > > > > filesystems DIR, and that Ubuntu comes with some understanding of > > > > > > how > > > > > > to mount a cephfs system. So, that explains how it can mount > > > > > > cephfs...but, what I don't understand is why I'm getting that -34 > > > > > > error with the 14.2.5 and 14.2.6 libs installed. I didn't have this > > > > > > issue with 14.2.3 or 14.2.4. > > > > > > > > > > This sounds like a regression in mount.ceph, probably due to something > > > > > that went in for v14.2.5. I can reproduce the problem on Fedora, and I > > > > > think it has something to do with the very long username you're using. > > > > > > > > > > I'll take a closer look and let you know. Stay tuned. > > > > > > > > > > > > > I think I see the issue. The SECRET_OPTION_BUFSIZE is just too small for > > > > your use case. We need to make that a little larger than the largest > > > > name= parameter can be. Prior to v14.2.5, it was ~1000 bytes, but I made > > > > it smaller in that set thinking that was too large. Mea culpa. > > > > > > > > The problem is determining how big that size can be. AFAICT EntityName > > > > is basically a std::string in the ceph code, which can be an arbitrary > > > > size (up to 4g or so). > > > > > > > > Aaron, would you mind opening a bug for this at tracker.ceph.com? We > > > > should be able to get it fixed up, once I do a bit more research to > > > > figure out how big to make this buffer. -- Jeff Layton ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
This debugging started because the ceph-provisioner from k8s was making those users...but what we found was doing something similar by hand caused the same issue. Just surprised no one else using k8s and ceph backed PVC/PVs ran into this issue. Thanks again for all your help! Cheers Aaron On Thu, Jan 16, 2020 at 8:21 PM Aaron wrote: > No worries, can definitely do that. > > Cheers > Aaron > > On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote: > >> On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote: >> > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote: >> > > Seeing a weird mount issue. Some info: >> > > >> > > No LSB modules are available. >> > > Distributor ID: Ubuntu >> > > Description: Ubuntu 18.04.3 LTS >> > > Release: 18.04 >> > > Codename: bionic >> > > >> > > Ubuntu 18.04.3 with kerne 4.15.0-74-generic >> > > Ceph 14.2.5 & 14.2.6 >> > > >> > > With ceph-common, ceph-base, etc installed: >> > > >> > > ceph/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-base/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] >> > > ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] >> > > ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] >> > > libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > python-ceph-argparse/stable,stable,now 14.2.6-1bionic all >> [installed,automatic] >> > > python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] >> > > >> > > I create a user via get-or-create cmd, and I have a users/secret now. >> > > When I try to mount on these Ubuntu nodes, >> > > >> > > The mount cmd I run for testing is: >> > > sudo mount -t ceph -o >> > > >> name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== >> > > 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 /tmp/test >> > > >> > > I get the error: >> > > couldn't finalize options: -34 >> > > >> > > From some tracking down, it's part of the get_secret_option() in >> > > common/secrets.c and the Linux System Error: >> > > >> > > #define ERANGE 34 /* Math result not representable */ >> > > >> > > Now the weird part...when I remove all the above libs above, the mount >> > > command works. I know that there are ceph.ko modules in the Ubuntu >> > > filesystems DIR, and that Ubuntu comes with some understanding of how >> > > to mount a cephfs system. So, that explains how it can mount >> > > cephfs...but, what I don't understand is why I'm getting that -34 >> > > error with the 14.2.5 and 14.2.6 libs installed. I didn't have this >> > > issue with 14.2.3 or 14.2.4. >> > >> > This sounds like a regression in mount.ceph, probably due to something >> > that went in for v14.2.5. I can reproduce the problem on Fedora, and I >> > think it has something to do with the very long username you're using. >> > >> > I'll take a closer look and let you know. Stay tuned. >> > >> >> I think I see the issue. The SECRET_OPTION_BUFSIZE is just too small for >> your use case. We need to make that a little larger than the largest >> name= parameter can be. Prior to v14.2.5, it was ~1000 bytes, but I made >> it smaller in that set thinking that was too large. Mea culpa. >> >> The problem is determining how big that size can be. AFAICT EntityName >> is basically a std::string in the ceph code, which can be an arbitrary >> size (up to 4g or so). >> >> Aaron, would you mind opening a bug for this at tracker.ceph.com? We >> should be able to get it fixed up, once I do a bit more research to >> figure out how big to make this buffer. >> -- >> Jeff Layton >> >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
No worries, can definitely do that. Cheers Aaron On Thu, Jan 16, 2020 at 8:08 PM Jeff Layton wrote: > On Thu, 2020-01-16 at 18:42 -0500, Jeff Layton wrote: > > On Wed, 2020-01-15 at 08:05 -0500, Aaron wrote: > > > Seeing a weird mount issue. Some info: > > > > > > No LSB modules are available. > > > Distributor ID: Ubuntu > > > Description: Ubuntu 18.04.3 LTS > > > Release: 18.04 > > > Codename: bionic > > > > > > Ubuntu 18.04.3 with kerne 4.15.0-74-generic > > > Ceph 14.2.5 & 14.2.6 > > > > > > With ceph-common, ceph-base, etc installed: > > > > > > ceph/stable,now 14.2.6-1bionic amd64 [installed] > > > ceph-base/stable,now 14.2.6-1bionic amd64 [installed] > > > ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] > > > ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] > > > ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] > > > ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] > > > libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > python-ceph-argparse/stable,stable,now 14.2.6-1bionic all > [installed,automatic] > > > python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] > > > > > > I create a user via get-or-create cmd, and I have a users/secret now. > > > When I try to mount on these Ubuntu nodes, > > > > > > The mount cmd I run for testing is: > > > sudo mount -t ceph -o > > > > name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== > > > 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 /tmp/test > > > > > > I get the error: > > > couldn't finalize options: -34 > > > > > > From some tracking down, it's part of the get_secret_option() in > > > common/secrets.c and the Linux System Error: > > > > > > #define ERANGE 34 /* Math result not representable */ > > > > > > Now the weird part...when I remove all the above libs above, the mount > > > command works. I know that there are ceph.ko modules in the Ubuntu > > > filesystems DIR, and that Ubuntu comes with some understanding of how > > > to mount a cephfs system. So, that explains how it can mount > > > cephfs...but, what I don't understand is why I'm getting that -34 > > > error with the 14.2.5 and 14.2.6 libs installed. I didn't have this > > > issue with 14.2.3 or 14.2.4. > > > > This sounds like a regression in mount.ceph, probably due to something > > that went in for v14.2.5. I can reproduce the problem on Fedora, and I > > think it has something to do with the very long username you're using. > > > > I'll take a closer look and let you know. Stay tuned. > > > > I think I see the issue. The SECRET_OPTION_BUFSIZE is just too small for > your use case. We need to make that a little larger than the largest > name= parameter can be. Prior to v14.2.5, it was ~1000 bytes, but I made > it smaller in that set thinking that was too large. Mea culpa. > > The problem is determining how big that size can be. AFAICT EntityName > is basically a std::string in the ceph code, which can be an arbitrary > size (up to 4g or so). > > Aaron, would you mind opening a bug for this at tracker.ceph.com? We > should be able to get it fixed up, once I do a bit more research to > figure out how big to make this buffer. > -- > Jeff Layton > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Weird mount issue (Ubuntu 18.04, Ceph 14.2.5 & 14.2.6)
Seeing a weird mount issue. Some info: No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic Ubuntu 18.04.3 with kerne 4.15.0-74-generic Ceph 14.2.5 & 14.2.6 With ceph-common, ceph-base, etc installed: ceph/stable,now 14.2.6-1bionic amd64 [installed] ceph-base/stable,now 14.2.6-1bionic amd64 [installed] ceph-common/stable,now 14.2.6-1bionic amd64 [installed,automatic] ceph-mds/stable,now 14.2.6-1bionic amd64 [installed] ceph-mgr/stable,now 14.2.6-1bionic amd64 [installed,automatic] ceph-mgr-dashboard/stable,stable,now 14.2.6-1bionic all [installed] ceph-mon/stable,now 14.2.6-1bionic amd64 [installed] ceph-osd/stable,now 14.2.6-1bionic amd64 [installed] libcephfs2/stable,now 14.2.6-1bionic amd64 [installed,automatic] python-ceph-argparse/stable,stable,now 14.2.6-1bionic all [installed,automatic] python-cephfs/stable,now 14.2.6-1bionic amd64 [installed,automatic] I create a user via get-or-create cmd, and I have a users/secret now. When I try to mount on these Ubuntu nodes, The mount cmd I run for testing is: sudo mount -t ceph -o name=user-20c5338c-34db-11ea-b27a-de7033e905f6,secret=AQC6dhpeyczkDxAAhRcr7oERUY4BcD2NCUkuNg== 10.10.10.10:6789:/work/20c5332d-34db-11ea-b27a-de7033e905f6 /tmp/test I get the error: couldn't finalize options: -34 >From some tracking down, it's part of the get_secret_option() in common/secrets.c and the Linux System Error: #define ERANGE 34 /* Math result not representable */ Now the weird part...when I remove all the above libs above, the mount command works. I know that there are ceph.ko modules in the Ubuntu filesystems DIR, and that Ubuntu comes with some understanding of how to mount a cephfs system. So, that explains how it can mount cephfs...but, what I don't understand is why I'm getting that -34 error with the 14.2.5 and 14.2.6 libs installed. I didn't have this issue with 14.2.3 or 14.2.4. Cheers, Aaron ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com