[ceph-users] Kubernetes support for rbd-nbd
Hi all, Heads up for any of you who are using Kubernetes or looking to use Kubernetes to schedule Ceph daemons running within containers (for block volumes). We have been working on adding support for rbd-nbd in the Kubernetes provisioner (as context, the current Kubernetes provisioner offers support only for the Ceph kernel client - krbd to provision Ceph block volumes). The changes to support rbd-nbd were merged into the upcoming 1.10 Kubernetes release. Once this is released, Kubernetes will be able to use rbd-nbd (with Ceph's user-space client librbd) to provision Ceph block volumes. The PR that was merged is here: https://github.com/kubernetes/kubernetes/pull/58916. Kubernetes 1.10 release timelines are here: https://github.com/kubernetes/sig-release/blob/master/releases/release-1.10/release-1.10.md . Thanks, Prashant -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 <http://smart.salesforce.com/sig/pmurthy//us_mb/default/link.html> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Obtaining perf counters/stats from krbd client
Thanks Greg. I thought as much :( We are looking into what stats are available and also for ways by which we can obtain stats from krbd, given this is going to be our current deployment in the near term. If this is useful to the community, we can share our findings and a proposal, once we have this available. Prashant On Wed, Jun 28, 2017 at 11:55 AM, Gregory Farnum <gfar...@redhat.com> wrote: > On Thu, Jun 22, 2017 at 11:27 AM Prashant Murthy <pmur...@salesforce.com> > wrote: > >> Hi Ceph users, >> >> We are currently using the Ceph kernel client module (krbd) in our >> deployment and we were looking to determine if there are ways by which we >> can obtain perf counters, log dumps, etc from such a deployment. Has >> anybody been able to obtain such stats? >> >> It looks like the libvirt interface allows for an admin socket to be >> configured on the client (http://docs.ceph.com/docs/master/rbd/libvirt/# >> configuring-ceph) into which you can issue commands, but is this >> specific to the librbd implementation? >> > > Sadly yes; I don't know what statistics are available from krbd, but it > doesn't work anything like the userspace stuff that is shared across > daemon/client types. > -Greg > > >> >> Thanks, >> Prashant >> >> -- >> Prashant Murthy >> Sr Director, Software Engineering | Salesforce >> Mobile: 919-961-3041 <(919)%20961-3041> >> >> >> -- >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Obtaining perf counters/stats from krbd client
Hi Ceph users, We are currently using the Ceph kernel client module (krbd) in our deployment and we were looking to determine if there are ways by which we can obtain perf counters, log dumps, etc from such a deployment. Has anybody been able to obtain such stats? It looks like the libvirt interface allows for an admin socket to be configured on the client ( http://docs.ceph.com/docs/master/rbd/libvirt/#configuring-ceph) into which you can issue commands, but is this specific to the librbd implementation? Thanks, Prashant -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD behavior for reads to a volume with no data written
Thanks for the detailed explanation, Jason. It makes sense that such operations would end up being a few metadata lookups only (and the metadata lookups will hit the disk only if they are not cached in-memory). Prashant On Tue, May 2, 2017 at 11:29 AM, Jason Dillaman <jdill...@redhat.com> wrote: > If the RBD object map feature is enabled, the read request would never > even be sent to the OSD if the client knows the backing object doesn't > exist. However, if the object map feature is disabled, the read request > will be sent to the OSD. > > The OSD isn't my area of expertise, but I can try to explain what occurs > to the best of my knowledge. There is a small in-memory cache for object > contexts with the OSD PG -- which includes a whiteout flag to indicate the > object is deleted. I believe that the whiteout flag is only really used on > a cache tier to avoid having to attempt to promote a known non-existent > object. Therefore, in the common case the OSD would query the non-existent > object from the object store (FileStore or BlueStore). > > In FileStore, it will attempt to open the associated backing file for > object. If the necessary dentries are cached in the kernel, I'd expect that > the -ENOENT error would be bubbled back to RBD w/o a disk hit. Otherwise, > the kernel would need to read the associated dentries from disk to > determine that the object is missing. > > In BlueStore, there is another in-memory cache for onodes that can quickly > detect a missing object. If the object isn't in the cache, the associated > onode will be looked up within the backing RocksDB. If the RocksDB metadata > scan for the object's onode fails since the object is missing, the -ENOENT > error would be bubbled back to the client. > > > > On Tue, May 2, 2017 at 1:24 PM, Prashant Murthy <pmur...@salesforce.com> > wrote: > >> I wanted to add that I was particularly interested about the behavior >> with filestore, but was also curious how this works on bluestore. >> >> Prashant >> >> On Mon, May 1, 2017 at 10:04 PM, Prashant Murthy <pmur...@salesforce.com> >> wrote: >> >>> Hi all, >>> >>> I was wondering what happens when reads are issued to an RBD device with >>> no previously written data. Can somebody explain how such requests flow >>> from rbd (client) into OSDs and whether any of these reads would hit the >>> disks at all or whether OSD metadata would recognize that there is no data >>> at the offsets requested and returns a bunch of zeros back to the client? >>> >>> Thanks, >>> Prashant >>> >>> -- >>> Prashant Murthy >>> Sr Director, Software Engineering | Salesforce >>> Mobile: 919-961-3041 <(919)%20961-3041> >>> >>> >>> -- >>> >> >> >> >> -- >> Prashant Murthy >> Sr Director, Software Engineering | Salesforce >> Mobile: 919-961-3041 <(919)%20961-3041> >> >> >> -- >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > > -- > Jason > -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD behavior for reads to a volume with no data written
I wanted to add that I was particularly interested about the behavior with filestore, but was also curious how this works on bluestore. Prashant On Mon, May 1, 2017 at 10:04 PM, Prashant Murthy <pmur...@salesforce.com> wrote: > Hi all, > > I was wondering what happens when reads are issued to an RBD device with > no previously written data. Can somebody explain how such requests flow > from rbd (client) into OSDs and whether any of these reads would hit the > disks at all or whether OSD metadata would recognize that there is no data > at the offsets requested and returns a bunch of zeros back to the client? > > Thanks, > Prashant > > -- > Prashant Murthy > Sr Director, Software Engineering | Salesforce > Mobile: 919-961-3041 <(919)%20961-3041> > > > -- > -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 <(919)%20961-3041> -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] RBD behavior for reads to a volume with no data written
Hi all, I was wondering what happens when reads are issued to an RBD device with no previously written data. Can somebody explain how such requests flow from rbd (client) into OSDs and whether any of these reads would hit the disks at all or whether OSD metadata would recognize that there is no data at the offsets requested and returns a bunch of zeros back to the client? Thanks, Prashant -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] librbd + rbd-nbd
Hi all, I wanted to ask if anybody is using librbd (user mode lib) with rbd-nbd (kernel module) on their Ceph clients. We're currently using krbd, but that doesn't support some of the features (such as rbd mirroring). So, I wanted to check if anybody has experience running with nbd + librbd on their clusters and can provide more details. Prashant -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Cannot search within ceph-users archives
Hi all, I tried to use the "Search" functionality on the ceph-users archives page ( http://lists.ceph.com/pipermail/ceph-users-ceph.com/), but each time I search for something, I get the following error: ht://Dig error htsearch detected an error. Please report this to the webmaster of this site by sending an e-mail to: mail...@listserver-dap.dreamhost.com The error message is: Unable to read word database file '/dh/mailman/dap/archives/private/ceph-users-ceph.com/htdig/db.words.db' Did you run htdig? Can you please help? Thanks, Prashant -- Prashant Murthy Sr Director, Software Engineering | Salesforce Mobile: 919-961-3041 -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
