Re: [ceph-users] questions about rbd image

[tim taler]

Well connecting an rbd to two servers would be like mapping a block
device from a storage array onto two different hosts,
that's is possible and (was) done.
(it would be much more difficult though to connect a single physical
harddisk to two computers)

The point is that as mentioned above you would need a cluster aware
filesystem on that rbd.
Not and ext4, not a zfs but gfs2 was mentioned above, I tried it once
just for fun using ocfs2 and it worked.

I wouldn't do that though nowadays (especially not in production).
AFAIK "cluster aware" filesysstem came a while BEFORE "clusterd and
distributed" filesystems.

With ocfs2 on rbd (one mount is read/write, second and following
mounts are read only right) you have to take care of two layers,
so two layers of prossible problems.

What you probably really want to have is a "clustered distributed
filesystem" such as cephfs or glusterfs (as to name two of them)
you can have as many replicas as you like, any number of host can
mount and read-write onto such a filesystem and - given the right set
up -
a bunch of hosts can even fail and the "clustered distributed"
filesystem is still available.

hth

On Mon, Dec 11, 2017 at 3:37 PM, David Turner  wrote:
> An RBD can only be mapped to a single client host.  There is no way around
> this.  An RBD at its core is a block device.  Connecting an RBD to 2 servers
> would be like connecting a harddrive to 2 servers.
>
> On Mon, Dec 11, 2017 at 9:13 AM 13605702596 <13605702...@163.com> wrote:
>>
>> hi Jason
>> thanks for your answer.
>> there is one more question, that is:
>> can we use rbd image to share data between two clients? one wirtes data,
>> another just reads?
>>
>> thanks
>>
>>
>> At 2017-12-11 21:52:54, "Jason Dillaman"  wrote:
>> >On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com
>> ><13605702...@163.com> wrote:
>> >> hi
>> >>
>> >> i'm testing on rbd image. the are TWO questions that confused me.
>> >> ceph -v
>> >> ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
>> >> uname -r
>> >> 3.10.0-514.el7.x86_64
>> >>
>> >> (1)  does rbd image supports multiple clients to write data
>> >> simultaneously?
>> >
>> >You would need to put a clustered file system like GFS2 on top of the
>> >block device to utilize it concurrently.
>> >
>> >> if it supports, how can share data between several clients using rbd
>> >> image?
>> >> client A: write data to rbd/test
>> >> client B: rbd map, and mount it to /mnt, file can be found in /mnt dir,
>> >> but
>> >> the content is miss.
>> >>
>> >> on monitor:
>> >> rbd create rbd/test -s 1024
>> >> rbd info rbd/test
>> >> rbd image 'test':
>> >> size 1024 MB in 256 objects
>> >> order 22 (4096 kB objects)
>> >> block_name_prefix: rbd_data.121d238e1f29
>> >> format: 2
>> >> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
>> >> flags:
>> >> then i disble the feature: object-map, fast-diff, deep-flatten
>> >>
>> >> on client A:
>> >> rbd map rbd/test
>> >> mkfs -t xfs /dev/rbd0
>> >> mount /dev/rbd/rbd/test /mnt/
>> >> echo 124 > /mnt/host124
>> >> cat host124
>> >> 124
>> >>
>> >> on client B:
>> >> rbd map rbd/test
>> >> mount /dev/rbd/rbd/test /mnt/
>> >> cat  host124  --> show nothing!
>> >>
>> >> echo 125 > /mnt/host125
>> >> cat /mnt/host125
>> >> 125
>> >>
>> >> then on client C:
>> >> rbd map rbd/test
>> >> mount /dev/rbd/rbd/test /mnt/
>> >> cd /mnt
>> >> cat host124 --> show nothing!
>> >> cat host125 --> show nothing!
>> >>
>> >> (2) does rbd image supports stripping? if does, howto?
>> >
>> >Not yet, but it's a work-in-progress for krbd to support "fancy"
>> >striping (librbd would support it via rbd-nbd).
>> >
>> >> on monitor, i create an image as following:
>> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock
>> >> --size
>> >> 1024 --object-size 4096 --stripe-unit 4096  --stripe-count 2
>> >> stripe unit is not a factor of the object size
>> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock
>> >> --size
>> >> 1024 --object-size 8M --stripe-unit 4M --stripe-count 2
>> >> rbd: the argument ('4M') for option '--unit' is invalid
>> >> i don't know why those cmd fails?
>> >
>> >Only Luminous and later releases support specifying the stripe unit
>> >with B/K/M suffixes.
>> >
>> >> finally, i successed with the following cmd:
>> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock
>> >> --size
>> >> 1024 --object-size 8388608 --stripe-unit 4194304  --stripe-count 2
>> >>
>> >> but whe i map it on client, it fails.
>> >> the error msg:
>> >> rbd: image test: unsupported stripe unit (got 4194304 want 8388608)
>> >>
>> >> best wishes
>> >> thanks
>> >>
>> >> 
>> >> 13605702...@163.com
>> >>
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >
>> >
>> >
>> >--
>> >Jason

Re: [ceph-users] questions about rbd image

[David Turner]

An RBD can only be mapped to a single client host.  There is no way around
this.  An RBD at its core is a block device.  Connecting an RBD to 2
servers would be like connecting a harddrive to 2 servers.

On Mon, Dec 11, 2017 at 9:13 AM 13605702596 <13605702...@163.com> wrote:

> hi Jason
> thanks for your answer.
> there is one more question, that is:
> can we use rbd image to share data between two clients? one wirtes data, 
> another just reads?
>
> thanks
>
>
> At 2017-12-11 21:52:54, "Jason Dillaman"  wrote:
> >On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com
> ><13605702...@163.com> wrote:
> >> hi
> >>
> >> i'm testing on rbd image. the are TWO questions that confused me.
> >> ceph -v
> >> ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
> >> uname -r
> >> 3.10.0-514.el7.x86_64
> >>
> >> (1)  does rbd image supports multiple clients to write data simultaneously?
> >
> >You would need to put a clustered file system like GFS2 on top of the
> >block device to utilize it concurrently.
> >
> >> if it supports, how can share data between several clients using rbd image?
> >> client A: write data to rbd/test
> >> client B: rbd map, and mount it to /mnt, file can be found in /mnt dir, but
> >> the content is miss.
> >>
> >> on monitor:
> >> rbd create rbd/test -s 1024
> >> rbd info rbd/test
> >> rbd image 'test':
> >> size 1024 MB in 256 objects
> >> order 22 (4096 kB objects)
> >> block_name_prefix: rbd_data.121d238e1f29
> >> format: 2
> >> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
> >> flags:
> >> then i disble the feature: object-map, fast-diff, deep-flatten
> >>
> >> on client A:
> >> rbd map rbd/test
> >> mkfs -t xfs /dev/rbd0
> >> mount /dev/rbd/rbd/test /mnt/
> >> echo 124 > /mnt/host124
> >> cat host124
> >> 124
> >>
> >> on client B:
> >> rbd map rbd/test
> >> mount /dev/rbd/rbd/test /mnt/
> >> cat  host124  --> show nothing!
> >>
> >> echo 125 > /mnt/host125
> >> cat /mnt/host125
> >> 125
> >>
> >> then on client C:
> >> rbd map rbd/test
> >> mount /dev/rbd/rbd/test /mnt/
> >> cd /mnt
> >> cat host124 --> show nothing!
> >> cat host125 --> show nothing!
> >>
> >> (2) does rbd image supports stripping? if does, howto?
> >
> >Not yet, but it's a work-in-progress for krbd to support "fancy"
> >striping (librbd would support it via rbd-nbd).
> >
> >> on monitor, i create an image as following:
> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> >> 1024 --object-size 4096 --stripe-unit 4096  --stripe-count 2
> >> stripe unit is not a factor of the object size
> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> >> 1024 --object-size 8M --stripe-unit 4M --stripe-count 2
> >> rbd: the argument ('4M') for option '--unit' is invalid
> >> i don't know why those cmd fails?
> >
> >Only Luminous and later releases support specifying the stripe unit
> >with B/K/M suffixes.
> >
> >> finally, i successed with the following cmd:
> >> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> >> 1024 --object-size 8388608 --stripe-unit 4194304  --stripe-count 2
> >>
> >> but whe i map it on client, it fails.
> >> the error msg:
> >> rbd: image test: unsupported stripe unit (got 4194304 want 8388608)
> >>
> >> best wishes
> >> thanks
> >>
> >> 
> >> 13605702...@163.com
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >
> >
> >
> >--
> >Jason
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] questions about rbd image

[13605702596]

hi Jasonthanks for your answer.there is one more question, that is: can we use 
rbd image to share data between two clients? one wirtes data, another just 
reads?

thanks


At 2017-12-11 21:52:54, "Jason Dillaman"  wrote:
>On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com
><13605702...@163.com> wrote:
>> hi
>>
>> i'm testing on rbd image. the are TWO questions that confused me.
>> ceph -v
>> ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
>> uname -r
>> 3.10.0-514.el7.x86_64
>>
>> (1)  does rbd image supports multiple clients to write data simultaneously?
>
>You would need to put a clustered file system like GFS2 on top of the
>block device to utilize it concurrently.
>
>> if it supports, how can share data between several clients using rbd image?
>> client A: write data to rbd/test
>> client B: rbd map, and mount it to /mnt, file can be found in /mnt dir, but
>> the content is miss.
>>
>> on monitor:
>> rbd create rbd/test -s 1024
>> rbd info rbd/test
>> rbd image 'test':
>> size 1024 MB in 256 objects
>> order 22 (4096 kB objects)
>> block_name_prefix: rbd_data.121d238e1f29
>> format: 2
>> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
>> flags:
>> then i disble the feature: object-map, fast-diff, deep-flatten
>>
>> on client A:
>> rbd map rbd/test
>> mkfs -t xfs /dev/rbd0
>> mount /dev/rbd/rbd/test /mnt/
>> echo 124 > /mnt/host124
>> cat host124
>> 124
>>
>> on client B:
>> rbd map rbd/test
>> mount /dev/rbd/rbd/test /mnt/
>> cat  host124  --> show nothing!
>>
>> echo 125 > /mnt/host125
>> cat /mnt/host125
>> 125
>>
>> then on client C:
>> rbd map rbd/test
>> mount /dev/rbd/rbd/test /mnt/
>> cd /mnt
>> cat host124 --> show nothing!
>> cat host125 --> show nothing!
>>
>> (2) does rbd image supports stripping? if does, howto?
>
>Not yet, but it's a work-in-progress for krbd to support "fancy"
>striping (librbd would support it via rbd-nbd).
>
>> on monitor, i create an image as following:
>> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
>> 1024 --object-size 4096 --stripe-unit 4096  --stripe-count 2
>> stripe unit is not a factor of the object size
>> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
>> 1024 --object-size 8M --stripe-unit 4M --stripe-count 2
>> rbd: the argument ('4M') for option '--unit' is invalid
>> i don't know why those cmd fails?
>
>Only Luminous and later releases support specifying the stripe unit
>with B/K/M suffixes.
>
>> finally, i successed with the following cmd:
>> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
>> 1024 --object-size 8388608 --stripe-unit 4194304  --stripe-count 2
>>
>> but whe i map it on client, it fails.
>> the error msg:
>> rbd: image test: unsupported stripe unit (got 4194304 want 8388608)
>>
>> best wishes
>> thanks
>>
>> 
>> 13605702...@163.com
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
>-- 
>Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] questions about rbd image

[Jason Dillaman]

On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com
<13605702...@163.com> wrote:
> hi
>
> i'm testing on rbd image. the are TWO questions that confused me.
> ceph -v
> ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
> uname -r
> 3.10.0-514.el7.x86_64
>
> (1)  does rbd image supports multiple clients to write data simultaneously?

You would need to put a clustered file system like GFS2 on top of the
block device to utilize it concurrently.

> if it supports, how can share data between several clients using rbd image?
> client A: write data to rbd/test
> client B: rbd map, and mount it to /mnt, file can be found in /mnt dir, but
> the content is miss.
>
> on monitor:
> rbd create rbd/test -s 1024
> rbd info rbd/test
> rbd image 'test':
> size 1024 MB in 256 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.121d238e1f29
> format: 2
> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
> flags:
> then i disble the feature: object-map, fast-diff, deep-flatten
>
> on client A:
> rbd map rbd/test
> mkfs -t xfs /dev/rbd0
> mount /dev/rbd/rbd/test /mnt/
> echo 124 > /mnt/host124
> cat host124
> 124
>
> on client B:
> rbd map rbd/test
> mount /dev/rbd/rbd/test /mnt/
> cat  host124  --> show nothing!
>
> echo 125 > /mnt/host125
> cat /mnt/host125
> 125
>
> then on client C:
> rbd map rbd/test
> mount /dev/rbd/rbd/test /mnt/
> cd /mnt
> cat host124 --> show nothing!
> cat host125 --> show nothing!
>
> (2) does rbd image supports stripping? if does, howto?

Not yet, but it's a work-in-progress for krbd to support "fancy"
striping (librbd would support it via rbd-nbd).

> on monitor, i create an image as following:
> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> 1024 --object-size 4096 --stripe-unit 4096  --stripe-count 2
> stripe unit is not a factor of the object size
> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> 1024 --object-size 8M --stripe-unit 4M --stripe-count 2
> rbd: the argument ('4M') for option '--unit' is invalid
> i don't know why those cmd fails?

Only Luminous and later releases support specifying the stripe unit
with B/K/M suffixes.

> finally, i successed with the following cmd:
> rbd create rbd/test --image-feature layering,striping,exclusive-lock --size
> 1024 --object-size 8388608 --stripe-unit 4194304  --stripe-count 2
>
> but whe i map it on client, it fails.
> the error msg:
> rbd: image test: unsupported stripe unit (got 4194304 want 8388608)
>
> best wishes
> thanks
>
> 
> 13605702...@163.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] questions about rbd image

[13605702...@163.com]

hi

i'm testing on rbd image. the are TWO questions that confused me. 
ceph -v
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
uname -r
3.10.0-514.el7.x86_64

(1)  does rbd image supports multiple clients to write data simultaneously?  if 
it supports, how can share data between several clients using rbd image?
client A: write data to rbd/test
client B: rbd map, and mount it to /mnt, file can be found in /mnt dir, but the 
content is miss.

on monitor:
rbd create rbd/test -s 1024
rbd info rbd/test
rbd image 'test':
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.121d238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags: 
then i disble the feature: object-map, fast-diff, deep-flatten

on client A:
rbd map rbd/test
mkfs -t xfs /dev/rbd0
mount /dev/rbd/rbd/test /mnt/
echo 124 > /mnt/host124
cat host124 
124

on client B:
rbd map rbd/test
mount /dev/rbd/rbd/test /mnt/
cat  host124  --> show nothing!

echo 125 > /mnt/host125
cat /mnt/host125 
125

then on client C: 
rbd map rbd/test
mount /dev/rbd/rbd/test /mnt/
cd /mnt
cat host124 --> show nothing!
cat host125 --> show nothing!

(2) does rbd image supports stripping? if does, howto?

on monitor, i create an image as following: 
rbd create rbd/test --image-feature layering,striping,exclusive-lock --size 
1024 --object-size 4096 --stripe-unit 4096  --stripe-count 2
stripe unit is not a factor of the object size
rbd create rbd/test --image-feature layering,striping,exclusive-lock --size 
1024 --object-size 8M --stripe-unit 4M --stripe-count 2
rbd: the argument ('4M') for option '--unit' is invalid
i don't know why those cmd fails?

finally, i successed with the following cmd:
rbd create rbd/test --image-feature layering,striping,exclusive-lock --size 
1024 --object-size 8388608 --stripe-unit 4194304  --stripe-count 2

but whe i map it on client, it fails.
the error msg: 
rbd: image test: unsupported stripe unit (got 4194304 want 8388608)

best wishes
thanks



13605702...@163.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Questions about rbd image features

[Jason Dillaman]

On Fri, Jan 13, 2017 at 5:11 AM, Vincent Godin  wrote:
> We are using a production cluster which started in Firefly, then moved to
> Giant, Hammer and finally Jewel. So our images have different features
> correspondind to the value of "rbd_default_features" of the version when
> they were created.
> We have actually three pack of features activated :
> image with :
> - layering ~ 1
> - layering, striping ~3
> - layering, exclusive-lock, object-map, fast-diff, deep-flatten ~ 61
>
> 1) Is it a good idea to try to give all images the same features ?

It isn't needed.

> 2) Is it possible to disable the striping feature on an already created
> image (we never specify any stripe-unit nor stripe-count) ?

Negative -- striping cannot be dynamically disabled because it would
result in potentially altering the structure and placement of the data
within the image. If your stripe-unit is the object size and the
stripe count is 1, that's a special case where the flag is essentially
ignored.

> 3) What is the behaviour of an already created image on which we activate
> the object-map feature ? Will a process try to rebuild a index of used
> blocks - if no, if we delete later the image, will ceph try to remove all
> the blocks or only the blocks refered by object-map index ?

You would need to run "rbd object-map rebuild " to rebuild
the object map. Until it is rebuilt, it will be considered invalid and
won't be used for reference. You can determine the object map state by
running "rbd info "

> 4) We are on Jewel but with tunables set to hammer (Centos 7.2). What are
> the best default features to set in that case ? (we use Ceph  under an
> Openstack for glance, nova and cinder

We feel like the current defaults are a good mix of features for
everyday use of non-shared images or non-krbd images. Most
importantly, all the default features can be dynamically disabled if
your needs for the image change.

>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Questions about rbd image features

[Vincent Godin]

We are using a production cluster which started in Firefly, then moved to
Giant, Hammer and finally Jewel. So our images have different features
correspondind to the value of "rbd_default_features" of the version when
they were created.
We have actually three pack of features activated :
image with :
- layering ~ 1
- layering, striping ~3
- layering, exclusive-lock, object-map, fast-diff, deep-flatten ~ 61

1) Is it a good idea to try to give all images the same features ?
2) Is it possible to disable the striping feature on an already created
image (we never specify any stripe-unit nor stripe-count) ?
3) What is the behaviour of an already created image on which we activate
the object-map feature ? Will a process try to rebuild a index of used
blocks - if no, if we delete later the image, will ceph try to remove all
the blocks or only the blocks refered by object-map index ?
4) We are on Jewel but with tunables set to hammer (Centos 7.2). What are
the best default features to set in that case ? (we use Ceph  under an
Openstack for glance, nova and cinder
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com