[ceph-users] clock skew detected

[Pavel V. Kaygorodov]

Hi!

Immediately after a reboot of mon.3 host its clock was unsynchronized and 
"clock skew detected on mon.3" warning is appeared.
But now (more then 1 hour of uptime) the clock is synced, but the warning still 
showing.
Is this ok?
Or I have to restart monitor after clock synchronization?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD images -- parent snapshot missing (help!)

[Pavel V. Kaygorodov]

Hi!

The program is in attachment.

WARNING: USE IN AT YOUR OWN RISK, NO WARRANTIES ARE PROVIDED!!!
IT MAY CORRUPT YOUR DATA!!!

You need to install rados-dev and compile the program with:

gcc ./upd_parent.c -o upd_parent -lrados

To change ID of parent image pool you need to run it with following arguments:

 ./upd_parent


for example (lost pool was 13, new pool is 20, image pool name is "volumes"):

 ./upd_parent 13 20 volumes volume-c083800d-7688-4fb8-9230-83643cbe8927

Hope, it will helps.
I recommend you to try to mount fixed image read-only, copy all important data 
(if possible) and only after that try to boot it or run fsck.

Pavel.



upd_parent.c
Description: Binary data



> 16 мая 2015 г., в 7:13, Tuomas Juntunen  
> написал(а):
> 
> Hey Pavel
> 
> Could you share your C program and the process how you were able to fix the 
> images.
> 
> Thanks
> 
> Tuomas
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Pavel V. Kaygorodov
> Sent: 13. toukokuuta 2015 18:24
> To: Jason Dillaman
> Cc: ceph-users
> Subject: Re: [ceph-users] RBD images -- parent snapshot missing (help!)
> 
> Hi!
> 
> Thank you for you effort, I think it will be very useful for many people!
> Now I have solved the problem, using C program and radios library, 2 images 
> from 3 is completely restored, one was corrupted, but I have rescued all 
> important data :)
> 
> Pavel.
> 
> 
> 
>> 13 мая 2015 г., в 17:20, Jason Dillaman  написал(а):
>> 
>> I just pushed an update to the rados CLI that allows the setomapval command 
>> to read the data from stdin.  In your example below, the command to use 
>> would be:
>> 
>> # cat ./rbd_header.9a3ab3d1382f3-parent | rados -p volumes setomapval 
>> rbd_header.9a3ab3d1382f3 parent
>> 
>> The change is currently under review in the wip-rados-binary-omapsetval 
>> branch.  If you wanted to test it out on a scratch machine, the packages 
>> should be available within the next hour or so from the gitbuilder.  See the 
>> Ceph website for documentation on how to install development packages [1].
>> 
>> [1] 
>> http://docs.ceph.com/docs/master/install/get-packages/#add-ceph-develo
>> pment
>> 
>> --
>> 
>> Jason Dillaman
>> Red Hat
>> dilla...@redhat.com
>> http://www.redhat.com
>> 
>> 
>> - Original Message -
>> From: "Pavel V. Kaygorodov" 
>> To: "Tuomas Juntunen" 
>> Cc: "ceph-users" 
>> Sent: Tuesday, May 12, 2015 3:55:21 PM
>> Subject: Re: [ceph-users] RBD images -- parent snapshot missing 
>> (help!)
>> 
>> Hi!
>> 
>> I have found a way to change a pool ID for image parent:
>> 
>> list images:
>> 
>> # rbd ls volumes
>> 5a4b167d-2588-4c06-904c-347abf91d788_disk.swap
>> volume-0ed965a0-53a5-4054-ad9c-3a432c8455d6
>> volume-1269b41a-4af0-499b-a16c-9bb6a5b98e70
>> volume-4094fbc1-9969-47aa-a0de-7026678b8e64
>> volume-5958295e-9623-4c46-b1e6-2017c6574805
>> volume-8448a317-d802-48fa-8bc1-ffd4c524fa43
>> volume-c083800d-7688-4fb8-9230-83643cbe8927
>> 
>> I want to repair volume volume-c083800d-7688-4fb8-9230-83643cbe8927, so get 
>> its internal ID:
>> 
>> #rados -p volumes get 
>> rbd_id.volume-c083800d-7688-4fb8-9230-83643cbe8927 ./image-id #cat 
>> ./image-id
>> 
>> # hexdump -C image-id   0d 00 00 00 39 61 33 61  62 33 64 31 33 38 
>> 32 66  |9a3ab3d1382f|
>> 0010  33|3|
>> 0011
>> 
>> So, ID is 9a3ab3d1382f3, get omap key/value pairs:
>> 
>> # rados -p volumes listomapvals rbd_header.9a3ab3d1382f3 features
>> value: (8 bytes) :
>>  : 01 00 00 00 00 00 00 00 : 
>> 
>> object_prefix
>> value: (26 bytes) :
>>  : 16 00 00 00 72 62 64 5f 64 61 74 61 2e 39 61 33 : rbd_data.9a3
>> 0010 : 61 62 33 64 31 33 38 32 66 33   : ab3d1382f3
>> 
>> order
>> value: (1 bytes) :
>>  : 17  : .
>> 
>> parent
>> value: (46 bytes) :
>>  : 01 01 28 00 00 00 0d 00 00 00 00 00 00 00 0c 00 : ..(.
>> 0010 : 00 00 38 61 65 63 32 65 37 34 65 62 30 36 22 00 : ..8aec2e74eb06".
>> 0020 : 00 00 00 00 00 00 00 00 00 40 00 00 00 00   : .@
>> 
>> size
>> value: (8 bytes) :
>>  : 00 00 00 80 00 00 00 00 : 
>> 
>> snap_seq
>> valu

Re: [ceph-users] RBD images -- parent snapshot missing (help!)

[Pavel V. Kaygorodov]

Hi!

Thank you for you effort, I think it will be very useful for many people!
Now I have solved the problem, using C program and radios library, 2 images 
from 3 is completely restored, one was corrupted, but I have rescued all 
important data :)

Pavel.



> 13 мая 2015 г., в 17:20, Jason Dillaman  написал(а):
> 
> I just pushed an update to the rados CLI that allows the setomapval command 
> to read the data from stdin.  In your example below, the command to use would 
> be:
> 
> # cat ./rbd_header.9a3ab3d1382f3-parent | rados -p volumes setomapval 
> rbd_header.9a3ab3d1382f3 parent
> 
> The change is currently under review in the wip-rados-binary-omapsetval 
> branch.  If you wanted to test it out on a scratch machine, the packages 
> should be available within the next hour or so from the gitbuilder.  See the 
> Ceph website for documentation on how to install development packages [1].
> 
> [1] 
> http://docs.ceph.com/docs/master/install/get-packages/#add-ceph-development
> 
> -- 
> 
> Jason Dillaman 
> Red Hat 
> dilla...@redhat.com 
> http://www.redhat.com 
> 
> 
> - Original Message -
> From: "Pavel V. Kaygorodov" 
> To: "Tuomas Juntunen" 
> Cc: "ceph-users" 
> Sent: Tuesday, May 12, 2015 3:55:21 PM
> Subject: Re: [ceph-users] RBD images -- parent snapshot missing (help!)
> 
> Hi!
> 
> I have found a way to change a pool ID for image parent:
> 
> list images:
> 
> # rbd ls volumes
> 5a4b167d-2588-4c06-904c-347abf91d788_disk.swap
> volume-0ed965a0-53a5-4054-ad9c-3a432c8455d6
> volume-1269b41a-4af0-499b-a16c-9bb6a5b98e70
> volume-4094fbc1-9969-47aa-a0de-7026678b8e64
> volume-5958295e-9623-4c46-b1e6-2017c6574805
> volume-8448a317-d802-48fa-8bc1-ffd4c524fa43
> volume-c083800d-7688-4fb8-9230-83643cbe8927
> 
> I want to repair volume volume-c083800d-7688-4fb8-9230-83643cbe8927, so get 
> its internal ID:
> 
> #rados -p volumes get rbd_id.volume-c083800d-7688-4fb8-9230-83643cbe8927 
> ./image-id
> #cat ./image-id
> 
> # hexdump -C image-id   0d 00 00 00 39 61 33 61  62 33 64 31 33 38 32 
> 66  |9a3ab3d1382f|
> 0010  33|3|
> 0011
> 
> So, ID is 9a3ab3d1382f3, get omap key/value pairs:
> 
> # rados -p volumes listomapvals rbd_header.9a3ab3d1382f3 features
> value: (8 bytes) :
>  : 01 00 00 00 00 00 00 00 : 
> 
> object_prefix
> value: (26 bytes) :
>  : 16 00 00 00 72 62 64 5f 64 61 74 61 2e 39 61 33 : rbd_data.9a3
> 0010 : 61 62 33 64 31 33 38 32 66 33   : ab3d1382f3
> 
> order
> value: (1 bytes) :
>  : 17  : .
> 
> parent
> value: (46 bytes) :
>  : 01 01 28 00 00 00 0d 00 00 00 00 00 00 00 0c 00 : ..(.
> 0010 : 00 00 38 61 65 63 32 65 37 34 65 62 30 36 22 00 : ..8aec2e74eb06".
> 0020 : 00 00 00 00 00 00 00 00 00 40 00 00 00 00   : .@
> 
> size
> value: (8 bytes) :
>  : 00 00 00 80 00 00 00 00 : 
> 
> snap_seq
> value: (8 bytes) :
>  : 00 00 00 00 00 00 00 00 : 
> 
> we see key "parent", where 0x0d might be an old pool id (13), new one is 20 
> (0x14), so we can dump this key into file:
> 
> # rados -p volumes getomapval rbd_header.9a3ab3d1382f3 parent 
> ./rbd_header.9a3ab3d1382f3-parent
> (length 46) : Writing to ./rbd_header.9a3ab3d1382f3-parent
> 
> # hexdump -C rbd_header.9a3ab3d1382f3-parent 
>   01 01 28 00 00 00 0d 00  00 00 00 00 00 00 0c 00  |..(.|
> 0010  00 00 38 61 65 63 32 65  37 34 65 62 30 36 22 00  |..8aec2e74eb06".|
> 0020  00 00 00 00 00 00 00 00  00 40 00 00 00 00|.@|
> 002e
> 
> Here is it! Now we can edit it with hexedit, replacing 0d to 14 and put the 
> value back:
> 
> # rados -p volumes setomapval rbd_header.9a3ab3d1382f3 parent ...
> 
> BUT!
> It seems to be impossible to put binary data with zeroes into bash argument, 
> so "rados setomapval" seems to be completely useless in this case.
> 
> I still need help!
> 
> Pavel.
> 
> 
> 
>> 12 мая 2015 г., в 20:54, Tuomas Juntunen  
>> написал(а):
>> 
>> Hi
>> 
>> I am having this exact same problem, for more than a week. I have not found
>> a way to do this either.
>> 
>> Any help would be appreciated. 
>> 
>> Basically all of our guests are now down, even though they are not in
>> production, we would still need to get the data out of them.
>> 
>> Br,
>> Tuomas
>> 
>> -Ori

Re: [ceph-users] RBD images -- parent snapshot missing (help!)

[Pavel V. Kaygorodov]

Hi!

I have found a way to change a pool ID for image parent:

list images:

# rbd ls volumes
5a4b167d-2588-4c06-904c-347abf91d788_disk.swap
volume-0ed965a0-53a5-4054-ad9c-3a432c8455d6
volume-1269b41a-4af0-499b-a16c-9bb6a5b98e70
volume-4094fbc1-9969-47aa-a0de-7026678b8e64
volume-5958295e-9623-4c46-b1e6-2017c6574805
volume-8448a317-d802-48fa-8bc1-ffd4c524fa43
volume-c083800d-7688-4fb8-9230-83643cbe8927

I want to repair volume volume-c083800d-7688-4fb8-9230-83643cbe8927, so get its 
internal ID:

#rados -p volumes get rbd_id.volume-c083800d-7688-4fb8-9230-83643cbe8927 
./image-id
#cat ./image-id

# hexdump -C image-id   0d 00 00 00 39 61 33 61  62 33 64 31 33 38 32 
66  |9a3ab3d1382f|
0010  33|3|
0011

So, ID is 9a3ab3d1382f3, get omap key/value pairs:

# rados -p volumes listomapvals rbd_header.9a3ab3d1382f3 features
value: (8 bytes) :
 : 01 00 00 00 00 00 00 00 : 

object_prefix
value: (26 bytes) :
 : 16 00 00 00 72 62 64 5f 64 61 74 61 2e 39 61 33 : rbd_data.9a3
0010 : 61 62 33 64 31 33 38 32 66 33   : ab3d1382f3

order
value: (1 bytes) :
 : 17  : .

parent
value: (46 bytes) :
 : 01 01 28 00 00 00 0d 00 00 00 00 00 00 00 0c 00 : ..(.
0010 : 00 00 38 61 65 63 32 65 37 34 65 62 30 36 22 00 : ..8aec2e74eb06".
0020 : 00 00 00 00 00 00 00 00 00 40 00 00 00 00   : .@

size
value: (8 bytes) :
 : 00 00 00 80 00 00 00 00 : 

snap_seq
value: (8 bytes) :
 : 00 00 00 00 00 00 00 00 : 

we see key "parent", where 0x0d might be an old pool id (13), new one is 20 
(0x14), so we can dump this key into file:

# rados -p volumes getomapval rbd_header.9a3ab3d1382f3 parent 
./rbd_header.9a3ab3d1382f3-parent
 (length 46) : Writing to ./rbd_header.9a3ab3d1382f3-parent

# hexdump -C rbd_header.9a3ab3d1382f3-parent 
  01 01 28 00 00 00 0d 00  00 00 00 00 00 00 0c 00  |..(.|
0010  00 00 38 61 65 63 32 65  37 34 65 62 30 36 22 00  |..8aec2e74eb06".|
0020  00 00 00 00 00 00 00 00  00 40 00 00 00 00|.@|
002e

Here is it! Now we can edit it with hexedit, replacing 0d to 14 and put the 
value back:

# rados -p volumes setomapval rbd_header.9a3ab3d1382f3 parent ...

BUT!
It seems to be impossible to put binary data with zeroes into bash argument, so 
"rados setomapval" seems to be completely useless in this case.

I still need help!

Pavel.



> 12 мая 2015 г., в 20:54, Tuomas Juntunen  
> написал(а):
> 
> Hi
> 
> I am having this exact same problem, for more than a week. I have not found
> a way to do this either.
> 
> Any help would be appreciated. 
> 
> Basically all of our guests are now down, even though they are not in
> production, we would still need to get the data out of them.
> 
> Br,
> Tuomas
> 
> -Original Message-----
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Pavel V. Kaygorodov
> Sent: 12. toukokuuta 2015 20:41
> To: ceph-users
> Subject: [ceph-users] RBD images -- parent snapshot missing (help!)
> 
> Hi!
> 
> I have an RBD image (in pool "volumes"), made by openstack from parent image
> (in pool "images").
> Recently, I have tried to decrease number of PG-s, to avoid new Hammer
> warning.
> I have copied pool "images" to another pool, deleted original pool and
> renamed new pool to "images". Ceph allowed me to do this without any
> warning.
> But, after a some time my VM-s was stopped and cannot be started again,
> because they cannot open volumes, which had parent snapshots on "images"
> pool.
> I have an original image unchanged, I can made a snapshot if it was not
> copied by "radios cppool", but RBD images linked to parent snapshot on pool
> with old ID, now this pool has new ID.
> So, I need to change the link to parent snapshot for RBD images, or change
> pool id of "images" to old one.
> How to do this?
> 
> Help, please!
> 
> Pavel.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD images -- parent snapshot missing (help!)

[Pavel V. Kaygorodov]

Hi!

I have an RBD image (in pool "volumes"), made by openstack from parent image 
(in pool "images").
Recently, I have tried to decrease number of PG-s, to avoid new Hammer warning.
I have copied pool "images" to another pool, deleted original pool and renamed 
new pool to "images". Ceph allowed me to do this without any warning.
But, after a some time my VM-s was stopped and cannot be started again, because 
they cannot open volumes, which had parent snapshots on "images" pool.
I have an original image unchanged, I can made a snapshot if it was not copied 
by "radios cppool", but RBD images linked to parent snapshot on pool with old 
ID, now this pool has new ID.
So, I need to change the link to parent snapshot for RBD images, or change pool 
id of "images" to old one.
How to do this?

Help, please!

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rados cppool

[Pavel V. Kaygorodov]

Hi!

I have copied two of my pools recently, because old ones has too many pgs.
Both of them contains RBD images, with 1GB and ~30GB of data.
Both pools was copied without errors, RBD images are mountable and seems to be 
fine.
CEPH version is 0.94.1

Pavel.
 

> 7 апр. 2015 г., в 18:29, Kapil Sharma  написал(а):
> 
> Hi folks,
> 
> I will really appreciate if someone could try "rados cppool  
> "
> command on their Hammer ceph cluster. It throws an error for me, not sure if 
> this is
> an upstream issue or something related to our distro only.
> 
> error trace- http://pastebin.com/gVkbiPLa
> 
> This works fine for me in my firefly cluster.
> 
> -- 
> Regards,
> Kapil.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] decrease pg number

[Pavel V. Kaygorodov]

Hi!

I have updated my cluster to Hammer and got a warning "too many PGs per OSD 
(2240 > max 300)".
I know, that there is no way to decrease number of page groups, so I want to 
re-create my pools with less pg number, move all my data to them, delete old 
pools and rename new pools as the old ones.
Also I want to preserve the user rights on new pools.
I have several pools with RBD images, some of them with snapshots.

Which is the proper way to do this?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph cluster on docker containers

[Pavel V. Kaygorodov]

Hi!

I'm using ceph cluster, packed to a number of docker containers.
There are two things, which you need to know:

1. Ceph OSDs are using FS attributes, which may not be supported by filesystem 
inside docker container, so you need to mount external directory inside a 
container to store OSD data.
2. Ceph monitors must have static external IP-s, so you have to use lxc-conf 
directives to use static IP-s inside containers.


With best regards,
  Pavel.


> 6 марта 2015 г., в 10:15, Sumit Gaur  написал(а):
> 
> Hi
> I need to know if Ceph has any Docker story. What I am not abel to find if 
> there are any predefined steps for ceph cluster to be deployed on Docker 
> containers.
> 
> Thanks
> sumit
>  
>  
> <201503061614748_BEI0XT4N.gif>
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question Blackout

[Pavel V. Kaygorodov]

Hi!

We have experienced several blackouts on our small ceph cluster.
Most annoying problem is time desync just after a blackout: mons are not 
starting to work before time sync, after resync and manual restart of monitors, 
some of pgs can stuck in "inactive" or "peering" state for a significant period 
of time, restarting of osds with such pgs can unstuck them.

Pavel.


> 18 марта 2015 г., в 6:32, Jesus Chavez (jeschave)  
> написал(а):
> 
> Hi everyone, I am ready to launch ceph on production but there is one thing 
> that keeps on my mind... If there was a Blackout where all the ceph nodes 
> went off what would really  happen with the filesystem? It would get corrupt? 
> Or ceph has any Kind of mechanism to survive to something like that?
> Thanks
> 
> 
> Jesus Chavez
> SYSTEMS ENGINEER-C.SALES
> 
> jesch...@cisco.com 
> Phone: +52 55 5267 3146 
> Mobile: +51 1 5538883255 
> 
> CCIE - 44433
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Advantages of using Ceph with LXC

[Pavel V. Kaygorodov]

Hi!

> What are a few advantages of using Ceph with LXC ?

I'm using ceph daemons, packed in docker containers (http://docker.io).
The main advantages is security and reliability, the software don't interact 
between each other, all daemons has different IP addresses, different 
filesystems, etc.
Also, almost all of configuration files are shared between containers, all 
containers has mounted configs read-only mode from host machine, so I'm always 
sure for config files consistency.
Main disadvantage -- you will have to install ceph by hands, without automation 
scripts provided.

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Federated gateways (our planning use case)

[Pavel V. Kaygorodov]

Hi!

Our institute now planning to deploy a set of robotic telescopes across a 
country.
Most of the telescopes will have low bandwidth and high latency, or even not 
permanent internet connectivity.
I think, we can set up synchronization of observational data with ceph, using 
federated gateways:

1. The main big storage ceph cluster will be set up in our institute main 
building
2. The small ceph clusters will be set up near each telescope, to store only 
the data from local telescope
3. VPN tunnels will be set up from each telescope site to our institute
4. Federated gateways mechanism will do all the magic to synchronize data

Is this a realistic plan?
What problems we can meet with this setup?

Thanks in advance,
  Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pgs stuck in active+clean+replay state

[Pavel V. Kaygorodov]

Hi!

> I imagine you aren't actually using the data/metadata pool that these
> PGs are in, but it's a previously-reported bug we haven't identified:
> http://tracker.ceph.com/issues/8758
> They should go away if you restart the OSDs that host them (or just
> remove those pools), but it's not going to hurt anything as long as
> you aren't using them.

Thanks a lot, restarting of osds helps!
BTW, I tried to delete data and metadata pools just after setup, but ceph 
refused me to do this.

With best regards,
  Pavel.



> On Thu, Sep 25, 2014 at 3:37 AM, Pavel V. Kaygorodov  wrote:
>> Hi!
>> 
>> 16 pgs in our ceph cluster are in active+clean+replay state more then one 
>> day.
>> All clients are working fine.
>> Is this ok?
>> 
>> root@bastet-mon1:/# ceph -w
>>cluster fffeafa2-a664-48a7-979a-517e3ffa0da1
>> health HEALTH_OK
>> monmap e3: 3 mons at 
>> {1=10.92.8.80:6789/0,2=10.92.8.81:6789/0,3=10.92.8.82:6789/0}, election 
>> epoch 2570, quorum 0,1,2 1,2,3
>> osdmap e3108: 16 osds: 16 up, 16 in
>>  pgmap v1419232: 8704 pgs, 6 pools, 513 GB data, 125 kobjects
>>2066 GB used, 10879 GB / 12945 GB avail
>>8688 active+clean
>>  16 active+clean+replay
>>  client io 3237 kB/s wr, 68 op/s
>> 
>> 
>> root@bastet-mon1:/# ceph pg dump | grep replay
>> dumped all in format plain
>> 0.fd0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:29.902766  0'0 3108:2628
>>[0,7,14,8] [0,7,14,8]   0   0'0 2014-09-23 
>> 02:23:49.463704  0'0 2014-09-23 02:23:49.463704
>> 0.e80   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:21.945082  0'0 3108:1823
>>[2,7,9,10] [2,7,9,10]   2   0'0 2014-09-22 
>> 14:37:32.910787  0'0 2014-09-22 14:37:32.910787
>> 0.aa0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:29.326607  0'0 3108:2451
>>[0,7,15,12][0,7,15,12]  0   0'0 2014-09-23 
>> 00:39:10.717363  0'0 2014-09-23 00:39:10.717363
>> 0.9c0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:29.325229  0'0 3108:1917
>>[0,7,9,12] [0,7,9,12]   0   0'0 2014-09-22 
>> 14:40:06.694479  0'0 2014-09-22 14:40:06.694479
>> 0.9a0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:29.325074  0'0 3108:2486
>>[0,7,14,11][0,7,14,11]  0   0'0 2014-09-23 
>> 01:14:55.825900  0'0 2014-09-23 01:14:55.825900
>> 0.910   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:28.839148  0'0 3108:1962
>>[0,7,9,10] [0,7,9,10]   0   0'0 2014-09-22 
>> 14:37:44.652796  0'0 2014-09-22 14:37:44.652796
>> 0.8c0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:28.838683  0'0 3108:2635
>>[0,2,9,11] [0,2,9,11]   0   0'0 2014-09-23 
>> 01:52:52.390529  0'0 2014-09-23 01:52:52.390529
>> 0.8b0   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:21.215964  0'0 3108:1636
>>[2,0,8,14] [2,0,8,14]   2   0'0 2014-09-23 
>> 01:31:38.134466  0'0 2014-09-23 01:31:38.134466
>> 0.500   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:35.869160  0'0 3108:1801
>>[7,2,15,10][7,2,15,10]  7   0'0 2014-09-20 
>> 08:38:53.963779  0'0 2014-09-13 10:27:26.977929
>> 0.440   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:35.871409  0'0 3108:1819
>>[7,2,15,10][7,2,15,10]  7   0'0 2014-09-20 
>> 11:59:05.208164  0'0 2014-09-20 11:59:05.208164
>> 0.390   0   0   0   0   0   0   
>> active+clean+replay 2014-09-24 02:38:28.653190  0'0 3108:1827
>>[0,2,9,10] [0,2,9,10]   0   0'0 2014-09-22 
>> 14:40:50.697850  0'0 2014-09-22 14:40:50.697850
>> 0.320

[ceph-users] pgs stuck in active+clean+replay state

[Pavel V. Kaygorodov]

Hi!

16 pgs in our ceph cluster are in active+clean+replay state more then one day.
All clients are working fine.
Is this ok?

root@bastet-mon1:/# ceph -w
cluster fffeafa2-a664-48a7-979a-517e3ffa0da1
 health HEALTH_OK
 monmap e3: 3 mons at 
{1=10.92.8.80:6789/0,2=10.92.8.81:6789/0,3=10.92.8.82:6789/0}, election epoch 
2570, quorum 0,1,2 1,2,3
 osdmap e3108: 16 osds: 16 up, 16 in
  pgmap v1419232: 8704 pgs, 6 pools, 513 GB data, 125 kobjects
2066 GB used, 10879 GB / 12945 GB avail
8688 active+clean
  16 active+clean+replay
  client io 3237 kB/s wr, 68 op/s


root@bastet-mon1:/# ceph pg dump | grep replay
dumped all in format plain
0.fd0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:29.902766  0'0 3108:2628   
[0,7,14,8] [0,7,14,8]   0   0'0 2014-09-23 02:23:49.463704  
0'0 2014-09-23 02:23:49.463704
0.e80   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:21.945082  0'0 3108:1823   
[2,7,9,10] [2,7,9,10]   2   0'0 2014-09-22 14:37:32.910787  
0'0 2014-09-22 14:37:32.910787
0.aa0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:29.326607  0'0 3108:2451   
[0,7,15,12][0,7,15,12]  0   0'0 2014-09-23 00:39:10.717363  
0'0 2014-09-23 00:39:10.717363
0.9c0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:29.325229  0'0 3108:1917   
[0,7,9,12] [0,7,9,12]   0   0'0 2014-09-22 14:40:06.694479  
0'0 2014-09-22 14:40:06.694479
0.9a0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:29.325074  0'0 3108:2486   
[0,7,14,11][0,7,14,11]  0   0'0 2014-09-23 01:14:55.825900  
0'0 2014-09-23 01:14:55.825900
0.910   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:28.839148  0'0 3108:1962   
[0,7,9,10] [0,7,9,10]   0   0'0 2014-09-22 14:37:44.652796  
0'0 2014-09-22 14:37:44.652796
0.8c0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:28.838683  0'0 3108:2635   
[0,2,9,11] [0,2,9,11]   0   0'0 2014-09-23 01:52:52.390529  
0'0 2014-09-23 01:52:52.390529
0.8b0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:21.215964  0'0 3108:1636   
[2,0,8,14] [2,0,8,14]   2   0'0 2014-09-23 01:31:38.134466  
0'0 2014-09-23 01:31:38.134466
0.500   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:35.869160  0'0 3108:1801   
[7,2,15,10][7,2,15,10]  7   0'0 2014-09-20 08:38:53.963779  
0'0 2014-09-13 10:27:26.977929
0.440   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:35.871409  0'0 3108:1819   
[7,2,15,10][7,2,15,10]  7   0'0 2014-09-20 11:59:05.208164  
0'0 2014-09-20 11:59:05.208164
0.390   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:28.653190  0'0 3108:1827   
[0,2,9,10] [0,2,9,10]   0   0'0 2014-09-22 14:40:50.697850  
0'0 2014-09-22 14:40:50.697850
0.320   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:10.970515  0'0 3108:1719   
[2,0,14,9] [2,0,14,9]   2   0'0 2014-09-20 12:06:23.716480  
0'0 2014-09-20 12:06:23.716480
0.2c0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:28.647268  0'0 3108:2540   
[0,7,12,8] [0,7,12,8]   0   0'0 2014-09-22 23:44:53.387815  
0'0 2014-09-22 23:44:53.387815
0.1f0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:28.651059  0'0 3108:2522   
[0,2,14,11][0,2,14,11]  0   0'0 2014-09-22 23:38:16.315755  
0'0 2014-09-22 23:38:16.315755
0.7 0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:35.848797  0'0 3108:1739   
[7,0,12,10][7,0,12,10]  7   0'0 2014-09-22 14:43:38.224718  
0'0 2014-09-22 14:43:38.224718
0.3 0   0   0   0   0   0   0   
active+clean+replay 2014-09-24 02:38:08.885066  0'0 3108:1640   
[2,0,11,15][2,0,11,15]  2   0'0 2014-09-20 06:18:55.987318  
0'0 2014-09-20 06:18:55.987318

With best regards,
  Pavel.

___
ceph-users mailing list
cep

[ceph-users] time out of sync after power failure

[Pavel V. Kaygorodov]

Hi!

We have experienced some problems with power supply and whole our ceph cluster 
was rebooted several times.
After a reboot the clocks on different monitor nodes becomes slightly 
desynchronized and ceph won't go up before time sync.
But even after a time sync the ceph cluster also shows that about a half 
(typically, sometimes more, sometimes less) of pgs are in peering state for 
several hours and ceph clients don't have an access to the data.
I have tried to speedup the process manually restarting monitors and osds, 
sometimes with success, sometimes without.

Is there a way to speedup cluster repair after a global reboot?

Thanks in advance,
  Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Pavel V. Kaygorodov]

Hi!

I'm trying to install ceph on Debian wheezy (from deb http://ceph.com/debian/ 
wheezy main) and getting following error:

# apt-get update && apt-get dist-upgrade -y && apt-get install -y ceph

...

The following packages have unmet dependencies:
 ceph : Depends: ceph-common (>= 0.78-500) but it is not going to be installed
Depends: libboost-system1.49.0 (>= 1.49.0-1) but it is not installable
Depends: libboost-thread1.49.0 (>= 1.49.0-1) but it is not installable
Recommends: btrfs-tools but it is not going to be installed
Recommends: ceph-mds but it is not going to be installed
Recommends: librados2 but it is not going to be installed
Recommends: librbd1 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Pavel.



16 июля 2014 г., в 3:45, Sage Weil  написал(а):

> This Firefly point release fixes an potential data corruption problem
> when ceph-osd daemons run on top of XFS and service Firefly librbd
> clients.  A recently added allocation hint that RBD utilizes triggers
> an XFS bug on some kernels (Linux 3.2, and likely others) that leads
> to data corruption and deep-scrub errors (and inconsistent PGs).  This
> release avoids the situation by disabling the allocation hint until we
> can validate which kernels are affected and/or are known to be safe to
> use the hint on.
> 
> We recommend that all v0.80.x Firefly users urgently upgrade,
> especially if they are using RBD.
> 
> Notable Changes
> ---
> 
> * osd: disable XFS extsize hint by default (#8830, Samuel Just)
> * rgw: fix extra data pool default name (Yehuda Sadeh)
> 
> For more detailed information, see:
> 
>  http://ceph.com/docs/master/_downloads/v0.80.4.txt
> 
> Getting Ceph
> 
> 
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
> * For packages, see http://ceph.com/docs/master/install/get-packages
> * For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error initializing cluster client: Error

[Pavel V. Kaygorodov]

07 июля 2014 г., в 22:07, Gregory Farnum  написал(а):

> Do you have a ceph.conf file that the "ceph" tool can access in a
> known location? Try specifying it manually with the "-c ceph.conf"

Genius! -c helped!

I have installed all ceph components (monitors and osds) in separate docker 
containers, single ceph.conf is mounted to each container as /ceph.conf and 
"ceph" tool sometimes can read it, sometimes not... May be it's a docker issue, 
not ceph itself.

Many thanks,
Pavel.



> argument. You can also add "--debug-ms 1, --debug-monc 10" and see if
> it outputs more useful error logs.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Sat, Jul 5, 2014 at 2:23 AM, Pavel V. Kaygorodov  wrote:
>> Hi!
>> 
>> I still have the same problem with "Error initializing cluster client: 
>> Error" on all monitor nodes:
>> 
>> root@bastet-mon2:~# ceph -w
>> Error initializing cluster client: Error
>> 
>> root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
>> mon_status
>> { "name": "2",
>> "rank": 1,
>> "state": "peon",
>> "election_epoch": 1566,
>> "quorum": [
>>   0,
>>   1,
>>   2],
>> "outside_quorum": [],
>> "extra_probe_peers": [],
>> "sync_provider": [],
>> "monmap": { "epoch": 3,
>> "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
>> "modified": "2014-06-19 18:16:01.074917",
>> "created": "2014-06-19 18:14:43.350501",
>> "mons": [
>>   { "rank": 0,
>> "name": "1",
>> "addr": "10.92.8.80:6789\/0"},
>>   { "rank": 1,
>> "name": "2",
>> "addr": "10.92.8.81:6789\/0"},
>>   { "rank": 2,
>> "name": "3",
>> "addr": "10.92.8.82:6789\/0"}]}}
>> 
>> root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
>> quorum_status
>> { "election_epoch": 1566,
>> "quorum": [
>>   0,
>>   1,
>>   2],
>> "quorum_names": [
>>   "1",
>>   "2",
>>   "3"],
>> "quorum_leader_name": "1",
>> "monmap": { "epoch": 3,
>> "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
>> "modified": "2014-06-19 18:16:01.074917",
>> "created": "2014-06-19 18:14:43.350501",
>> "mons": [
>>   { "rank": 0,
>> "name": "1",
>> "addr": "10.92.8.80:6789\/0"},
>>   { "rank": 1,
>> "name": "2",
>> "addr": "10.92.8.81:6789\/0"},
>>   { "rank": 2,
>> "name": "3",
>> "addr": "10.92.8.82:6789\/0"}]}}
>> 
>> root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
>> version{"version":"0.80.1"}
>> 
>> /
>> 
>> The same situation on all 3 monitor nodes, but the cluster is alive and all 
>> clients works fine.
>> Any ideas how to fix this?
>> 
>> Pavel.
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error initializing cluster client: Error

[Pavel V. Kaygorodov]

Hi!

I still have the same problem with "Error initializing cluster client: Error" 
on all monitor nodes:

root@bastet-mon2:~# ceph -w
Error initializing cluster client: Error

root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
mon_status 
{ "name": "2",
  "rank": 1,
  "state": "peon",
  "election_epoch": 1566,
  "quorum": [
0,
1,
2],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 3,
  "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
  "modified": "2014-06-19 18:16:01.074917",
  "created": "2014-06-19 18:14:43.350501",
  "mons": [
{ "rank": 0,
  "name": "1",
  "addr": "10.92.8.80:6789\/0"},
{ "rank": 1,
  "name": "2",
  "addr": "10.92.8.81:6789\/0"},
{ "rank": 2,
  "name": "3",
  "addr": "10.92.8.82:6789\/0"}]}}

root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
quorum_status
{ "election_epoch": 1566,
  "quorum": [
0,
1,
2],
  "quorum_names": [
"1",
"2",
"3"],
  "quorum_leader_name": "1",
  "monmap": { "epoch": 3,
  "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
  "modified": "2014-06-19 18:16:01.074917",
  "created": "2014-06-19 18:14:43.350501",
  "mons": [
{ "rank": 0,
  "name": "1",
  "addr": "10.92.8.80:6789\/0"},
{ "rank": 1,
  "name": "2",
  "addr": "10.92.8.81:6789\/0"},
{ "rank": 2,
  "name": "3",
  "addr": "10.92.8.82:6789\/0"}]}}

root@bastet-mon2:~# ceph --admin-daemon /var/run/ceph/ceph-mon.2.asok 
version{"version":"0.80.1"}

/

The same situation on all 3 monitor nodes, but the cluster is alive and all 
clients works fine.
Any ideas how to fix this?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error 95: Operation not supported

[Pavel V. Kaygorodov]

Hi!

I'm getting a strange error, trying to create rbd image:

# rbd -p images create --size 10 test
rbd: create error: (95) Operation not supported
2014-06-20 18:28:39.537889 7f32af795780 -1 librbd: error adding image to 
directory: (95) Operation not supported

The images -- erasure encoded pool and I can create image in replicated pool.

The erasure coding are not supported by rbd?

Pavel.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] erasure pool & crush ruleset

[Pavel V. Kaygorodov]

> You need:
> 
> type erasure
> 

It works!
Thanks a lot!

Pavel.


   min_size 6
   max_size 6
   step take host1
   step chooseleaf firstn 3 type osd
   step emit
   step take host2
   step chooseleaf firstn 3 type osd
   step emit
 }
> 
> Cheers
> 
>>> Hi,
>>> 
>>> I suggest you test the ruleset created with crushtool to check if what 
>>> comes out of it is what you expect. It's quite convenient to use multiple 
>>> of 10 to visually match the result. For instance 
>>> 
>>>   crushtool -o /tmp/t.map --num_osds 500 --build node straw 10 datacenter 
>>> straw 10 root straw 0
>>> 
>>> then you can
>>> 
>>>   crushtool -c /tmp/t.txt -o /tmp/t.map ; crushtool -i /tmp/t.map 
>>> --show-bad-mappings --show-statistics --test --rule 1 --x 1 --num-rep 12
>>> 
>>> This is the general idea and you can find details about this in the 
>>> crushtool help and the test scripts at
>>> 
>>>   https://github.com/ceph/ceph/tree/master/src/test/cli/crushtool
>>> 
>>> for instance
>>> 
>>>   
>>> https://github.com/ceph/ceph/blob/master/src/test/cli/crushtool/bad-mappings.t
>>> 
>>> which shows what happens when there is a "bad mapping", i.e. the crushmap 
>>> could not be used to get the number of OSD you want. This is most probably 
>>> why pg get stuck.
>>> 
>>> Cheers
>>> 
 I have created an erasure code profile:
 
 ceph osd erasure-code-profile set def33 k=3 m=3
 
 I have created a pool:
 
 ceph osd pool create images 2048 2048 erasure def33 ruleset_3_3
 
 Now I see 2048 pgs permanently in "creating" state.
 
 What is wrong?
 
 Pavel.
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
>>> 
>>> -- 
>>> Loïc Dachary, Artisan Logiciel Libre
>>> 
>> 
> 
> -- 
> Loïc Dachary, Artisan Logiciel Libre

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] erasure pool & crush ruleset

[Pavel V. Kaygorodov]

This ruleset works well for replicated pools with size 6 (I have tested it on 
data and metadata pools, which I cannot delete). 
The erasure pool with k=3 and m=3 must have size 6?

Pavel.

> On 19/06/2014 18:17, Pavel V. Kaygorodov wrote:
>> Hi!
>> 
>> I want to make erasure-coded pool with k=3 and m=3. Also, I want to 
>> distribute data between two hosts, having 3 osd from host1 and 3 from host2.
>> I have created a ruleset:
>> 
>> rule ruleset_3_3 {
>>ruleset 0
>>type replicated
>>min_size 6
>>max_size 6
>>step take host1
>>step chooseleaf firstn 3 type osd
>>step emit
>>step take host2
>>step chooseleaf firstn 3 type osd
>>step emit
>> }
> 
> Hi,
> 
> I suggest you test the ruleset created with crushtool to check if what comes 
> out of it is what you expect. It's quite convenient to use multiple of 10 to 
> visually match the result. For instance 
> 
>crushtool -o /tmp/t.map --num_osds 500 --build node straw 10 datacenter 
> straw 10 root straw 0
> 
> then you can
> 
>crushtool -c /tmp/t.txt -o /tmp/t.map ; crushtool -i /tmp/t.map 
> --show-bad-mappings --show-statistics --test --rule 1 --x 1 --num-rep 12
> 
> This is the general idea and you can find details about this in the crushtool 
> help and the test scripts at
> 
>https://github.com/ceph/ceph/tree/master/src/test/cli/crushtool
> 
> for instance
> 
>
> https://github.com/ceph/ceph/blob/master/src/test/cli/crushtool/bad-mappings.t
> 
> which shows what happens when there is a "bad mapping", i.e. the crushmap 
> could not be used to get the number of OSD you want. This is most probably 
> why pg get stuck.
> 
> Cheers
> 
>> I have created an erasure code profile:
>> 
>> ceph osd erasure-code-profile set def33 k=3 m=3
>> 
>> I have created a pool:
>> 
>> ceph osd pool create images 2048 2048 erasure def33 ruleset_3_3
>> 
>> Now I see 2048 pgs permanently in "creating" state.
>> 
>> What is wrong?
>> 
>> Pavel.
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
> 
> -- 
> Loïc Dachary, Artisan Logiciel Libre
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] erasure pool & crush ruleset

[Pavel V. Kaygorodov]

Hi!

I want to make erasure-coded pool with k=3 and m=3. Also, I want to distribute 
data between two hosts, having 3 osd from host1 and 3 from host2.
I have created a ruleset:

rule ruleset_3_3 {
ruleset 0
type replicated
min_size 6
max_size 6
step take host1
step chooseleaf firstn 3 type osd
step emit
step take host2
step chooseleaf firstn 3 type osd
step emit
}

I have created an erasure code profile:

ceph osd erasure-code-profile set def33 k=3 m=3

I have created a pool:

ceph osd pool create images 2048 2048 erasure def33 ruleset_3_3

Now I see 2048 pgs permanently in "creating" state.

What is wrong?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] switch pool from replicated to erasure coded

[Pavel V. Kaygorodov]

Hi!

May be I have missed something in docs, but is there a way to switch a pool 
from replicated to erasure coded?
Or I have to create a new pool an somehow manually transfer data from old pool 
to new one?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Advanced CRUSH map rules

[Pavel V. Kaygorodov]

Hi!

> CRUSH can do this. You'd have two choose ...emit sequences;
> the first of which would descend down to a host and then choose n-1
> devices within the host; the second would descend once. I think
> something like this should work:
> 
> step take default
> step choose firstn 1 datacenter
> step chooseleaf firstn -1 room
> step emit
> step chooseleaf firstn 1 datacenter
> step emit
> 

May be I'm wrong, but this will not guarantee choice of different datacenters 
for n-1 and remaining replica. 
I have experimented with rules like this, trying to put one replica to "main 
host" and other replicas to some other hosts.
Some OSDs was referenced two times in some of generated pg's.

Pavel.



> Would pick one datacenter, and put R-1 copies of the data in separate
> rooms. Then it would pick another datacenter and put 1 copy of the
> data somewhere in it. I haven't tested this and it's been a while so
> there might be some sharp edges, though (I *think* that should work
> just fine, but you might need to use choose statements instead of
> chooseleaf all the way down or something).
> -Greg
> 
>> And it's even possible to specifiy the "primary affinity" in the sense
>> that for specified clients (or for specified pools) ceph has to store
>> the primary replica in the closest (to the client) datacenter and the
>> other replica in another datacenter.
> 
> To do something like this you'd want to set up pools with special
> rules to do that. Instead of "step take default" you'd do "step take
> ".
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Replace journals disk

[Pavel V. Kaygorodov]

Hi!

I'm not a specialist, but I think it will be better to move journals to other 
place first (stopping each OSD, moving it journal file to a HDD, and starting 
again), replace SSD and move journals to a new drive, again, one-by-one. The 
"no-out" mode can help.

Pavel.



06 мая 2014 г., в 14:34, Gandalf Corvotempesta 
 написал(а):

> Hi to all,
> I would like to replace a disk used as journal (one partition for each OSD)
> 
> Which is the safest method to do so?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD on Mac OS X

[Pavel V. Kaygorodov]

Hi!

I want to use ceph for time machine backups on Mac OS X.
Is it possible to map RBD or mount CephFS on mac directly, for example, using 
osxfuse?
Or it is only way to do this -- make an intermediate linux server?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD as a hot spare

[Pavel V. Kaygorodov]

17 апр. 2014 г., в 16:41, Wido den Hollander  написал(а):

> On 04/17/2014 02:37 PM, Pavel V. Kaygorodov wrote:
>> Hi!
>> 
>> How do you think, is it a good idea, to add RBD block device as a hot spare 
>> drive to a linux software raid?
>> 
> 
> Well, it could work, but why? What is the total setup going to be?
> 
> RAID over a couple of physical disks with RBD as hotspare?

Yes. If it works, it can reduce overall cost of disk subsystem, because I can 
use all physical drives for raid on each host, and add RBD hot spares, which 
(due to thin provisioning) will not consume any real space until hot spares 
become active.

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD as a hot spare

[Pavel V. Kaygorodov]

Hi!

How do you think, is it a good idea, to add RBD block device as a hot spare 
drive to a linux software raid?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph cluster health monitoring

[Pavel V. Kaygorodov]

Hi!

I want to receive email notifications for any ceph errors/warnings and for 
osd/mon disk full/near_full states. For example, I want to know it immediately 
if free space on any osd/mon becomes less then 10%.
How to properly monitor ceph cluster health?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD kernel module / Centos 6.5

[Pavel V. Kaygorodov]

29 марта 2014 г., в 14:37, Arne Wiebalck  написал(а):

> You'll need a newer kernel, try kernel-ml or -lt from elrepo: 
> http://elrepo.org/tiki/tiki-index.php

Thank you, I will try it.
Does rbd module already compiled in the kernels or I need to install additional 
packages?

Pavel.





> HTH,
>  Arne 
> 
> On Mar 29, 2014, at 10:36 AM, "Pavel V. Kaygorodov" 
>  wrote:
> 
>> Hi!
>> 
>> I have followed the instructions on 
>> http://ceph.com/docs/master/start/quick-rbd/ , "ceph-deploy install 
>> localhost" finished without errors, but modprobe rbd returns "FATAL: Module 
>> rbd not found.".
>> 
>> How to install the module?
>> 
>> [root@taurus ~]# lsb_release -a
>> LSB Version:
>> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
>> Distributor ID: CentOS
>> Description:CentOS release 6.5 (Final)
>> Release:6.5
>> Codename:   Final
>> [root@taurus ~]# uname -r
>> 2.6.32-431.5.1.el6.x86_64
>> [root@taurus ~]# ceph --version
>> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
>> 
>> WIth best regards,
>>  Pavel.
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD kernel module / Centos 6.5

[Pavel V. Kaygorodov]

Hi!

I have followed the instructions on 
http://ceph.com/docs/master/start/quick-rbd/ , "ceph-deploy install localhost" 
finished without errors, but modprobe rbd returns "FATAL: Module rbd not 
found.".

How to install the module?

[root@taurus ~]# lsb_release -a
LSB Version:
:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:CentOS release 6.5 (Final)
Release:6.5
Codename:   Final
[root@taurus ~]# uname -r
2.6.32-431.5.1.el6.x86_64
[root@taurus ~]# ceph --version
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)

WIth best regards,
  Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error initializing cluster client: Error

[Pavel V. Kaygorodov]

Hi!

Now I have the same situation on al monitors without any reboot:

root@bes-mon3:~# ceph --verbose -w
Error initializing cluster client: Error

root@bes-mon3:~# ceph --admin-daemon /var/run/ceph/ceph-mon.3.asok mon_status
{ "name": "3",
  "rank": 2,
  "state": "peon",
  "election_epoch": 86,
  "quorum": [
0,
1,
2],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 3,
  "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
  "modified": "2014-03-15 11:52:21.182767",
  "created": "2014-03-15 11:51:42.321256",
  "mons": [
{ "rank": 0,
  "name": "1",
  "addr": "10.92.8.80:6789\/0"},
{ "rank": 1,
  "name": "2",
  "addr": "10.92.8.81:6789\/0"},
{ "rank": 2,
  "name": "3",
  "addr": "10.92.8.82:6789\/0"}]}}

root@bes-mon3:~# ceph --admin-daemon /var/run/ceph/ceph-mon.3.asok quorum_status
{ "election_epoch": 86,
  "quorum": [
0,
1,
2],
  "quorum_names": [
"1",
"2",
"3"],
  "quorum_leader_name": "1",
  "monmap": { "epoch": 3,
  "fsid": "fffeafa2-a664-48a7-979a-517e3ffa0da1",
  "modified": "2014-03-15 11:52:21.182767",
  "created": "2014-03-15 11:51:42.321256",
  "mons": [
{ "rank": 0,
  "name": "1",
  "addr": "10.92.8.80:6789\/0"},
{ "rank": 1,
  "name": "2",
  "addr": "10.92.8.81:6789\/0"},
{ "rank": 2,
  "name": "3",
  "addr": "10.92.8.82:6789\/0"}]}}


root@bes-mon3:~# ceph --admin-daemon /var/run/ceph/ceph-mon.3.asok version
{"version":"0.72.2"}

The rbd image mounted from this cluster seems to be ok, reading and writing 
don't hangs.

Pavel.


23 марта 2014 г., в 8:49, Kyle Bader  написал(а):

>> I have two nodes with 8 OSDs on each. First node running 2 monitors on 
>> different virtual machines (mon.1 and mon.2), second node runing mon.3
>> After several reboots (I have tested power failure scenarios) "ceph -w" on 
>> node 2 always fails with message:
>> 
>> root@bes-mon3:~# ceph --verbose -w
>> Error initializing cluster client: Error
> 
> The cluster is simply protecting itself from a split brain situation.
> Say you have:
> 
> mon.1  mon.2  mon.3
> 
> If mon.1 fails, no big deal, you still have 2/3 so no problem.
> 
> Now instead, say mon.1 is separated from mon.2 and mon.3 because of a
> network partition (trunk failure, whatever). If one monitor of the
> three could elect itself as leader then you might have divergence
> between your monitors. Self-elected mon.1 thinks it's the leader and
> mon.{2,3} have elected a leader amongst themselves. The harsh reality
> is you really need to have monitors on 3 distinct physical hosts to
> protect against the failure of a physical host.
> 
> -- 
> 
> Kyle

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error initializing cluster client: Error

[Pavel V. Kaygorodov]

> You have file config sync?
> 
ceph.conf are same on all servers, keys also not differs.
I have checked the problem now and see ceph -w working fine on all hosts.
Mysterious :-/

Pavel.



> 22 марта 2014 г. 16:11 пользователь "Pavel V. Kaygorodov"  
> написал:
> Hi!
> 
> I have two nodes with 8 OSDs on each. First node running 2 monitors on 
> different virtual machines (mon.1 and mon.2), second node runing mon.3
> After several reboots (I have tested power failure scenarios) "ceph -w" on 
> node 2 always fails with message:
> 
> root@bes-mon3:~# ceph --verbose -w
> Error initializing cluster client: Error
> 
> Logs files are not show any error:
> 
> 2014-03-22 16:05:51.288526 osd.3 10.92.8.103:6800/7492 3510 : [INF] 0.262 
> deep-scrub ok
> 2014-03-22 16:05:54.997444 osd.1 10.92.8.101:6800/7688 3288 : [INF] 1.22b 
> deep-scrub ok
> 2014-03-22 16:06:09.350377 mon.0 10.92.8.80:6789/0 11104 : [INF] pgmap 
> v28682: 12288 pgs: 12288 active+clean; 246 MB data, 18131 MB used, 12928 GB / 
> 12945 GB avail
> 
> 2014-03-22 16:07:24.795144 7f7bf42b4700  1 mon.3@2(peon).paxos(paxos active c 
> 67771..68517) is_readable now=2014-03-22 16:07:24.795145 
> lease_expire=2014-03-22 16:07:29.791889 has v0 lc 68517
> 2014-03-22 16:07:27.795042 7f7bf42b4700  1 mon.3@2(peon).paxos(paxos active c 
> 67771..68517) is_readable now=2014-03-22 16:07:27.795044 
> lease_expire=2014-03-22 16:07:32.792003 has v0 lc 68517
> 
> On the node 1 I have got the same error just after reboots, but now 
> everything seems to be ok:
> 
> root@bastet-mon2:/# ceph -w
> cluster fffeafa2-a664-48a7-979a-517e3ffa0da1
>  health HEALTH_OK
>  monmap e3: 3 mons at 
> {1=10.92.8.80:6789/0,2=10.92.8.81:6789/0,3=10.92.8.82:6789/0}, election epoch 
> 62, quorum 0,1,2 1,2,3
>  osdmap e680: 16 osds: 16 up, 16 in
>   pgmap v28692: 12288 pgs, 6 pools, 246 MB data, 36 objects
> 18131 MB used, 12928 GB / 12945 GB avail
>12288 active+clean
> 
> 
> 2014-03-22 16:08:10.611578 mon.0 [INF] pgmap v28692: 12288 pgs: 12288 
> active+clean; 246 MB data, 18131 MB used, 12928 GB / 12945 GB avail
> 
> 
> 
> How to debug and fix "Error initializing cluster client: Error" problem ?
> 
> With best regards,
>   Pavel.
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error initializing cluster client: Error

[Pavel V. Kaygorodov]

Hi!

I have two nodes with 8 OSDs on each. First node running 2 monitors on 
different virtual machines (mon.1 and mon.2), second node runing mon.3
After several reboots (I have tested power failure scenarios) "ceph -w" on node 
2 always fails with message:

root@bes-mon3:~# ceph --verbose -w
Error initializing cluster client: Error

Logs files are not show any error:

2014-03-22 16:05:51.288526 osd.3 10.92.8.103:6800/7492 3510 : [INF] 0.262 
deep-scrub ok
2014-03-22 16:05:54.997444 osd.1 10.92.8.101:6800/7688 3288 : [INF] 1.22b 
deep-scrub ok
2014-03-22 16:06:09.350377 mon.0 10.92.8.80:6789/0 11104 : [INF] pgmap v28682: 
12288 pgs: 12288 active+clean; 246 MB data, 18131 MB used, 12928 GB / 12945 GB 
avail

2014-03-22 16:07:24.795144 7f7bf42b4700  1 mon.3@2(peon).paxos(paxos active c 
67771..68517) is_readable now=2014-03-22 16:07:24.795145 
lease_expire=2014-03-22 16:07:29.791889 has v0 lc 68517
2014-03-22 16:07:27.795042 7f7bf42b4700  1 mon.3@2(peon).paxos(paxos active c 
67771..68517) is_readable now=2014-03-22 16:07:27.795044 
lease_expire=2014-03-22 16:07:32.792003 has v0 lc 68517

On the node 1 I have got the same error just after reboots, but now everything 
seems to be ok:

root@bastet-mon2:/# ceph -w
cluster fffeafa2-a664-48a7-979a-517e3ffa0da1
 health HEALTH_OK
 monmap e3: 3 mons at 
{1=10.92.8.80:6789/0,2=10.92.8.81:6789/0,3=10.92.8.82:6789/0}, election epoch 
62, quorum 0,1,2 1,2,3
 osdmap e680: 16 osds: 16 up, 16 in
  pgmap v28692: 12288 pgs, 6 pools, 246 MB data, 36 objects
18131 MB used, 12928 GB / 12945 GB avail
   12288 active+clean


2014-03-22 16:08:10.611578 mon.0 [INF] pgmap v28692: 12288 pgs: 12288 
active+clean; 246 MB data, 18131 MB used, 12928 GB / 12945 GB avail



How to debug and fix "Error initializing cluster client: Error" problem ?

With best regards,
  Pavel.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Encryption/Multi-tennancy

[Pavel V. Kaygorodov]

Hi!

I think, it is impossible to hide crypto keys from admin, who have access to 
host machine where VM guest running. Admin can always make snapshot of running 
VM and extract all keys just from memory. May be, you can achieve enough level 
of security providing a dedicated real server holding crypto keys in RAM only 
and somehow guarantee that the server will not be substituted at one fine day 
with VM by malicious admin :)

Pavel.


10 марта 2014 г., в 5:09, Mark s2c  написал(а):

> Ceph is seriously badass, but my requirements are to create a cluster in 
> which I can host my customer's data in separate areas which are independently 
> encrypted, with passphrases which we as cloud admins do not have access to.  
> 
> My current thoughts are:
> 1. Create an OSD per machine stretching over all installed disks, then create 
> a user-sized block device per customer.  Mount this block device on an access 
> VM and create a LUKS container in to it followed by a zpool and then I can 
> allow the users to create separate bins of data as separate ZFS filesystems 
> in the container which is actually a blockdevice striped across the OSDs. 
> 2. Create an OSD per customer and use dm-crypt, then store the dm-crypt key 
> somewhere which is rendered in some way so that we cannot access it, such as 
> a pgp-encrypted file using a passphrase which only the customer knows. 
> 
> My questions are:
> 1. What are people's comments regarding this problem (irrespective of my 
> thoughts)
> 2. Which would be the most efficient of (1) and (2) above?
> 3. As per (1), would it be easy to stretch a created block dev over more OSDs 
> dynamically should we increase the size of one or more? Also, what if we had 
> millions of customers/block devices?
> 
> Any advice on the above would be deluxe.
> 
> M 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] questions about monitor data and ceph recovery

[Pavel V. Kaygorodov]

Hi!

> 2. One node (with 8 osds) goes offline. Will ceph automatically replicate all 
> objects on the remaining node to maintain number of replicas = 2?
> No, because it can no longer satisfy your CRUSH rules. Your crush rule states 
> 1x copy pr. node and it will keep it that way. The cluster will go into a 
> degraded state until you can bring up another node (ie all your data now is 
> very vulnerable). I think it is often suggested to run with 3x replica if 
> possible - or at the very least nr_nodes = replicas + 1. If you had to make 
> it replicate on the remaining node you'd have to change your CRUSH rule to 
> replicate based on OSD and not node. But then you'll most likely have 
> problems when 1 node dies because objects could easily be on 2x OSD on the 
> failed node. 

Is it possible to define "fallback" crush rule, which must work if main rule 
cannot obtain needed number of replicas?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading ceph

[Pavel V. Kaygorodov]

25 февр. 2014 г., в 14:13, Srinivasa Rao Ragolu  написал(а):

> always better to have same version in all the nodes of cluster to avoid 
> integration issues rule out.

But, while updating, some nodes will run on older version for a some period. Is 
this ok?

Pavel.






> On Tue, Feb 25, 2014 at 2:40 PM, Pavel V. Kaygorodov  wrote:
> Hi!
> 
> Is it possible to have monitors and osd daemons running different versions of 
> ceph in one cluster?
> 
> Pavel.
> 
> 
> 
> 
> 25 февр. 2014 г., в 10:56, Srinivasa Rao Ragolu  
> написал(а):
> 
> > Yes Sahana,
> >
> > First of all uninstall ceph packages from your node.
> > then
> > Approach for rpm based:
> > You just open /etc/yum.repos.d/ceph.repo
> > Replace the {ceph-stable-release} with emperor and {distro} with rpm based 
> > distro
> > baseurl=http://ceph.com/rpm-{ceph-stable-release}/{distro}/noarch
> >
> >
> > Now:
> > sudo yum update; sudo yum install {ceph-***whatever**}
> >
> >
> > Thanks,
> > Srinivas.
> >
> >
> >
> >
> >
> > On Tue, Feb 25, 2014 at 11:56 AM, Sahana  wrote:
> > Hi ,
> >
> > May I know how to upgrade ceph from one version to another(Eg : Dumpling to 
> > emperor). Is there any specific steps ?
> >
> > Thanks
> > Sahana
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] questions about monitor data and ceph recovery

[Pavel V. Kaygorodov]

Hi!

My first question will be about monitor data directory. How much space I need 
to reserve for it? Can monitor-fs be corrupted if monitor goes out of storage 
space? 

I also have questions about ceph auto-recovery process.
For example, I have two nodes with 8 drives on each, each drive is presented as 
separate osd. The number of replicas = 2. I have wrote a crush ruleset, which 
picks two nodes and one osd on each to store replicas. Which will happens on 
following scenarios:

1. One drive in one node failed. Will ceph automatically re-replicate affected 
objects? Where replicas will be stored?

1.1 The failed osd will appears online again with all of it's data. How ceph 
cluster will deal with it?

2. One node (with 8 osds) goes offline. Will ceph automatically replicate all 
objects on the remaining node to maintain number of replicas = 2?

2.1 The failed node goes online again with all data. How ceph cluster will deal 
with it?

Thanks in advance,
  Pavel.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph osd create with uuid & ceph-osd --mkfs

[Pavel V. Kaygorodov]

Hi!

I have found strange behavior of ceph-osd, which must be documented, in my 
opinion:

While creating osd fs (with ceph-osd --mkfs), ceph-osd looking for UUID in 
ceph.conf only, if there are no "osd uuid = ..." line, it not asking monitor 
for uuid and just generates random one.
If one has previously created osd by "ceph osd create ", the created 
osd uuid will conflict with uuid stored by monitor and osd will never appears 
"up".

May be it will be better for ceph-osd to ask monitor for uuid for given id and 
use it, if no uuid provided in ceph.conf, and display warning if uuid, returned 
by monitor conflicts with uuid, stored on the disk.

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph osd create with uuid & ceph-osd --mkfs

[Pavel V. Kaygorodov]

Hi!

I have found strange behavior of ceph-osd, which must be documented, in my 
opinion:

While creating osd fs (with ceph-osd --mkfs), ceph-osd looking for UUID in 
ceph.conf only, if there are no "osd uuid = ..." line, it not asking monitor 
for uuid and just generates random one.
If one has previously created osd by "ceph osd create ", the created 
osd uuid will conflict with uuid stored by monitor and osd will never appears 
"up".

May be it will be better for ceph-osd to ask monitor for uuid for given id and 
use it, if no uuid provided in ceph.conf, and display warning if uuid, returned 
by monitor conflicts with uuid, stored on the disk.

Pavel.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon segmentation fault

[Pavel V. Kaygorodov]

Hi!

>>> I created a ticket: http://tracker.ceph.com/issues/7487
>>> 
>>> But my guess is that this is a result of having 0 CRUSH weight for the
>>> entire tree while linking them up. Can you give the OSD a weight and
>>> see if it works after that?
>> 
>> How to do this?
>> I still not very familiar with ceph tools yet :)
> 
> See http://ceph.com/docs/master/rados/operations/add-or-rm-osds/#adding-osds
> In particular you'll want to use "ceph osd reweight 
> ". (The weight should probably just be 1, or the disk size in
> TB, or similar.)
> 

I have tried following command:
root@ceph-base:/# ceph osd reweight 0 1

nothing is changed:

root@ceph-base:/# ceph osd tree
# idweight  type name   up/down reweight
-3  0   osd osd.0
-1  0   root default
-2  0   host odd-host

The following command caused a error:

root@ceph-base:/# ceph osd reweight osd.0 1
Invalid command:  osd.0 doesn't represent an int
osd reweight   :  reweight osd to 0.0 <  < 1.0
Error EINVAL: invalid command

The problem still exists:

root@ceph-base:/# ceph osd reweight 0 1
root@ceph-base:/# ceph osd crush move osd.0 host=osd-host
2014-02-20 21:52:30.580751 7f2cf92f2700  0 monclient: hunting for new mon
2014-02-20 21:52:30.580943 7f2cf81ef700  0 -- 172.17.0.223:0/1000358 >> 
172.17.0.222:6789/0 pipe(0x7f2ce80046e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f2ce8004940).fault

> I assumed you were basically following those steps already!


I just have played with ceph, trying to resolve my "osd down" problem.
Finally, just right now I have found the solution:

If I use "ceph odd create" command with UUID, ceph-osd --mkfs created fs not 
taking this UUID into account, so it cannot connect to the monitor after all. 
Removing uuid parameter from "ceph osd create" fixes the problem.
If this is not a bug, may be it will be better to document this behavior.

With best regards,
  Pavel.




>> Pavel.
>> 
>> 
>>> -Greg
>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>> 
>>> 
>>> On Tue, Feb 18, 2014 at 4:21 AM, Pavel V. Kaygorodov  
>>> wrote:
>>>> Hi!
>>>> 
>>>> Playing with ceph, I found a bug:
>>>> 
>>>> I have compiled and installed ceph from sources on debian/jessie:
>>>> 
>>>> git clone --recursive -b v0.75 https://github.com/ceph/ceph.git
>>>> cd ceph/ && ./autogen.sh && ./configure && make && make install
>>>> 
>>>> /usr/local/bin/ceph-authtool --create-keyring /data/ceph.mon.keyring 
>>>> --gen-key -n mon. --cap mon 'allow *'
>>>> /usr/local/bin/ceph-authtool --create-keyring /ceph.client.admin.keyring 
>>>> --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow 
>>>> *' --cap mds 'allow'
>>>> /usr/local/bin/ceph-authtool /data/ceph.mon.keyring --import-keyring 
>>>> /ceph.client.admin.keyring
>>>> /usr/local/bin/monmaptool --create --fsid 
>>>> e90dfd37-98d1-45bb-a847-8590a5ed8e71 /data/monmap
>>>> /usr/local/bin/ceph-mon --mkfs -i ceph-mon.dkctl --monmap /data/monmap 
>>>> --keyring /data/ceph.mon.keyring
>>>> 
>>>> my ceph.conf is (I have configured local TLD dkctl. with ceph-mon 
>>>> A-record):
>>>> 
>>>> [global]
>>>> 
>>>> fsid = e90dfd37-98d1-45bb-a847-8590a5ed8e71
>>>> mon initial members = ceph-mon.dkctl
>>>> 
>>>> auth cluster required = cephx
>>>> auth service required = cephx
>>>> auth client required = cephx
>>>> 
>>>> keyring = /ceph.client.admin.keyring
>>>> 
>>>> osd pool default size = 2
>>>> osd pool default min size = 2
>>>> osd pool default pg num = 333
>>>> osd pool default pgp num = 333
>>>> osd crush chooseleaf type = 1
>>>> osd journal size = 1000
>>>> 
>>>> filestore xattr use omap = true
>>>> 
>>>> mon host = ceph-mon.dkctl
>>>> mon addr = ceph-mon.dkctl
>>>> 
>>>> log file = /data/logs/ceph.log
>>>> 
>>>> [mon]
>>>> mon data = /data/mon
>>>> keyring = /data/ceph.mon.keyring
>>>> log file = /data/logs/mon.log
>>>> 
>>>> [osd.0]
>>>> osd host= osd0
>>>> osd data= /data/osd
>>>> osd journal = /data/osd.journal
>>>> log file= /data/logs/osd.log
>>>

[ceph-users] monitor data

[Pavel V. Kaygorodov]

Hi!

May be it is a dumb question, but anyway: 

If I lose all monitors (mon data dirs), does it possible to recover cluster 
with data from OSDs only?

Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] smart replication

[Pavel V. Kaygorodov]

Hi!

I have two sorts of storage hosts: small number of reliable hosts with a number 
of big drives on each (reliable zone of the cluster), and a much larger set of 
less reliable hosts, some with big drives, some with relatively small ones 
(non-reliable zone of the cluster). Non-reliable hosts are bad-admined, can be 
connected and disconnected, drives can be crashed without replacement, etc. 
Reliable hosts are well-admined, have a good power and network connectivity, 
faulty drives are quickly replaced etc.

I also have a two sorts of data: important (virtual machines, mailboxes, etc) 
and unimportant -- snapshots, backups, etc.
I want to have two copies of important data objects in reliable zone and some 
(1-2) copies on non-reliable zone. If there are no space in non-releable zone 
are available, I want to have exactly two copies in reliable zone.
For non-important data I want to have two copies in non-releable zone. If there 
are no space for two copies in non-releable zone, I want to have exactly one 
copy of this data in reliable zone and, if possible, one copy in non-reliable 
zone.

Is it possible to do this with ceph?
If yes, how to configure this?

Thanks in advance,
  Pavel.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-mon segmentation fault

[Pavel V. Kaygorodov]

Hi!

Playing with ceph, I found a bug:

I have compiled and installed ceph from sources on debian/jessie:

git clone --recursive -b v0.75 https://github.com/ceph/ceph.git
cd ceph/ && ./autogen.sh && ./configure && make && make install

/usr/local/bin/ceph-authtool --create-keyring /data/ceph.mon.keyring --gen-key 
-n mon. --cap mon 'allow *'
/usr/local/bin/ceph-authtool --create-keyring /ceph.client.admin.keyring 
--gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' 
--cap mds 'allow'
/usr/local/bin/ceph-authtool /data/ceph.mon.keyring --import-keyring 
/ceph.client.admin.keyring
/usr/local/bin/monmaptool --create --fsid e90dfd37-98d1-45bb-a847-8590a5ed8e71 
/data/monmap
/usr/local/bin/ceph-mon --mkfs -i ceph-mon.dkctl --monmap /data/monmap 
--keyring /data/ceph.mon.keyring

my ceph.conf is (I have configured local TLD dkctl. with ceph-mon A-record):

[global]

fsid = e90dfd37-98d1-45bb-a847-8590a5ed8e71
mon initial members = ceph-mon.dkctl

auth cluster required = cephx
auth service required = cephx
auth client required = cephx

keyring = /ceph.client.admin.keyring

osd pool default size = 2
osd pool default min size = 2
osd pool default pg num = 333 
osd pool default pgp num = 333
osd crush chooseleaf type = 1
osd journal size = 1000

filestore xattr use omap = true

mon host = ceph-mon.dkctl
mon addr = ceph-mon.dkctl

log file = /data/logs/ceph.log

[mon]
mon data = /data/mon
keyring = /data/ceph.mon.keyring
log file = /data/logs/mon.log

[osd.0]
osd host= osd0
osd data= /data/osd
osd journal = /data/osd.journal
log file= /data/logs/osd.log
keyring = /data/ceph.osd.keyring

started ceph-mon:

/usr/local/bin/ceph-mon -c /ceph.conf --public-addr `grep ceph-mon /etc/hosts | 
awk '{print $1}'` -i ceph-mon.dkctl

After that following commands crushed ceph-mon daemon:

root@ceph-mon:/# ceph osd crush add-bucket osd-host host
added bucket osd-host type host to crush map
root@ceph-mon:/# ceph osd crush move osd-host root=default
moved item id -2 name 'osd-host' to location {root=default} in crush map
root@ceph-mon:/# ceph osd crush add-bucket osd.0 osd
added bucket osd.0 type osd to crush map
root@ceph-mon:/# ceph osd tree
# idweight  type name   up/down reweight
-3  0   osd osd.0
-1  0   root default
-2  0   host osd-host

root@ceph-mon:/# ceph osd crush move osd.0 host=osd-host
2014-02-18 16:00:14.093243 7ff077fff700  0 monclient: hunting for new mon
2014-02-18 16:00:14.093781 7ff07c130700  0 -- 172.17.0.160:0/1000148 >> 
172.17.0.160:6789/0 pipe(0x7ff06c004770 sd=4 :0 s=1 pgs=0 cs=0 l=1 
c=0x7ff06c0049d0).fault
2014-02-18 16:00:16.996981 7ff07c231700  0 -- 172.17.0.160:0/1000148 >> 
172.17.0.160:6789/0 pipe(0x7ff06c00 sd=5 :0 s=1 pgs=0 cs=0 l=1 
c=0x7ff06e60).fault
2014-02-18 16:00:19.998108 7ff07c130700  0 -- 172.17.0.160:0/1000148 >> 
172.17.0.160:6789/0 pipe(0x7ff060003010 sd=5 :0 s=1 pgs=0 cs=0 l=1 
c=0x7ff060001e70).fault

Log file of ceph mon shows:

*** Caught signal (Segmentation fault) **
 in thread 7f09109dd700
 ceph version 0.75 (946d60369589d6a269938edd65c0a6a7b1c3ef5c)
 1: /usr/local/bin/ceph-mon() [0x83457e]
 2: (()+0xf210) [0x7f0915772210]
 3: /usr/local/bin/ceph-mon() [0x7c398a]
 4: /usr/local/bin/ceph-mon() [0x7c3c9c]
 5: /usr/local/bin/ceph-mon() [0x7c3d31]
 6: (crush_do_rule()+0x20a) [0x7c448a]
 7: (OSDMap::_pg_to_osds(pg_pool_t const&, pg_t, std::vector >&) const+0xdd) [0x725add]
 8: (OSDMap::pg_to_acting_osds(pg_t, std::vector >&) 
const+0x81) [0x725da1]
 9: (PGMonitor::map_pg_creates()+0x15f) [0x610abf]
 10: (PGMonitor::post_paxos_update()+0x25) [0x611205]
 11: (Monitor::refresh_from_paxos(bool*)+0x95) [0x543205]
 12: (Paxos::do_refresh()+0x24) [0x590c24]
 13: (Paxos::begin(ceph::buffer::list&)+0x99e) [0x59b54e]
 14: (Paxos::propose_queued()+0xdd) [0x59b92d]
 15: (Paxos::propose_new_value(ceph::buffer::list&, Context*)+0x150) [0x59ca30]
 16: (PaxosService::propose_pending()+0x6d9) [0x5a3099]
 17: (PaxosService::dispatch(PaxosServiceMessage*)+0xd77) [0x5a4347]
 18: (Monitor::handle_command(MMonCommand*)+0x1073) [0x56e253]
 19: (Monitor::dispatch(MonSession*, Message*, bool)+0x2e8) [0x571168]
 20: (Monitor::_ms_dispatch(Message*)+0x1e4) [0x571774]
 21: (Monitor::ms_dispatch(Message*)+0x20) [0x590050]
 22: (DispatchQueue::entry()+0x56a) [0x80a65a]
 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x73e75d]
 24: (()+0x7e0e) [0x7f091576ae0e]
25: (clone()+0x6d) [0x7f0913d1c0fd]
2014-02-18 16:00:14.088851 7f09109dd700 -1 *** Caught signal (Segmentation fault
) **
 in thread 7f09109dd700

 ceph version 0.75 (946d60369589d6a269938edd65c0a6a7b1c3ef5c)
 1: /usr/local/bin/ceph-mon() [0x83457e]
 2: (()+0xf210) [0x7f0915772210]
 3: /usr/local/bin/ceph-mon() [0x7c398a]
 4: /usr/local/bin/ceph-mon() [0x7c3c9c]
 5: /usr/local/bin/ceph-mon() [0x7c3d31]
 6: (crush_do_rule()+0x20a) [0x7c448a]
 7: (OSDMap::_pg_to_osds(pg_pool_t const&, pg_t, std::vector >&) const+0xdd) [0x725add]
 8: (OSDMap::pg_to_acting_osds(pg_t, st

Re: [ceph-users] osd down

[Pavel V. Kaygorodov]

Hi!

I have tried, but situation not changed significantly:

# ceph -w
cluster e90dfd37-98d1-45bb-a847-8590a5ed8e71
 health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean; 2/2 in 
osds are down
 monmap e1: 1 mons at {host1=172.17.0.4:6789/0}, election epoch 1, quorum 0 
host1
 osdmap e9: 2 osds: 0 up, 2 in
  pgmap v10: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
 192 creating
2014-02-16 17:25:29.872538 mon.0 [INF] osdmap e9: 2 osds: 0 up, 2 in

# ceph osd tree
# idweight  type name   up/down reweight
-1  2   root default
-2  2   host host1
0   1   osd.0   down1
1   1   osd.1   down1

# ceph health
HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean; 2/2 in osds are down

ps showed both osd daemons running.

Pavel.

17 февр. 2014 г., в 1:50, Karan Singh  написал(а):

> Hi Pavel
> 
> Try to add at least 1 more OSD ( bare minimum ) and set pool replication to 2 
> after that.
> For osd.0  try  ,   # ceph osd in osd.0   , once the osd is IN , try to bring 
> up osd.0 services up 
> 
> 
> Finally your both the OSD should be  IN  and UP , so that your cluster can 
> store data.
> 
> Regards
> Karan
> 
> 
> On 16 Feb 2014, at 20:06, Pavel V. Kaygorodov  wrote:
> 
>> Hi, All!
>> 
>> I am trying to setup ceph from scratch, without dedicated drive, with one 
>> mon and one osd.
>> After all, I see following output of ceph osd tree:
>> 
>> # idweight  type name   up/down reweight
>> -1  1   root default
>> -2  1   host host1
>> 0   1   osd.0   down0
>> 
>> ceph -w:
>> 
>>cluster e90dfd37-98d1-45bb-a847-8590a5ed8e71
>> health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
>> monmap e1: 1 mons at {host1=172.17.0.4:6789/0}, election epoch 1, quorum 
>> 0 host1
>> osdmap e5: 1 osds: 0 up, 0 in
>>  pgmap v6: 192 pgs, 3 pools, 0 bytes data, 0 objects
>>0 kB used, 0 kB / 0 kB avail
>> 192 creating
>> 
>> 2014-02-16 13:27:30.095938 mon.0 [INF] osdmap e5: 1 osds: 0 up, 0 in
>> 
>> What can be wrong?
>> I see working daemons, and nothing bad in log files.
>> 
>> 
>> 
>> How to reproduce:
>> I have cloned and compiled sources on debian/jessie:
>> 
>> git clone --recursive -b v0.75 https://github.com/ceph/ceph.git
>> cd /ceph/ && ./autogen.sh && ./configure && make && make install
>> 
>> Everything seems ok.
>> 
>> I have created ceph.conf:
>> 
>> [global]
>> 
>> fsid = e90dfd37-98d1-45bb-a847-8590a5ed8e71
>> mon initial members = host1
>> 
>> auth cluster required = cephx
>> auth service required = cephx
>> auth client required = cephx
>> 
>> keyring = /data/ceph.client.admin.keyring
>> 
>> osd pool default size = 1
>> osd pool default min size = 1
>> osd pool default pg num = 333
>> osd pool default pgp num = 333
>> osd crush chooseleaf type = 0   
>> osd journal size = 1000
>> 
>> filestore xattr use omap = true
>> 
>> ;journal dio = false
>> ;journal aio = false
>> 
>> mon addr = ceph.dkctl
>> mon host = ceph.dkctl
>> 
>> log file = /data/logs/ceph.log
>> 
>> [mon]
>> mon data = /data/mon0
>> keyring = /data/ceph.mon.keyring
>> log file = /data/logs/mon0.log
>> 
>> [osd.0]
>> osd host= host1
>> osd data= /data/osd0
>> osd journal = /data/osd0.journal
>> log file= /data/logs/osd0.log
>> keyring = /data/ceph.osd0.keyring
>> 
>> ///
>> 
>> I have initialized mon and osd using following script:
>> 
>> /usr/local/bin/ceph-authtool --create-keyring /data/ceph.mon.keyring 
>> --gen-key -n mon. --cap mon 'allow *'
>> /usr/local/bin/ceph-authtool --create-keyring 
>> /data/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap 
>> mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
>> /usr/local/bin/ceph-authtool /data/ceph.mon.keyring --import-keyring 
>> /data/ceph.client.admin.keyring 
>> /usr/local/bin/monmaptool --create --add host1 `grep ceph /etc/hosts | awk 
>> '{print $1}'` --fsid de90dfd37-98d1-45bb-a847-8590a5ed8e71 /data/monmap
>> /usr/local/bin/ceph-mon --mkfs -i host1 --monmap /data/monmap --keyring 
>> /data/ceph.mon.keyring
>> /usr/local

[ceph-users] osd down

[Pavel V. Kaygorodov]

Hi, All!

I am trying to setup ceph from scratch, without dedicated drive, with one mon 
and one osd.
After all, I see following output of ceph osd tree:

# idweight  type name   up/down reweight
-1  1   root default
-2  1   host host1
0   1   osd.0   down0

ceph -w:

cluster e90dfd37-98d1-45bb-a847-8590a5ed8e71
 health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
 monmap e1: 1 mons at {host1=172.17.0.4:6789/0}, election epoch 1, quorum 0 
host1
 osdmap e5: 1 osds: 0 up, 0 in
  pgmap v6: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
 192 creating

2014-02-16 13:27:30.095938 mon.0 [INF] osdmap e5: 1 osds: 0 up, 0 in

What can be wrong?
I see working daemons, and nothing bad in log files.



How to reproduce:
I have cloned and compiled sources on debian/jessie:

git clone --recursive -b v0.75 https://github.com/ceph/ceph.git
cd /ceph/ && ./autogen.sh && ./configure && make && make install

Everything seems ok.

I have created ceph.conf:

[global]

fsid = e90dfd37-98d1-45bb-a847-8590a5ed8e71
mon initial members = host1

auth cluster required = cephx
auth service required = cephx
auth client required = cephx

keyring = /data/ceph.client.admin.keyring

osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 0   
osd journal size = 1000

filestore xattr use omap = true

;journal dio = false
;journal aio = false

mon addr = ceph.dkctl
mon host = ceph.dkctl

log file = /data/logs/ceph.log

[mon]
mon data = /data/mon0
keyring = /data/ceph.mon.keyring
log file = /data/logs/mon0.log

[osd.0]
osd host= host1
osd data= /data/osd0
osd journal = /data/osd0.journal
log file= /data/logs/osd0.log
keyring = /data/ceph.osd0.keyring

///

I have initialized mon and osd using following script:

/usr/local/bin/ceph-authtool --create-keyring /data/ceph.mon.keyring --gen-key 
-n mon. --cap mon 'allow *'
/usr/local/bin/ceph-authtool --create-keyring /data/ceph.client.admin.keyring 
--gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' 
--cap mds 'allow'
/usr/local/bin/ceph-authtool /data/ceph.mon.keyring --import-keyring 
/data/ceph.client.admin.keyring 
/usr/local/bin/monmaptool --create --add host1 `grep ceph /etc/hosts | awk 
'{print $1}'` --fsid de90dfd37-98d1-45bb-a847-8590a5ed8e71 /data/monmap
/usr/local/bin/ceph-mon --mkfs -i host1 --monmap /data/monmap --keyring 
/data/ceph.mon.keyring
/usr/local/bin/ceph-mon -c /ceph.conf --public-addr `grep ceph /etc/hosts | awk 
'{print $1}'` -i host1
/usr/local/bin/ceph osd create e90dfd37-98d1-45bb-a847-8590a5ed8e71
/usr/local/bin/ceph osd create e90dfd37-98d1-45bb-a847-8590a5ed8e71
/usr/local/bin/ceph-osd -i 0 --mkfs --mkkey
/usr/local/bin/ceph auth add osd.0 osd 'allow *' mon 'allow rwx'  -i 
/data/ceph.osd0.keyring
/usr/local/bin/ceph osd crush add-bucket host1 host
/usr/local/bin/ceph osd crush move host1 root=default
/usr/local/bin/ceph osd crush add osd.0 1.0 host=host1



Script ouptut seems to be ok:

creating /data/ceph.mon.keyring
creating /data/ceph.client.admin.keyring
importing contents of /data/ceph.client.admin.keyring into 
/data/ceph.mon.keyring
/usr/local/bin/monmaptool: monmap file /data/monmap
/usr/local/bin/monmaptool: set fsid to e90dfd37-98d1-45bb-a847-8590a5ed8e71
/usr/local/bin/monmaptool: writing epoch 0 to /data/monmap (1 monitors)
/usr/local/bin/ceph-mon: set fsid to e90dfd37-98d1-45bb-a847-8590a5ed8e71
/usr/local/bin/ceph-mon: created monfs at /data/mon0 for mon.host1
0
2014-02-16 13:24:37.833469 7f5ef61747c0 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio to force use of aio 
anyway
2014-02-16 13:24:37.94 7f5ef61747c0 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio to force use of aio 
anyway
2014-02-16 13:24:37.948704 7f5ef61747c0 -1 filestore(/data/osd0) could not find 
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-02-16 13:24:38.054345 7f5ef61747c0 -1 created object store /data/osd0 
journal /data/osd0.journal for osd.0 fsid e90dfd37-98d1-45bb-a847-8590a5ed8e71
2014-02-16 13:24:38.054427 7f5ef61747c0 -1 auth: error reading file: 
/data/ceph.osd0.keyring: can't open /data/ceph.osd0.keyring: (2) No such file 
or directory
2014-02-16 13:24:38.054529 7f5ef61747c0 -1 created new key in keyring 
/data/ceph.osd0.keyring
added key for osd.0
added bucket host1 type host to crush map
moved item id -2 name 'host1' to location {root=default} in crush map
add item id 0 name 'osd.0' weight 1 at location {host=host1} to crush map

///

I am started daemons, using commands:

/usr/local/bin/ceph-mon -