Re: [ceph-users] How can we repair OSD leveldb?

[Christian Balzer]

Hello,

On Wed, 17 Aug 2016 16:54:41 -0500 Dan Jakubiec wrote:

> Hi Wido,
> 
> Thank you for the response:
> 
> > On Aug 17, 2016, at 16:25, Wido den Hollander  wrote:
> > 
> > 
> >> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec :
> >> 
> >> 
> >> Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 
> >> machines.  We've managed to recover the XFS filesystems on 7 of the 
> >> machines, but the OSD service is only starting on 1 of them.
> >> 
> >> The other 5 machines all have complaints similar to the following:
> >> 
> >>2016-08-17 09:32:15.549588 7fa2f4666800 -1 
> >> filestore(/var/lib/ceph/osd/ceph-1) Error initializing leveldb : 
> >> Corruption: 6 missing files; e.g.: 
> >> /var/lib/ceph/osd/ceph-1/current/omap/042421.ldb
> >>

That looks bad. 
And as Wido said, this shouldn't happen.
What are your XFS mount options for that FS?
I tend to remember seeing "nobarrier" in many OSD examples...

> >> How can we repair the leveldb to allow the OSDs to startup?  

Hopefully somebody with a leveldb clue will pipe up, but I have grave
doubts.

> >> 
> > 
> > My first question would be: How did this happen?
> > 
> > What hardware are you using underneath? Is there a RAID controller which is 
> > not flushing properly? Since this should not happen during a power failure.
> > 
> 
> Each OSD drive is connected to an onboard hardware RAID controller and 
> configured in RAID 0 mode as individual virtual disks.  The RAID controller 
> is an LSI 3108.
> 
What are the configuration options?
If there is no BBU and the controller is forcibly set to writeback
caching, this would explain it, too.

> I agree -- I am finding it bizarre that 7 of our 8 OSDs (one per machine) did 
> not survive the power outage.  
>
My philosophy on this is that if any of DCs we're in should suffer a total
and abrupt power loss I won't care, as I'll be buried below tons of
concrete (this being Tokyo).

In a place were power outages are more likely, I'd put local APU in front
of stuff and issues a remote shutdown from it when it starts to run out
of juice.

Having a HW/SW combo that can survive a sudden power loss is nice, having
something in place that softly shuts down things before that is a lot
better.
 
> We did have some problems with the stock Ubunut xfs_repair (3.1.9) seg 
> faulting, which eventually we overcame by building a newer version of 
> xfs_repair (4.7.0).  But it did finally repair clean.
> 
That also doesn't instill me with confidence, both Ubuntu and XFS wise.

> We actually have some different errors on other OSDs.  A few of them are 
> failing with "Missing map in load_pgs" errors.  But generally speaking it 
> appears to be missing files of various types causing different kinds of 
> failures.
> 
> I'm really nervous now about the OSD's inability to start with any 
> inconsistencies and no repair utilities (that I can find).  Any advice on how 
> to recover?
>
What I've seen in the past assumes that you have at least a running
cluster of sorts, just trashed PGs. 
This is far worse.

Christian 
> > I don't know the answer to your question, but lost files are not good.
> > 
> > You might find them in a lost+found directory if XFS repair worked?
> > 
> 
> Sadly this directory is empty.
> 
> -- Dan
> 
> > Wido
> > 
> >> Thanks,
> >> 
> >> -- Dan J___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to Local Disks

[Gaurav Goyal]

As it is a lab environment, can i install the setup in a way to achieve
less redundancy (replication factor) and more capacity?

How can i achieve that?




On Wed, Aug 17, 2016 at 7:47 PM, Gaurav Goyal 
wrote:

> Hello,
>
> Awaiting any suggestion please!
>
>
>
>
> Regards
>
> On Wed, Aug 17, 2016 at 9:59 AM, Gaurav Goyal 
> wrote:
>
>> Hello Brian,
>>
>> Thanks for your response!
>>
>> Can you please elaborate on this.
>>
>> Do you mean i must use
>>
>> 4 x 1TB HDD on each nodes rather than 2 x 2TB?
>>
>> This is going to be a lab environment. Can you please suggest to have
>> best possible design for my lab environment.
>>
>>
>>
>> On Wed, Aug 17, 2016 at 9:54 AM, Brian ::  wrote:
>>
>>> You're going to see pretty slow performance on a cluster this size
>>> with spinning disks...
>>>
>>> Ceph scales very very well but at this type of size cluster it can be
>>> challenging to get nice throughput and iops..
>>>
>>> for something small like this either use all ssd osds or consider
>>> having more spinning osds per node backed by nvme or ssd journals..
>>>
>>>
>>>
>>> On Wed, Aug 17, 2016 at 1:14 PM, Gaurav Goyal 
>>> wrote:
>>> > Dear Ceph Users,
>>> >
>>> > Can you please address my scenario and suggest me a solution.
>>> >
>>> > Regards
>>> > Gaurav Goyal
>>> >
>>> > On Tue, Aug 16, 2016 at 11:13 AM, Gaurav Goyal <
>>> er.gauravgo...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hello
>>> >>
>>> >>
>>> >> I need your help to redesign my ceph storage network.
>>> >>
>>> >> As suggested in earlier discussions, i must not use SAN storage. So we
>>> >> have decided to removed it.
>>> >>
>>> >> Now we are ordering Local HDDs.
>>> >>
>>> >> My Network would be
>>> >>
>>> >> Host1 --> Controller + COmpute --> Local Disk 600GB Host 2-->
>>> Compute2 -->
>>> >> Local Disk 600GB Host 3 --> Compute2
>>> >>
>>> >> Is it right setup for ceph network? For Host1 and Host2 , we are
>>> using 1
>>> >> 600GB disk for basic filesystem.
>>> >>
>>> >> Should we use same size storage disks for ceph environment or i can
>>> order
>>> >> Disks in size of 2TB for ceph cluster?
>>> >>
>>> >> Making it
>>> >>
>>> >> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>>> >>
>>> >> 12TB in total. replication factor 2 should make it 6 TB?
>>> >>
>>> >>
>>> >> Regards
>>> >>
>>> >> Gaurav Goyal
>>> >>
>>> >>
>>> >> On Thu, Aug 4, 2016 at 1:52 AM, Bharath Krishna <
>>> bkris...@walmartlabs.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Gaurav,
>>> >>>
>>> >>> There are several ways to do it depending on how you deployed your
>>> ceph
>>> >>> cluster. Easiest way to do it is using ceph-ansible with
>>> purge-cluster yaml
>>> >>> ready made to wipe off CEPH.
>>> >>>
>>> >>> https://github.com/ceph/ceph-ansible/blob/master/purge-cluster.yml
>>> >>>
>>> >>> You may need to configure ansible inventory with ceph hosts.
>>> >>>
>>> >>> Else if you want to purge manually, you can do it using:
>>> >>> http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-purge/
>>> >>>
>>> >>>
>>> >>> Thanks
>>> >>> Bharath
>>> >>>
>>> >>> From: ceph-users  on behalf of
>>> Gaurav
>>> >>> Goyal 
>>> >>> Date: Thursday, August 4, 2016 at 8:19 AM
>>> >>> To: David Turner 
>>> >>> Cc: ceph-users 
>>> >>> Subject: Re: [ceph-users] Fwd: Ceph Storage Migration from SAN
>>> storage to
>>> >>> Local Disks
>>> >>>
>>> >>> Please suggest a procedure for this uninstallation process?
>>> >>>
>>> >>>
>>> >>> Regards
>>> >>> Gaurav Goyal
>>> >>>
>>> >>> On Wed, Aug 3, 2016 at 5:58 PM, Gaurav Goyal
>>> >>> > wrote:
>>> >>>
>>> >>> Thanks for your  prompt
>>> >>> response!
>>> >>>
>>> >>> Situation is bit different now. Customer want us to remove the ceph
>>> >>> storage configuration from scratch. Let is openstack system work
>>> without
>>> >>> ceph. Later on install ceph with local disks.
>>> >>>
>>> >>> So I need to know a procedure to uninstall ceph and unconfigure it
>>> from
>>> >>> openstack.
>>> >>>
>>> >>> Regards
>>> >>> Gaurav Goyal
>>> >>> On 03-Aug-2016 4:59 pm, "David Turner"
>>> >>> >
>>> wrote:
>>> >>> If I'm understanding your question correctly that you're asking how
>>> to
>>> >>> actually remove the SAN osds from ceph, then it doesn't matter what
>>> is using
>>> >>> the storage (ie openstack, cephfs, krbd, etc) as the steps are the
>>> same.
>>> >>>
>>> >>> I'm going to assume that you've already added the new storage/osds
>>> to the
>>> >>> cluster, weighted the SAN osds to 0.0 and that the backfilling has
>>> finished.
>>> >>> If that is true, then your disk used space on the SAN's should be
>>> basically
>>> >>> empty while the new osds on the local disks should have a fair
>>> amount of
>>> >>> data.  If that is the case, then 

Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to Local Disks

[Gaurav Goyal]

Hello,

Awaiting any suggestion please!




Regards

On Wed, Aug 17, 2016 at 9:59 AM, Gaurav Goyal 
wrote:

> Hello Brian,
>
> Thanks for your response!
>
> Can you please elaborate on this.
>
> Do you mean i must use
>
> 4 x 1TB HDD on each nodes rather than 2 x 2TB?
>
> This is going to be a lab environment. Can you please suggest to have best
> possible design for my lab environment.
>
>
>
> On Wed, Aug 17, 2016 at 9:54 AM, Brian ::  wrote:
>
>> You're going to see pretty slow performance on a cluster this size
>> with spinning disks...
>>
>> Ceph scales very very well but at this type of size cluster it can be
>> challenging to get nice throughput and iops..
>>
>> for something small like this either use all ssd osds or consider
>> having more spinning osds per node backed by nvme or ssd journals..
>>
>>
>>
>> On Wed, Aug 17, 2016 at 1:14 PM, Gaurav Goyal 
>> wrote:
>> > Dear Ceph Users,
>> >
>> > Can you please address my scenario and suggest me a solution.
>> >
>> > Regards
>> > Gaurav Goyal
>> >
>> > On Tue, Aug 16, 2016 at 11:13 AM, Gaurav Goyal <
>> er.gauravgo...@gmail.com>
>> > wrote:
>> >>
>> >> Hello
>> >>
>> >>
>> >> I need your help to redesign my ceph storage network.
>> >>
>> >> As suggested in earlier discussions, i must not use SAN storage. So we
>> >> have decided to removed it.
>> >>
>> >> Now we are ordering Local HDDs.
>> >>
>> >> My Network would be
>> >>
>> >> Host1 --> Controller + COmpute --> Local Disk 600GB Host 2--> Compute2
>> -->
>> >> Local Disk 600GB Host 3 --> Compute2
>> >>
>> >> Is it right setup for ceph network? For Host1 and Host2 , we are using
>> 1
>> >> 600GB disk for basic filesystem.
>> >>
>> >> Should we use same size storage disks for ceph environment or i can
>> order
>> >> Disks in size of 2TB for ceph cluster?
>> >>
>> >> Making it
>> >>
>> >> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>> >>
>> >> 12TB in total. replication factor 2 should make it 6 TB?
>> >>
>> >>
>> >> Regards
>> >>
>> >> Gaurav Goyal
>> >>
>> >>
>> >> On Thu, Aug 4, 2016 at 1:52 AM, Bharath Krishna <
>> bkris...@walmartlabs.com>
>> >> wrote:
>> >>>
>> >>> Hi Gaurav,
>> >>>
>> >>> There are several ways to do it depending on how you deployed your
>> ceph
>> >>> cluster. Easiest way to do it is using ceph-ansible with
>> purge-cluster yaml
>> >>> ready made to wipe off CEPH.
>> >>>
>> >>> https://github.com/ceph/ceph-ansible/blob/master/purge-cluster.yml
>> >>>
>> >>> You may need to configure ansible inventory with ceph hosts.
>> >>>
>> >>> Else if you want to purge manually, you can do it using:
>> >>> http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-purge/
>> >>>
>> >>>
>> >>> Thanks
>> >>> Bharath
>> >>>
>> >>> From: ceph-users  on behalf of
>> Gaurav
>> >>> Goyal 
>> >>> Date: Thursday, August 4, 2016 at 8:19 AM
>> >>> To: David Turner 
>> >>> Cc: ceph-users 
>> >>> Subject: Re: [ceph-users] Fwd: Ceph Storage Migration from SAN
>> storage to
>> >>> Local Disks
>> >>>
>> >>> Please suggest a procedure for this uninstallation process?
>> >>>
>> >>>
>> >>> Regards
>> >>> Gaurav Goyal
>> >>>
>> >>> On Wed, Aug 3, 2016 at 5:58 PM, Gaurav Goyal
>> >>> > wrote:
>> >>>
>> >>> Thanks for your  prompt
>> >>> response!
>> >>>
>> >>> Situation is bit different now. Customer want us to remove the ceph
>> >>> storage configuration from scratch. Let is openstack system work
>> without
>> >>> ceph. Later on install ceph with local disks.
>> >>>
>> >>> So I need to know a procedure to uninstall ceph and unconfigure it
>> from
>> >>> openstack.
>> >>>
>> >>> Regards
>> >>> Gaurav Goyal
>> >>> On 03-Aug-2016 4:59 pm, "David Turner"
>> >>> >
>> wrote:
>> >>> If I'm understanding your question correctly that you're asking how to
>> >>> actually remove the SAN osds from ceph, then it doesn't matter what
>> is using
>> >>> the storage (ie openstack, cephfs, krbd, etc) as the steps are the
>> same.
>> >>>
>> >>> I'm going to assume that you've already added the new storage/osds to
>> the
>> >>> cluster, weighted the SAN osds to 0.0 and that the backfilling has
>> finished.
>> >>> If that is true, then your disk used space on the SAN's should be
>> basically
>> >>> empty while the new osds on the local disks should have a fair amount
>> of
>> >>> data.  If that is the case, then for every SAN osd, you just run the
>> >>> following commands replacing OSD_ID with the osd's id:
>> >>>
>> >>> # On the server with the osd being removed
>> >>> sudo stop ceph-osd id=OSD_ID
>> >>> ceph osd down OSD_ID
>> >>> ceph osd out OSD_ID
>> >>> ceph osd crush remove osd.OSD_ID
>> >>> ceph auth del osd.OSD_ID
>> >>> ceph osd rm OSD_ID
>> >>>
>> >>> Test running those commands on a test osd and if you 

Re: [ceph-users] Designing ceph cluster

[Gaurav Goyal]

Dear Ceph Users,

Awaiting some suggestion please!



On Wed, Aug 17, 2016 at 11:15 AM, Gaurav Goyal 
wrote:

> Hello Mart,
>
> Thanks a lot for the detailed information!
> Please find my response inline and help me to get more knowledge on it
>
>
> Ceph works best with more hardware. It is not really designed for small
> scale setups. Of course small setups can work for a PoC or testing, but I
> would not advise this for production.
>
> [Gaurav] : We need this setup for PoC or testing.
>
> If you want to proceed however, have a good look the manuals or this
> mailinglist archive and do invest some time to understand the logic and
> workings of ceph before working or ordering hardware
>
> At least you want:
> - 3 monitors, preferable on dedicated servers
> [Gaurav] : With my current setup, can i install MON on Host 1 -->
> Controller + Compute1, Host 2 and Host 3
>
> - Per disk you will be running an ceph-osd instance. So a host with 2
> disks will run 2 osd instances. More OSD process is better performance, but
> also more memory and cpu usage.
>
> [Gaurav] : Understood, That means having 1T x 4 would be better than 2T x
> 2.
>
> - Per default ceph uses a replication factor of 3 (it is possible to set
> this to 2, but is not advised)
> - You can not fill up disks to 100%, also data will not distribute even
> over all disks, expect disks to be filled up (on average) maximum to
> 60-70%. You want to add more disks once you reach this limit.
>
> All on all, with a setup of 3 hosts, with 2x2TB disks, this will result in
> a net data availablity of (3x2x2TBx0.6)/3 = 2.4 TB
>
> [Gaurav] : As this is going to be a test lab environment, can we change
> the configuration to have more capacity rather than redundancy? How can we
> achieve it?
>
> If speed is required, consider SSD's (for data & journals, or only
> journals).
>
> In you email you mention "compute1/2/3", please note, if you use the rbd
> kernel driver, this can interfere with the OSD process and is not advised
> to run OSD and Kernel driver on the same hardware. If you still want to do
> that, split it up using VMs (we have a small testing cluster where we do
> mix compute and storage, there we have the OSDs running in VMs)
>
> [Gaurav] : within my mentioned environment, How can we split rbd kernel
> driver and OSD process? Should it be like rbd kernel driver on controller
> and OSD processes on compute hosts?
>
> Since my host 1 is controller + Compute1, Can you please share the steps
> to split it up using VMs and suggested by you.
>
> Regards
> Gaurav Goyal
>
>
> On Wed, Aug 17, 2016 at 9:28 AM, Mart van Santen 
> wrote:
>
>>
>> Dear Gaurav,
>>
>> Ceph works best with more hardware. It is not really designed for small
>> scale setups. Of course small setups can work for a PoC or testing, but I
>> would not advise this for production.
>>
>> If you want to proceed however, have a good look the manuals or this
>> mailinglist archive and do invest some time to understand the logic and
>> workings of ceph before working or ordering hardware
>>
>> At least you want:
>> - 3 monitors, preferable on dedicated servers
>> - Per disk you will be running an ceph-osd instance. So a host with 2
>> disks will run 2 osd instances. More OSD process is better performance, but
>> also more memory and cpu usage.
>> - Per default ceph uses a replication factor of 3 (it is possible to set
>> this to 2, but is not advised)
>> - You can not fill up disks to 100%, also data will not distribute even
>> over all disks, expect disks to be filled up (on average) maximum to
>> 60-70%. You want to add more disks once you reach this limit.
>>
>> All on all, with a setup of 3 hosts, with 2x2TB disks, this will result
>> in a net data availablity of (3x2x2TBx0.6)/3 = 2.4 TB
>>
>>
>> If speed is required, consider SSD's (for data & journals, or only
>> journals).
>>
>> In you email you mention "compute1/2/3", please note, if you use the rbd
>> kernel driver, this can interfere with the OSD process and is not advised
>> to run OSD and Kernel driver on the same hardware. If you still want to do
>> that, split it up using VMs (we have a small testing cluster where we do
>> mix compute and storage, there we have the OSDs running in VMs)
>>
>> Hope this helps,
>>
>> regards,
>>
>> mart
>>
>>
>>
>>
>> On 08/17/2016 02:21 PM, Gaurav Goyal wrote:
>>
>> Dear Ceph Users,
>>
>> I need your help to redesign my ceph storage network.
>>
>> As suggested in earlier discussions, i must not use SAN storage. So we
>> have decided to removed it.
>>
>> Now we are ordering Local HDDs.
>>
>> My Network would be
>>
>> Host1 --> Controller + Compute1 Host 2--> Compute2 Host 3 --> Compute3
>>
>> Is it right setup for ceph network? For Host1 and Host2 , we are using 1
>> 500GB disk for OS on each host .
>>
>> Should we use same size storage disks 500GB *8 for ceph environment or i
>> can order Disks in size of 2TB for ceph cluster?
>>
>> Making it
>>
>> 2T X 2 on 

Re: [ceph-users] build and Compile ceph in development mode takes an hour

[Brad Hubbard]

On Thu, Aug 18, 2016 at 1:12 AM, agung Laksono  wrote:
> Hi Ceph User,
>
> When I make change inside ceph codes in the development mode,
> I found that recompiling takes around an hour because I have to remove
> a build folder and all the contest and then reproduce it.
>
> Is there a way to make the compiling process be faster? something like only
> compile a particular code that I change.

Sure, just use the same build directory and run "make" again after you make code
changes and it should only re-compile the binaries that are effected
by your code
changes.

You can use "make -jX" if you aren't already where 'X' is usually
number of CPUs / 2
which may speed up the build.

HTH,
Brad

>
> Thanks before
>
>
> --
> Cheers,
>
> Agung Laksono
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How can we repair OSD leveldb?

[Dan Jakubiec]

Hi Wido,

Thank you for the response:

> On Aug 17, 2016, at 16:25, Wido den Hollander  wrote:
> 
> 
>> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec :
>> 
>> 
>> Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 
>> machines.  We've managed to recover the XFS filesystems on 7 of the 
>> machines, but the OSD service is only starting on 1 of them.
>> 
>> The other 5 machines all have complaints similar to the following:
>> 
>>  2016-08-17 09:32:15.549588 7fa2f4666800 -1 
>> filestore(/var/lib/ceph/osd/ceph-1) Error initializing leveldb : Corruption: 
>> 6 missing files; e.g.: /var/lib/ceph/osd/ceph-1/current/omap/042421.ldb
>> 
>> How can we repair the leveldb to allow the OSDs to startup?  
>> 
> 
> My first question would be: How did this happen?
> 
> What hardware are you using underneath? Is there a RAID controller which is 
> not flushing properly? Since this should not happen during a power failure.
> 

Each OSD drive is connected to an onboard hardware RAID controller and 
configured in RAID 0 mode as individual virtual disks.  The RAID controller is 
an LSI 3108.

I agree -- I am finding it bizarre that 7 of our 8 OSDs (one per machine) did 
not survive the power outage.  

We did have some problems with the stock Ubunut xfs_repair (3.1.9) seg 
faulting, which eventually we overcame by building a newer version of 
xfs_repair (4.7.0).  But it did finally repair clean.

We actually have some different errors on other OSDs.  A few of them are 
failing with "Missing map in load_pgs" errors.  But generally speaking it 
appears to be missing files of various types causing different kinds of 
failures.

I'm really nervous now about the OSD's inability to start with any 
inconsistencies and no repair utilities (that I can find).  Any advice on how 
to recover?

> I don't know the answer to your question, but lost files are not good.
> 
> You might find them in a lost+found directory if XFS repair worked?
> 

Sadly this directory is empty.

-- Dan

> Wido
> 
>> Thanks,
>> 
>> -- Dan J___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How can we repair OSD leveldb?

[Wido den Hollander]

> Op 17 augustus 2016 om 17:44 schreef Dan Jakubiec :
> 
> 
> Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 
> machines.  We've managed to recover the XFS filesystems on 7 of the machines, 
> but the OSD service is only starting on 1 of them.
> 
> The other 5 machines all have complaints similar to the following:
> 
>   2016-08-17 09:32:15.549588 7fa2f4666800 -1 
> filestore(/var/lib/ceph/osd/ceph-1) Error initializing leveldb : Corruption: 
> 6 missing files; e.g.: /var/lib/ceph/osd/ceph-1/current/omap/042421.ldb
> 
> How can we repair the leveldb to allow the OSDs to startup?  
> 

My first question would be: How did this happen?

What hardware are you using underneath? Is there a RAID controller which is not 
flushing properly? Since this should not happen during a power failure.

I don't know the answer to your question, but lost files are not good.

You might find them in a lost+found directory if XFS repair worked?

Wido

> Thanks,
> 
> -- Dan J___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Reading payload from rados_watchcb2_t callback

[Nick Fisk]

Hi All,

I'm writing a small piece of code to call fsfreeze/unfreeze that can be invoked 
by a RADOS notify. I have the basic watch/notify
functionality working but I need to be able to determine if the notify message 
is to freeze or unfreeze, or maybe something
completely unrelated.

I'm looking at the rados_watchcb2_t callback and can see that the data payload 
is returned as a void pointer. This is where it all
starts to go a little pear shaped for my basic C skills. I think I have to cast 
the pointer to a (char *) but I still can't seem to
get anything useful from it.

I've been following some of the tests in the Ceph source and they seem to use 
some sort of typedef called a bufferlist, is this what
I need to try and look into?

Does anyone have any pointers (excuse the pun) as to how I would read the text 
part of the payload from it?

void watch_notify2_cb(void *arg, uint64_t notify_id, uint64_t cookie, uint64_t 
notifier_gid, void *data, size_t data_len)

Many Thanks,
Nick

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Tech Talk - Next Week

[Patrick McGarry]

Hey cephers,

Just a reminder that the August Ceph Tech Talk is on for next Thursday
@ 1p EDT.

http://ceph.com/ceph-tech-talks/

Alfredo Deza will be talking about ‘Unified CI: transitioning away
from gitbuilders’ and a special guest moderator will be recording the
event while I’m running around doing APAC Ceph Days. I hope you all
can attend and ask brilliant questions.


-- 

Best Regards,

Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com  ||  http://community.redhat.com
@scuttlemonkey || @ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] openATTIC 2.0.13 beta has been released

[Lenz Grimmer]

Hi,

On 08/16/2016 02:16 PM, Lenz Grimmer wrote:

> I blogged about the state of Ceph support a few months ago [1], a 
> followup posting is currently in the works.
> 
> [1] 
> https://blog.openattic.org/posts/update-the-state-of-ceph-support-in-openattic/

FWIW, the update has been published now:
https://blog.openattic.org/posts/the-state-of-ceph-support-in-openattic-august-2016/

Lenz



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How can we repair OSD leveldb?

[Dan Jakubiec]

Hello, we have a Ceph cluster with 8 OSD that recently lost power to all 8 
machines.  We've managed to recover the XFS filesystems on 7 of the machines, 
but the OSD service is only starting on 1 of them.

The other 5 machines all have complaints similar to the following:

2016-08-17 09:32:15.549588 7fa2f4666800 -1 
filestore(/var/lib/ceph/osd/ceph-1) Error initializing leveldb : Corruption: 6 
missing files; e.g.: /var/lib/ceph/osd/ceph-1/current/omap/042421.ldb

How can we repair the leveldb to allow the OSDs to startup?  

Thanks,

-- Dan J___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Designing ceph cluster

[Gaurav Goyal]

Hello Mart,

Thanks a lot for the detailed information!
Please find my response inline and help me to get more knowledge on it


Ceph works best with more hardware. It is not really designed for small
scale setups. Of course small setups can work for a PoC or testing, but I
would not advise this for production.

[Gaurav] : We need this setup for PoC or testing.

If you want to proceed however, have a good look the manuals or this
mailinglist archive and do invest some time to understand the logic and
workings of ceph before working or ordering hardware

At least you want:
- 3 monitors, preferable on dedicated servers
[Gaurav] : With my current setup, can i install MON on Host 1 -->
Controller + Compute1, Host 2 and Host 3

- Per disk you will be running an ceph-osd instance. So a host with 2 disks
will run 2 osd instances. More OSD process is better performance, but also
more memory and cpu usage.

[Gaurav] : Understood, That means having 1T x 4 would be better than 2T x
2.

- Per default ceph uses a replication factor of 3 (it is possible to set
this to 2, but is not advised)
- You can not fill up disks to 100%, also data will not distribute even
over all disks, expect disks to be filled up (on average) maximum to
60-70%. You want to add more disks once you reach this limit.

All on all, with a setup of 3 hosts, with 2x2TB disks, this will result in
a net data availablity of (3x2x2TBx0.6)/3 = 2.4 TB

[Gaurav] : As this is going to be a test lab environment, can we change the
configuration to have more capacity rather than redundancy? How can we
achieve it?

If speed is required, consider SSD's (for data & journals, or only
journals).

In you email you mention "compute1/2/3", please note, if you use the rbd
kernel driver, this can interfere with the OSD process and is not advised
to run OSD and Kernel driver on the same hardware. If you still want to do
that, split it up using VMs (we have a small testing cluster where we do
mix compute and storage, there we have the OSDs running in VMs)

[Gaurav] : within my mentioned environment, How can we split rbd kernel
driver and OSD process? Should it be like rbd kernel driver on controller
and OSD processes on compute hosts?

Since my host 1 is controller + Compute1, Can you please share the steps to
split it up using VMs and suggested by you.

Regards
Gaurav Goyal


On Wed, Aug 17, 2016 at 9:28 AM, Mart van Santen  wrote:

>
> Dear Gaurav,
>
> Ceph works best with more hardware. It is not really designed for small
> scale setups. Of course small setups can work for a PoC or testing, but I
> would not advise this for production.
>
> If you want to proceed however, have a good look the manuals or this
> mailinglist archive and do invest some time to understand the logic and
> workings of ceph before working or ordering hardware
>
> At least you want:
> - 3 monitors, preferable on dedicated servers
> - Per disk you will be running an ceph-osd instance. So a host with 2
> disks will run 2 osd instances. More OSD process is better performance, but
> also more memory and cpu usage.
> - Per default ceph uses a replication factor of 3 (it is possible to set
> this to 2, but is not advised)
> - You can not fill up disks to 100%, also data will not distribute even
> over all disks, expect disks to be filled up (on average) maximum to
> 60-70%. You want to add more disks once you reach this limit.
>
> All on all, with a setup of 3 hosts, with 2x2TB disks, this will result in
> a net data availablity of (3x2x2TBx0.6)/3 = 2.4 TB
>
>
> If speed is required, consider SSD's (for data & journals, or only
> journals).
>
> In you email you mention "compute1/2/3", please note, if you use the rbd
> kernel driver, this can interfere with the OSD process and is not advised
> to run OSD and Kernel driver on the same hardware. If you still want to do
> that, split it up using VMs (we have a small testing cluster where we do
> mix compute and storage, there we have the OSDs running in VMs)
>
> Hope this helps,
>
> regards,
>
> mart
>
>
>
>
> On 08/17/2016 02:21 PM, Gaurav Goyal wrote:
>
> Dear Ceph Users,
>
> I need your help to redesign my ceph storage network.
>
> As suggested in earlier discussions, i must not use SAN storage. So we
> have decided to removed it.
>
> Now we are ordering Local HDDs.
>
> My Network would be
>
> Host1 --> Controller + Compute1 Host 2--> Compute2 Host 3 --> Compute3
>
> Is it right setup for ceph network? For Host1 and Host2 , we are using 1
> 500GB disk for OS on each host .
>
> Should we use same size storage disks 500GB *8 for ceph environment or i
> can order Disks in size of 2TB for ceph cluster?
>
> Making it
>
> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>
> 12TB in total. replication factor 2 should make it 6 TB?
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Mart van Santen
> Greenhost
> E: 

[ceph-users] build and Compile ceph in development mode takes an hour

[agung Laksono]

Hi Ceph User,

When I make change inside ceph codes in the development mode,
I found that recompiling takes around an hour because I have to remove
a build folder and all the contest and then reproduce it.

Is there a way to make the compiling process be faster? something like only
compile a particular code that I change.

Thanks before


-- 
Cheers,

Agung Laksono
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to Local Disks

[Gaurav Goyal]

Hello Brian,

Thanks for your response!

Can you please elaborate on this.

Do you mean i must use

4 x 1TB HDD on each nodes rather than 2 x 2TB?

This is going to be a lab environment. Can you please suggest to have best
possible design for my lab environment.



On Wed, Aug 17, 2016 at 9:54 AM, Brian ::  wrote:

> You're going to see pretty slow performance on a cluster this size
> with spinning disks...
>
> Ceph scales very very well but at this type of size cluster it can be
> challenging to get nice throughput and iops..
>
> for something small like this either use all ssd osds or consider
> having more spinning osds per node backed by nvme or ssd journals..
>
>
>
> On Wed, Aug 17, 2016 at 1:14 PM, Gaurav Goyal 
> wrote:
> > Dear Ceph Users,
> >
> > Can you please address my scenario and suggest me a solution.
> >
> > Regards
> > Gaurav Goyal
> >
> > On Tue, Aug 16, 2016 at 11:13 AM, Gaurav Goyal  >
> > wrote:
> >>
> >> Hello
> >>
> >>
> >> I need your help to redesign my ceph storage network.
> >>
> >> As suggested in earlier discussions, i must not use SAN storage. So we
> >> have decided to removed it.
> >>
> >> Now we are ordering Local HDDs.
> >>
> >> My Network would be
> >>
> >> Host1 --> Controller + COmpute --> Local Disk 600GB Host 2--> Compute2
> -->
> >> Local Disk 600GB Host 3 --> Compute2
> >>
> >> Is it right setup for ceph network? For Host1 and Host2 , we are using 1
> >> 600GB disk for basic filesystem.
> >>
> >> Should we use same size storage disks for ceph environment or i can
> order
> >> Disks in size of 2TB for ceph cluster?
> >>
> >> Making it
> >>
> >> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
> >>
> >> 12TB in total. replication factor 2 should make it 6 TB?
> >>
> >>
> >> Regards
> >>
> >> Gaurav Goyal
> >>
> >>
> >> On Thu, Aug 4, 2016 at 1:52 AM, Bharath Krishna <
> bkris...@walmartlabs.com>
> >> wrote:
> >>>
> >>> Hi Gaurav,
> >>>
> >>> There are several ways to do it depending on how you deployed your ceph
> >>> cluster. Easiest way to do it is using ceph-ansible with purge-cluster
> yaml
> >>> ready made to wipe off CEPH.
> >>>
> >>> https://github.com/ceph/ceph-ansible/blob/master/purge-cluster.yml
> >>>
> >>> You may need to configure ansible inventory with ceph hosts.
> >>>
> >>> Else if you want to purge manually, you can do it using:
> >>> http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-purge/
> >>>
> >>>
> >>> Thanks
> >>> Bharath
> >>>
> >>> From: ceph-users  on behalf of
> Gaurav
> >>> Goyal 
> >>> Date: Thursday, August 4, 2016 at 8:19 AM
> >>> To: David Turner 
> >>> Cc: ceph-users 
> >>> Subject: Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage
> to
> >>> Local Disks
> >>>
> >>> Please suggest a procedure for this uninstallation process?
> >>>
> >>>
> >>> Regards
> >>> Gaurav Goyal
> >>>
> >>> On Wed, Aug 3, 2016 at 5:58 PM, Gaurav Goyal
> >>> > wrote:
> >>>
> >>> Thanks for your  prompt
> >>> response!
> >>>
> >>> Situation is bit different now. Customer want us to remove the ceph
> >>> storage configuration from scratch. Let is openstack system work
> without
> >>> ceph. Later on install ceph with local disks.
> >>>
> >>> So I need to know a procedure to uninstall ceph and unconfigure it from
> >>> openstack.
> >>>
> >>> Regards
> >>> Gaurav Goyal
> >>> On 03-Aug-2016 4:59 pm, "David Turner"
> >>> >
> wrote:
> >>> If I'm understanding your question correctly that you're asking how to
> >>> actually remove the SAN osds from ceph, then it doesn't matter what is
> using
> >>> the storage (ie openstack, cephfs, krbd, etc) as the steps are the
> same.
> >>>
> >>> I'm going to assume that you've already added the new storage/osds to
> the
> >>> cluster, weighted the SAN osds to 0.0 and that the backfilling has
> finished.
> >>> If that is true, then your disk used space on the SAN's should be
> basically
> >>> empty while the new osds on the local disks should have a fair amount
> of
> >>> data.  If that is the case, then for every SAN osd, you just run the
> >>> following commands replacing OSD_ID with the osd's id:
> >>>
> >>> # On the server with the osd being removed
> >>> sudo stop ceph-osd id=OSD_ID
> >>> ceph osd down OSD_ID
> >>> ceph osd out OSD_ID
> >>> ceph osd crush remove osd.OSD_ID
> >>> ceph auth del osd.OSD_ID
> >>> ceph osd rm OSD_ID
> >>>
> >>> Test running those commands on a test osd and if you had set the weight
> >>> of the osd to 0.0 previously and if the backfilling had finished, then
> what
> >>> you should see is that your cluster has 1 less osd than it used to,
> and no
> >>> pgs should be backfilling.
> >>>
> >>> HOWEVER, if my assumptions above are incorrect, please provide the
> output
> 

Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to Local Disks

[Brian ::]

You're going to see pretty slow performance on a cluster this size
with spinning disks...

Ceph scales very very well but at this type of size cluster it can be
challenging to get nice throughput and iops..

for something small like this either use all ssd osds or consider
having more spinning osds per node backed by nvme or ssd journals..



On Wed, Aug 17, 2016 at 1:14 PM, Gaurav Goyal  wrote:
> Dear Ceph Users,
>
> Can you please address my scenario and suggest me a solution.
>
> Regards
> Gaurav Goyal
>
> On Tue, Aug 16, 2016 at 11:13 AM, Gaurav Goyal 
> wrote:
>>
>> Hello
>>
>>
>> I need your help to redesign my ceph storage network.
>>
>> As suggested in earlier discussions, i must not use SAN storage. So we
>> have decided to removed it.
>>
>> Now we are ordering Local HDDs.
>>
>> My Network would be
>>
>> Host1 --> Controller + COmpute --> Local Disk 600GB Host 2--> Compute2 -->
>> Local Disk 600GB Host 3 --> Compute2
>>
>> Is it right setup for ceph network? For Host1 and Host2 , we are using 1
>> 600GB disk for basic filesystem.
>>
>> Should we use same size storage disks for ceph environment or i can order
>> Disks in size of 2TB for ceph cluster?
>>
>> Making it
>>
>> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>>
>> 12TB in total. replication factor 2 should make it 6 TB?
>>
>>
>> Regards
>>
>> Gaurav Goyal
>>
>>
>> On Thu, Aug 4, 2016 at 1:52 AM, Bharath Krishna 
>> wrote:
>>>
>>> Hi Gaurav,
>>>
>>> There are several ways to do it depending on how you deployed your ceph
>>> cluster. Easiest way to do it is using ceph-ansible with purge-cluster yaml
>>> ready made to wipe off CEPH.
>>>
>>> https://github.com/ceph/ceph-ansible/blob/master/purge-cluster.yml
>>>
>>> You may need to configure ansible inventory with ceph hosts.
>>>
>>> Else if you want to purge manually, you can do it using:
>>> http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-purge/
>>>
>>>
>>> Thanks
>>> Bharath
>>>
>>> From: ceph-users  on behalf of Gaurav
>>> Goyal 
>>> Date: Thursday, August 4, 2016 at 8:19 AM
>>> To: David Turner 
>>> Cc: ceph-users 
>>> Subject: Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to
>>> Local Disks
>>>
>>> Please suggest a procedure for this uninstallation process?
>>>
>>>
>>> Regards
>>> Gaurav Goyal
>>>
>>> On Wed, Aug 3, 2016 at 5:58 PM, Gaurav Goyal
>>> > wrote:
>>>
>>> Thanks for your  prompt
>>> response!
>>>
>>> Situation is bit different now. Customer want us to remove the ceph
>>> storage configuration from scratch. Let is openstack system work without
>>> ceph. Later on install ceph with local disks.
>>>
>>> So I need to know a procedure to uninstall ceph and unconfigure it from
>>> openstack.
>>>
>>> Regards
>>> Gaurav Goyal
>>> On 03-Aug-2016 4:59 pm, "David Turner"
>>> > wrote:
>>> If I'm understanding your question correctly that you're asking how to
>>> actually remove the SAN osds from ceph, then it doesn't matter what is using
>>> the storage (ie openstack, cephfs, krbd, etc) as the steps are the same.
>>>
>>> I'm going to assume that you've already added the new storage/osds to the
>>> cluster, weighted the SAN osds to 0.0 and that the backfilling has finished.
>>> If that is true, then your disk used space on the SAN's should be basically
>>> empty while the new osds on the local disks should have a fair amount of
>>> data.  If that is the case, then for every SAN osd, you just run the
>>> following commands replacing OSD_ID with the osd's id:
>>>
>>> # On the server with the osd being removed
>>> sudo stop ceph-osd id=OSD_ID
>>> ceph osd down OSD_ID
>>> ceph osd out OSD_ID
>>> ceph osd crush remove osd.OSD_ID
>>> ceph auth del osd.OSD_ID
>>> ceph osd rm OSD_ID
>>>
>>> Test running those commands on a test osd and if you had set the weight
>>> of the osd to 0.0 previously and if the backfilling had finished, then what
>>> you should see is that your cluster has 1 less osd than it used to, and no
>>> pgs should be backfilling.
>>>
>>> HOWEVER, if my assumptions above are incorrect, please provide the output
>>> of the following commands and try to clarify your question.
>>>
>>> ceph status
>>> ceph osd tree
>>>
>>> I hope this helps.
>>>
>>> > Hello David,
>>> >
>>> > Can you help me with steps/Procedure to uninstall Ceph storage from
>>> > openstack environment?
>>> >
>>> >
>>> > Regards
>>> > Gaurav Goyal
>>> 
>>> [cid:image001.jpg@01D1EE42.88EF6E60]
>>>
>>> David Turner | Cloud Operations Engineer | StorageCraft Technology
>>> Corporation
>>> 380 Data Drive Suite 300 | Draper | Utah | 84020
>>> Office: 801.871.2760 | Mobile: 385.224.2943
>>>

Re: [ceph-users] Designing ceph cluster

[Mart van Santen]

Dear Gaurav,

Ceph works best with more hardware. It is not really designed for small
scale setups. Of course small setups can work for a PoC or testing, but
I would not advise this for production.

If you want to proceed however, have a good look the manuals or this
mailinglist archive and do invest some time to understand the logic and
workings of ceph before working or ordering hardware

At least you want:
- 3 monitors, preferable on dedicated servers
- Per disk you will be running an ceph-osd instance. So a host with 2
disks will run 2 osd instances. More OSD process is better performance,
but also more memory and cpu usage.
- Per default ceph uses a replication factor of 3 (it is possible to set
this to 2, but is not advised)
- You can not fill up disks to 100%, also data will not distribute even
over all disks, expect disks to be filled up (on average) maximum to
60-70%. You want to add more disks once you reach this limit.

All on all, with a setup of 3 hosts, with 2x2TB disks, this will result
in a net data availablity of (3x2x2TBx0.6)/3 = 2.4 TB


If speed is required, consider SSD's (for data & journals, or only
journals).

In you email you mention "compute1/2/3", please note, if you use the rbd
kernel driver, this can interfere with the OSD process and is not
advised to run OSD and Kernel driver on the same hardware. If you still
want to do that, split it up using VMs (we have a small testing cluster
where we do mix compute and storage, there we have the OSDs running in VMs)

Hope this helps,

regards,

mart



On 08/17/2016 02:21 PM, Gaurav Goyal wrote:
>
> Dear Ceph Users,
>
> I need your help to redesign my ceph storage network.
>
> As suggested in earlier discussions, i must not use SAN storage. So we
> have decided to removed it.
>
> Now we are ordering Local HDDs.
>
> My Network would be
>
> Host1 --> Controller + Compute1 Host 2--> Compute2 Host 3 --> Compute3
>
> Is it right setup for ceph network? For Host1 and Host2 , we are using
> 1 500GB disk for OS on each host .
>
> Should we use same size storage disks 500GB *8 for ceph environment or
> i can order Disks in size of 2TB for ceph cluster?
>
> Making it
>
> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>
> 12TB in total. replication factor 2 should make it 6 TB?
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Mart van Santen
Greenhost
E: m...@greenhost.nl
T: +31 20 4890444
W: https://greenhost.nl

A PGP signature can be attached to this e-mail,
you need PGP software to verify it. 
My public key is available in keyserver(s)
see: http://tinyurl.com/openpgp-manual

PGP Fingerprint: CA85 EB11 2B70 042D AF66  B29A 6437 01A1 10A3 D3A5



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Designing ceph cluster

[Gaurav Goyal]

Dear Ceph Users,


I need your help to redesign my ceph storage network.

As suggested in earlier discussions, i must not use SAN storage. So we have
decided to removed it.

Now we are ordering Local HDDs.

My Network would be

Host1 --> Controller + Compute1 Host 2--> Compute2 Host 3 --> Compute3

Is it right setup for ceph network? For Host1 and Host2 , we are using 1
500GB disk for OS on each host .

Should we use same size storage disks 500GB *8 for ceph environment or i
can order Disks in size of 2TB for ceph cluster?

Making it

2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3

12TB in total. replication factor 2 should make it 6 TB?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Re: (no subject)

[Gaurav Goyal]

Dear Ceph Users,


I need your help to redesign my ceph storage network.

As suggested in earlier discussions, i must not use SAN storage. So we have
decided to removed it.

Now we are ordering Local HDDs.

My Network would be

Host1 --> Controller + Compute1 Host 2--> Compute2 Host 3 --> Compute3

Is it right setup for ceph network? For Host1 and Host2 , we are using 1
500GB disk for OS on each host .

Should we use same size storage disks 500GB *8 for ceph environment or i
can order Disks in size of 2TB for ceph cluster?

Making it

2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3

12TB in total. replication factor 2 should make it 6 TB?


Regards

On Tue, Aug 2, 2016 at 11:16 AM, Gaurav Goyal 
wrote:

>
> Hello Jason/Kees,
>
> I am trying to take snapshot of my instance.
>
> Image was stuck up in Queued state and instance is stuck up in Image
> Pending Upload state.
>
> I had to manually quit the job as it was not working since last 1 hour ..
> my instance is still in Image Pending Upload state.
>
> Is it something wrong with my ceph configuration?
> can i take snapshots with ceph storage? How?
>
> Regards
> Gaurav Goyal
>
> On Wed, Jul 13, 2016 at 9:44 AM, Jason Dillaman 
> wrote:
>
>> The RAW file will appear to be the exact image size but the filesystem
>> will know about the holes in the image and it will be sparsely
>> allocated on disk.  For example:
>>
>> # dd if=/dev/zero of=sparse-file bs=1 count=1 seek=2GiB
>> # ll sparse-file
>> -rw-rw-r--. 1 jdillaman jdillaman 2147483649 Jul 13 09:20 sparse-file
>> # du -sh sparse-file
>> 4.0K sparse-file
>>
>> Now, running qemu-img to copy the image into the backing RBD pool:
>>
>> # qemu-img convert -f raw -O raw ~/sparse-file rbd:rbd/sparse-file
>> # rbd disk-usage sparse-file
>> NAMEPROVISIONED USED
>> sparse-file   2048M0
>>
>>
>> On Wed, Jul 13, 2016 at 3:31 AM, Fran Barrera 
>> wrote:
>> > Yes, but is the same problem isn't? The image will be too large because
>> the
>> > format is raw.
>> >
>> > Thanks.
>> >
>> > 2016-07-13 9:24 GMT+02:00 Kees Meijs :
>> >>
>> >> Hi Fran,
>> >>
>> >> Fortunately, qemu-img(1) is able to directly utilise RBD (supporting
>> >> sparse block devices)!
>> >>
>> >> Please refer to http://docs.ceph.com/docs/hammer/rbd/qemu-rbd/ for
>> >> examples.
>> >>
>> >> Cheers,
>> >> Kees
>> >>
>> >> On 13-07-16 09:18, Fran Barrera wrote:
>> >> > Can you explain how you do this procedure? I have the same problem
>> >> > with the large images and snapshots.
>> >> >
>> >> > This is what I do:
>> >> >
>> >> > # qemu-img convert -f qcow2 -O raw image.qcow2 image.img
>> >> > # openstack image create image.img
>> >> >
>> >> > But the image.img is too large.
>> >>
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
>>
>> --
>> Jason
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to Local Disks

[Gaurav Goyal]

Dear Ceph Users,

Can you please address my scenario and suggest me a solution.

Regards
Gaurav Goyal

On Tue, Aug 16, 2016 at 11:13 AM, Gaurav Goyal 
wrote:

> Hello
>
>
> I need your help to redesign my ceph storage network.
>
> As suggested in earlier discussions, i must not use SAN storage. So we
> have decided to removed it.
>
> Now we are ordering Local HDDs.
>
> My Network would be
>
> Host1 --> Controller + COmpute --> Local Disk 600GB Host 2--> Compute2 -->
> Local Disk 600GB Host 3 --> Compute2
>
> Is it right setup for ceph network? For Host1 and Host2 , we are using 1
> 600GB disk for basic filesystem.
>
> Should we use same size storage disks for ceph environment or i can order
> Disks in size of 2TB for ceph cluster?
>
> Making it
>
> 2T X 2 on Host1 2T X 2 on Host 2 2T X 2 on Host 3
>
> 12TB in total. replication factor 2 should make it 6 TB?
>
>
> Regards
>
> Gaurav Goyal
>
> On Thu, Aug 4, 2016 at 1:52 AM, Bharath Krishna 
> wrote:
>
>> Hi Gaurav,
>>
>> There are several ways to do it depending on how you deployed your ceph
>> cluster. Easiest way to do it is using ceph-ansible with purge-cluster yaml
>> ready made to wipe off CEPH.
>>
>> https://github.com/ceph/ceph-ansible/blob/master/purge-cluster.yml
>>
>> You may need to configure ansible inventory with ceph hosts.
>>
>> Else if you want to purge manually, you can do it using:
>> http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-purge/
>>
>>
>> Thanks
>> Bharath
>>
>> From: ceph-users  on behalf of Gaurav
>> Goyal 
>> Date: Thursday, August 4, 2016 at 8:19 AM
>> To: David Turner 
>> Cc: ceph-users 
>> Subject: Re: [ceph-users] Fwd: Ceph Storage Migration from SAN storage to
>> Local Disks
>>
>> Please suggest a procedure for this uninstallation process?
>>
>>
>> Regards
>> Gaurav Goyal
>>
>> On Wed, Aug 3, 2016 at 5:58 PM, Gaurav Goyal > > wrote:
>>
>> Thanks for your  prompt
>> response!
>>
>> Situation is bit different now. Customer want us to remove the ceph
>> storage configuration from scratch. Let is openstack system work without
>> ceph. Later on install ceph with local disks.
>>
>> So I need to know a procedure to uninstall ceph and unconfigure it from
>> openstack.
>>
>> Regards
>> Gaurav Goyal
>> On 03-Aug-2016 4:59 pm, "David Turner" > > wrote:
>> If I'm understanding your question correctly that you're asking how to
>> actually remove the SAN osds from ceph, then it doesn't matter what is
>> using the storage (ie openstack, cephfs, krbd, etc) as the steps are the
>> same.
>>
>> I'm going to assume that you've already added the new storage/osds to the
>> cluster, weighted the SAN osds to 0.0 and that the backfilling has
>> finished.  If that is true, then your disk used space on the SAN's should
>> be basically empty while the new osds on the local disks should have a fair
>> amount of data.  If that is the case, then for every SAN osd, you just run
>> the following commands replacing OSD_ID with the osd's id:
>>
>> # On the server with the osd being removed
>> sudo stop ceph-osd id=OSD_ID
>> ceph osd down OSD_ID
>> ceph osd out OSD_ID
>> ceph osd crush remove osd.OSD_ID
>> ceph auth del osd.OSD_ID
>> ceph osd rm OSD_ID
>>
>> Test running those commands on a test osd and if you had set the weight
>> of the osd to 0.0 previously and if the backfilling had finished, then what
>> you should see is that your cluster has 1 less osd than it used to, and no
>> pgs should be backfilling.
>>
>> HOWEVER, if my assumptions above are incorrect, please provide the output
>> of the following commands and try to clarify your question.
>>
>> ceph status
>> ceph osd tree
>>
>> I hope this helps.
>>
>> > Hello David,
>> >
>> > Can you help me with steps/Procedure to uninstall Ceph storage from
>> openstack environment?
>> >
>> >
>> > Regards
>> > Gaurav Goyal
>> 
>> [cid:image001.jpg@01D1EE42.88EF6E60]
>>
>> David Turner | Cloud Operations Engineer | StorageCraft Technology
>> Corporation
>> 380 Data Drive Suite 300 | Draper | Utah | 84020
>> Office: 801.871.2760 | Mobile: 385.224.2943
>>
>> 
>> If you are not the intended recipient of this message or received it
>> erroneously, please notify the sender and delete it, together with any
>> attachments, and be advised that any dissemination or copying of this
>> message is prohibited.
>>
>> 
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph admin socket from non root

[Alexandre DERUMIER]

Hi Stefan,

I have same problem than you, trying to monitor ceph through the socket with a 
non-root user.

Do you have find a clean way to add write permissions to ceph group to the 
socket ?


- Mail original -
De: "Stefan Priebe, Profihost AG" 
À: "Gregory Farnum" 
Cc: "ceph-users" 
Envoyé: Mardi 19 Juillet 2016 12:51:59
Objet: Re: [ceph-users] ceph admin socket from non root

Am 18.07.2016 um 20:14 schrieb Gregory Farnum: 
> I'm not familiar with how it's set up but skimming and searching 
> through the code I'm not seeing anything, no. We've got a chown but no 
> chmod. 

That's odd ;-) how do all the people do their monitoring? running as root? 

> That's a reasonably feature idea though, and presumably you 
> could add a chmod it to your init scripts? 

Yes i could hack that into the init script. I just had the feeling that 
the feature must exist and i'm just missing something. 

Greets, 
Stefan 

> -Greg 
> 
> On Mon, Jul 18, 2016 at 3:02 AM, Stefan Priebe - Profihost AG 
>  wrote: 
>> 
>> Nobody? Is it at least possible with jewel to give the sockets group 
>> write permissions? 
>> 
>> Am 10.07.2016 um 23:51 schrieb Stefan Priebe - Profihost AG: 
>>> Hi, 
>>> 
>>> is there a proposed way how to connect from non root f.e. a monitoring 
>>> system to the ceph admin socket? 
>>> 
>>> In the past they were created with 777 permissions but now they're 755 
>>> which prevents me from connecting from our monitoring daemon. I don't 
>>> like to set CAP_DAC_OVERRIDE for the monitoring agent. 
>>> 
>>> Greets, 
>>> Stefan 
>>> 
>>> ___ 
>>> ceph-users mailing list 
>>> ceph-users@lists.ceph.com 
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>>> 
>> ___ 
>> ceph-users mailing list 
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph cluster not reposnd

[Niv Azriel]

sorry guys, I solved the problem.
The issue because high io wait on the local disk of the monitor servers.
I migrate the local disks from raid1 to raid5 to get more io .

(The leveldb stores on the local disks of the monitor servers and each
change in the map requires update of the database)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] is it possible to get and set zonegroup , zone through admin rest api?

[Leo Yu]

hi,cepher

is it possible to set zonegroup and zone through admin rest api?
i can get and set the zonegroup and zone through radosgw-admin command like
the following :


[root@ceph04 src]# ./radosgw-admin zone get  --rgw-zone=us-east-2
#dump those to an file and inject after modify
2016-08-17 13:40:10.382984 7fad488bc9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
2016-08-17 13:40:10.384066 7fad488bc9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
2016-08-17 13:40:10.409597 7fad488bc9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
{
"id": "c825c777-8e75-41a7-8e10-36fdc68d6d02",
"name": "us-east-2",
"domain_root": "us-east-2.rgw.data.root",
"control_pool": "us-east-2.rgw.control",
"gc_pool": "us-east-2.rgw.gc",
"log_pool": "us-east-2.rgw.log",
"intent_log_pool": "us-east-2.rgw.intent-log",
"usage_log_pool": "us-east-2.rgw.usage",
"user_keys_pool": "us-east-2.rgw.users.keys",
"user_email_pool": "us-east-2.rgw.users.email",
"user_swift_pool": "us-east-2.rgw.users.swift",
"user_uid_pool": "us-east-2.rgw.users.uid",
"system_key": {
"access_key": "admin",
"secret_key": "admin"
},
"placement_pools": [
{
"key": "beijing-placement",
"val": {
"index_pool": "us-east-2.rgw.buckets.index",
"data_pool": "us-east-2.beijing",
"data_extra_pool": "us-east-2.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "default-placement",
"val": {
"index_pool": "us-east-2.rgw.buckets.index",
"data_pool": "us-east-2.rgw.buckets.data",
"data_extra_pool": "us-east-2.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "shanghai-placement",
"val": {
"index_pool": "us-east-2.rgw.buckets.index",
"data_pool": "us-east-2.shanghai",
"data_extra_pool": "us-east-2.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "suzhou-placement",
"val": {
"index_pool": "us-east-2.rgw.buckets.index",
"data_pool": "us-east-2.suzhou",
"data_extra_pool": "us-east-2.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": "us-east-2.rgw.meta",
"realm_id": "d3ef8d11-5621-4e7a-b563-1348efe41a3c"
}


and

[root@ceph04 src]# ./radosgw-admin zonegroup get  --rgw-zonegroup=us #dump
those to an file and inject after modify
2016-08-17 13:40:07.059401 7fb8ed67d9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
2016-08-17 13:40:07.060382 7fb8ed67d9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
2016-08-17 13:40:07.085429 7fb8ed67d9c0 -1 WARNING: the following dangerous
and experimental features are enabled: *
{
"id": "0e78f19b-b64f-48c9-ac60-5a055f648264",
"name": "us",
"api_name": "us",
"is_master": "true",
"endpoints": [
"http:\/\/10.254.3.81:8000"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "c825c777-8e75-41a7-8e10-36fdc68d6d02",
"zones": [
{
"id": "c825c777-8e75-41a7-8e10-36fdc68d6d02",
"name": "us-east-2",
"endpoints": [
"http:\/\/10.254.3.81:8000"
],
"log_meta": "true",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "beijing-placement",
"tags": []
},
{
"name": "default-placement",
"tags": []
},
{
"name": "shanghai-placement",
"tags": []
},
{
"name": "suzhou-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "d3ef8d11-5621-4e7a-b563-1348efe41a3c"
}



is it possible to get and set those information though admin rest api?
i have a user have the zone caps *，so i can get the realm like Get some
usefull message though the following python code:

import requests
import logging
from requests_toolbelt.utils import dump
from aws_requests_auth.aws_auth import AWSRequestsAuth
from awsauth import S3Auth
logging.basicConfig(level=logging.DEBUG)
host = '10.254.3.81:8000'
access_key = 'admin'
secret_key = 'admin'
cmd = '/admin/config'
cmd = '/admin/realm/period'
url = 'http://%s%s' % (host,cmd)
# url = 'http://%s' % host
response = requests.get(url, auth=S3Auth(access_key,
secret_key,service_url=host))
data = dump.dump_all(response)
print(data.decode('utf-8'))


the out put of /admin/config