Re: [ceph-users] Help with systemd

[Jeffrey Ollie]

I put the systemd service files that I use to map a RBD and mount the
filesystem before starting up PostgreSQL into the following gist. It's
probably not perfect, but it seems to work for me. Personally, I like
using a native service to accomplish this rather than using fstab and
the generator.

https://gist.github.com/jcollie/60f8b278d1ac5eadb4794db1f4c0e87d

On Mon, Aug 22, 2016 at 1:16 PM, K.C. Wong  wrote:
> Folks,
>
> I have some services that depends on RBD images getting
> mounted prior to service start-up. I am having a really
> hard time getting out of systemd dependency hell.
>
> * I create a run-once systemd service that basically does
>   the rbd map operation, and set it start after network.target,
>   network-online.target, and ceph.target (probably overkill)
> * I added 'x-systemd.requires=' to the
>   mount-point in /etc/fstab
>
> And when the system reboot, it'd complain about ordering
> cycle and sometimes resulting in rescue mode. Because the
> filesystem is 'xfs', I believe systemd-fstab-generator
> classifies the mount-point as 'local-fs'. Is there a way
> to force a 'remote-fs' reclassification? Or is there some
> other way to get out of this ordering nightmare... Old
> school 'S' and 'K' numbers are *so* simple; I'd trade
> consistency for speed any day.
>
> Thanks for any suggestion or insight.
>
> -kc
> BTW, I disable NetworkManager which, I know, kind of breaks
> network-online.target.
>
> K.C. Wong
> kcw...@verseon.com
> 4096R/B8995EDE  E527 CBE8 023E 79EA 8BBB  5C77 23A6 92E9 B899 5EDE
> hkps://hkps.pool.sks-keyservers.net
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph - Health and Monitoring

[Jeffrey Ollie]

I can definitely recommend Prometheus but I prefer the exporter for Ceph
that I wrote :)

https://github.com/jcollie/ceph_exporter


On Mon, Jan 2, 2017 at 7:55 PM, Craig Chi  wrote:

> Hello,
>
> I suggest Prometheus with ceph_exporter
>  and Grafana (UI). It can
> also monitor the node's health and any other services you want.
> And it has a beautiful UI.
>
> Sincerely,
> Craig Chi
>
> On 2017-01-02 21:32, ulem...@polarzone.de wrote:
>
> Hi Andre,
> I use check_ceph_dash on top of ceph-dash for this (is an nagios/icinga
> Plugin).
> https://github.com/Crapworks/ceph-dashhttps://github.com/Crapworks/check_ceph_dash
>
> ceph-dash provide an simple clear overview as web-dashbord.
>
>
> Udo
>
> Am 2017-01-02 12:42, schrieb Andre Forigato:
> > Hello,
> >
> > I am responsible with the health of the servers and the entire Ceph
> > system.
> > What should I use to monitor the entire Celp environment?
> > Monitor all objects.
> >
> > Which one is the best?
> > Is it SNMP only?
> >
> >
> > Thanks.
> >
> > Andre
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS support?

[Jeffrey Ollie]

CoreOS doesn't appear to have the user space binaries necessary to mount
CephFS, but does appear to have the necessary kernel support (I think).
Does anyone have a documented procedure for using a helper container that
has the necessary user space components to mount a CephFS filesystem?  Or
is this not possible until the Ceph user space binaries are added?  I'm
running CoreOS alpha (976.0.0) and Ceph 0.94.6 (Hammer) on my Ceph cluster.

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS support?

[Jeffrey Ollie]

Gah, meant to send this to the CoreOS users list...

On Sat, Mar 5, 2016 at 4:26 PM, Jeffrey Ollie  wrote:

> CoreOS doesn't appear to have the user space binaries necessary to mount
> CephFS, but does appear to have the necessary kernel support (I think).
> Does anyone have a documented procedure for using a helper container that
> has the necessary user space components to mount a CephFS filesystem?  Or
> is this not possible until the Ceph user space binaries are added?  I'm
> running CoreOS alpha (976.0.0) and Ceph 0.94.6 (Hammer) on my Ceph cluster.
>
> --
> Jeff Ollie
>
>


-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Weird cluster restart behavior

[Jeffrey Ollie]

On Tue, Mar 31, 2015 at 3:05 PM, Gregory Farnum  wrote:

> On Tue, Mar 31, 2015 at 12:56 PM, Quentin Hartman
> >
> > My understanding is that the "right" method to take an entire cluster
> > offline is to set noout and then shutting everything down. Is there a
> better
> > way?
>
> That's probably the best way to do it. Like I said, there was also a
> bug here that I think is fixed for Hammer but that might not have been
> backported to Giant. Unfortunately I don't remember the right keywords
> as I wasn't involved in the fix.


I'd hope that the complete shutdown scenario would get some more testing in
the future...  I know that Ceph is targeted more at "enterprise" situations
where things like generators and properly sized battery backups aren't
extravagant luxuries, but there are probably a lot of clusters out there
that will get shut down completely, planned or unplanned.

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Renaming pools used by CephFS

[Jeffrey Ollie]

I've got a Ceph system (running 0.80.5) at home that I've been messing
around with, partly to learn Ceph, but also as reliable storage for all of
my media.  During the process I deleted the data and metadata pools used by
CephFS and recreated them.  However, when I recreated the filesystem, the
pool called "data" got assigned as a metadata pool and the pool called
"metadata" got assigned as a data pool.

Is there a safe way to rename the pools?  It's purely an aesthetic thing (I
think), so if it's difficult/dangerous to do I'll leave it be.

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Problems starting up OSD

[Jeffrey Ollie]

-- One of my OSDs lost network connectivity for a short while.  The OSD
crashed and now when I try and start it back up the process is killed
because of an illegal instruction.  Is there anything that I can do to
get this going again or am I going to need to rebuild it from scratch
(which wouldn't be a completely terrible idea as I set this up with
the journal on the same drive).  This particular OSD is running on
Fedora 21 Beta.

Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 0> 2014-11-22
12:23:26.908700 7fdab90ae7c0 -1 *** Caught signal (Illegal
instruction) **
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: in thread 7fdab90ae7c0
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: ceph version 0.87
(
c51c8f9d80fa4e0168aa52685b8de40e42758578)
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 1:
/usr/bin/ceph-osd() [0x9edd55]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 2: (()+0x100d0)
[0x7fdab80740d0]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 3:
(pthread_rwlock_unlock()+0x13) [0x7fdab8070153]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 4:
(IndexManager::init_index(coll_t, char const*, unsigned int)+0x513)
[0x8da3b3]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 5:
(FileStore::init_index(coll_t)+0x84) [0x88a464]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 6:
(FileStore::_create_collection(coll_t, SequencerPosition
const&)+0x12b) [0x89d13b]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 7:
(FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
int, ThreadPool::TPHandle*)+0x2988) [0x8c2158]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 8:
(FileStore::_do_transactions(std::list >&, unsigned long,
ThreadPool::TPHandle*)+0x64) [0x8c51c4]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 9:
(JournalingObjectStore::journal_replay(unsigned long)+0x8da)
[0x8df81a]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 10:
(FileStore::mount()+0x2fc2) [0x8ae142]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 11:
(OSD::do_convertfs(ObjectStore*)+0x2d) [0x5faccd]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 12: (main()+0x2495)
[0x5e3975]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 13:
(__libc_start_main()+0xf0) [0x7fdab6c71fe0]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 14:
/usr/bin/ceph-osd() [0x5e9417]
Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: NOTE: a copy of the
executable, or `objdump -rdS ` is needed to interpret
this.

Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems starting up OSD

[Jeffrey Ollie]

On Sat, Nov 22, 2014 at 1:22 PM, Gregory Farnum  wrote:
> Can you post the OSD log somewhere? It should have a few more details
> about what's going on here. (This backtrace looks like it's crashing
> in a call to phreads, which is a little unusual.)

Uploaded to Google Drive:

https://drive.google.com/file/d/0B5VwdTUBhU7UNXFlR1FRRHRVNm8/view?usp=sharing

>
> On Sat, Nov 22, 2014 at 1:01 PM, Jeffrey Ollie  wrote:
>> -- One of my OSDs lost network connectivity for a short while.  The OSD
>> crashed and now when I try and start it back up the process is killed
>> because of an illegal instruction.  Is there anything that I can do to
>> get this going again or am I going to need to rebuild it from scratch
>> (which wouldn't be a completely terrible idea as I set this up with
>> the journal on the same drive).  This particular OSD is running on
>> Fedora 21 Beta.
>>
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 0> 2014-11-22
>> 12:23:26.908700 7fdab90ae7c0 -1 *** Caught signal (Illegal
>> instruction) **
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: in thread 7fdab90ae7c0
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: ceph version 0.87
>> (
>> c51c8f9d80fa4e0168aa52685b8de40e42758578)
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 1:
>> /usr/bin/ceph-osd() [0x9edd55]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 2: (()+0x100d0)
>> [0x7fdab80740d0]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 3:
>> (pthread_rwlock_unlock()+0x13) [0x7fdab8070153]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 4:
>> (IndexManager::init_index(coll_t, char const*, unsigned int)+0x513)
>> [0x8da3b3]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 5:
>> (FileStore::init_index(coll_t)+0x84) [0x88a464]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 6:
>> (FileStore::_create_collection(coll_t, SequencerPosition
>> const&)+0x12b) [0x89d13b]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 7:
>> (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
>> int, ThreadPool::TPHandle*)+0x2988) [0x8c2158]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 8:
>> (FileStore::_do_transactions(std::list> std::allocator >&, unsigned long,
>> ThreadPool::TPHandle*)+0x64) [0x8c51c4]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 9:
>> (JournalingObjectStore::journal_replay(unsigned long)+0x8da)
>> [0x8df81a]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 10:
>> (FileStore::mount()+0x2fc2) [0x8ae142]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 11:
>> (OSD::do_convertfs(ObjectStore*)+0x2d) [0x5faccd]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 12: (main()+0x2495)
>> [0x5e3975]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 13:
>> (__libc_start_main()+0xf0) [0x7fdab6c71fe0]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: 14:
>> /usr/bin/ceph-osd() [0x5e9417]
>> Nov 22 12:23:26 home01.ocjtech.us ceph-osd[22977]: NOTE: a copy of the
>> executable, or `objdump -rdS ` is needed to interpret
>> this.
>>
>> Jeff Ollie
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems starting up OSD

[Jeffrey Ollie]

On Sat, Nov 22, 2014 at 1:59 PM, Gregory Farnum  wrote:
>
> Looks to me like this is the result of us being naughty with rwlock handling:
> http://tracker.ceph.com/issues/10085
> https://github.com/ceph/ceph/pull/2937
>
> It should be fixed soon, and was probably triggered by the disk
> snapshot state being not quite what the OSD expected. If you're able
> to build your own packages you can apply the linked patch and it
> should start up again for you.
> -Greg

Sweet.  Thanks for the quick diagnosis.  I'll start building a patched
package now.

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems starting up OSD

[Jeffrey Ollie]

On Sat, Nov 22, 2014 at 2:39 PM, Jeffrey Ollie  wrote:
> On Sat, Nov 22, 2014 at 1:59 PM, Gregory Farnum  wrote:
>>
>> Looks to me like this is the result of us being naughty with rwlock handling:
>> http://tracker.ceph.com/issues/10085
>> https://github.com/ceph/ceph/pull/2937
>>
>> It should be fixed soon, and was probably triggered by the disk
>> snapshot state being not quite what the OSD expected. If you're able
>> to build your own packages you can apply the linked patch and it
>> should start up again for you.
>> -Greg
>
> Sweet.  Thanks for the quick diagnosis.  I'll start building a patched
> package now.

Just wanted to say thanks to Gregory for looking at my problem and
coming up with a solution so quickly!  After building a package with
that patch applied the OSD is back up and running.  I hadn't lost any
data but getting the OSD back so quickly definitely saved me a LOT of
time because I didn't have to rebuild the OSD from scratch!

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems running ceph commands.on custom linux system

[Jeffrey Ollie]

On Tue, Dec 9, 2014 at 10:15 AM, Patrick Darley
 wrote:
>
> I'm having a problem running commands such as `ceph --help` and `ceph -s`.
> These commands output the expected information, but then they hang
> indefinitely.

If you're using Python 2.7.8 it's probably this issue:

http://tracker.ceph.com/issues/8797

-- 
Jeff Ollie
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com