Re: [ceph-users] RBD hard crash on kernel 3.10

[Christian Balzer]

On Thu, 09 Apr 2015 02:33:44 + Shawn Edwards wrote:

> On Wed, Apr 8, 2015 at 9:23 PM Christian Balzer  wrote:
> 
> > On Wed, 08 Apr 2015 14:25:36 + Shawn Edwards wrote:
> >
> > > We've been working on a storage repository for xenserver 6.5, which
> > > uses the 3.10 kernel (ug).  I got the xenserver guys to include the
> > > rbd and libceph kernel modules into the 6.5 release, so that's at
> > > least available.
> > >
> > Woah, hold on for a moment here.
> >
> > AFAIK no version of XenServer supports Ceph/RBD, how are you planning
> > to integrate that?
> >
> > Another group in my company uses XenServer and I was hoping to supply
> > them with cheap(er) storage than what they have now, but the local
> > Citrix reseller (we have a huge spendy license) wasn't helpful at all
> > when it came to get a feature request for upcoming versions into the
> > pipeline.
> >
> > And anything with a NFS or iSCSI head on top of Ceph is another can of
> > worms in terms of complexity and reliability, never mind that cost and
> > performance will also be impacted by such a kludge.
> >
> > So is this some work in progress then?
> >
> 
> Yes, this is work being done at my employer's.  We've been looking at
> this as a possibility for a while, and finally had the resources to
> dedicate to it.
> 
That's very welcome news, anywhere I should look for updates on that,
other than any missives by you on this ML?

> 
> >
> > > Where things go bad is when we have many (>10 or so) VMs on one
> > > host, all using RBD clones for the storage mapped using the rbd
> > > kernel module.  The Xenserver crashes so badly that it doesn't even
> > > get a chance to kernel panic.  The whole box just hangs.
> > >
> > > Has anyone else seen this sort of behavior?
> > >
> > I think that's very much expected with that kernel, you'll probably
> > want 3.18 when using the kernel module, as that is the first to
> > support TRIM as well.
> > Also the host you're mapping things to isn't also a Ceph OSD node,
> > right?
> >
> >
> Correct.  The Xenservers and Ceph OSD are completely separate machines.
> 
So that common cause for Ceph fireworks is ruled out then.
 
> 
> > > We have a lot of ways to try to work around this, but none of them
> > > are very pretty:
> > >
> > > * move the code to user space, ditch the kernel driver:  The build
> > > tools for Xenserver are all CentOS5 based, and it is painful to get
> > > all of the deps built to get the ceph user space libs built.
> > >
> > I feel your pain, however experience tells that the user space is
> > better and most importantly faster maintained when it comes to vital
> > features.
> >
> > This is the option I'd go for, everything else being equal.
> >
> >
> Yeah, agreed.  This is the best option.  Grumble.  The recent source for
> Ceph really doesn't like to play well with the older build tools, and the
> dependency list is staggering.
> 
I can only begin to imagine that mess, one would have hoped for something
a little more modern on the XenServer side given it was just released this
year.

> 
> > > * backport the ceph and rbd kernel modules to 3.10.  Has proven
> > > painful, as the block device code changed somewhere in the 3.14-3.16
> > > timeframe.
> > >
> > Madness seems to lie down that path.
> >
> >
> Agreed.
> 
Never mind that only starting with 3.14 Ceph can map more than 250 RBD
devices using the kernel module. 
Which was a crucial milestone that happened just in time for a ganeti
cluster I deployed when that kernel became available.

> 
> > > * forward-port the xen kernel patches from 3.10 to a newer driver
> > > (3.18 preferred) and run that on xenserver.  Painful for the same
> > > reasons as above, but in the opposite direction.
> > >
> > Probably the better of the 2 choices, as you gain many other
> > improvements as well. Including support of newer hardware. ^o^
> >
> >
> Agreed as well.  At this point, it may be the 'best' option, as it would
> require the least rewrite of our most current bits.  Although it would
> make maintaining Xen painful, since we'd need to be extra careful with
> vendor patches.
> 
Yeah, there's certainly no way this won't involve a lot of heavy lifting
while being very careful on the other side.


> 
> > Regards,
> >
> > Christian
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com   Global OnLine Japan/Fusion Communications
> > http://www.gol.com/
> >


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cascading Failure of OSDs

[Carl-Johan Schenström]

Francois Lafont wrote:

> Just in case it could be useful, I have noticed the -s option (on my
> Ubuntu) that offer an output probably easier to parse:
> 
> # "column -t" is just to make it's nice for the human eyes.
> ifconfig -s | column -t

Since ifconfig is deprecated, one should use iproute2 instead.

ip -s link show p2p1 | awk '/(RX|TX):/{getline; print $3;}'

However, the sysfs interface is probably a better alternative. See 

 and .

-- 
Carl-Johan Schenström
Driftansvarig / System Administrator
Språkbanken & Svensk nationell datatjänst /
The Swedish Language Bank & Swedish National Data Service
Göteborgs universitet / University of Gothenburg
carl-johan.schenst...@gu.se / +46 709 116769
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] live migration fails with image on ceph

[Yuming Ma (yumima)]

Josh,

I think we are using plain live migration and not mirroring block drives as the 
other test did. What are the chances or scenario that disk image can be 
corrupted during the live migration for both source and target are connected to 
the same volume and RBD caches is turned on:

rbd cache = true

rbd cache writethrough until flush = true

rbd cache size = 33554432

rbd cache max dirty = 25165824

rbd cache target dirty = 16777216

rbd cache max dirty age = 1

— Yuming
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] long blocking with writes on rbds

[Jeff Epstein]

Our workload involves creating and destroying a lot of pools. Each pool
has 100 pgs, so it adds up. Could this be causing the problem? What
would you suggest instead?


...this is most likely the cause. Deleting a pool causes the data and
pgs associated with it to be deleted asynchronously, which can be a lot
of background work for the osds.

If you're using the cfq scheduler you can try decreasing the priority 
of these operations with the "osd disk thread ioprio..." options:


http://ceph.com/docs/master/rados/configuration/osd-config-ref/#operations 



If that doesn't help enough, deleting data from pools before deleting
the pools might help, since you can control the rate more finely. And of
course not creating/deleting so many pools would eliminate the hidden
background cost of deleting the pools.


Thanks for your answer. Some follow-up questions:

- I wouldn't expect that pool deletion is the problem, since our pools, 
although many, don't contain much data. Typically, we will have one rbd 
per pool, several GB in size, but in practice containing little data. 
Would you expect that performance penalty from deleting pool to be 
relative to the requested size of the rbd, or relative to the quantity 
of data actually stored in it?


- Rather than creating and deleting multiple pools, each containing a 
single rbd, do you think we would see a speed-up if we were to instead 
have one pool, containing multiple (frequently created and deleted) 
rbds? Does the performance penalty stem only from deleting pools 
themselves, or from deleting objects within the pool as well?


- Somewhat off-topic, but for my own curiosity: Why is deleting data so 
slow, in terms of ceph's architecture? Shouldn't it just be a matter of 
flagging a region as available and allowing it to be overwritten, as 
would a traditional file system?


Jeff
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MDS unmatched rstat after upgrade hammer

[Yan, Zheng]

On Thu, Apr 9, 2015 at 7:09 AM, Scottix  wrote:
> I was testing the upgrade on our dev environment and after I restarted the
> mds I got the following errors.
>
> 2015-04-08 15:58:34.056470 mds.0 [ERR] unmatched rstat on 605, inode has
> n(v70 rc2015-03-16 09:11:34.390905), dirfrags have n(v0 rc2015-03-16
> 09:11:34.390905 1=0+1)
> 2015-04-08 15:58:34.056530 mds.0 [ERR] unmatched rstat on 604, inode has
> n(v69 rc2015-03-31 08:07:09.265241), dirfrags have n(v0 rc2015-03-31
> 08:07:09.265241 1=0+1)
> 2015-04-08 15:58:34.056581 mds.0 [ERR] unmatched rstat on 606, inode has
> n(v67 rc2015-03-16 08:54:36.314790), dirfrags have n(v0 rc2015-03-16
> 08:54:36.314790 1=0+1)
> 2015-04-08 15:58:34.056633 mds.0 [ERR] unmatched rstat on 607, inode has
> n(v57 rc2015-03-16 08:54:46.797240), dirfrags have n(v0 rc2015-03-16
> 08:54:46.797240 1=0+1)
> 2015-04-08 15:58:34.056687 mds.0 [ERR] unmatched rstat on 608, inode has
> n(v23 rc2015-03-16 08:54:59.634299), dirfrags have n(v0 rc2015-03-16
> 08:54:59.634299 1=0+1)
> 2015-04-08 15:58:34.056737 mds.0 [ERR] unmatched rstat on 609, inode has
> n(v62 rc2015-03-16 08:55:06.598286), dirfrags have n(v0 rc2015-03-16
> 08:55:06.598286 1=0+1)
> 2015-04-08 15:58:34.056789 mds.0 [ERR] unmatched rstat on 600, inode has
> n(v101 rc2015-03-16 08:55:16.153175), dirfrags have n(v0 rc2015-03-16
> 08:55:16.153175 1=0+1)

These errors are likely caused by the bug that rstats are not set to
correct values
when creating new fs. Nothing to worry about, the MDS automatically fixes rstat
errors.

>
> I am not sure if this is an issue or got fixed or something I should worry
> about. But would just like some context around this issue since it came up
> in the ceph -w and other users might see it as well.
>
> I have done a lot of "unsafe" stuff on this mds so not to freak anyone out
> if that is the issue.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD hard crash on kernel 3.10

[Shawn Edwards]

On Wed, Apr 8, 2015 at 9:23 PM Christian Balzer  wrote:

> On Wed, 08 Apr 2015 14:25:36 + Shawn Edwards wrote:
>
> > We've been working on a storage repository for xenserver 6.5, which uses
> > the 3.10 kernel (ug).  I got the xenserver guys to include the rbd and
> > libceph kernel modules into the 6.5 release, so that's at least
> > available.
> >
> Woah, hold on for a moment here.
>
> AFAIK no version of XenServer supports Ceph/RBD, how are you planning to
> integrate that?
>
> Another group in my company uses XenServer and I was hoping to supply them
> with cheap(er) storage than what they have now, but the local Citrix
> reseller (we have a huge spendy license) wasn't helpful at all when it
> came to get a feature request for upcoming versions into the pipeline.
>
> And anything with a NFS or iSCSI head on top of Ceph is another can of
> worms in terms of complexity and reliability, never mind that cost and
> performance will also be impacted by such a kludge.
>
> So is this some work in progress then?
>

Yes, this is work being done at my employer's.  We've been looking at this
as a possibility for a while, and finally had the resources to dedicate to
it.


>
> > Where things go bad is when we have many (>10 or so) VMs on one host, all
> > using RBD clones for the storage mapped using the rbd kernel module.  The
> > Xenserver crashes so badly that it doesn't even get a chance to kernel
> > panic.  The whole box just hangs.
> >
> > Has anyone else seen this sort of behavior?
> >
> I think that's very much expected with that kernel, you'll probably want
> 3.18 when using the kernel module, as that is the first to support TRIM as
> well.
> Also the host you're mapping things to isn't also a Ceph OSD node, right?
>
>
Correct.  The Xenservers and Ceph OSD are completely separate machines.


> > We have a lot of ways to try to work around this, but none of them are
> > very pretty:
> >
> > * move the code to user space, ditch the kernel driver:  The build tools
> > for Xenserver are all CentOS5 based, and it is painful to get all of the
> > deps built to get the ceph user space libs built.
> >
> I feel your pain, however experience tells that the user space is better
> and most importantly faster maintained when it comes to vital features.
>
> This is the option I'd go for, everything else being equal.
>
>
Yeah, agreed.  This is the best option.  Grumble.  The recent source for
Ceph really doesn't like to play well with the older build tools, and the
dependency list is staggering.


> > * backport the ceph and rbd kernel modules to 3.10.  Has proven painful,
> > as the block device code changed somewhere in the 3.14-3.16 timeframe.
> >
> Madness seems to lie down that path.
>
>
Agreed.


> > * forward-port the xen kernel patches from 3.10 to a newer driver (3.18
> > preferred) and run that on xenserver.  Painful for the same reasons as
> > above, but in the opposite direction.
> >
> Probably the better of the 2 choices, as you gain many other improvements
> as well. Including support of newer hardware. ^o^
>
>
Agreed as well.  At this point, it may be the 'best' option, as it would
require the least rewrite of our most current bits.  Although it would make
maintaining Xen painful, since we'd need to be extra careful with vendor
patches.


> Regards,
>
> Christian
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Global OnLine Japan/Fusion Communications
> http://www.gol.com/
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD hard crash on kernel 3.10

[Christian Balzer]

On Wed, 08 Apr 2015 14:25:36 + Shawn Edwards wrote:

> We've been working on a storage repository for xenserver 6.5, which uses
> the 3.10 kernel (ug).  I got the xenserver guys to include the rbd and
> libceph kernel modules into the 6.5 release, so that's at least
> available.
> 
Woah, hold on for a moment here.

AFAIK no version of XenServer supports Ceph/RBD, how are you planning to
integrate that?

Another group in my company uses XenServer and I was hoping to supply them
with cheap(er) storage than what they have now, but the local Citrix
reseller (we have a huge spendy license) wasn't helpful at all when it
came to get a feature request for upcoming versions into the pipeline.

And anything with a NFS or iSCSI head on top of Ceph is another can of
worms in terms of complexity and reliability, never mind that cost and
performance will also be impacted by such a kludge.

So is this some work in progress then?

> Where things go bad is when we have many (>10 or so) VMs on one host, all
> using RBD clones for the storage mapped using the rbd kernel module.  The
> Xenserver crashes so badly that it doesn't even get a chance to kernel
> panic.  The whole box just hangs.
> 
> Has anyone else seen this sort of behavior?
> 
I think that's very much expected with that kernel, you'll probably want
3.18 when using the kernel module, as that is the first to support TRIM as
well.
Also the host you're mapping things to isn't also a Ceph OSD node, right?

> We have a lot of ways to try to work around this, but none of them are
> very pretty:
> 
> * move the code to user space, ditch the kernel driver:  The build tools
> for Xenserver are all CentOS5 based, and it is painful to get all of the
> deps built to get the ceph user space libs built.
> 
I feel your pain, however experience tells that the user space is better
and most importantly faster maintained when it comes to vital features.

This is the option I'd go for, everything else being equal.

> * backport the ceph and rbd kernel modules to 3.10.  Has proven painful,
> as the block device code changed somewhere in the 3.14-3.16 timeframe.
>
Madness seems to lie down that path.
 
> * forward-port the xen kernel patches from 3.10 to a newer driver (3.18
> preferred) and run that on xenserver.  Painful for the same reasons as
> above, but in the opposite direction.
>
Probably the better of the 2 choices, as you gain many other improvements
as well. Including support of newer hardware. ^o^
 
Regards,

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

[Christian Balzer]

On Wed, 08 Apr 2015 14:59:21 +0200 Francois Lafont wrote:

> Hi,
> 
> Sorry in advance for this thread not directly linked to Ceph. ;)
> We are thinking about buying servers to build a ceph cluster and we
> would like to have, if possible, a *approximative* power usage
> estimation of these servers (this parameter could be important in
> your choice):
> 
In short, way, way, way too many variables.
Which CPUs, HDDs/SSDs, PSUs. 
And a lightly loaded cluster/node will consume like 1/3rd of the power CPU
wise than a very busy one does.

> 1. the 12xbays supermicro OSD node
>(here https://www.supermicro.com/solutions/datasheet_Ceph.pdf,
>page 2, model SSG-6027R-OSD040H in the table)
> 
I'd really wish SM would revise that pamphlet, for nearly all the roles in
there they have better suited models. 
And models that fill requirements not really covered in that sheet.

If you're willing to take the 1:5 SSD journal to OSD ratio risk, as
proposed by that configuration, why not go all out to a chassis that has 2
hotswap bays in the back and 1:6. Much better density and you'll have
journals and HDDs on different SATA buses.

> 2. SC216-based chassis 2U, 24xbays 2.5" (like this one for instance
>http://www.supermicro.com/products/chassis/2U/216/SC216BA-R1K28LP.cfm)
> 

At this level of density, you'd need about 24GHz combined CPU power to
fully utilize the IOPS potentioal of a pure HDD based node. 
The moment you add SSD journals to this picture, that number at _least_
doubles, making it a potentially very power hungry unit.

You'll also need a HBA/RAID card to connect up those 6 mini-SAS ports on
the backplane.

If you're concerned about power, look at their X10 offerings with Titanium
level PSUs and pick CPUs that are energy efficient while still having
enough capacity to satisfy your IOPS needs.

> If someone here has a server as above, we would be curious to have
> a appromative power usage estimation (for instance in volt-ampere).
> 
A SM server (not running Ceph, but as a mailbox server being somewhat
comparable) here with Platinum (94% efficiency supposedly) PSUs consumes
while being basically idle 105W on the input side (100V in Japan) and 95W
on the output side.
This triples basically during peak utilization times.

Christian

> Thanks in advance for your help.
> Regards
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Michael Kidd]

I don't think this came through the first time.. resending.. If it's a
dupe, my apologies..

For Firefly / Giant installs, I've had success with the following:

yum install ceph ceph-common --disablerepo=base --disablerepo=epel

Let us know if this works for you as well.

Thanks,

Michael J. Kidd
Sr. Storage Consultant
Inktank Professional Services
 - by Red Hat

On Wed, Apr 8, 2015 at 9:07 PM, Michael Kidd  wrote:

> For Firefly / Giant installs, I've had success with the following:
>
> yum install ceph ceph-common --disablerepo=base --disablerepo=epel
>
> Let us know if this works for you as well.
>
> Thanks,
>
> Michael J. Kidd
> Sr. Storage Consultant
> Inktank Professional Services
>  - by Red Hat
>
> On Wed, Apr 8, 2015 at 8:55 PM, Travis Rhoden  wrote:
>
>> I did also confirm that, as Ken mentioned, this is not a problem on
>> Hammer since Hammer includes the package split (python-ceph became
>> python-rados and python-rbd).
>>
>>  - Travis
>>
>> On Wed, Apr 8, 2015 at 5:00 PM, Travis Rhoden  wrote:
>>
>>> Hi Vickey,
>>>
>>> The easiest way I know of to get around this right now is to add the
>>> following line in section for epel in /etc/yum.repos.d/epel.repo
>>>
>>> exclude=python-rados python-rbd
>>>
>>> So this is what my epel.repo file looks like: http://fpaste.org/208681/
>>>
>>> It is those two packages in EPEL that are causing problems.  I also
>>> tried enabling epel-testing, but that didn't work either.
>>>
>>> Unfortunately you would need to add this line on each node where Ceph
>>> Giant is being installed.
>>>
>>>  - Travis
>>>
>>> On Wed, Apr 8, 2015 at 4:11 PM, Vickey Singh <
>>> vickey.singh22...@gmail.com> wrote:
>>>
 Community , need help.


 -VS-

 On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh <
 vickey.singh22...@gmail.com> wrote:

> Any suggestion  geeks
>
>
> VS
>
> On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh <
> vickey.singh22...@gmail.com> wrote:
>
>>
>> Hi
>>
>>
>> The below suggestion also didn’t worked
>>
>>
>> Full logs here : http://paste.ubuntu.com/10771939/
>>
>>
>>
>>
>> [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph
>>
>> Loaded plugins: fastestmirror, priorities
>>
>> Loading mirror speeds from cached hostfile
>>
>>  * base: mirror.zetup.net
>>
>>  * epel: ftp.fi.muni.cz
>>
>>  * extras: mirror.zetup.net
>>
>>  * updates: mirror.zetup.net
>>
>> 25 packages excluded due to repository priority protections
>>
>> Available Packages
>>
>> ceph.x86_64
>> 0.80.6-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.7-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.8-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.9-0.el7.centos
>> Ceph
>>
>> [root@rgw-node1 yum.repos.d]#
>>
>>
>>
>>
>>
>> Its not able to install latest available package , yum is getting
>> confused with other DOT releases.
>>
>>
>> Any other suggestion to fix this ???
>>
>>
>>
>> --> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>
>> --> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>
>> --> Finished Dependency Resolution
>>
>> Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libaio.so.1(LIBAIO_0.4)(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.6-0.el7.centos
>>
>>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.8-0.el7.centos
>>
>>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.9-0.el7.centos
>>
>> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: python-request

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Travis Rhoden]

I did also confirm that, as Ken mentioned, this is not a problem on Hammer
since Hammer includes the package split (python-ceph became python-rados
and python-rbd).

 - Travis

On Wed, Apr 8, 2015 at 5:00 PM, Travis Rhoden  wrote:

> Hi Vickey,
>
> The easiest way I know of to get around this right now is to add the
> following line in section for epel in /etc/yum.repos.d/epel.repo
>
> exclude=python-rados python-rbd
>
> So this is what my epel.repo file looks like: http://fpaste.org/208681/
>
> It is those two packages in EPEL that are causing problems.  I also tried
> enabling epel-testing, but that didn't work either.
>
> Unfortunately you would need to add this line on each node where Ceph
> Giant is being installed.
>
>  - Travis
>
> On Wed, Apr 8, 2015 at 4:11 PM, Vickey Singh 
> wrote:
>
>> Community , need help.
>>
>>
>> -VS-
>>
>> On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh > > wrote:
>>
>>> Any suggestion  geeks
>>>
>>>
>>> VS
>>>
>>> On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh <
>>> vickey.singh22...@gmail.com> wrote:
>>>

 Hi


 The below suggestion also didn’t worked


 Full logs here : http://paste.ubuntu.com/10771939/




 [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph

 Loaded plugins: fastestmirror, priorities

 Loading mirror speeds from cached hostfile

  * base: mirror.zetup.net

  * epel: ftp.fi.muni.cz

  * extras: mirror.zetup.net

  * updates: mirror.zetup.net

 25 packages excluded due to repository priority protections

 Available Packages

 ceph.x86_64
 0.80.6-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.7-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.8-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.9-0.el7.centos
 Ceph

 [root@rgw-node1 yum.repos.d]#





 Its not able to install latest available package , yum is getting
 confused with other DOT releases.


 Any other suggestion to fix this ???



 --> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 --> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 --> Finished Dependency Resolution

 Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.4)(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.6-0.el7.centos

Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.8-0.el7.centos

Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.9-0.el7.centos

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-ceph = 0.80.7-0.el7.centos

Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.6-0.el7.centos

Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>

[ceph-users] MDS unmatched rstat after upgrade hammer

[Scottix]

I was testing the upgrade on our dev environment and after I restarted the
mds I got the following errors.

2015-04-08 15:58:34.056470 mds.0 [ERR] unmatched rstat on 605, inode has
n(v70 rc2015-03-16 09:11:34.390905), dirfrags have n(v0 rc2015-03-16
09:11:34.390905 1=0+1)
2015-04-08 15:58:34.056530 mds.0 [ERR] unmatched rstat on 604, inode has
n(v69 rc2015-03-31 08:07:09.265241), dirfrags have n(v0 rc2015-03-31
08:07:09.265241 1=0+1)
2015-04-08 15:58:34.056581 mds.0 [ERR] unmatched rstat on 606, inode has
n(v67 rc2015-03-16 08:54:36.314790), dirfrags have n(v0 rc2015-03-16
08:54:36.314790 1=0+1)
2015-04-08 15:58:34.056633 mds.0 [ERR] unmatched rstat on 607, inode has
n(v57 rc2015-03-16 08:54:46.797240), dirfrags have n(v0 rc2015-03-16
08:54:46.797240 1=0+1)
2015-04-08 15:58:34.056687 mds.0 [ERR] unmatched rstat on 608, inode has
n(v23 rc2015-03-16 08:54:59.634299), dirfrags have n(v0 rc2015-03-16
08:54:59.634299 1=0+1)
2015-04-08 15:58:34.056737 mds.0 [ERR] unmatched rstat on 609, inode has
n(v62 rc2015-03-16 08:55:06.598286), dirfrags have n(v0 rc2015-03-16
08:55:06.598286 1=0+1)
2015-04-08 15:58:34.056789 mds.0 [ERR] unmatched rstat on 600, inode has
n(v101 rc2015-03-16 08:55:16.153175), dirfrags have n(v0 rc2015-03-16
08:55:16.153175 1=0+1)

I am not sure if this is an issue or got fixed or something I should worry
about. But would just like some context around this issue since it came up
in the ceph -w and other users might see it as well.

I have done a lot of "unsafe" stuff on this mds so not to freak anyone out
if that is the issue.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] object size in rados bench write

[Deneau, Tom]

Ah, I see there is an osd parameter for this

osd max write size
Description:The maximum size of a write in megabytes.
Default:90

> -Original Message-
> From: Deneau, Tom
> Sent: Wednesday, April 08, 2015 3:57 PM
> To: 'ceph-users@lists.ceph.com'
> Subject: object size in rados bench write
> 
> I've noticed when I use large object sizes like 100M with rados bench write,
> I get
> rados -p data2 bench 60 write --no-cleanup -b 100M
>  Maintaining 16 concurrent writes of 104857600 bytes for up to 60 seconds or
> 0 objects
>sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>  0   0 0 0 0 0 - 0
>  1   3 3 0 0 0 - 0
>  2   5 5 0 0 0 - 0
>  3   8 8 0 0 0 - 0
>  4  1010 0 0 0 - 0
>  5  1313 0 0 0 - 0
>  6  1515 0 0 0 - 0
> error during benchmark: -5
> error 5: (5) Input/output error
> 
> An object_size of 32M works fine and the cluster seems otherwise fine.
> 
> Seems related to this issue http://lists.ceph.com/pipermail/ceph-users-
> ceph.com/2014-March/028288.html
> But I didn't see a resolution for that.
> 
> Is there a timeout that is kicking in?
> 
> -- Tom Deneau

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Travis Rhoden]

Hi Vickey,

The easiest way I know of to get around this right now is to add the
following line in section for epel in /etc/yum.repos.d/epel.repo

exclude=python-rados python-rbd

So this is what my epel.repo file looks like: http://fpaste.org/208681/

It is those two packages in EPEL that are causing problems.  I also tried
enabling epel-testing, but that didn't work either.

Unfortunately you would need to add this line on each node where Ceph Giant
is being installed.

 - Travis

On Wed, Apr 8, 2015 at 4:11 PM, Vickey Singh 
wrote:

> Community , need help.
>
>
> -VS-
>
> On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh 
> wrote:
>
>> Any suggestion  geeks
>>
>>
>> VS
>>
>> On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh > > wrote:
>>
>>>
>>> Hi
>>>
>>>
>>> The below suggestion also didn’t worked
>>>
>>>
>>> Full logs here : http://paste.ubuntu.com/10771939/
>>>
>>>
>>>
>>>
>>> [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph
>>>
>>> Loaded plugins: fastestmirror, priorities
>>>
>>> Loading mirror speeds from cached hostfile
>>>
>>>  * base: mirror.zetup.net
>>>
>>>  * epel: ftp.fi.muni.cz
>>>
>>>  * extras: mirror.zetup.net
>>>
>>>  * updates: mirror.zetup.net
>>>
>>> 25 packages excluded due to repository priority protections
>>>
>>> Available Packages
>>>
>>> ceph.x86_64
>>> 0.80.6-0.el7.centos
>>> Ceph
>>>
>>> ceph.x86_64
>>> 0.80.7-0.el7.centos
>>> Ceph
>>>
>>> ceph.x86_64
>>> 0.80.8-0.el7.centos
>>> Ceph
>>>
>>> ceph.x86_64
>>> 0.80.9-0.el7.centos
>>> Ceph
>>>
>>> [root@rgw-node1 yum.repos.d]#
>>>
>>>
>>>
>>>
>>>
>>> Its not able to install latest available package , yum is getting
>>> confused with other DOT releases.
>>>
>>>
>>> Any other suggestion to fix this ???
>>>
>>>
>>>
>>> --> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
>>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>>
>>> --> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
>>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>>
>>> --> Finished Dependency Resolution
>>>
>>> Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>>
>>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>>
>>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libaio.so.1(LIBAIO_0.4)(64bit)
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: librados2 = 0.80.7-0.el7.centos
>>>
>>>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>>>
>>>librados2 = 0.80.6-0.el7.centos
>>>
>>>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>librados2 = 0.80.7-0.el7.centos
>>>
>>>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>>>
>>>librados2 = 0.80.8-0.el7.centos
>>>
>>>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>>>
>>>librados2 = 0.80.9-0.el7.centos
>>>
>>> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: python-requests
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: librbd1 = 0.80.7-0.el7.centos
>>>
>>>Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)
>>>
>>>librbd1 = 0.80.6-0.el7.centos
>>>
>>>Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>librbd1 = 0.80.7-0.el7.centos
>>>
>>>Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)
>>>
>>>librbd1 = 0.80.8-0.el7.centos
>>>
>>>Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)
>>>
>>>librbd1 = 0.80.9-0.el7.centos
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>>
>>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: python-ceph = 0.80.7-0.el7.centos
>>>
>>>Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)
>>>
>>>python-ceph = 0.80.6-0.el7.centos
>>>
>>>Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>python-ceph = 0.80.7-0.el7.centos
>>>
>>>Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)
>>>
>>>python-ceph = 0.80.8-0.el7.centos
>>>
>>>Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)
>>>
>>>python-ceph = 0.80.9-0.el7.centos
>>>
>>> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>>
>>>Requires: libboost_system-mt.so.1.53.0()(64bi

[ceph-users] object size in rados bench write

[Deneau, Tom]

I've noticed when I use large object sizes like 100M with rados bench write, I 
get 
rados -p data2 bench 60 write --no-cleanup -b 100M
 Maintaining 16 concurrent writes of 104857600 bytes for up to 60 seconds or 0 
objects
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
 0   0 0 0 0 0 - 0
 1   3 3 0 0 0 - 0
 2   5 5 0 0 0 - 0
 3   8 8 0 0 0 - 0
 4  1010 0 0 0 - 0
 5  1313 0 0 0 - 0
 6  1515 0 0 0 - 0
error during benchmark: -5
error 5: (5) Input/output error

An object_size of 32M works fine and the cluster seems otherwise fine.

Seems related to this issue 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-March/028288.html
But I didn't see a resolution for that.

Is there a timeout that is kicking in?

-- Tom Deneau

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] long blocking with writes on rbds

[Josh Durgin]

On 04/08/2015 11:40 AM, Jeff Epstein wrote:

Hi, thanks for answering. Here are the answers to your questions.
Hopefully they will be helpful.

On 04/08/2015 12:36 PM, Lionel Bouton wrote:

I probably won't be able to help much, but people knowing more will
need at least: - your Ceph version, - the kernel version of the host
on which you are trying to format /dev/rbd1, - which hardware and
network you are using for this cluster (CPU, RAM, HDD or SSD models,
network cards, jumbo frames, ...).


ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)

Linux 3.18.4pl2 #3 SMP Thu Jan 29 21:11:23 CET 2015 x86_64 GNU/Linux

The hardware is an Amazon AWS c3.large. So, a (virtual) Xeon(R) CPU
E5-2680 v2 @ 2.80GHz, 3845992 kB RAM, plus whatever other virtual
hardware Amazon provides.


AWS will cause some extra perf variance, but...


There's only one thing surprising me here: you have only 6 OSDs, 1504GB
(~ 250G / osd) and a total of 4400 pgs ? With a replication of 3 this is
2200 pgs / OSD, which might be too much and unnecessarily increase the
load on your OSDs.

Best regards,

Lionel Bouton


Our workload involves creating and destroying a lot of pools. Each pool
has 100 pgs, so it adds up. Could this be causing the problem? What
would you suggest instead?


...this is most likely the cause. Deleting a pool causes the data and
pgs associated with it to be deleted asynchronously, which can be a lot
of background work for the osds.

If you're using the cfq scheduler you can try decreasing the priority of 
these operations with the "osd disk thread ioprio..." options:


http://ceph.com/docs/master/rados/configuration/osd-config-ref/#operations

If that doesn't help enough, deleting data from pools before deleting
the pools might help, since you can control the rate more finely. And of
course not creating/deleting so many pools would eliminate the hidden
background cost of deleting the pools.

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Community , need help.


-VS-

On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh 
wrote:

> Any suggestion  geeks
>
>
> VS
>
> On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh 
> wrote:
>
>>
>> Hi
>>
>>
>> The below suggestion also didn’t worked
>>
>>
>> Full logs here : http://paste.ubuntu.com/10771939/
>>
>>
>>
>>
>> [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph
>>
>> Loaded plugins: fastestmirror, priorities
>>
>> Loading mirror speeds from cached hostfile
>>
>>  * base: mirror.zetup.net
>>
>>  * epel: ftp.fi.muni.cz
>>
>>  * extras: mirror.zetup.net
>>
>>  * updates: mirror.zetup.net
>>
>> 25 packages excluded due to repository priority protections
>>
>> Available Packages
>>
>> ceph.x86_64
>> 0.80.6-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.7-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.8-0.el7.centos
>> Ceph
>>
>> ceph.x86_64
>> 0.80.9-0.el7.centos
>> Ceph
>>
>> [root@rgw-node1 yum.repos.d]#
>>
>>
>>
>>
>>
>> Its not able to install latest available package , yum is getting
>> confused with other DOT releases.
>>
>>
>> Any other suggestion to fix this ???
>>
>>
>>
>> --> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>
>> --> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
>> package: librbd1-0.80.9-0.el7.centos.x86_64
>>
>> --> Finished Dependency Resolution
>>
>> Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libaio.so.1(LIBAIO_0.4)(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.6-0.el7.centos
>>
>>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.8-0.el7.centos
>>
>>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.9-0.el7.centos
>>
>> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: python-requests
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: librbd1 = 0.80.7-0.el7.centos
>>
>>Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 0.80.6-0.el7.centos
>>
>>Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 0.80.7-0.el7.centos
>>
>>Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 0.80.8-0.el7.centos
>>
>>Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 0.80.9-0.el7.centos
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: python-ceph = 0.80.7-0.el7.centos
>>
>>Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)
>>
>>python-ceph = 0.80.6-0.el7.centos
>>
>>Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>python-ceph = 0.80.7-0.el7.centos
>>
>>Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)
>>
>>python-ceph = 0.80.8-0.el7.centos
>>
>>Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)
>>
>>python-ceph = 0.80.9-0.el7.centos
>>
>> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: libboost_system-mt.so.1.53.0()(64bit)
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: python-requests
>>
>> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>Requires: librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.6-0.el7.centos
>>
>>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.7-0.el7.centos
>>
>>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.8-0.el7.centos
>>
>>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>>
>>librados2 = 0.80.9-0.el7.

Re: [ceph-users] long blocking with writes on rbds

[Jeff Epstein]

Hi, thanks for answering. Here are the answers to your questions. 
Hopefully they will be helpful.


On 04/08/2015 12:36 PM, Lionel Bouton wrote:
I probably won't be able to help much, but people knowing more will 
need at least: - your Ceph version, - the kernel version of the host 
on which you are trying to format /dev/rbd1, - which hardware and 
network you are using for this cluster (CPU, RAM, HDD or SSD models, 
network cards, jumbo frames, ...). 


ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)

Linux 3.18.4pl2 #3 SMP Thu Jan 29 21:11:23 CET 2015 x86_64 GNU/Linux

The hardware is an Amazon AWS c3.large. So, a (virtual) Xeon(R) CPU 
E5-2680 v2 @ 2.80GHz, 3845992 kB RAM, plus whatever other virtual 
hardware Amazon provides.

There's only one thing surprising me here: you have only 6 OSDs, 1504GB
(~ 250G / osd) and a total of 4400 pgs ? With a replication of 3 this is
2200 pgs / OSD, which might be too much and unnecessarily increase the
load on your OSDs.

Best regards,

Lionel Bouton


Our workload involves creating and destroying a lot of pools. Each pool 
has 100 pgs, so it adds up. Could this be causing the problem? What 
would you suggest instead?


Jeff

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

Mike, yeah, I wouldn't switch to rdma until it is fully supported in a stable 
release ))) 

Andrei 

- Original Message -

> From: "Andrei Mikhailovsky" 
> To: "Somnath Roy" 
> Cc: ceph-users@lists.ceph.com, "ceph-devel"
> 
> Sent: Wednesday, 8 April, 2015 7:16:40 PM
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> Somnath,

> Sounds very promising! I can't wait to try it on my cluster as I am
> currently using IPOIB instread of the native rdma.

> Cheers

> Andrei

> - Original Message -

> > From: "Somnath Roy" 
> 
> > To: "Andrei Mikhailovsky" , "Andrey Korolyov"
> > 
> 
> > Cc: ceph-users@lists.ceph.com, "ceph-devel"
> > 
> 
> > Sent: Wednesday, 8 April, 2015 5:23:23 PM
> 
> > Subject: RE: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > Andrei,
> 
> > Yes, I see it has lot of potential and I believe fixing the
> > performance bottlenecks inside XIO messenger it should go further.
> 
> > We are working on it and will keep community posted..
> 

> > Thanks & Regards
> 
> > Somnath
> 

> > From: Andrei Mikhailovsky [mailto:and...@arhont.com]
> 
> > Sent: Wednesday, April 08, 2015 2:22 AM
> 
> > To: Andrey Korolyov
> 
> > Cc: ceph-users@lists.ceph.com; ceph-devel; Somnath Roy
> 
> > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > Hi,
> 

> > Am I the only person noticing disappointing results from the
> > preliminary RDMA testing, or am I reading the numbers wrong?
> 

> > Yes, it's true that on a very small cluster you do see a great
> > improvement in rdma, but in real life rdma is used in large
> > infrastructure projects, not on a few servers with a handful of
> > osds. In fact, from what i've seen from the slides, the rdma
> > implementation scales horribly to the point that it becomes slower
> > the more osds you through at it.
> 

> > From my limited knowledge, i have expected a much higher
> > performance
> > gains with rdma, taking into account that you should have much
> > lower
> > latency and overhead and lower cpu utilisation when using this
> > transport in comparison with tcp.
> 

> > Are we likely to see a great deal of improvement with ceph and rdma
> > in a near future? Is there a roadmap for having a stable and
> > reliable rdma protocol support?
> 

> > Thanks
> 

> > Andrei
> 
> > - Original Message -
> 

> > > From: "Andrey Korolyov" < and...@xdel.ru >
> > 
> 
> > > To: "Somnath Roy" < somnath@sandisk.com >
> > 
> 
> > > Cc: ceph-users@lists.ceph.com , "ceph-devel" <
> > > ceph-de...@vger.kernel.org >
> > 
> 
> > > Sent: Wednesday, 8 April, 2015 9:28:12 AM
> > 
> 
> > > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> > 
> 

> > > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <
> > > somnath@sandisk.com > wrote:
> > 
> 
> > > >
> > 
> 
> > > > Hi,
> > 
> 
> > > > Please find the preliminary performance numbers of TCP Vs RDMA
> > > > (XIO) implementation (on top of SSDs) in the following link.
> > 
> 
> > > >
> > 
> 
> > > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> > 
> 
> > > >
> > 
> 
> > > > The attachment didn't go through it seems, so, I had to use
> > > > slideshare.
> > 
> 
> > > >
> > 
> 
> > > > Mark,
> > 
> 
> > > > If we have time, I can present it in tomorrow's performance
> > > > meeting.
> > 
> 
> > > >
> > 
> 
> > > > Thanks & Regards
> > 
> 
> > > > Somnath
> > 
> 
> > > >
> > 
> 

> > > Those numbers are really impressive (for small numbers at least)!
> > > What
> > 
> 
> > > are TCP settings you using?For example, difference can be lowered
> > > on
> > 
> 
> > > scale due to less intensive per-connection acceleration on CUBIC
> > > on
> > > a
> > 
> 
> > > larger number of nodes, though I do not believe that it was a
> > > main
> > 
> 
> > > reason for an observed TCP catchup on a relatively flat workload
> > > such
> > 
> 
> > > as fio generates.
> > 
> 
> > > ___
> > 
> 
> > > ceph-users mailing list
> > 
> 
> > > ceph-users@lists.ceph.com
> > 
> 
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> 
> > PLEASE NOTE: The information contained in this electronic mail
> > message is intended only for the use of the designated recipient(s)
> > named above. If the reader of this message is not the intended
> > recipient, you are hereby notified that you have received this
> > message in error and that any review, dissemination, distribution,
> > or copying of this message is strictly prohibited. If you have
> > received this communication in error, please notify the sender by
> > telephone or e-mail (as shown above) immediately and destroy any
> > and
> > all copies of this message in your possession (whether hard copies
> > or electronically stored copies).
> 

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
ht

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Mark Nelson]

Please do keep in mind that this is *very* experimental still and likely 
to destroy all data and life within a 2 mile radius. ;)


Mark

On 04/08/2015 01:16 PM, Andrei Mikhailovsky wrote:

Somnath,

Sounds very promising! I can't wait to try it on my cluster as I am
currently using IPOIB instread of the native rdma.

Cheers

Andrei





*From: *"Somnath Roy" 
*To: *"Andrei Mikhailovsky" , "Andrey Korolyov"

*Cc: *ceph-users@lists.ceph.com, "ceph-devel"

*Sent: *Wednesday, 8 April, 2015 5:23:23 PM
*Subject: *RE: [ceph-users] Preliminary RDMA vs TCP numbers

Andrei,

Yes, I see it has lot of potential and I believe fixing the
performance bottlenecks inside XIO messenger it should go further.

We are working on it and will keep community posted..

Thanks & Regards

Somnath

*From:*Andrei Mikhailovsky [mailto:and...@arhont.com]
*Sent:* Wednesday, April 08, 2015 2:22 AM
*To:* Andrey Korolyov
*Cc:* ceph-users@lists.ceph.com; ceph-devel; Somnath Roy
*Subject:* Re: [ceph-users] Preliminary RDMA vs TCP numbers

Hi,

Am I the only person noticing disappointing results from the
preliminary RDMA testing, or am I reading the numbers wrong?

Yes, it's true that on a very small cluster you do see a great
improvement in rdma, but in real life rdma is used in large
infrastructure projects, not on a few servers with a handful of
osds. In fact, from what i've seen from the slides, the rdma
implementation scales horribly to the point that it becomes slower
the more osds you through at it.

 From my limited knowledge, i have expected a much higher
performance gains with rdma, taking into account that you should
have much lower latency and overhead and lower cpu utilisation when
using this transport in comparison with tcp.

Are we likely to see a great deal of improvement with ceph and rdma
in a near future? Is there a roadmap for having a stable and
reliable rdma protocol support?

Thanks

Andrei



*From: *"Andrey Korolyov" mailto:and...@xdel.ru>>
*To: *"Somnath Roy" mailto:somnath@sandisk.com>>
*Cc: *ceph-users@lists.ceph.com
, "ceph-devel"
mailto:ceph-de...@vger.kernel.org>>
*Sent: *Wednesday, 8 April, 2015 9:28:12 AM
*Subject: *Re: [ceph-users] Preliminary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy
mailto:somnath@sandisk.com>> wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
implementation (on top of SSDs) in the following link.
>
>http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use 
slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at
least)! What
are TCP settings you using?For example, difference can be lowered on
scale due to less intensive per-connection acceleration on CUBIC
on a
larger number of nodes, though I do not believe that it was a main
reason for an observed TCP catchup on a relatively flat workload
such
as fio generates.
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




PLEASE NOTE: The information contained in this electronic mail
message is intended only for the use of the designated recipient(s)
named above. If the reader of this message is not the intended
recipient, you are hereby notified that you have received this
message in error and that any review, dissemination, distribution,
or copying of this message is strictly prohibited. If you have
received this communication in error, please notify the sender by
telephone or e-mail (as shown above) immediately and destroy any and
all copies of this message in your possession (whether hard copies
or electronically stored copies).




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

Somnath, 

Sounds very promising! I can't wait to try it on my cluster as I am currently 
using IPOIB instread of the native rdma. 

Cheers 

Andrei 

- Original Message -

> From: "Somnath Roy" 
> To: "Andrei Mikhailovsky" , "Andrey Korolyov"
> 
> Cc: ceph-users@lists.ceph.com, "ceph-devel"
> 
> Sent: Wednesday, 8 April, 2015 5:23:23 PM
> Subject: RE: [ceph-users] Preliminary RDMA vs TCP numbers

> Andrei,
> Yes, I see it has lot of potential and I believe fixing the
> performance bottlenecks inside XIO messenger it should go further.
> We are working on it and will keep community posted..

> Thanks & Regards
> Somnath

> From: Andrei Mikhailovsky [mailto:and...@arhont.com]
> Sent: Wednesday, April 08, 2015 2:22 AM
> To: Andrey Korolyov
> Cc: ceph-users@lists.ceph.com; ceph-devel; Somnath Roy
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> Hi,

> Am I the only person noticing disappointing results from the
> preliminary RDMA testing, or am I reading the numbers wrong?

> Yes, it's true that on a very small cluster you do see a great
> improvement in rdma, but in real life rdma is used in large
> infrastructure projects, not on a few servers with a handful of
> osds. In fact, from what i've seen from the slides, the rdma
> implementation scales horribly to the point that it becomes slower
> the more osds you through at it.

> From my limited knowledge, i have expected a much higher performance
> gains with rdma, taking into account that you should have much lower
> latency and overhead and lower cpu utilisation when using this
> transport in comparison with tcp.

> Are we likely to see a great deal of improvement with ceph and rdma
> in a near future? Is there a roadmap for having a stable and
> reliable rdma protocol support?

> Thanks

> Andrei
> - Original Message -

> > From: "Andrey Korolyov" < and...@xdel.ru >
> 
> > To: "Somnath Roy" < somnath@sandisk.com >
> 
> > Cc: ceph-users@lists.ceph.com , "ceph-devel" <
> > ceph-de...@vger.kernel.org >
> 
> > Sent: Wednesday, 8 April, 2015 9:28:12 AM
> 
> > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <
> > somnath@sandisk.com > wrote:
> 
> > >
> 
> > > Hi,
> 
> > > Please find the preliminary performance numbers of TCP Vs RDMA
> > > (XIO) implementation (on top of SSDs) in the following link.
> 
> > >
> 
> > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> 
> > >
> 
> > > The attachment didn't go through it seems, so, I had to use
> > > slideshare.
> 
> > >
> 
> > > Mark,
> 
> > > If we have time, I can present it in tomorrow's performance
> > > meeting.
> 
> > >
> 
> > > Thanks & Regards
> 
> > > Somnath
> 
> > >
> 

> > Those numbers are really impressive (for small numbers at least)!
> > What
> 
> > are TCP settings you using?For example, difference can be lowered
> > on
> 
> > scale due to less intensive per-connection acceleration on CUBIC on
> > a
> 
> > larger number of nodes, though I do not believe that it was a main
> 
> > reason for an observed TCP catchup on a relatively flat workload
> > such
> 
> > as fio generates.
> 
> > ___
> 
> > ceph-users mailing list
> 
> > ceph-users@lists.ceph.com
> 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this
> message in error and that any review, dissemination, distribution,
> or copying of this message is strictly prohibited. If you have
> received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any and
> all copies of this message in your possession (whether hard copies
> or electronically stored copies).
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Interesting problem: 2 pgs stuck in EC pool with missing OSDs

[Loic Dachary]

Hi Paul,

Contrary to what the documentation states at

http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#crush-gives-up-too-soon

the crush ruleset can be modified (an update at 
https://github.com/ceph/ceph/pull/4306 will fix that). Placement groups will 
move around, but that's to be expected.

Cheers

On 06/04/2015 20:40, Paul Evans wrote:
> Thanks for the insights, Greg.  It would be great if the CRUSH rule for an EC 
> pool can be dynamically changed…but if that’s not the case, the 
> troubleshooting doc also offers up the idea of adding more OSDs, and we have 
> another 8 OSDs (one from each node) we can move into the default root.  
> However: just to clarify the point of adding OSDs: the current EC profile has 
> a failure domain of ‘host’... will adding more OSDs still improve the odds of 
> CRUSH finding a good mapping within the given timeout period? 
> 
> BTW, I’m a little concerned about moving all 8 OSDs at once, as we’re skinny 
> on RAM and the EC pools seem to like more RAM that replicated pools do. 
> Considering the RAM issue, is adding 2-4 OSDs at a time the recommendation? 
> (other than adding more RAM). 
> 
> -- 
> *Paul Evans
> *
> *
> *
>> This looks like it's just the standard risk of using a pseudo-random
>> algorithm: you need to "randomly" map 8 pieces into 8 slots. Sometimes
>> the CRUSH calculation will return the same 7 slots so many times in a
>> row that it simply fails to get all 8 of them inside of the time
>> bounds that are currently set.
>>
>> If you look through the list archives we've discussed this a few
>> times, especially Loïc in the context of erasure coding. See
>> http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#crush-gives-up-too-soon
>> for the fix.
>> But I think that doc is wrong and you can change the CRUSH rule in use
>> without creating a new pool — right, Loïc?
>> -Greg
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of ioctx per rados connection

[Josh Durgin]

Yes, you can use multiple ioctxs with the same underlying rados connection. 
There's no hard limit on how many, it depends on your usage if/when a single 
rados connection becomes a bottleneck.

It's safe to use different ioctxs from multiple threads. IoCtxs have some local 
state like namespace, object locator key and snapshot that limit what you can 
do safely with multiple threads using the same IoCtx. librados.h has more 
details, but it's simplest to use a separate ioctx for each thread.

Josh


From: Michel Hollands 
Sent: Apr 8, 2015 6:54 AM
To: ceph-us...@ceph.com
Subject: [ceph-users] Number of ioctx per rados connection

> Hello,
>
> This is a question about the C API for librados. Can you use multiple “IO 
> contexts” (ioctx) per rados connection and if so how many ? Can these then be 
> used by multiple threads ? 
>
> Thanks in advance,
>
> Michel___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting placement groups to place evenly (again)

[J David]

On Wed, Apr 8, 2015 at 11:40 AM, Gregory Farnum  wrote:
> "ceph pg dump" will output the size of each pg, among other things.

Among many other things. :)

Here is the raw output, in case I'm misinterpreting it:

http://pastebin.com/j4ySNBdQ

It *looks* like the pg's are roughly uniform in size.  They range from
2.27GiB to 2.91GiB with an average of 2.58GiB and a standard deviation
of only 0.1GiB, and it looks like about 95% are within two standard
deviations.  The difference between the least used and most used OSDs
is on the order of 100+GB.  A few placement groups being a few hundred
megs bigger or smaller doesn't seem like it would account for that.

Breaking out the placement groups per OSD was a bit trickier, but this
seems to do the trick:

egrep '^2\.' ceph-pg-dump.txt  | awk '{print$14}' | tr -d '[]' | awk
-F, '{print$1"\n"$2}' | sort | uniq -c | sort -n | awk '{print$2"
"$1}' | sort -n

That showed that the OSD with the fewest PG's has 85 (and indeed that
is the lowest space-utilized OSD), and the OSD with the most PG's has
118.  That's not the OSD with the highest utiization, but the OSD with
the highest utilization does check in with 117 PG's.

So it does seem more of an issue of allocating placement groups
unevenly between OSDs than it does of unevenly sized placement groups.

Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rados bench seq read with single "thread"

[Deneau, Tom]

Say I have a single node "cluster" with 5 disks.
And using dd iflag=direct on that node, I can see disk read bandwidth at 160 
MB/s

I populate a pool with 4MB objects.
And then on that same single node, I run
$ drop-caches using /proc/sys/vm/drop_caches
$ rados -p mypool bench nn seq -t 1

What bandwidth should I expect for the rados bench seq command here?
(I am seeing approximately 70 MB/s with -t 1)

-- Tom Deneau


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] long blocking with writes on rbds

[Lionel Bouton]

On 04/08/15 18:24, Jeff Epstein wrote:
> Hi, I'm having sporadic very poor performance running ceph. Right now
> mkfs, even with nodiscard, takes 30 mintes or more. These kind of
> delays happen often but irregularly .There seems to be no common
> denominator. Clearly, however, they make it impossible to deploy ceph
> in production.
>
> I reported this problem earlier on ceph's IRC, and was told to add
> nodiscard to mkfs. That didn't help. Here is the command that I'm
> using to format an rbd:
>
> For example: mkfs.ext4 -text4 -m0 -b4096 -E nodiscard /dev/rbd1

I probably won't be able to help much, but people knowing more will need
at least:
- your Ceph version,
- the kernel version of the host on which you are trying to format
/dev/rbd1,
- which hardware and network you are using for this cluster (CPU, RAM,
HDD or SSD models, network cards, jumbo frames, ...).

>
> Ceph says everything is okay:
>
> cluster e96e10d3-ad2b-467f-9fe4-ab5269b70206
>  health HEALTH_OK
>  monmap e1: 3 mons at
> {a=192.168.224.4:6789/0,b=192.168.232.4:6789/0,c=192.168.240.4:6789/0}, 
> election
> epoch 12, quorum 0,1,2 a,b,c
>  osdmap e972: 6 osds: 6 up, 6 in
>   pgmap v4821: 4400 pgs, 44 pools, 5157 MB data, 1654 objects
> 46138 MB used, 1459 GB / 1504 GB avail
> 4400 active+clean
>

There's only one thing surprising me here: you have only 6 OSDs, 1504GB
(~ 250G / osd) and a total of 4400 pgs ? With a replication of 3 this is
2200 pgs / OSD, which might be too much and unnecessarily increase the
load on your OSDs.

Best regards,

Lionel Bouton
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] long blocking with writes on rbds

[Jeff Epstein]

Hi, I'm having sporadic very poor performance running ceph. Right now 
mkfs, even with nodiscard, takes 30 mintes or more. These kind of delays 
happen often but irregularly .There seems to be no common denominator. 
Clearly, however, they make it impossible to deploy ceph in production.


I reported this problem earlier on ceph's IRC, and was told to add 
nodiscard to mkfs. That didn't help. Here is the command that I'm using 
to format an rbd:


For example: mkfs.ext4 -text4 -m0 -b4096 -E nodiscard /dev/rbd1

Ceph says everything is okay:

cluster e96e10d3-ad2b-467f-9fe4-ab5269b70206
 health HEALTH_OK
 monmap e1: 3 mons at 
{a=192.168.224.4:6789/0,b=192.168.232.4:6789/0,c=192.168.240.4:6789/0}, 
election epoch 12, quorum 0,1,2 a,b,c

 osdmap e972: 6 osds: 6 up, 6 in
  pgmap v4821: 4400 pgs, 44 pools, 5157 MB data, 1654 objects
46138 MB used, 1459 GB / 1504 GB avail
4400 active+clean

And here's my crush map:

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5

# types
type 0 osd
type 1 district
type 2 region

# buckets
district district-1 {
id -1# do not change unnecessarily
# weight 3.000
alg straw
hash 0# rjenkins1
item osd.1 weight 1.000
item osd.2 weight 1.000
item osd.5 weight 1.000
}
district district-2 {
id -2# do not change unnecessarily
# weight 3.000
alg straw
hash 0# rjenkins1
item osd.0 weight 1.000
item osd.3 weight 1.000
item osd.4 weight 1.000
}
region ec2 {
id -3# do not change unnecessarily
# weight 2.000
alg straw
hash 0# rjenkins1
item district-1 weight 1.000
item district-2 weight 1.000
}

# rules
rule rule-district-1 {
ruleset 0
type replicated
min_size 2
max_size 3
step take district-1
step chooseleaf firstn 0 type osd
step emit
}
rule rule-district-2 {
ruleset 1
type replicated
min_size 2
max_size 3
step take district-2
step chooseleaf firstn 0 type osd
step emit
}

# end crush map

Does anyone have any insight into diagnosing this problem?

Jeff
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Somnath Roy]

Andrei,
Yes, I see it has lot of potential and I believe fixing the performance 
bottlenecks inside XIO messenger it should go further.
We are working on it and will keep community posted..

Thanks & Regards
Somnath

From: Andrei Mikhailovsky [mailto:and...@arhont.com]
Sent: Wednesday, April 08, 2015 2:22 AM
To: Andrey Korolyov
Cc: ceph-users@lists.ceph.com; ceph-devel; Somnath Roy
Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

Hi,

Am I the only person noticing disappointing results from the preliminary RDMA 
testing, or am I reading the numbers wrong?

Yes, it's true that on a very small cluster you do see a great improvement in 
rdma, but in real life rdma is used in large infrastructure projects, not on a 
few servers with a handful of osds. In fact, from what i've seen from the 
slides, the rdma implementation scales horribly to the point that it becomes 
slower the more osds you through at it.

From my limited knowledge, i have expected a much higher performance gains with 
rdma, taking into account that you should have much lower latency and overhead 
and lower cpu utilisation when using this transport in comparison with tcp.

Are we likely to see a great deal of improvement with ceph and rdma in a near 
future? Is there a roadmap for having a stable and reliable rdma protocol 
support?

Thanks

Andrei

From: "Andrey Korolyov" mailto:and...@xdel.ru>>
To: "Somnath Roy" mailto:somnath@sandisk.com>>
Cc: ceph-users@lists.ceph.com, "ceph-devel" 
mailto:ceph-de...@vger.kernel.org>>
Sent: Wednesday, 8 April, 2015 9:28:12 AM
Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy 
mailto:somnath@sandisk.com>> wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
> implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What
are TCP settings you using?For example, difference can be lowered on
scale due to less intensive per-connection acceleration on CUBIC on a
larger number of nodes, though I do not believe that it was a main
reason for an observed TCP catchup on a relatively flat workload such
as fio generates.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Somnath Roy]

I am using Mellanox 40GbE, I think it is TCP offloaded.

From: Viral Mehta [mailto:viral@gmail.com]
Sent: Wednesday, April 08, 2015 1:30 AM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-devel
Subject: Re: Preliminary RDMA vs TCP numbers

I am sorry, I am new to the discussion.
But, is it TCP offloaded (TOE) or otherwise ?

On Wed, Apr 8, 2015 at 1:47 PM, Somnath Roy 
mailto:somnath@sandisk.com>> wrote:

Hi,
Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
implementation (on top of SSDs) in the following link.

http://www.slideshare.net/somnathroy7568/ceph-on-rdma

The attachment didn't go through it seems, so, I had to use slideshare.

Mark,
If we have time, I can present it in tomorrow's performance meeting.

Thanks & Regards
Somnath



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to 
majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Thanks,
Viral Mehta
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Somnath Roy]

I used the default TCP setting in Ubuntu 14.04.

-Original Message-
From: Andrey Korolyov [mailto:and...@xdel.ru]
Sent: Wednesday, April 08, 2015 1:28 AM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-devel
Subject: Re: Preliminary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy  wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
> implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What are TCP 
settings you using?For example, difference can be lowered on scale due to less 
intensive per-connection acceleration on CUBIC on a larger number of nodes, 
though I do not believe that it was a main reason for an observed TCP catchup 
on a relatively flat workload such as fio generates.



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSDs not coming up on one host

[Gregory Farnum]

Im on my phone so can't check exactly what those threads are trying to do,
but the osd has several threads which are stuck. The FileStore threads are
certainly trying to access the disk/local filesystem. You may not have a
hardware fault, but it looks like something in your stack is not behaving
when the osd asks the filesystem to do something. Check dmesg, etc.
-Greg
On Wed, Apr 8, 2015 at 7:55 AM Jacob Reid 
wrote:

> I have a cluster of 3 servers (recently updated from 0.80.5 to 0.80.9),
> each running 4-6 osds as single disks, journaled to a partition each on an
> SSD, with 3 mons on separate hosts. Recently, I started taking the hosts
> down to move disks between controllers and add extra disk capacity before
> bringing them back into the cluster. This worked fine on the first host,
> which is now back in with 6 osds. However, on doing the same with the
> second, the osds are (re-)created properly, but never join the cluster in
> an 'up' state; they just stay as 'down'. The actual osd processes
> themselves show a status of 'booting' they never leave.
>
> $ ceph osd tree
> # idweighttype nameup/downreweight
> -116root default
> -26host ceph01
> 01osd.0up1
> 31osd.3up1
> 11osd.1up1
> 21osd.2up1
> 121osd.12up1
> 131osd.13up1
> -36host ceph02
> 71osd.7down0
> 41osd.4down0
> 61osd.6down0
> 141osd.14down0
> 51osd.5down0
> 151osd.15down0
> -44host ceph03
> 81osd.8up1
> 91osd.9up1
> 101osd.10up1
> 111osd.11up1
>
> In the log for an osd that stays down, I see the normal starting up
> messages, then just the following:
>
> 2015-04-08 14:22:38.324037 7f683b90c780  0 osd.15 3808 done with init,
> starting boot process
> 2015-04-08 14:26:23.614375 7f68235fe700  1 heartbeat_map reset_timeout
> 'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
> 2015-04-08 14:26:23.614397 7f6824e01700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
> 2015-04-08 14:26:23.614387 7f6824600700  1 heartbeat_map reset_timeout
> 'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
> 2015-04-08 14:26:23.614404 7f6825602700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6825602700' had timed out after 4
> 2015-04-08 14:26:23.614410 7f6823dff700  1 heartbeat_map reset_timeout
> 'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
> 2015-04-08 14:26:23.614431 7f682f55e700  1 heartbeat_map reset_timeout
> 'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
> 2015-04-08 14:26:23.615003 7f6837d6f700  1 heartbeat_map is_healthy
> 'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
> 2015-04-08 14:26:23.615299 7f682fd5f700  1 heartbeat_map reset_timeout
> 'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
> 2015-04-08 14:26:53.066311 7f682f55e700  1 heartbeat_map reset_timeout
> 'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
> 2015-04-08 14:26:53.066777 7f682fd5f700  1 heartbeat_map reset_timeout
> 'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
> 2015-04-08 14:28:23.237330 7f6823dff700  1 heartbeat_map reset_timeout
> 'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
> 2015-04-08 14:28:23.237646 7f6824600700  1 heartbeat_map reset_timeout
> 'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
> 2015-04-08 14:28:23.237690 7f68235fe700  1 heartbeat_map reset_timeout
> 'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
> 2015-04-08 14:28:23.238010 7f6824e01700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
> 2015-04-08 14:28:23.238051 7f6825602700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6825602700' had timed out after 4
> 2015-04-08 14:29:53.469859 7f6824e01700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
> 2015-04-08 14:29:53.469882 7f68235fe700  1 heartbeat_map reset_timeout
> 'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
> 2015-04-08 14:29:53.469895 7f6825602700  1 heartbeat_map reset_timeout
> 'OSD::op_tp thread 0x7f6825602700' had timed out after 4
> 2015-04-08 14:29:53.469900 7f6823dff700  1 heartbeat_map reset_timeout
> 'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
> 2015-04-08 14:29:53.469947 7f6824600700  1 heartbeat_map reset_timeout
> 'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
> 2015-04-08 14:31:45.141384 7f682f55e700  1 heartbeat_map reset_timeout
> 'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
> 2015-04-08 14:31:45.141425 7f6824600700  1 heartbeat_map reset_timeou

Re: [ceph-users] Getting placement groups to place evenly (again)

[Gregory Farnum]

"ceph pg dump" will output the size of each pg, among other things.
On Wed, Apr 8, 2015 at 8:34 AM J David  wrote:

> On Wed, Apr 8, 2015 at 11:33 AM, Gregory Farnum  wrote:
> > Is this a problem with your PGs being placed unevenly, with your PGs
> being
> > sized very differently, or both?
>
> Please forgive the silly question, but how would one check that?
>
> Thanks!
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting placement groups to place evenly (again)

[J David]

On Wed, Apr 8, 2015 at 11:33 AM, Gregory Farnum  wrote:
> Is this a problem with your PGs being placed unevenly, with your PGs being
> sized very differently, or both?

Please forgive the silly question, but how would one check that?

Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Getting placement groups to place evenly (again)

[Gregory Farnum]

Is this a problem with your PGs being placed unevenly, with your PGs being
sized very differently, or both?

CRUSH is never going to balance perfectly, but the numbers you're quoting
look a bit worse than usual at first glance.
-Greg
On Tue, Apr 7, 2015 at 8:16 PM J David  wrote:

> Getting placement groups to be placed evenly continues to be a major
> challenge for us, bordering on impossible.
>
> When we first reported trouble with this, the ceph cluster had 12
> OSD's (each Intel DC S3700 400GB) spread across three nodes.  Since
> then, it has grown to 8 nodes with 38 OSD's.
>
> The average utilization is 80%.  With weights all set to 1, utlization
> varies from 53% to 96%.  Immediately after "ceph osd
> reweight-by-utilization 105" it varies from 61% to 90%.  Essentially,
> once utilization goes over 75%, managing the osd weights to keep all
> of them under 90% becomes a full-time job.
>
> This is on 0.80.9 with optimal tunables (including chooseleaf_vary_r=1
> and straw_calc_version=1 setting.  The pool has 2048 placement groups
> and has size=2.
>
> What, if anything, can we do about this?  The goals are twofold, and
> in priority order:
>
> 1) Guarantee that the cluster can survive the loss of a node without
> dying because one "unlucky" OSD overfills.
>
> 2) Utilize the available space as efficiently as possible.  We are
> targeting 85% utilization, but currently things to get ugly pretty
> quickly over 75%.
>
> Thanks for any advice!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Inconsistent "ceph-deploy disk list" command results

[f...@univ-lr.fr]

Hi Travis,

Thanks for your advice, Issue #11347 created
   http://tracker.ceph.com/issues/11347

Frederic

Travis Rhoden  a écrit le 8/04/15 16:44 :

Hi Frederic,

Thanks for the report!  Do you mind throwing this details into a bug
report at http://tracker.ceph.com/ ?

I have seen the same thing once before, but at the time didn't have
the chance to check if the inconsistency was coming from ceph-deploy
or from ceph-disk.  This certainly seems to point at ceph-deploy!

 - Travis

On Wed, Apr 8, 2015 at 4:15 AM, f...@univ-lr.fr  wrote:
  

Hi all,

I want to alert on a command we've learned to avoid for its inconsistent
results.

on Giant 0.87.1 and Hammer 0.93.0 (ceph-deploy-1.5.22-0.noarch was used in
both cases) "ceph-deploy disk list" command has a problem.

We should get an exhaustive list of devices entries, like this one :
../..
/dev/sdk :
/dev/sdk1 ceph data, active, cluster ceph, osd.34, journal /dev/sda9
../..

But from the admin node,
when we count how many disks we have on our nodes , results are incorrect
and differ each time :
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
 8
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
12
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
10
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
15
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
12

From the nodes,
results are correct (15) and always the same :
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15

but a pretty similar 'ceph-deploy osd list' command works fine

Frederic
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSDs not coming up on one host

[Jacob Reid]

I have a cluster of 3 servers (recently updated from 0.80.5 to 0.80.9), each 
running 4-6 osds as single disks, journaled to a partition each on an SSD, with 
3 mons on separate hosts. Recently, I started taking the hosts down to move 
disks between controllers and add extra disk capacity before bringing them back 
into the cluster. This worked fine on the first host, which is now back in with 
6 osds. However, on doing the same with the second, the osds are (re-)created 
properly, but never join the cluster in an 'up' state; they just stay as 
'down'. The actual osd processes themselves show a status of 'booting' they 
never leave.

$ ceph osd tree
# idweighttype nameup/downreweight
-116root default
-26host ceph01
01osd.0up1   
31osd.3up1   
11osd.1up1   
21osd.2up1   
121osd.12up1   
131osd.13up1   
-36host ceph02
71osd.7down0   
41osd.4down0   
61osd.6down0   
141osd.14down0   
51osd.5down0   
151osd.15down0   
-44host ceph03
81osd.8up1   
91osd.9up1   
101osd.10up1   
111osd.11up1   

In the log for an osd that stays down, I see the normal starting up messages, 
then just the following:

2015-04-08 14:22:38.324037 7f683b90c780  0 osd.15 3808 done with init, starting 
boot process
2015-04-08 14:26:23.614375 7f68235fe700  1 heartbeat_map reset_timeout 
'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
2015-04-08 14:26:23.614397 7f6824e01700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
2015-04-08 14:26:23.614387 7f6824600700  1 heartbeat_map reset_timeout 
'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
2015-04-08 14:26:23.614404 7f6825602700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6825602700' had timed out after 4
2015-04-08 14:26:23.614410 7f6823dff700  1 heartbeat_map reset_timeout 
'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
2015-04-08 14:26:23.614431 7f682f55e700  1 heartbeat_map reset_timeout 
'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
2015-04-08 14:26:23.615003 7f6837d6f700  1 heartbeat_map is_healthy 
'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
2015-04-08 14:26:23.615299 7f682fd5f700  1 heartbeat_map reset_timeout 
'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
2015-04-08 14:26:53.066311 7f682f55e700  1 heartbeat_map reset_timeout 
'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
2015-04-08 14:26:53.066777 7f682fd5f700  1 heartbeat_map reset_timeout 
'FileStore::op_tp thread 0x7f682fd5f700' had timed out after 4
2015-04-08 14:28:23.237330 7f6823dff700  1 heartbeat_map reset_timeout 
'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
2015-04-08 14:28:23.237646 7f6824600700  1 heartbeat_map reset_timeout 
'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
2015-04-08 14:28:23.237690 7f68235fe700  1 heartbeat_map reset_timeout 
'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
2015-04-08 14:28:23.238010 7f6824e01700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
2015-04-08 14:28:23.238051 7f6825602700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6825602700' had timed out after 4
2015-04-08 14:29:53.469859 7f6824e01700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6824e01700' had timed out after 4
2015-04-08 14:29:53.469882 7f68235fe700  1 heartbeat_map reset_timeout 
'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
2015-04-08 14:29:53.469895 7f6825602700  1 heartbeat_map reset_timeout 
'OSD::op_tp thread 0x7f6825602700' had timed out after 4
2015-04-08 14:29:53.469900 7f6823dff700  1 heartbeat_map reset_timeout 
'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
2015-04-08 14:29:53.469947 7f6824600700  1 heartbeat_map reset_timeout 
'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
2015-04-08 14:31:45.141384 7f682f55e700  1 heartbeat_map reset_timeout 
'FileStore::op_tp thread 0x7f682f55e700' had timed out after 4
2015-04-08 14:31:45.141425 7f6824600700  1 heartbeat_map reset_timeout 
'OSD::recovery_tp thread 0x7f6824600700' had timed out after 4
2015-04-08 14:31:45.141603 7f6837d6f700  1 heartbeat_map is_healthy 
'OSD::command_tp thread 0x7f68235fe700' had timed out after 4
2015-04-08 14:31:45.141614 7f6837d6f700  1 heartbeat_map is_healthy 
'OSD::disk_tp thread 0x7f6823dff700' had timed out after 4
2015-04-08 14:31:45.141616 7f6837d6f700  1 heartbeat_map is_healthy 'OSD::op_tp 
thread 0x7f6824e01700' had timed out after 4
2015-04-08 14:31:45.141619 7f6837d6f700  1 heartbea

Re: [ceph-users] Inconsistent "ceph-deploy disk list" command results

[Travis Rhoden]

Hi Frederic,

Thanks for the report!  Do you mind throwing this details into a bug
report at http://tracker.ceph.com/ ?

I have seen the same thing once before, but at the time didn't have
the chance to check if the inconsistency was coming from ceph-deploy
or from ceph-disk.  This certainly seems to point at ceph-deploy!

 - Travis

On Wed, Apr 8, 2015 at 4:15 AM, f...@univ-lr.fr  wrote:
> Hi all,
>
> I want to alert on a command we've learned to avoid for its inconsistent
> results.
>
> on Giant 0.87.1 and Hammer 0.93.0 (ceph-deploy-1.5.22-0.noarch was used in
> both cases) "ceph-deploy disk list" command has a problem.
>
> We should get an exhaustive list of devices entries, like this one :
> ../..
> /dev/sdk :
> /dev/sdk1 ceph data, active, cluster ceph, osd.34, journal /dev/sda9
> ../..
>
> But from the admin node,
> when we count how many disks we have on our nodes , results are incorrect
> and differ each time :
> $ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
>  8
> $ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
> 12
> $ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
> 10
> $ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
> 15
> $ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
> 12
>
> From the nodes,
> results are correct (15) and always the same :
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
> $ ceph-disk list 2>&1|grep "active," |wc -l
> 15
>
> but a pretty similar 'ceph-deploy osd list' command works fine
>
> Frederic
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [ANN] ceph-deploy 1.5.23 released

[Travis Rhoden]

Hi All,

This is a new release of ceph-deploy that includes a new feature for
Hammer and bugfixes.  ceph-deploy can be installed from the ceph.com
hosted repos for Firefly, Giant, Hammer, or testing, and is also
available on PyPI.

ceph-deploy now defaults to installing the Hammer release. If you need
to install a different release, use the --release flag.

To go along with the Hammer release, ceph-deploy now includes support
for a drastically simplified deployment for RGW.  See further details
at [1] and [2].

This release also fixes an issue where keyrings pushed to remote nodes
ended up with world-readable permissions.

The full changelog can be seen at [3].

Please update!

Cheers,

 - Travis

[1] http://ceph.com/docs/master/start/quick-ceph-deploy/#add-an-rgw-instance
[2] http://ceph.com/ceph-deploy/docs/rgw.html
[3] http://ceph.com/ceph-deploy/docs/changelog.html#id2
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] when recovering start

[Stéphane DUGRAVOT]

- Le 8 Avr 15, à 14:21, lijian  a écrit : 

> Hi Stephane,

> I dump from a osd deamon

You have to apply the mon_osd_down_out_interval value to monitor and not osd. 
What is the value on the mon ? 
Stephane. 

> Thanks
> Jian LI

> At 2015-04-08 16:05:04, "Stéphane DUGRAVOT" 
> 
> wrote:

>> - Le 7 Avr 15, à 14:57, lijian < blacker1...@163.com > a écrit :

>>> Haomai Wang,

>>> the mon_osd_down_out_interval is 300, please refer to my settings, and I 
>>> use the
>>> cli 'service ceph stop osd.X' to stop a osd
>>> the pg status change to remap,backfill and recovering ... immediately
>>> so other something wrong with my settings or operation?

>> Hi,
>> From what daemon you get the mon_osd_down_out_interval value ?
>> Is it from mon or osd ?
>> Stephane.

>>> Thanks,

>>> Jian Ji

>>> At 2015-04-07 20:38:29, "Haomai Wang" < haomaiw...@gmail.com > wrote:
>>> >Whatever the version you tested, ceph won't recover data when you
>>> >manually stop osd immediately. And it will trigger mark down osd out
>>> >when it reach "mon_osd_down_out_interval" seconds.

>>> >On Tue, Apr 7, 2015 at 8:33 PM, lijian < blacker1...@163.com > wrote:
>>> >> Hi,
>>> >> The recovering start delay 300s after I stop a osd and the osd status 
>>> >> change
>>> >> from in to out,  the test ENV is Ceph 0.80.7

>>> >> But I test in ceph 0.87.1, the recovering start immediately after I stop 
>>> >> a
>>> >> OSD,all the settings is the default value,the following is mon_osd* 
>>> >> settings
>>> >> in my test ENV:
>>> >>   "mon_osd_laggy_halflife": "3600",
>>> >>   "mon_osd_laggy_weight": "0.3",
>>> >>   "mon_osd_adjust_heartbeat_grace": "true",
>>> >>   "mon_osd_adjust_down_out_interval": "true",
>>> >>   "mon_osd_auto_mark_in": "false",
>>> >>   "mon_osd_auto_mark_auto_out_in": "true",
>>> >>   "mon_osd_auto_mark_new_in": "true",
>>> >>   "mon_osd_down_out_interval": "300",
>>> >>   "mon_osd_down_out_subtree_limit": "rack",
>>> >>   "mon_osd_min_up_ratio": "0.3",
>>> >>   "mon_osd_min_in_ratio": "0.3",
>>> >>   "mon_osd_max_op_age": "32",
>>> >>   "mon_osd_max_split_count": "32",
>>> >>   "mon_osd_allow_primary_temp": "false",
>>> >>   "mon_osd_allow_primary_affinity": "false",
>>> >>   "mon_osd_full_ratio": "0.95",
>>> >>   "mon_osd_nearfull_ratio": "0.85",
>>> >>   "mon_osd_report_timeout": "45000",
>>> >>   "mon_osd_min_down_reporters": "50",
>>> >>   "mon_osd_min_down_reports": "150",
>>> >>   "mon_osd_force_trim_to": "0",

>>> >> so when the recovering start? why they are different with the two Ceph
>>> >> version, or someting wrong with my settings

>>> >> Thanks!
>>> >> Jian Li



>>> >> ___
>>> >> ceph-users mailing list
> ceph-users@lists.ceph.com >>
>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




>>> >--
>>> >Best Regards,

>>> >Wheat

>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw GC parallelization

[LOPEZ Jean-Charles]

Hi,

the following parameters can be used to have more GC processing and more 
efficient
- rgw_gc_max_objs defaults to 32
- rgw_gc_obj_min_wait defaults to 2 * 3600
- rgw_gc_processor_max_time defaults to 3600 
- rgw_gc_processor_period defaults to 3600

It is recommended to set rgw_gc_max_objs with a prime number rather than the 
default value of 32 to have a better dispersion on the objects across the GC 
buckets. The minimum action here is to set this value to either 31 or to a 
higher prime number such as 97 for example.

You can then play with the other parameters so that objects are picked up 
faster by the GC process and let the GC process run for longer period of time 
rather than the default values.

This way you can adjust to your particular environment

JC

> On 8 Apr 2015, at 07:12, c...@jack.fr.eu.org wrote:
> 
> Hi,
> 
> I have a Ceph cluster, used through radosgw.
> In that cluster, I write files each seconds: input files are known,
> predictible and stable, there is always the same number of new
> fiexd-size files, each second.
> 
> Theses files are kept a few days, then remove after a fixed duration.
> And thus, I want to remove the same stable number of files each second
> (I want to remove them faster than store them, or my cluster will grow
> to death)
> 
> On a daily-basis, I look at my files, remove the useless ones, and force
> the gc process.
> The first part of the process is quite fast (couple of minutes).
> The last part (gc process) is, however, a bit slow.
> 
> Is there a way to speed up the garbage collection ?
> Is the GC dedicated to a radosgw (meaning : if I remove & gc process
> from multiple radosgw, will the process be faster ?)
> From my experiment, multithreading the gc process using some stuff like
> parallel is not efficient (I guess gc processes are locking each others
> or something): is that true ?
> 
> Thanks for advises!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD hard crash on kernel 3.10

[Shawn Edwards]

We've been working on a storage repository for xenserver 6.5, which uses
the 3.10 kernel (ug).  I got the xenserver guys to include the rbd and
libceph kernel modules into the 6.5 release, so that's at least available.

Where things go bad is when we have many (>10 or so) VMs on one host, all
using RBD clones for the storage mapped using the rbd kernel module.  The
Xenserver crashes so badly that it doesn't even get a chance to kernel
panic.  The whole box just hangs.

Has anyone else seen this sort of behavior?

We have a lot of ways to try to work around this, but none of them are very
pretty:

* move the code to user space, ditch the kernel driver:  The build tools
for Xenserver are all CentOS5 based, and it is painful to get all of the
deps built to get the ceph user space libs built.

* backport the ceph and rbd kernel modules to 3.10.  Has proven painful, as
the block device code changed somewhere in the 3.14-3.16 timeframe.

* forward-port the xen kernel patches from 3.10 to a newer driver (3.18
preferred) and run that on xenserver.  Painful for the same reasons as
above, but in the opposite direction.

Any and all suggestions are welcome.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

[Dan Ryder (daryder)]

Yes, the unit is in seconds for those latencies. The sum/avgcount is the 
average since the daemon was (re)started. 

If you're interested, I've co-authored a collectd plugin which captures data 
from Ceph daemons - built into the plugin I give the option to calculate either 
the long-run avg (sum/avgcount) or the last-poll delta 
(sum_now-sum_last_poll/avgcount_now-avgcount_last_poll). It's been added to the 
latest collectd branch (https://github.com/collectd/collectd).


Dan Ryder

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Francois Lafont
Sent: Wednesday, April 08, 2015 10:11 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] What are you doing to locate performance issues in a 
Ceph cluster?

Chris Kitzmiller wrote:

>> ~# ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok perf
>>
>>  [...]
>>
>>  "osd": { "opq": 0,
>>  "op_wip": 0,
>>  "op": 3566,
>>  "op_in_bytes": 208803635,
>>  "op_out_bytes": 146962506,
>>  "op_latency": { "avgcount": 3566,
>>  "sum": 100.330695000},
>>  "op_process_latency": { "avgcount": 3566,
>>  "sum": 84.702772000},
>>  "op_r": 471,
>>  "op_r_out_bytes": 146851024,
>>  "op_r_latency": { "avgcount": 471,
>>  "sum": 1.329795000},
>>
>>   [...]
>>
>> Is the value of "op_r_latency" (ie 1.329ms above)?
>> In this case, I don't understand the meaning of "avgcount"
>> and "sum".
>>
>> "sum" is the sum of what?
>> "avgcount" is the average of what?
> 
> There are a bunch of these avgcount/sum pairs and, from what I've gleaned, 
> you're to simply divide sum by avgcount to get the mean of that particular 
> stat over whatever time frame it is measuring.

Err..., I'm sorry, I'm not sure to well understand. If I take the values of 
op_r_latency above, I have:

sum/avgcount = 1.329795000/471 = 0.002823344

0,002823344ms would be my latency of read operation?
It seems to me impossible (unfortunately ;)) or maybe the unit is in seconds?
In this case 2.823344ms could be a plausible value. In any case, I don't 
understand the name "avgcount". The name "count" seems to me more logical (but 
maybe I don't really have understand its meaning).

If I see the source code ./src/common/perf_counters.cc, it seems to me that, 
indeed, the number is in seconds but I'm (really) not a c++ expert.
Is possible to confirm to me that?

Another thing: if I understand well, the value sum/avgcount is an average of 
latency, average computed from the start of the osd daemon. So, after lot of 
times, the average will be more stable and it no longer incur variation.
Is it possible to restart the counters? I noticed that restarting the daemon 
works but it's a little drastic.

--
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Radosgw GC parallelization

[ceph]

Hi,

I have a Ceph cluster, used through radosgw.
In that cluster, I write files each seconds: input files are known,
predictible and stable, there is always the same number of new
fiexd-size files, each second.

Theses files are kept a few days, then remove after a fixed duration.
And thus, I want to remove the same stable number of files each second
(I want to remove them faster than store them, or my cluster will grow
to death)

On a daily-basis, I look at my files, remove the useless ones, and force
the gc process.
The first part of the process is quite fast (couple of minutes).
The last part (gc process) is, however, a bit slow.

Is there a way to speed up the garbage collection ?
Is the GC dedicated to a radosgw (meaning : if I remove & gc process
from multiple radosgw, will the process be faster ?)
>From my experiment, multithreading the gc process using some stuff like
parallel is not efficient (I guess gc processes are locking each others
or something): is that true ?

Thanks for advises!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

[Francois Lafont]

Chris Kitzmiller wrote:

>> ~# ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok perf
>>
>>  [...]
>>
>>  "osd": { "opq": 0,
>>  "op_wip": 0,
>>  "op": 3566,
>>  "op_in_bytes": 208803635,
>>  "op_out_bytes": 146962506,
>>  "op_latency": { "avgcount": 3566,
>>  "sum": 100.330695000},
>>  "op_process_latency": { "avgcount": 3566,
>>  "sum": 84.702772000},
>>  "op_r": 471,
>>  "op_r_out_bytes": 146851024,
>>  "op_r_latency": { "avgcount": 471,
>>  "sum": 1.329795000},
>>
>>   [...]
>>
>> Is the value of "op_r_latency" (ie 1.329ms above)?
>> In this case, I don't understand the meaning of "avgcount"
>> and "sum".
>>
>> "sum" is the sum of what?
>> "avgcount" is the average of what?
> 
> There are a bunch of these avgcount/sum pairs and, from what I've gleaned, 
> you're to simply divide sum by avgcount to get the mean of that particular 
> stat over whatever time frame it is measuring.

Err..., I'm sorry, I'm not sure to well understand. If I take the values
of op_r_latency above, I have:

sum/avgcount = 1.329795000/471 = 0.002823344

0,002823344ms would be my latency of read operation?
It seems to me impossible (unfortunately ;)) or maybe the unit is in seconds?
In this case 2.823344ms could be a plausible value. In any case,
I don't understand the name "avgcount". The name "count" seems to me
more logical (but maybe I don't really have understand its meaning).

If I see the source code ./src/common/perf_counters.cc, it seems to me
that, indeed, the number is in seconds but I'm (really) not a c++ expert.
Is possible to confirm to me that?

Another thing: if I understand well, the value sum/avgcount is an average
of latency, average computed from the start of the osd daemon. So, after lot of
times, the average will be more stable and it no longer incur variation.
Is it possible to restart the counters? I noticed that restarting the daemon
works but it's a little drastic.

-- 
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Number of ioctx per rados connection

[Michel Hollands]

Hello,

This is a question about the C API for librados. Can you use multiple “IO 
contexts” (ioctx) per rados connection and if so how many ? Can these then be 
used by multiple threads ?

Thanks in advance,

Michel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Any suggestion  geeks


VS

On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh 
wrote:

>
> Hi
>
>
> The below suggestion also didn’t worked
>
>
> Full logs here : http://paste.ubuntu.com/10771939/
>
>
>
>
> [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph
>
> Loaded plugins: fastestmirror, priorities
>
> Loading mirror speeds from cached hostfile
>
>  * base: mirror.zetup.net
>
>  * epel: ftp.fi.muni.cz
>
>  * extras: mirror.zetup.net
>
>  * updates: mirror.zetup.net
>
> 25 packages excluded due to repository priority protections
>
> Available Packages
>
> ceph.x86_64
> 0.80.6-0.el7.centos
> Ceph
>
> ceph.x86_64
> 0.80.7-0.el7.centos
> Ceph
>
> ceph.x86_64
> 0.80.8-0.el7.centos
> Ceph
>
> ceph.x86_64
> 0.80.9-0.el7.centos
> Ceph
>
> [root@rgw-node1 yum.repos.d]#
>
>
>
>
>
> Its not able to install latest available package , yum is getting confused
> with other DOT releases.
>
>
> Any other suggestion to fix this ???
>
>
>
> --> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
> package: librbd1-0.80.9-0.el7.centos.x86_64
>
> --> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
> package: librbd1-0.80.9-0.el7.centos.x86_64
>
> --> Finished Dependency Resolution
>
> Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_system-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_system-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libaio.so.1(LIBAIO_0.4)(64bit)
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: librados2 = 0.80.7-0.el7.centos
>
>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.6-0.el7.centos
>
>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.7-0.el7.centos
>
>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.8-0.el7.centos
>
>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.9-0.el7.centos
>
> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_thread-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: python-requests
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: librbd1 = 0.80.7-0.el7.centos
>
>Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 0.80.6-0.el7.centos
>
>Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 0.80.7-0.el7.centos
>
>Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 0.80.8-0.el7.centos
>
>Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 0.80.9-0.el7.centos
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_system-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: python-ceph = 0.80.7-0.el7.centos
>
>Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)
>
>python-ceph = 0.80.6-0.el7.centos
>
>Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>python-ceph = 0.80.7-0.el7.centos
>
>Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)
>
>python-ceph = 0.80.8-0.el7.centos
>
>Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)
>
>python-ceph = 0.80.9-0.el7.centos
>
> Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_system-mt.so.1.53.0()(64bit)
>
> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: python-requests
>
> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: librados2 = 0.80.7-0.el7.centos
>
>Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.6-0.el7.centos
>
>Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.7-0.el7.centos
>
>Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.8-0.el7.centos
>
>Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 0.80.9-0.el7.centos
>
> Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)
>
>Requires: libaio.so.1(LIBAIO_0.1)(64bit)
>
> Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)
>
>Requires: libboost_system-mt.so.1.53.0()(64bit)
>
> Error: Package: librados2-0

Re: [ceph-users] Cascading Failure of OSDs

[Francois Lafont]

Hi,

01/04/2015 17:28, Quentin Hartman wrote:

> Right now we're just scraping the output of ifconfig:
> 
> ifconfig p2p1 | grep -e 'RX\|TX' | grep packets | awk '{print $3}'
> 
> It clunky, but it works. I'm sure there's a cleaner way, but this was
> expedient.
> 
> QH

Ok, thx for the information Quentin.
Just in case it could be useful, I have noticed the -s option (on my
Ubuntu) that offer an output probably easier to parse:

# "column -t" is just to make it's nice for the human eyes.
ifconfig -s | column -t

Bye.

-- 
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [a bit off-topic] Power usage estimation of hardware for Ceph

[Francois Lafont]

Hi,

Sorry in advance for this thread not directly linked to Ceph. ;)
We are thinking about buying servers to build a ceph cluster and we
would like to have, if possible, a *approximative* power usage
estimation of these servers (this parameter could be important in
your choice):

1. the 12xbays supermicro OSD node
   (here https://www.supermicro.com/solutions/datasheet_Ceph.pdf,
   page 2, model SSG-6027R-OSD040H in the table)

2. SC216-based chassis 2U, 24xbays 2.5" (like this one for instance
   http://www.supermicro.com/products/chassis/2U/216/SC216BA-R1K28LP.cfm)

If someone here has a server as above, we would be curious to have
a appromative power usage estimation (for instance in volt-ampere).

Thanks in advance for your help.
Regards

-- 
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What are you doing to locate performance issues in a Ceph cluster?

[Chris Kitzmiller]

On Apr 7, 2015, at 7:44 PM, Francois Lafont wrote:
> Chris Kitzmiller wrote:
> I graph aggregate stats for `ceph --admin-daemon 
>> /var/run/ceph/ceph-osd.$osdid.asok perf dump`. If the max latency strays too 
>> far 
>> outside of my mean latency I know to go look for the troublemaker. My graphs 
>> look something like this:
>> 
>> [...]
> 
> Thanks Chris for these interesting explanations.
> Sorry for my basic question but which is the entry in the output that gives
> you the read latency?
> 
> Here is an example from my cluster (Firefly):
> 
> ~# ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok perf
> 
>  [...]
> 
>  "osd": { "opq": 0,
>  "op_wip": 0,
>  "op": 3566,
>  "op_in_bytes": 208803635,
>  "op_out_bytes": 146962506,
>  "op_latency": { "avgcount": 3566,
>  "sum": 100.330695000},
>  "op_process_latency": { "avgcount": 3566,
>  "sum": 84.702772000},
>  "op_r": 471,
>  "op_r_out_bytes": 146851024,
>  "op_r_latency": { "avgcount": 471,
>  "sum": 1.329795000},
> 
>   [...]
> 
> Is the value of "op_r_latency" (ie 1.329ms above)?
> In this case, I don't understand the meaning of "avgcount"
> and "sum".
> 
> "sum" is the sum of what?
> "avgcount" is the average of what?

There are a bunch of these avgcount/sum pairs and, from what I've gleaned, 
you're to simply divide sum by avgcount to get the mean of that particular stat 
over whatever time frame it is measuring.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] when recovering start

[lijian]

Hi Stephane,

I dump from a osd deamon

Thanks
Jian LI







At 2015-04-08 16:05:04, "Stéphane DUGRAVOT" 
 wrote:





- Le 7 Avr 15, à 14:57, lijian  a écrit :



Haomai Wang,

the mon_osd_down_out_interval is 300, please refer to my settings, and I use 
the cli 'service ceph stop osd.X' to stop a osd
the pg status change to remap,backfill and recovering ... immediately
so  other something wrong with my settings or operation?
Hi,

From what daemon you get the mon_osd_down_out_interval value ?

Is it from mon or osd ?

Stephane.


Thanks,

Jian Ji






At 2015-04-07 20:38:29, "Haomai Wang"  wrote:
>Whatever the version you tested, ceph won't recover data when you
>manually stop osd immediately. And it will trigger mark down osd out
>when it reach "mon_osd_down_out_interval" seconds.
>
>On Tue, Apr 7, 2015 at 8:33 PM, lijian  wrote:
>> Hi,
>> The recovering start delay 300s after I stop a osd and the osd status change
>> from in to out,  the test ENV is Ceph 0.80.7
>>
>> But I test in ceph 0.87.1, the recovering start immediately after I stop a
>> OSD,all the settings is the default value,the following is mon_osd* settings
>> in my test ENV:
>>   "mon_osd_laggy_halflife": "3600",
>>   "mon_osd_laggy_weight": "0.3",
>>   "mon_osd_adjust_heartbeat_grace": "true",
>>   "mon_osd_adjust_down_out_interval": "true",
>>   "mon_osd_auto_mark_in": "false",
>>   "mon_osd_auto_mark_auto_out_in": "true",
>>   "mon_osd_auto_mark_new_in": "true",
>>   "mon_osd_down_out_interval": "300",
>>   "mon_osd_down_out_subtree_limit": "rack",
>>   "mon_osd_min_up_ratio": "0.3",
>>   "mon_osd_min_in_ratio": "0.3",
>>   "mon_osd_max_op_age": "32",
>>   "mon_osd_max_split_count": "32",
>>   "mon_osd_allow_primary_temp": "false",
>>   "mon_osd_allow_primary_affinity": "false",
>>   "mon_osd_full_ratio": "0.95",
>>   "mon_osd_nearfull_ratio": "0.85",
>>   "mon_osd_report_timeout": "45000",
>>   "mon_osd_min_down_reporters": "50",
>>   "mon_osd_min_down_reports": "150",
>>   "mon_osd_force_trim_to": "0",
>>
>> so when the recovering start? why they are different with the two Ceph
>> version, or someting wrong with my settings
>>
>> Thanks!
>> Jian Li
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
>-- 
>Best Regards,
>
>Wheat




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Hi


The below suggestion also didn’t worked


Full logs here : http://paste.ubuntu.com/10771939/




[root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph

Loaded plugins: fastestmirror, priorities

Loading mirror speeds from cached hostfile

 * base: mirror.zetup.net

 * epel: ftp.fi.muni.cz

 * extras: mirror.zetup.net

 * updates: mirror.zetup.net

25 packages excluded due to repository priority protections

Available Packages

ceph.x86_64
0.80.6-0.el7.centos
Ceph

ceph.x86_64
0.80.7-0.el7.centos
Ceph

ceph.x86_64
0.80.8-0.el7.centos
Ceph

ceph.x86_64
0.80.9-0.el7.centos
Ceph

[root@rgw-node1 yum.repos.d]#





Its not able to install latest available package , yum is getting confused
with other DOT releases.


Any other suggestion to fix this ???



--> Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
package: librbd1-0.80.9-0.el7.centos.x86_64

--> Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
package: librbd1-0.80.9-0.el7.centos.x86_64

--> Finished Dependency Resolution

Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libaio.so.1(LIBAIO_0.4)(64bit)

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: librados2 = 0.80.7-0.el7.centos

   Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.6-0.el7.centos

   Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.7-0.el7.centos

   Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.8-0.el7.centos

   Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.9-0.el7.centos

Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: python-requests

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: librbd1 = 0.80.7-0.el7.centos

   Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.6-0.el7.centos

   Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.7-0.el7.centos

   Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.8-0.el7.centos

   Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.9-0.el7.centos

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: python-ceph = 0.80.7-0.el7.centos

   Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)

   python-ceph = 0.80.6-0.el7.centos

   Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   python-ceph = 0.80.7-0.el7.centos

   Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)

   python-ceph = 0.80.8-0.el7.centos

   Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)

   python-ceph = 0.80.9-0.el7.centos

Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: python-requests

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: librados2 = 0.80.7-0.el7.centos

   Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.6-0.el7.centos

   Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.7-0.el7.centos

   Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.8-0.el7.centos

   Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

   librados2 = 0.80.9-0.el7.centos

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libaio.so.1(LIBAIO_0.1)(64bit)

Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libaio.so.1()(64bit)

Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Irek Fasikhov]

I use Centos 7.1. The problem is that in the basic package repository has
"ceph-common".

[root@ceph01p24 cluster]# yum --showduplicates list ceph-common
Loaded plugins: dellsysid, etckeeper, fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: centos-mirror.rbc.ru
 * epel: be.mirror.eurid.eu
 * extras: ftp.funet.fi
 * updates: centos-mirror.rbc.ru
Installed Packages
ceph-common.x86_64

 0.80.7-0.el7.centos
@Ceph
Available Packages
ceph-common.x86_64

 0.80.6-0.el7.centos
Ceph
ceph-common.x86_64

 0.80.7-0.el7.centos
Ceph
ceph-common.x86_64

 0.80.8-0.el7.centos
Ceph
ceph-common.x86_64

 0.80.9-0.el7.centos
Ceph
ceph-common.x86_64
 1:0.80.7-0.4.el7

   epel
ceph-common.x86_64
 1:0.80.7-2.el7

   base

I make the installation as follows:

rpm -ivh
http://ceph.com/rpm-firefly/el7/noarch/ceph-release-1-0.el7.noarch.rpm
yum install redhat-lsb-core-4.1-27.el7.centos.1.x86_64
gperftools-libs.x86_64 yum-plugin-priorities.noarch ntp -y
yum install librbd1-0.80.7-0.el7.centos
librados2-0.80.7-0.el7.centos.x86_64.rpm -y
yum install gdisk cryptsetup leveldb python-jinja2 hdparm -y

yum install --disablerepo=base --disablerepo=epel
ceph-common-0.80.7-0.el7.centos.x86_64 -y
yum install --disablerepo=base --disablerepo=epel ceph-0.80.7-0.el7.centos
-y

2015-04-08 12:40 GMT+03:00 Vickey Singh :

> Hello Everyone
>
>
> I also tried setting higher priority as suggested by SAM but no luck
>
>
> Please see the Full logs here http://paste.ubuntu.com/10771358/
>
>
> While installing yum searches for correct Ceph repository but it founds 3
> versions of python-ceph under http://ceph.com/rpm-giant/el7/x86_64/
>
>
> How can i instruct yum to install latest version of ceph from giant
> repository ?? FYI i have this setting already
>
>
> [root@rgw-node1 yum.repos.d]# cat /etc/yum/pluginconf.d/priorities.conf
>
> [main]
>
> enabled = 1
>
> check_obsoletes = 1
>
> [root@rgw-node1 yum.repos.d]#
>
>
>
>
> This issue can be easily reproduced, just now i tried on a fresh server
> centos 7.0.1406 but it still fails.
>
> Please help.
>
> Please help.
>
> Please help.
>
>
> # cat /etc/redhat-release
>
> CentOS Linux release 7.0.1406 (Core)
>
> #
>
> # uname -r
>
> 3.10.0-123.20.1.el7.x86_64
>
> #
>
>
> Regards
>
> VS
>
>
> On Wed, Apr 8, 2015 at 11:10 AM, Sam Wouters  wrote:
>
>>  Hi Vickey,
>>
>> we had a similar issue and we resolved it by giving the centos base and
>> update repo a higher priority (ex 10) then the epel repo.
>> The ceph-deploy tool only sets a prio of 1 for the ceph repo's, but the
>> centos and epel repo's stay on the default of 99.
>>
>> regards,
>> Sam
>>
>> On 08-04-15 09:32, Vickey Singh wrote:
>>
>>  Hi Ken
>>
>>
>>  As per your suggestion , i tried enabling epel-testing repository but
>> still no luck.
>>
>>
>>  Please check the below output. I would really appreciate  any help
>> here.
>>
>>
>>
>>  # yum install ceph --enablerepo=epel-testing
>>
>>
>>  ---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed
>>
>> --> Processing Dependency: librbd1 = 1:0.80.7 for package:
>> 1:python-rbd-0.80.7-0.5.el7.x86_64
>>
>> --> Finished Dependency Resolution
>>
>> Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)
>>
>>Requires: libcephfs1 = 1:0.80.7
>>
>>Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)
>>
>>libcephfs1 = 1:0.86-0.el7.centos
>>
>>Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)
>>
>>libcephfs1 = 1:0.87-0.el7.centos
>>
>>Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)
>>
>>libcephfs1 = 1:0.87.1-0.el7.centos
>>
>> *Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*
>>
>>Requires: librbd1 = 1:0.80.7
>>
>>Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)
>>
>>librbd1 = 0.80.9-0.el7.centos
>>
>>Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 1:0.87.1-0.el7.centos
>>
>>Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 1:0.86-0.el7.centos
>>
>>Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)
>>
>>librbd1 = 1:0.87-0.el7.centos
>>
>> *Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*
>>
>>Requires: librados2 = 1:0.80.7
>>
>>Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)
>>
>>librados2 = 0.80.9-0.el7.centos
>>
>>Updat

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Hello Everyone


I also tried setting higher priority as suggested by SAM but no luck


Please see the Full logs here http://paste.ubuntu.com/10771358/


While installing yum searches for correct Ceph repository but it founds 3
versions of python-ceph under http://ceph.com/rpm-giant/el7/x86_64/


How can i instruct yum to install latest version of ceph from giant
repository ?? FYI i have this setting already


[root@rgw-node1 yum.repos.d]# cat /etc/yum/pluginconf.d/priorities.conf

[main]

enabled = 1

check_obsoletes = 1

[root@rgw-node1 yum.repos.d]#




This issue can be easily reproduced, just now i tried on a fresh server
centos 7.0.1406 but it still fails.

Please help.

Please help.

Please help.


# cat /etc/redhat-release

CentOS Linux release 7.0.1406 (Core)

#

# uname -r

3.10.0-123.20.1.el7.x86_64

#


Regards

VS


On Wed, Apr 8, 2015 at 11:10 AM, Sam Wouters  wrote:

>  Hi Vickey,
>
> we had a similar issue and we resolved it by giving the centos base and
> update repo a higher priority (ex 10) then the epel repo.
> The ceph-deploy tool only sets a prio of 1 for the ceph repo's, but the
> centos and epel repo's stay on the default of 99.
>
> regards,
> Sam
>
> On 08-04-15 09:32, Vickey Singh wrote:
>
>  Hi Ken
>
>
>  As per your suggestion , i tried enabling epel-testing repository but
> still no luck.
>
>
>  Please check the below output. I would really appreciate  any help here.
>
>
>
>
>  # yum install ceph --enablerepo=epel-testing
>
>
>  ---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed
>
> --> Processing Dependency: librbd1 = 1:0.80.7 for package:
> 1:python-rbd-0.80.7-0.5.el7.x86_64
>
> --> Finished Dependency Resolution
>
> Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)
>
>Requires: libcephfs1 = 1:0.80.7
>
>Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.86-0.el7.centos
>
>Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.87-0.el7.centos
>
>Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.87.1-0.el7.centos
>
> *Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*
>
>Requires: librbd1 = 1:0.80.7
>
>Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)
>
>librbd1 = 0.80.9-0.el7.centos
>
>Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 1:0.87.1-0.el7.centos
>
>Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 1:0.86-0.el7.centos
>
>Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)
>
>librbd1 = 1:0.87-0.el7.centos
>
> *Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*
>
>Requires: librados2 = 1:0.80.7
>
>Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)
>
>librados2 = 0.80.9-0.el7.centos
>
>Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 1:0.87.1-0.el7.centos
>
>Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 1:0.86-0.el7.centos
>
>Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)
>
>librados2 = 1:0.87-0.el7.centos
>
>  You could try using --skip-broken to work around the problem
>
>  You could try running: rpm -Va --nofiles --nodigest
>
>
>
>
>
>  # yum install ceph --enablerepo=epel-testing --disablerepo=ceph*
>
>
>
>  ---> Package spax.x86_64 0:1.5.2-11.el7 will be installed
>
> ---> Package time.x86_64 0:1.7-45.el7 will be installed
>
> --> Running transaction check
>
> ---> Package cups-libs.x86_64 1:1.6.3-17.el7 will be installed
>
> ---> Package python-cephfs.x86_64 1:0.80.7-0.4.el7 will be installed
>
> --> Processing Dependency: libcephfs1 = 1:0.80.7 for package:
> 1:python-cephfs-0.80.7-0.4.el7.x86_64
>
> ---> Package python-rados.x86_64 1:0.80.7-0.5.el7 will be installed
>
> --> Processing Dependency: librados2 = 1:0.80.7 for package:
> 1:python-rados-0.80.7-0.5.el7.x86_64
>
> ---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed
>
> --> Processing Dependency: librbd1 = 1:0.80.7 for package:
> 1:python-rbd-0.80.7-0.5.el7.x86_64
>
> --> Finished Dependency Resolution
>
> Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)
>
>Requires: libcephfs1 = 1:0.80.7
>
>Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.86-0.el7.centos
>
>Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.87-0.el7.centos
>
>Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)
>
>libcephfs1 = 1:0.87.1-0.el7.centos
>
> *Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*
>
>Requires: librbd1 = 1:0.80.7
>
>Removing: librbd1-

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

Hi, 

Am I the only person noticing disappointing results from the preliminary RDMA 
testing, or am I reading the numbers wrong? 

Yes, it's true that on a very small cluster you do see a great improvement in 
rdma, but in real life rdma is used in large infrastructure projects, not on a 
few servers with a handful of osds. In fact, from what i've seen from the 
slides, the rdma implementation scales horribly to the point that it becomes 
slower the more osds you through at it. 

>From my limited knowledge, i have expected a much higher performance gains 
>with rdma, taking into account that you should have much lower latency and 
>overhead and lower cpu utilisation when using this transport in comparison 
>with tcp. 

Are we likely to see a great deal of improvement with ceph and rdma in a near 
future? Is there a roadmap for having a stable and reliable rdma protocol 
support? 

Thanks 

Andrei 
- Original Message -

> From: "Andrey Korolyov" 
> To: "Somnath Roy" 
> Cc: ceph-users@lists.ceph.com, "ceph-devel"
> 
> Sent: Wednesday, 8 April, 2015 9:28:12 AM
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy
>  wrote:
> >
> > Hi,
> > Please find the preliminary performance numbers of TCP Vs RDMA
> > (XIO) implementation (on top of SSDs) in the following link.
> >
> > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> >
> > The attachment didn't go through it seems, so, I had to use
> > slideshare.
> >
> > Mark,
> > If we have time, I can present it in tomorrow's performance
> > meeting.
> >
> > Thanks & Regards
> > Somnath
> >

> Those numbers are really impressive (for small numbers at least)!
> What
> are TCP settings you using?For example, difference can be lowered on
> scale due to less intensive per-connection acceleration on CUBIC on a
> larger number of nodes, though I do not believe that it was a main
> reason for an observed TCP catchup on a relatively flat workload such
> as fio generates.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Preliminary RDMA vs TCP numbers

[Andrey Korolyov]

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy  wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
> implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What
are TCP settings you using?For example, difference can be lowered on
scale due to less intensive per-connection acceleration on CUBIC on a
larger number of nodes, though I do not believe that it was a main
reason for an observed TCP catchup on a relatively flat workload such
as fio generates.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Preliminary RDMA vs TCP numbers

[Somnath Roy]

Hi,
Please find the preliminary performance numbers of TCP Vs RDMA (XIO) 
implementation (on top of SSDs) in the following link.

http://www.slideshare.net/somnathroy7568/ceph-on-rdma

The attachment didn't go through it seems, so, I had to use slideshare.

Mark,
If we have time, I can present it in tomorrow's performance meeting.

Thanks & Regards
Somnath



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Inconsistent "ceph-deploy disk list" command results

[f...@univ-lr.fr]

Hi all,

I want to alert on a command we've learned to avoid for its inconsistent 
results.


on Giant 0.87.1 and Hammer 0.93.0 (ceph-deploy-1.5.22-0.noarch was used 
in both cases) "ceph-deploy disk list" command has a problem.


We should get an exhaustive list of devices entries, like this one :
../..
/dev/sdk :
/dev/sdk1 ceph data, active, cluster ceph, osd.34, journal /dev/sda9
../..

But from the admin node,
when we count how many disks we have on our nodes , results are 
incorrect and differ each time :

$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
 8
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
12
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
10
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
15
$ ceph-deploy disk list osdnode1 2>&1|grep "active," |wc -l
12

From the nodes,
results are correct (15) and always the same :
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15
$ ceph-disk list 2>&1|grep "active," |wc -l
15

but a pretty similar 'ceph-deploy osd list' command works fine

Frederic
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Sam Wouters]

Hi Vickey,

we had a similar issue and we resolved it by giving the centos base and 
update repo a higher priority (ex 10) then the epel repo.
The ceph-deploy tool only sets a prio of 1 for the ceph repo's, but the 
centos and epel repo's stay on the default of 99.


regards,
Sam

On 08-04-15 09:32, Vickey Singh wrote:


Hi Ken


As per your suggestion , i tried enabling epel-testing repository but 
still no luck.



Please check the below output. I would really appreciate  any help here.



# yum install ceph --enablerepo=epel-testing


---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librbd1 = 1:0.80.7 for package: 
1:python-rbd-0.80.7-0.5.el7.x86_64


--> Finished Dependency Resolution

Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

  Requires: libcephfs1 = 1:0.80.7

  Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.86-0.el7.centos

  Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.87-0.el7.centos

  Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.87.1-0.el7.centos

*Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

  Requires: librbd1 = 1:0.80.7

  Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

  librbd1 = 0.80.9-0.el7.centos

  Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.87.1-0.el7.centos

  Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.86-0.el7.centos

  Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.87-0.el7.centos

*Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*

  Requires: librados2 = 1:0.80.7

  Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)

  librados2 = 0.80.9-0.el7.centos

  Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.87.1-0.el7.centos

  Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.86-0.el7.centos

  Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.87-0.el7.centos

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest





# yum install ceph --enablerepo=epel-testing --disablerepo=ceph*



---> Package spax.x86_64 0:1.5.2-11.el7 will be installed

---> Package time.x86_64 0:1.7-45.el7 will be installed

--> Running transaction check

---> Package cups-libs.x86_64 1:1.6.3-17.el7 will be installed

---> Package python-cephfs.x86_64 1:0.80.7-0.4.el7 will be installed

--> Processing Dependency: libcephfs1 = 1:0.80.7 for package: 
1:python-cephfs-0.80.7-0.4.el7.x86_64


---> Package python-rados.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librados2 = 1:0.80.7 for package: 
1:python-rados-0.80.7-0.5.el7.x86_64


---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librbd1 = 1:0.80.7 for package: 
1:python-rbd-0.80.7-0.5.el7.x86_64


--> Finished Dependency Resolution

Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

  Requires: libcephfs1 = 1:0.80.7

  Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.86-0.el7.centos

  Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.87-0.el7.centos

  Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

  libcephfs1 = 1:0.87.1-0.el7.centos

*Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

  Requires: librbd1 = 1:0.80.7

  Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

  librbd1 = 0.80.9-0.el7.centos

  Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.87.1-0.el7.centos

  Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.86-0.el7.centos

  Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)

  librbd1 = 1:0.87-0.el7.centos

*Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*

  Requires: librados2 = 1:0.80.7

  Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)

  librados2 = 0.80.9-0.el7.centos

  Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.87.1-0.el7.centos

  Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.86-0.el7.centos

  Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)

  librados2 = 1:0.87-0.el7.centos

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest

[root@rgw-node1 yum.repos.d]#




[root@rgw-node1 yum.repos.d]# yum install --enablerepo=epel-testing 
ceph-0.80.7-0.5.el7


Loaded plugins: fastestmirror, priorities

Loading mirror speeds from cached hostfile

 * base: ftp.funet.fi 

Re: [ceph-users] when recovering start

[Stéphane DUGRAVOT]

- Le 7 Avr 15, à 14:57, lijian  a écrit : 

> Haomai Wang,

> the mon_osd_down_out_interval is 300, please refer to my settings, and I use 
> the
> cli 'service ceph stop osd.X' to stop a osd
> the pg status change to remap,backfill and recovering ... immediately
> so other something wrong with my settings or operation?

Hi, 
>From what daemon you get the mon_osd_down_out_interval value ? 
Is it from mon or osd ? 
Stephane. 

> Thanks,

> Jian Ji

> At 2015-04-07 20:38:29, "Haomai Wang"  wrote:
> >Whatever the version you tested, ceph won't recover data when you
> >manually stop osd immediately. And it will trigger mark down osd out
> >when it reach "mon_osd_down_out_interval" seconds.

> >On Tue, Apr 7, 2015 at 8:33 PM, lijian  wrote:
> >> Hi,
> >> The recovering start delay 300s after I stop a osd and the osd status 
> >> change
> >> from in to out,  the test ENV is Ceph 0.80.7

> >> But I test in ceph 0.87.1, the recovering start immediately after I stop a
> >> OSD,all the settings is the default value,the following is mon_osd* 
> >> settings
> >> in my test ENV:
> >>   "mon_osd_laggy_halflife": "3600",
> >>   "mon_osd_laggy_weight": "0.3",
> >>   "mon_osd_adjust_heartbeat_grace": "true",
> >>   "mon_osd_adjust_down_out_interval": "true",
> >>   "mon_osd_auto_mark_in": "false",
> >>   "mon_osd_auto_mark_auto_out_in": "true",
> >>   "mon_osd_auto_mark_new_in": "true",
> >>   "mon_osd_down_out_interval": "300",
> >>   "mon_osd_down_out_subtree_limit": "rack",
> >>   "mon_osd_min_up_ratio": "0.3",
> >>   "mon_osd_min_in_ratio": "0.3",
> >>   "mon_osd_max_op_age": "32",
> >>   "mon_osd_max_split_count": "32",
> >>   "mon_osd_allow_primary_temp": "false",
> >>   "mon_osd_allow_primary_affinity": "false",
> >>   "mon_osd_full_ratio": "0.95",
> >>   "mon_osd_nearfull_ratio": "0.85",
> >>   "mon_osd_report_timeout": "45000",
> >>   "mon_osd_min_down_reporters": "50",
> >>   "mon_osd_min_down_reports": "150",
> >>   "mon_osd_force_trim_to": "0",

> >> so when the recovering start? why they are different with the two Ceph
> >> version, or someting wrong with my settings

> >> Thanks!
> >> Jian Li



> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




> >--
> >Best Regards,

> >Wheat

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Hi Ken


As per your suggestion , i tried enabling epel-testing repository but still
no luck.


Please check the below output. I would really appreciate  any help here.



# yum install ceph --enablerepo=epel-testing


---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librbd1 = 1:0.80.7 for package:
1:python-rbd-0.80.7-0.5.el7.x86_64

--> Finished Dependency Resolution

Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

   Requires: libcephfs1 = 1:0.80.7

   Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.86-0.el7.centos

   Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.87-0.el7.centos

   Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.87.1-0.el7.centos

*Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

   Requires: librbd1 = 1:0.80.7

   Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

   librbd1 = 0.80.9-0.el7.centos

   Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.87.1-0.el7.centos

   Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.86-0.el7.centos

   Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.87-0.el7.centos

*Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*

   Requires: librados2 = 1:0.80.7

   Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)

   librados2 = 0.80.9-0.el7.centos

   Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.87.1-0.el7.centos

   Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.86-0.el7.centos

   Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.87-0.el7.centos

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest





# yum install ceph --enablerepo=epel-testing --disablerepo=ceph*



---> Package spax.x86_64 0:1.5.2-11.el7 will be installed

---> Package time.x86_64 0:1.7-45.el7 will be installed

--> Running transaction check

---> Package cups-libs.x86_64 1:1.6.3-17.el7 will be installed

---> Package python-cephfs.x86_64 1:0.80.7-0.4.el7 will be installed

--> Processing Dependency: libcephfs1 = 1:0.80.7 for package:
1:python-cephfs-0.80.7-0.4.el7.x86_64

---> Package python-rados.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librados2 = 1:0.80.7 for package:
1:python-rados-0.80.7-0.5.el7.x86_64

---> Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

--> Processing Dependency: librbd1 = 1:0.80.7 for package:
1:python-rbd-0.80.7-0.5.el7.x86_64

--> Finished Dependency Resolution

Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

   Requires: libcephfs1 = 1:0.80.7

   Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.86-0.el7.centos

   Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.87-0.el7.centos

   Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

   libcephfs1 = 1:0.87.1-0.el7.centos

*Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

   Requires: librbd1 = 1:0.80.7

   Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

   librbd1 = 0.80.9-0.el7.centos

   Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.87.1-0.el7.centos

   Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.86-0.el7.centos

   Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)

   librbd1 = 1:0.87-0.el7.centos

*Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*

   Requires: librados2 = 1:0.80.7

   Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)

   librados2 = 0.80.9-0.el7.centos

   Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.87.1-0.el7.centos

   Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.86-0.el7.centos

   Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)

   librados2 = 1:0.87-0.el7.centos

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest

[root@rgw-node1 yum.repos.d]#




[root@rgw-node1 yum.repos.d]# yum install --enablerepo=epel-testing
ceph-0.80.7-0.5.el7

Loaded plugins: fastestmirror, priorities

Loading mirror speeds from cached hostfile

 * base: ftp.funet.fi

 * epel: fedora.uib.no

 * epel-debuginfo: fedora.uib.no

 * epel-source: fedora.uib.no

 * epe