[ceph-users] Regarding ceph osd setmaxosd

[Anand Bhat]

Hi,

I have question on intention of Ceph setmaxosd command. From source code, it 
appears as if this is present as a way to limit the number of OSDs in the Ceph 
cluster.  Can this be used to shrink the number of OSDs in the cluster without 
gracefully shutting down OSDs and letting recovery/remapping to happen?

Regards,
Anand



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] EU mirror now supports rsync

[David Moreau Simard]

(taking this back to ceph-users, not sure why I posted to ceph-devel?)

Thanks for the info, I sent them a message to inquire about access.
In the meantime, the mirror is already synchronized (sync every 4 hours) and 
available on http://mirror.iweb.ca or directly on http://ceph.mirror.iweb.ca.

David Moreau Simard

On Jul 17, 2014, at 5:21 AM, Wido den Hollander 
mailto:w...@42on.com>> wrote:

On 07/16/2014 09:48 PM, David Moreau Simard wrote:
Hi,

Thanks for making this available.
I am currently synchronizing off of it and will make it available on our 4 Gbps 
mirror on the Canadian east coast by the end of this week.


Cool! More mirrors is always better.

Are you able to share how you are synchronizing from the Ceph repositories ?
It would probably be better for us to synchronize from the source rather than 
the Europe mirror.


I have a SSH account into the ceph.com server and use that to 
sync the packages. I've set this up the the community guys Ross and Patrick, 
you might want to ping them.

There is no official distribution mechanism for this right now. I simply set up 
a rsyncd to provide this the community.

Wido

--
David Moreau Simard

On Apr 9, 2014, at 2:04 AM, Wido den Hollander 
mailto:w...@42on.com>> wrote:

Hi,

I just enabled rsync on the eu.ceph.com mirror.

eu.ceph.com mirrors from Ceph.com every 3 
hours.

Feel free to rsync all the contents to your local environment, might be useful
for some large deployments where you want to save external bandwidth by not
having each machine fetch the Deb/RPM packages from the internet.

Rsync is available over IPv4 and IPv6, simply sync with this command:
$ mkdir cephmirror
$ rsync -avr --stats --progress eu.ceph.com::ceph cephmirror

I ask you all to be gentle. It's a free service, so don't start hammering the
server by setting your Cron to sync every 5 minutes. Once every couple of hours
should be sufficient.

Also, please don't all start syncing at the first minute of the hour. When
setting up the Cron, select a random minute from the hour. This way the load on
the system can be spread out.

Should you have any questions or issues, let me know!

--
Wido den Hollander
42on B.V.
Ceph trainer and consultant
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Wido den Hollander
Ceph consultant and trainer
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Jake Young]

There are two command line tools for Linux for LSI cards: megacli and
storcli

You can do pretty much everything from those tools.

Jake

On Thursday, July 17, 2014, Dennis Kramer (DT)  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi,
>
> What do you recommend in case of a disk failure in this kind of
> configuration? Are you bringing down the host when you replace the
> disk and re-create the raid-0 for the replaced disk? I reckon that
> linux doesn't automatically get the disk replacement either...
>
> Dennis
>
> On 07/16/2014 11:02 PM, Shain Miley wrote:
> > Robert, We use those cards here in our Dell R-720 servers.
> >
> > We just ended up creating a bunch of single disk RAID-0 units,
> > since there was no jbod option available.
> >
> > Shain
> >
> >
> > On 07/16/2014 04:55 PM, Robert Fantini wrote:
> >> I've 2 dell systems with PERC H710 raid cards. Those are very
> >> good end cards , but do not support jbod .
> >>
> >> They support raid 0, 1, 5, 6, 10, 50, 60 .
> >>
> >> lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208
> >>  [Thunderbolt] (rev 05)
> >>
> >> The firmware Dell uses on the card does not support jbod.
> >>
> >> My question is how can this be best used for Ceph? Or should it
> >> not be used?
> >>
> >>
> >>
> >>
> >>
> >> ___ ceph-users
> >> mailing list ceph-users@lists.ceph.com 
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> > -- Shain Miley | Manager of Systems and Infrastructure, Digital
> > Media | smi...@npr.org  | 202.513.3649
> >
>
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.15 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlPHZ1MACgkQiJDTKUBxIRusogCeJ+jnADW/KBoQAxnDSz62yT3P
> FNoAnin3A52AqiA+KlFJQoc5bdQRoyYe
> =/MPE
> -END PGP SIGNATURE-
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pg repair info

[Wido den Hollander]

On 07/17/2014 09:44 PM, Caius Howcroft wrote:

I wonder if someone can just clarify something for me.

I have a cluster which I have upgraded to firefly. I'm having pg
inconsistencies due to the recent reported xfs bug. However, I'm
running pg repair X.YYY and I would like to just understand what,
exactly this is doing. It looks like its copying from the primary to
the other two (if size=3), but is it still doing this if the primary
is odd one out? i.e. what happens if the primary get corrupted? I
thought pg repair should fail in this case, but now I'm not so sure.



If the primary OSD is down CRUSH will select a different OSD as the primary.

Ceph doesn't know which object is corrupted. It simply knows that the 
secondary OSD does not have the same copy as the primary.


btrfs could help here since it has online checksumming, XFS doesn't.


Also is there a way to get the information about which objects and on
which osd are inconsistent, basically the stuff I see in the mon logs
but get it from a json dump  or from admin socket ? I would like to
track these errors better by feeding into our metrics collection.


$ ceph pg  query

That should tell you more.



Thanks
Caius





--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] pg repair info

[Caius Howcroft]

I wonder if someone can just clarify something for me.

I have a cluster which I have upgraded to firefly. I'm having pg
inconsistencies due to the recent reported xfs bug. However, I'm
running pg repair X.YYY and I would like to just understand what,
exactly this is doing. It looks like its copying from the primary to
the other two (if size=3), but is it still doing this if the primary
is odd one out? i.e. what happens if the primary get corrupted? I
thought pg repair should fail in this case, but now I'm not so sure.

Also is there a way to get the information about which objects and on
which osd are inconsistent, basically the stuff I see in the mon logs
but get it from a json dump  or from admin socket ? I would like to
track these errors better by feeding into our metrics collection.
Thanks
Caius


-- 
Caius Howcroft
@caiushowcroft
http://www.linkedin.com/in/caius
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to scale out RADOSgw?

[Wido den Hollander]

On 07/17/2014 02:27 PM, Riccardo Murri wrote:

Hi Wido, all,

thanks for the quick reply.  One more question:

On 16 July 2014 17:02, Wido den Hollander  wrote:

Op 16 jul. 2014 om 16:54 heeft "Riccardo Murri"  het 
volgende geschreven:

Since RADOSgw is a FastCGI module, can one scale it by just adding
more HTTP servers behind a load-balancer?


Yes, that's the way it works. Simply add more machines when you need to.


Then would it be better to co-locate RADOSgw servers with OSDs in the
same machine (say, we buy larger/fatter nodes), or would it be better
to have a cluster of separate "thin" RADOSgw servers?



You can run RGW on the same machines as the OSD, no problem. There is no 
"better" way, but you just have to be aware that both services are 
running on the same machine.


Most deployments I've done had separate RGW machines though.


Thanks,
Riccardo




--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed disks

[Dmitry Borodaenko]

The meeting is in 2 hours, so you still have a chance to particilate
or at least lurk :)

On Wed, Jul 16, 2014 at 11:55 PM, Somhegyi Benjamin
 wrote:
> Hi Dmitry,
>
> Will you please share with us how things went on the meeting?
>
> Many thanks,
> Benjamin
>
>
>
>> -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>> Dmitry Borodaenko
>> Sent: Wednesday, July 16, 2014 11:18 PM
>> To: ceph-users@lists.ceph.com
>> Cc: OpenStack Development Mailing List (not for usage questions)
>> Subject: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed
>> disks
>>
>> I've got a bit of good news and bad news about the state of landing the
>> rbd-ephemeral-clone patch series for Nova in Juno.
>>
>> The good news is that the first patch in the series
>> (https://review.openstack.org/91722 fixing a data loss inducing bug with
>> live migrations of instances with RBD backed ephemeral drives) was
>> merged yesterday.
>>
>> The bad news is that after 2 months of sitting in review queue and only
>> getting its first a +1 from a core reviewer on the spec approval freeze
>> day, the spec for the blueprint rbd-clone-image-handler
>> (https://review.openstack.org/91486) wasn't approved in time. Because of
>> that, today the blueprint was rejected along with the rest of the
>> commits in the series, even though the code itself was reviewed and
>> approved a number of times.
>>
>> Our last chance to avoid putting this work on hold for yet another
>> OpenStack release cycle is to petition for a spec freeze exception in
>> the next Nova team meeting:
>> https://wiki.openstack.org/wiki/Meetings/Nova
>>
>> If you're using Ceph RBD as backend for ephemeral disks in Nova and are
>> interested this patch series, please speak up. Since the biggest concern
>> raised about this spec so far has been lack of CI coverage, please let
>> us know if you're already using this patch series with Juno, Icehouse,
>> or Havana.
>>
>> I've put together an etherpad with a summary of where things are with
>> this patch series and how we got here:
>> https://etherpad.openstack.org/p/nova-ephemeral-rbd-clone-status
>>
>> Previous thread about this patch series on ceph-users ML:
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-
>> March/028097.html
>>
>> --
>> Dmitry Borodaenko
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Dmitry Borodaenko
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed disks

[Dmitry Borodaenko]

In case of Icehouse on Ubuntu 14.04, you should be able to test this
patch series by grabbing this branch from github:
https://github.com/angdraug/nova/tree/rbd-ephemeral-clone-stable-icehouse

and replacing contents of /usr/share/pyshared/nova with contents of
nova/ from that branch. You may also need to clean out related .pyc
files from /usr/lib/python2.7/dist-packages/nova/.


On Wed, Jul 16, 2014 at 11:22 PM, Dennis Kramer (DT)  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Dmitry,
>
> I've been using Ubuntu 14.04LTS + Icehouse /w CEPH as a storage
> backend for glance, cinder and nova (kvm/libvirt). I *really* would
> love to see this patch cycle in Juno. It's been a real performance
> issue because of the unnecessary re-copy from-and-to CEPH when using
> the default "boot from image"-option. It seems that the your fix would
> be the solution to all. IMHO this is one of the most important
> features when using CEPH RBD as a backend for Openstack Nova.
>
> Can you point me in the right direction in how to apply this patch of
> yours on a default Ubuntu14.04LTS + Icehouse installation? I'm using
> the default ubuntu packages since Icehouse lives in core and I'm not
> sure how to apply the patch series. I would love to test and review it.
>
> With regards,
>
> Dennis
>
> On 07/16/2014 11:18 PM, Dmitry Borodaenko wrote:
>> I've got a bit of good news and bad news about the state of
>> landing the rbd-ephemeral-clone patch series for Nova in Juno.
>>
>> The good news is that the first patch in the series
>> (https://review.openstack.org/91722 fixing a data loss inducing
>> bug with live migrations of instances with RBD backed ephemeral
>> drives) was merged yesterday.
>>
>> The bad news is that after 2 months of sitting in review queue and
>> only getting its first a +1 from a core reviewer on the spec
>> approval freeze day, the spec for the blueprint
>> rbd-clone-image-handler (https://review.openstack.org/91486) wasn't
>> approved in time. Because of that, today the blueprint was rejected
>> along with the rest of the commits in the series, even though the
>> code itself was reviewed and approved a number of times.
>>
>> Our last chance to avoid putting this work on hold for yet another
>> OpenStack release cycle is to petition for a spec freeze exception
>> in the next Nova team meeting:
>> https://wiki.openstack.org/wiki/Meetings/Nova
>>
>> If you're using Ceph RBD as backend for ephemeral disks in Nova
>> and are interested this patch series, please speak up. Since the
>> biggest concern raised about this spec so far has been lack of CI
>> coverage, please let us know if you're already using this patch
>> series with Juno, Icehouse, or Havana.
>>
>> I've put together an etherpad with a summary of where things are
>> with this patch series and how we got here:
>> https://etherpad.openstack.org/p/nova-ephemeral-rbd-clone-status
>>
>> Previous thread about this patch series on ceph-users ML:
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-March/028097.html
>>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.15 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlPHa6kACgkQiJDTKUBxIRtpOwCeNjTlYlyypOsaGeI/+HRxZ6nt
> Y2kAoNLckOlSaEfw+dwSBacXP3JGkcAj
> =0Ez1
> -END PGP SIGNATURE-



-- 
Dmitry Borodaenko
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Craig Lewis]

I'd like to see some way to cap recovery IOPS per OSD.  Don't allow
backfill to do no more than 50 operations per second.  It will slow
backfill down, but reserve plenty of IOPS for normal operation.  I know
that implementing this well is not a simple task.


I know I did some stupid things that caused a lot of my problems.  Most of
my problems can be traced back to
  osd mkfs options xfs = -l size=1024m -n size=64k -i size=2048 -s size=4096
 and the kernel malloc problems it caused.

Reformatting all of the disks fixed a lot of my issues, but it didn't fix
them all.




While I was reformatting my secondary cluster, I tested the stability by
reformatting all of the disks on the last node at once.  I didn't mark them
out and wait for the rebuild; I removed the OSDs, reformatted, and added
them back to the cluster.  It was 10 disks out of 36 total, in a 4 node
cluster (I'm waiting for hardware to free up to build the 5th node).
 Everything was fine for the first hour or so.  After several hours, there
was enough latency that the HTTP load balancer was marking RadosGW nodes
down.  My load balancer has a 30s timeout.  Since the latency was cluster
wide, all RadosGW nodes were marked down together.  When the latency spike
subsided, they'd all get marked up again.  This continued until the
backfill completed.  They were mostly up.  I don't have numbers, but I
think they were marked down about 5 times an hour, for less than a minute
each time.  That really messes with radosgw-agent.


I had recovery tuned down:
  osd max backfills = 1
  osd recovery max active = 1
  osd recovery op priority = 1

I have journals on SSD, and single GigE public and cluster networks.  This
cluster has 2x replication (I'm waiting for the 5th node to go to 3x).  The
cluster network was pushing 950 Mbps.  The SSDs and OSDs had plenty of
write bandwidth, but the HDDs were saturating their IOPs.  These are
consumer class 7200 RPM SATA disks, so they don't have very many IOPS.

The average write latency on these OSDs is normally ~10ms.  While this
backfill was going on, the average write latency was 100ms, with plenty of
times when the latency was 200ms.  The average read latency increased, but
not as bad.  It averaged 50ms, with occasional spikes up to 400ms.  Since I
formatted a 27% of my cluster, I was seeing higher latency on 55% of my
OSDs (readers and writers).

Instead, if I trickle in the disks, everything works fine.  I was able to
reformat 2 OSDs at a time without a problem.  The cluster latency increase
was barely noticeable, even though the IOPS on those two disks were
saturated.  A bit of latency here and there (5% of the time) doesn't hurt
much.  When it's 55% of the time, it hurts a lot more.


When I finally get the 5th node, and increase replication from 2x to 3x, I
expect this cluster to be unusable for about a week.







On Thu, Jul 17, 2014 at 9:02 AM, Andrei Mikhailovsky 
wrote:

> Comments inline
>
>
> --
> *From: *"Sage Weil" 
> *To: *"Quenten Grasso" 
> *Cc: *ceph-users@lists.ceph.com
> *Sent: *Thursday, 17 July, 2014 4:44:45 PM
>
> *Subject: *Re: [ceph-users] ceph osd crush tunables optimal AND add new
> OSD at the same time
>
> On Thu, 17 Jul 2014, Quenten Grasso wrote:
>
> > Hi Sage & List
> >
> > I understand this is probably a hard question to answer.
> >
> > I mentioned previously our cluster is co-located MON?s on OSD servers,
> which
> > are R515?s w/ 1 x AMD 6 Core processor & 11 3TB OSD?s w/ dual 10GBE.
> >
> > When our cluster is doing these busy operations and IO has stopped as in
> my
> > case, I mentioned earlier running/setting tuneable to optimal or heavy
> > recovery
> >
> > operations is there a way to ensure our IO doesn?t get completely
> > blocked/stopped/frozen in our vms?
> >
> > Could it be as simple as putting all 3 of our mon servers on baremetal
> >  w/ssd?s? (I recall reading somewhere that a mon disk was doing several
> > thousand IOPS during a recovery operation)
> >
> > I assume putting just one on baremetal won?t help because our mon?s will
> only
> > ever be as fast as our slowest mon server?
>
> I don't think this is related to where the mons are (most likely).  The
> big question for me is whether IO is getting completely blocked, or just
> slowed enough that the VMs are all timing out.
>
>
> AM: I was looking at the cluster status while the rebalancing was taking
> place and I was seeing very little client IO reported by ceph -s output.
> The numbers were around 20-100 whereas our typical IO for the cluster is
> around 1000. Having said that, this was not enough as _all_ of our vms
> become unresponsive and didn't recover after rebalancing finished.
>
>
> What slow request messages
> did you see during the rebalance?
>
> AM: As I was experimenting with different options while trying to gain
> some client IO back i've noticed that when I am limiting the options to 1
> per osd ( osd max backfills = 1, osd recovery max active = 1, osd
> recovery threads

Re: [ceph-users] Some OSD and MDS crash

[Pierre BLONDEAU]

Hi

0 Brilliant  I recovered my data.

1 Gregory, Joao, John, Samuel : Thank a lot for all the help and to have 
responded at each time.


2 It's my fault, if i am pass to 0.82. And it's good, if that helped you 
to find some bugs ;)


3 With this fear, we will recreate our cluster in firefly. I think we'll 
change our infrastructure mode to NFS over RDB. But the transfer time 
risks to be long and we will lost the advantage of the performances of 
CephFS for computation.


There is a better technique than rsync to transfer data between CephFS 
and RDB ?


For the moment, I have restart only one MDS. Can I restart other MDSs or 
it's dangerous ?


I always have two pages in inconsistence mode, how can i solved that ?

Regards,
Pierre

PS : the action :

# cephfs-journal-tool journal reset
old journal was 25356971019142~18446718717516473070
new journal start will be 780140544 (2199948 bytes past old end)
writing journal head
writing EResetJournal entry
done

Le 17/07/2014 15:38, John Spray a écrit :

Hi Pierre,

Unfortunately it looks like we had a bug in 0.82 that could lead to
journal corruption of the sort you're seeing here.  A new journal
format was added, and on the first start after an update the MDS would
re-write the journal to the new format.  This should only have been
happening on the single active MDS for a given rank, but it was
actually being done by standby-replay MDS daemons too.  As a result,
if there were standby-replay daemons configured, they could try to
rewrite the journal at the same time, resulting in a corrupt journal.

In your case, I think the probability of the condition occurring was
increased by the OSD issues you were having, because at some earlier
stage the rewrite process had been stopped partway through.  Without
standby MDSs this would be recovered from cleanly, but with the
standbys in play the danger of corruption is high while the journal is
in the partly-rewritten state.

The ticket is here: http://tracker.ceph.com/issues/8811
The candidate fix is here: https://github.com/ceph/ceph/pull/2115

If you have recent backups then I would suggest recreating the
filesystem and restoring from backups.  You can also try using the
"cephfs-journal-tool journal reset" command, which will wipe out the
journal entirely, losing the most recent writes to the filesystem and
potentially leaving some stray objects in the data pool.

Sorry that this has bitten you, even though 0.82 was not a named
release this was a pretty nasty bug to let out there, and I'm going to
improve our automated tests in this area.

Regards,
John


On Wed, Jul 16, 2014 at 11:57 PM, Pierre BLONDEAU
 wrote:

Le 16/07/2014 22:40, Gregory Farnum a écrit :


On Wed, Jul 16, 2014 at 6:21 AM, Pierre BLONDEAU
 wrote:


Hi,

After the repair process, i have :
1926 active+clean
 2 active+clean+inconsistent

This two PGs seem to be on the same osd ( #34 ):
# ceph pg dump | grep inconsistent
dumped all in format plain
0.2e4   0   0   0   8388660 4   4
active+clean+inconsistent   2014-07-16 11:39:43.819631  9463'4
438411:133968   [34,4]  34  [34,4]  34  9463'4  2014-07-16
04:52:54.417333  9463'4  2014-07-11 09:29:22.041717
0.1ed   5   0   0   0   8388623 10  10
active+clean+inconsistent   2014-07-16 11:39:45.820142  9712'10
438411:144792   [34,2]  34  [34,2]  34  9712'10 2014-07-16
09:12:44.742488  9712'10 2014-07-10 21:57:11.345241

It's can explain why my MDS won't to start ? If i remove ( or shutdown )
this OSD, it's can solved my problem ?



You want to figure out why they're inconsistent (if they're still
going inconsistent, or maybe just need to be repaired), but this
shouldn't be causing your MDS troubles.
Can you dump the MDS journal and put it somewhere accessible? (You can
use ceph-post-file to upload it.) John has been trying to reproduce
this crash but hasn't succeeded yet.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com



Hi,

I try to do :
cephfs-journal-tool journal export ceph-journal.bin 2>
cephfs-journal-tool.log

But the program crash. I upload log file :
e069c6ac-3cb4-4a52-8950-da7c600e2b01

There is a mistake in
http://ceph.com/docs/master/cephfs/cephfs-journal-tool/ in "Example: journal
inspect". The good syntax seems to be :
# cephfs-journal-tool  journal inspect
2014-07-17 00:54:14.155382 7ff89d239780 -1 Header is invalid (inconsistent
offsets)
Overall journal integrity: DAMAGED
Header could not be decoded

Regards


--
--
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique

tel : 02 31 56 75 42
bureau  : Campus 2, Science 3, 406
--




--
--
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique

tel : 02 31 56 75 42
bureau  : Ca

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Andrei Mikhailovsky]

Comments inline 


- Original Message -

From: "Sage Weil"  
To: "Quenten Grasso"  
Cc: ceph-users@lists.ceph.com 
Sent: Thursday, 17 July, 2014 4:44:45 PM 
Subject: Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at 
the same time 

On Thu, 17 Jul 2014, Quenten Grasso wrote: 

> Hi Sage & List 
> 
> I understand this is probably a hard question to answer. 
> 
> I mentioned previously our cluster is co-located MON?s on OSD servers, which 
> are R515?s w/ 1 x AMD 6 Core processor & 11 3TB OSD?s w/ dual 10GBE. 
> 
> When our cluster is doing these busy operations and IO has stopped as in my 
> case, I mentioned earlier running/setting tuneable to optimal or heavy 
> recovery 
> 
> operations is there a way to ensure our IO doesn?t get completely 
> blocked/stopped/frozen in our vms? 
> 
> Could it be as simple as putting all 3 of our mon servers on baremetal 
> w/ssd?s? (I recall reading somewhere that a mon disk was doing several 
> thousand IOPS during a recovery operation) 
> 
> I assume putting just one on baremetal won?t help because our mon?s will only 
> ever be as fast as our slowest mon server? 

I don't think this is related to where the mons are (most likely). The 
big question for me is whether IO is getting completely blocked, or just 
slowed enough that the VMs are all timing out. 


AM: I was looking at the cluster status while the rebalancing was taking place 
and I was seeing very little client IO reported by ceph -s output. The numbers 
were around 20-100 whereas our typical IO for the cluster is around 1000. 
Having said that, this was not enough as _all_ of our vms become unresponsive 
and didn't recover after rebalancing finished. 


What slow request messages 
did you see during the rebalance? 

AM: As I was experimenting with different options while trying to gain some 
client IO back i've noticed that when I am limiting the options to 1 per osd ( 
osd max backfills = 1, osd recovery max active = 1, osd recovery threads = 1), 
I did not have any slow or blocked requests at all. Increasing these values did 
produce some blocked requests occasionally, but they were being quickly 
cleared. 


What were the op latencies? 

AM: In general, the latencies were around 5-10 higher compared to the normal 
cluster ops. The second column of the "ceph osd perf" was around 50s where as 
it is typically between 3-10. It did occasionally jump to some crazy numbers 
like 2000-3000 on several osds, but only for 5-10 seconds. 

It's 
possible there is a bug here, but it's also possible the cluster is just 
operating close enough to capacity that the additional rebalancing work 
pushes it into a place where it can't keep up and the IO latencies are 
too high. 


AM: My cluster in particular is under-utilised for the majority of time. I do 
not typically see osds more than 20-30% utilised and our ssd journals are 
usually less than 10% utilised. 


Or that we just have more work to do prioritizing requests.. 
but it's hard to say without more info. 

sage 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Sage Weil]

On Thu, 17 Jul 2014, Quenten Grasso wrote:

> Hi Sage & List
> 
> I understand this is probably a hard question to answer.
> 
> I mentioned previously our cluster is co-located MON?s on OSD servers, which
> are R515?s w/ 1 x AMD 6 Core processor & 11 3TB OSD?s w/ dual 10GBE.
> 
> When our cluster is doing these busy operations and IO has stopped as in my
> case, I mentioned earlier running/setting tuneable to optimal or heavy
> recovery
> 
> operations is there a way to ensure our IO doesn?t get completely
> blocked/stopped/frozen in our vms?
> 
> Could it be as simple as putting all 3 of our mon servers on baremetal
>  w/ssd?s? (I recall reading somewhere that a mon disk was doing several
> thousand IOPS during a recovery operation)
> 
> I assume putting just one on baremetal won?t help because our mon?s will only
> ever be as fast as our slowest mon server?

I don't think this is related to where the mons are (most likely).  The 
big question for me is whether IO is getting completely blocked, or just 
slowed enough that the VMs are all timing out.  What slow request messages 
did you see during the rebalance?  What were the op latencies?  It's 
possible there is a bug here, but it's also possible the cluster is just 
operating close enough to capacity that the additional rebalancing work 
pushes it into a place where it can't keep up and the IO latencies are 
too high.  Or that we just have more work to do prioritizing requests.. 
but it's hard to say without more info.

sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-fuse couldn't be connect.

[Jaemyoun Lee]

Thank you, Greg!

I solved it through creating MDS.

- Jae


On Wed, Jul 16, 2014 at 8:36 PM, Gregory Farnum  wrote:

> Your MDS isn't running or isn't active.
> -Greg
>
>
> On Wednesday, July 16, 2014, Jaemyoun Lee  wrote:
>
>>
>> The result is same.
>>
>> # ceph-fuse --debug-ms 1 --debug-client 10 -m 192.168.122.106:6789 /mnt
>> ceph-fuse[3296] :  starting ceph client
>>
>> And the log file is
>>
>> # cat /var/log/ceph/ceph-client.admin.log
>> 2014-07-16 17:08:13.146032 7f9a212f87c0  0 ceph version 0.80.1
>> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-fuse, pid 3294
>> 2014-07-16 17:08:13.156429 7f9a212f87c0  1 -- :/0 messenger.start
>> 2014-07-16 17:08:13.157537 7f9a212f87c0  1 -- :/3296 -->
>> 192.168.122.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
>> 0x7f9a23c0e6c0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.158198 7f9a212f6700  1 -- 192.168.122.166:0/3296
>> learned my addr 192.168.122.166:0/3296
>> 2014-07-16 17:08:13.158505 7f9a167fc700 10 client.-1 ms_handle_connect on
>> 192.168.122.106:6789/0
>> 2014-07-16 17:08:13.159083 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 1  mon_map v1  193+0+0 (4132823754
>> 0 0) 0x7f9a0ab0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.159182 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 2  auth_reply(proto 2 0 (0) Success) v1
>>  33+0+0 (1915318666 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.159375 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
>> 192.168.122.106:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
>> 0x7f9a0c0013a0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.159845 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 3  auth_reply(proto 2 0 (0) Success) v1
>>  206+0+0 (2967970554 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.159976 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
>> 192.168.122.106:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0
>> 0x7f9a0c001ec0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.160810 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 4  auth_reply(proto 2 0 (0) Success) v1
>>  409+0+0 (3799435439 0 0) 0x7f9a11d0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.160945 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
>> 192.168.122.106:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0
>> 0x7f9a23c102c0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.160979 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
>> 192.168.122.106:6789/0 -- mon_subscribe({mdsmap=0+,osdmap=0}) v2 -- ?+0
>> 0x7f9a23c10630 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.161033 7f9a212f87c0  2 client.4705 mounted: have
>> osdmap 0 and mdsmap 0
>> 2014-07-16 17:08:13.161056 7f9a212f87c0 10 client.4705 did not get mds
>> through better means, so chose random mds -1
>> 2014-07-16 17:08:13.161059 7f9a212f87c0 10 client.4705  target mds.-1 not
>> active, waiting for new mdsmap
>> 2014-07-16 17:08:13.161668 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 5  osd_map(45..45 src has 1..45) v3
>>  3907+0+0 (2386867192 0 0) 0x7f9a2060 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.161843 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 6  mdsmap(e 1) v1  396+0+0
>> (394292161 0 0) 0x7f9a2500 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.161861 7f9a167fc700  1 client.4705 handle_mds_map
>> epoch 1
>> 2014-07-16 17:08:13.161884 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 7  osd_map(45..45 src has 1..45) v3
>>  3907+0+0 (2386867192 0 0) 0x7f9a37a0 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.161900 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 8  mon_subscribe_ack(300s) v1 
>> 20+0+0 (4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30
>> 2014-07-16 17:08:13.161932 7f9a212f87c0 10 client.4705 did not get mds
>> through better means, so chose random mds -1
>> 2014-07-16 17:08:13.161942 7f9a212f87c0 10 client.4705  target mds.-1 not
>> active, waiting for new mdsmap
>> 2014-07-16 17:08:14.161453 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:08:34.166977 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:08:54.171234 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:09:14.174106 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:09:34.177062 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:09:54.179365 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:10:14.181731 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:10:34.184270 7f9a177fe700 10 client.4705 renew_caps()
>> 2014-07-16 17:10:46.161158 7f9a15ffb700  1 -- 192.168.122.166:0/3296 -->
>> 192.168.122.106:6789/0 -- mon_subscribe({mdsmap=2+,monmap=2+}) v2 -- ?+0
>> 0x7f99f8002c50 con 0x7f9a23c0dd30
>> 2014-07-16 17:10:46.161770 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
>> mon.0 192.168.122.106:6789/0 9  mon_subscribe_ack(300

[ceph-users] Warning for Ceph FS users (bug in 0.82)

[John Spray]

Hi,

If you are using the experimental filesystem component of Ceph, and
you use the less stable "numbered" Ceph releases, you should be aware
of the following issue affecting the 0.82 development release:
http://tracker.ceph.com/issues/8811

This issue introduces a risk of corruption when first starting MDS
daemons after installing the 0.82 packages.  A filesystem may be
affected if:
 * You are upgrading an existing filesystem from an earlier version of
Ceph (not if you are creating a new filesystem)
 * You have one or more MDS daemons configured with "mds standby replay = true"

There are several ways to avoid this issue:
 1 Ensure that the standby-replay MDS daemons are stopped while
starting a single active MDS for the first time after upgrade.
 2 Set "mds journal format = 0" in your configuration file before updating
 3 Do not install 0.82, wait for the next development release

Option 3 will be the best, unless you have a compelling need to
install the latest development release.

Don't forget: Ceph FS is currently not recommended for production data.

Thanks,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] row geo-replication to another data store?

[Guang Yang]

Hi cephers,
We are investigating a backup solution for Ceph, in short, we would like a 
solution to backup a Ceph cluster to another data store (not Ceph cluster, 
assume it has SWIFT API). We would like to have both full backup and 
incremental backup on top of the full backup.

After going through the geo-replication blueprint [1], I am thinking that we 
can leverage the effort and instead of replicate the data into another ceph 
cluster, we make it replicate to another data store. At the same time, I have a 
couple of questions which need your help:

1) How does the ragosgw-agent scale to multiple hosts? Our first investigation 
shows it only works on a single host but I would like to confirm.
2) Can we configure the interval  to do incremental backup like 1 hour / 1 day 
/ 1 month?

[1] 
https://wiki.ceph.com/Planning/Blueprints/Dumpling/RGW_Geo-Replication_and_Disaster_Recovery

Thanks,
Guang
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Some OSD and MDS crash

[John Spray]

Hi Pierre,

Unfortunately it looks like we had a bug in 0.82 that could lead to
journal corruption of the sort you're seeing here.  A new journal
format was added, and on the first start after an update the MDS would
re-write the journal to the new format.  This should only have been
happening on the single active MDS for a given rank, but it was
actually being done by standby-replay MDS daemons too.  As a result,
if there were standby-replay daemons configured, they could try to
rewrite the journal at the same time, resulting in a corrupt journal.

In your case, I think the probability of the condition occurring was
increased by the OSD issues you were having, because at some earlier
stage the rewrite process had been stopped partway through.  Without
standby MDSs this would be recovered from cleanly, but with the
standbys in play the danger of corruption is high while the journal is
in the partly-rewritten state.

The ticket is here: http://tracker.ceph.com/issues/8811
The candidate fix is here: https://github.com/ceph/ceph/pull/2115

If you have recent backups then I would suggest recreating the
filesystem and restoring from backups.  You can also try using the
"cephfs-journal-tool journal reset" command, which will wipe out the
journal entirely, losing the most recent writes to the filesystem and
potentially leaving some stray objects in the data pool.

Sorry that this has bitten you, even though 0.82 was not a named
release this was a pretty nasty bug to let out there, and I'm going to
improve our automated tests in this area.

Regards,
John


On Wed, Jul 16, 2014 at 11:57 PM, Pierre BLONDEAU
 wrote:
> Le 16/07/2014 22:40, Gregory Farnum a écrit :
>
>> On Wed, Jul 16, 2014 at 6:21 AM, Pierre BLONDEAU
>>  wrote:
>>>
>>> Hi,
>>>
>>> After the repair process, i have :
>>> 1926 active+clean
>>> 2 active+clean+inconsistent
>>>
>>> This two PGs seem to be on the same osd ( #34 ):
>>> # ceph pg dump | grep inconsistent
>>> dumped all in format plain
>>> 0.2e4   0   0   0   8388660 4   4
>>> active+clean+inconsistent   2014-07-16 11:39:43.819631  9463'4
>>> 438411:133968   [34,4]  34  [34,4]  34  9463'4  2014-07-16
>>> 04:52:54.417333  9463'4  2014-07-11 09:29:22.041717
>>> 0.1ed   5   0   0   0   8388623 10  10
>>> active+clean+inconsistent   2014-07-16 11:39:45.820142  9712'10
>>> 438411:144792   [34,2]  34  [34,2]  34  9712'10 2014-07-16
>>> 09:12:44.742488  9712'10 2014-07-10 21:57:11.345241
>>>
>>> It's can explain why my MDS won't to start ? If i remove ( or shutdown )
>>> this OSD, it's can solved my problem ?
>>
>>
>> You want to figure out why they're inconsistent (if they're still
>> going inconsistent, or maybe just need to be repaired), but this
>> shouldn't be causing your MDS troubles.
>> Can you dump the MDS journal and put it somewhere accessible? (You can
>> use ceph-post-file to upload it.) John has been trying to reproduce
>> this crash but hasn't succeeded yet.
>> -Greg
>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
> Hi,
>
> I try to do :
> cephfs-journal-tool journal export ceph-journal.bin 2>
> cephfs-journal-tool.log
>
> But the program crash. I upload log file :
> e069c6ac-3cb4-4a52-8950-da7c600e2b01
>
> There is a mistake in
> http://ceph.com/docs/master/cephfs/cephfs-journal-tool/ in "Example: journal
> inspect". The good syntax seems to be :
> # cephfs-journal-tool  journal inspect
> 2014-07-17 00:54:14.155382 7ff89d239780 -1 Header is invalid (inconsistent
> offsets)
> Overall journal integrity: DAMAGED
> Header could not be decoded
>
> Regards
>
>
> --
> --
> Pierre BLONDEAU
> Administrateur Systèmes & réseaux
> Université de Caen
> Laboratoire GREYC, Département d'informatique
>
> tel : 02 31 56 75 42
> bureau  : Campus 2, Science 3, 406
> --
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to scale out RADOSgw?

[Riccardo Murri]

Hi Wido, all,

thanks for the quick reply.  One more question:

On 16 July 2014 17:02, Wido den Hollander  wrote:
>> Op 16 jul. 2014 om 16:54 heeft "Riccardo Murri"  het 
>> volgende geschreven:
>>
>> Since RADOSgw is a FastCGI module, can one scale it by just adding
>> more HTTP servers behind a load-balancer?
>
> Yes, that's the way it works. Simply add more machines when you need to.

Then would it be better to co-locate RADOSgw servers with OSDs in the
same machine (say, we buy larger/fatter nodes), or would it be better
to have a cluster of separate "thin" RADOSgw servers?

Thanks,
Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Andrei Mikhailovsky]

Sage, 

would it help if you add a cache pool to your cluster? Let's say if you add a 
few TBs of ssds acting as a cache pool to your cluster, would this help with 
retaining IO to the guest vms during data recovery or reshuffling? 

Over the past year and a half that we've been using ceph we had a positive 
experience for the majority of time. The only downtime we had for our vms was 
when ceph is doing recovery. It seems that regardless of the tuning options 
we've used, our vms are still unable to get IO, they get to 98-99% iowait and 
freeze. This has happened on dumpling, emperor and now firefly releases. 
Because of this I've set noout flag on my cluster and have to keep an eye on 
the osds for manual intervention, which is far from ideal case (((. 

Andrei 

-- 
Andrei Mikhailovsky 
Director 
Arhont Information Security 

Web: http://www.arhont.com 
http://www.wi-foo.com 
Tel: +44 (0)870 4431337 
Fax: +44 (0)208 429 3111 
PGP: Key ID - 0x2B3438DE 
PGP: Server - keyserver.pgp.com 

DISCLAIMER 

The information contained in this email is intended only for the use of the 
person(s) to whom it is addressed and may be confidential or contain legally 
privileged information. If you are not the intended recipient you are hereby 
notified that any perusal, use, distribution, copying or disclosure is strictly 
prohibited. If you have received this email in error please immediately advise 
us by return email at and...@arhont.com and delete and purge the email and any 
attachments without making a copy. 


- Original Message -

From: "Sage Weil"  
To: "Gregory Farnum"  
Cc: ceph-users@lists.ceph.com 
Sent: Thursday, 17 July, 2014 1:06:52 AM 
Subject: Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at 
the same time 

On Wed, 16 Jul 2014, Gregory Farnum wrote: 
> On Wed, Jul 16, 2014 at 4:45 PM, Craig Lewis  
> wrote: 
> > One of the things I've learned is that many small changes to the cluster 
> > are 
> > better than one large change. Adding 20% more OSDs? Don't add them all at 
> > once, trickle them in over time. Increasing pg_num & pgp_num from 128 to 
> > 1024? Go in steps, not one leap. 
> > 
> > I try to avoid operations that will touch more than 20% of the disks 
> > simultaneously. When I had journals on HDD, I tried to avoid going over 10% 
> > of the disks. 
> > 
> > 
> > Is there a way to execute `ceph osd crush tunables optimal` in a way that 
> > takes smaller steps? 
> 
> Unfortunately not; the crush tunables are changes to the core 
> placement algorithms at work. 

Well, there is one way, but it is only somewhat effective. If you 
decompile the crush maps for bobtail vs firefly the actual difference is 

tunable chooseleaf_vary_r 1 

and this is written such that a value of 1 is the optimal 'new' way, 0 is 
the legacy old way, but values > 1 are less-painful steps between the two 
(though mostly closer to the firefly value of 1). So, you could set 

tunable chooseleaf_vary_r 4 

wait for it to settle, and then do 

tunable chooseleaf_vary_r 3 

...and so forth down to 1. I did some limited testing of the data 
movement involved and noted it here: 

https://github.com/ceph/ceph/commit/37f840b499da1d39f74bfb057cf2b92ef4e84dc6 

In my test case, going from 0 to 4 was about 1/10th as bad as going 
straight from 0 to 1, but the final step from 2 to 1 is still about 1/2 as 
bad. I'm not sure if that means it's not worth the trouble of not just 
jumping straight to the firefly tunables, or whether it means legacy users 
should just set (and leave) this at 2 or 3 or 4 and get almost all the 
benefit without the rebalance pain. 

sage 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] problem in ceph installation

[pragya jain]

Hi all,

I am installing ceph on ubuntu 14.04 desktop 64-bit VM using the link 
http://eu.ceph.com/docs/wip-6919/start/quick-start/

But I got following error while installing ceph

-
root@prag2648-VirtualBox:~# sudo apt-get update && sudo apt-get install ceph
Ign http://security.ubuntu.com trusty-security InRelease  
Ign http://in.archive.ubuntu.com trusty InRelease 
Hit http://security.ubuntu.com trusty-security Release.gpg    
Ign http://in.archive.ubuntu.com trusty-updates InRelease 
Hit http://security.ubuntu.com trusty-security Release    
Ign http://in.archive.ubuntu.com trusty-backports InRelease   
Hit http://in.archive.ubuntu.com trusty Release.gpg   
Hit http://security.ubuntu.com trusty-security/main Sources   
Hit http://in.archive.ubuntu.com trusty-updates Release.gpg   
Hit http://security.ubuntu.com trusty-security/restricted Sources
Ign http://extras.ubuntu.com trusty InRelease 
Hit http://in.archive.ubuntu.com trusty-backports Release.gpg 
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://extras.ubuntu.com trusty Release.gpg   
Hit http://in.archive.ubuntu.com trusty Release   
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit http://extras.ubuntu.com trusty Release   
Hit http://in.archive.ubuntu.com trusty-updates Release   
Hit http://ceph.com trusty InRelease  
Hit http://extras.ubuntu.com trusty/main Sources  
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://in.archive.ubuntu.com trusty-backports Release 
Hit http://extras.ubuntu.com trusty/main amd64 Packages   
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://ceph.com trusty/main amd64 Packages    
Hit http://extras.ubuntu.com trusty/main i386 Packages    
Hit http://in.archive.ubuntu.com trusty/main Sources  
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://ceph.com trusty/main i386 Packages 
Hit http://in.archive.ubuntu.com trusty/restricted Sources    
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://in.archive.ubuntu.com trusty/universe Sources  
Hit http://security.ubuntu.com trusty-security/main i386 Packages
Hit http://in.archive.ubuntu.com trusty/multiverse Sources    
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages
Hit http://in.archive.ubuntu.com trusty/main amd64 Packages   
Hit http://security.ubuntu.com trusty-security/universe i386 Packages
Hit http://in.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en_IN    
Hit http://in.archive.ubuntu.com trusty/universe amd64 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en   
Hit http://in.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://in.archive.ubuntu.com trusty/main i386 Packages    
Hit http://in.archive.ubuntu.com trusty/restricted i386 Packages
Hit http://in.archive.ubuntu.com trusty/universe i386 Packages
Hit http://in.archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://in.archive.ubuntu.com trusty/main Translation-en   
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://in.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://in.archive.ubuntu.com trusty/restricted Translation-en
Hit http://in.archive.ubuntu.com trusty/universe Translation-en
Hit http://in.archive.ubuntu.com trusty-updates/main Sources  
Hit http://in.archive.ubuntu.com trusty-updates/restricted Sources
Ign http://ceph.com trusty/main Translation-en_IN 
Hit http://in.archive.ubuntu.com trusty-updates/universe Sources
Hit http://in.archive.ubuntu.com trusty-updates/multiverse Sources
Ign http://ceph.com trusty/main Translation-en    
Get:1 http://in.archive.ubuntu.com trusty-updates/main amd64 Packages [218 kB]
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Ign http://security.ubuntu.com trusty-security/main Translation-en_IN
Ign http://security.ubuntu.com trusty-security/multiverse Translation-en_IN
Ign http://security.ubuntu.com trusty-security/restricted Translation-en_IN
Ign http://security.ubuntu.com trusty-security/universe Translation-en_IN
Get:2 http://in.archive.ubuntu.com trusty-updates/restricted amd64 Packages [14 
B]
Get:3 http://in.archive.ubuntu.com trusty-updates/universe amd64 Packages [155 
kB]
Get:4 http://in.archive.ubuntu.com trusty-updates/multiverse amd64 Packages 
[7,392 B]
Get:5 http://in.archive.ubuntu.com trusty-updates/main i386 Packages [218 kB]
Get:6 http://in.archive.ubuntu.com trust