[ceph-users] Luminous 12.2.4: CephFS kernel client (4.15/4.16) shows up as jewel

[Linh Vu]

Hi all,


On my test Luminous 12.2.4 cluster, with this set (initially so I could use 
upmap in the mgr balancer module):


# ceph osd set-require-min-compat-client luminous

# ceph osd dump | grep client
require_min_compat_client luminous
min_compat_client jewel


Not quite sure why min_compat_client is still jewel.


I have created cephfs on the cluster, and use a mix of fuse and kernel clients 
to test it. The fuse clients are on ceph-fuse 12.2.5 and show up as luminous 
clients.


The kernel client (just one mount) either on kernel 4.15.13 or 4.16.13 (the 
latest, just out) is showing up as jewel, seen in `ceph features`:


"client": {
"group": {
"features": "0x7010fb86aa42ada",
"release": "jewel",
"num": 1
},
"group": {
"features": "0x1ffddff8eea4fffb",
"release": "luminous",
"num": 8
}
}

I thought I read somewhere here that kernel 4.13+ should have full support for 
Luminous, so I don't know why this is showing up as jewel. I'm also surprised 
that it could mount and write to my cephfs share just fine despite that. It 
also doesn't seem to matter when I run ceph balancer with upmap mode despite 
this client being connected and writing files.


I can't see anything in mount.ceph options to specify jewel vs luminous either.


Is this just a mislabel i.e my kernel client is actually fully Luminous 
supported but showing up as Jewel? Or is the kernel client a bit behind still?


Currently we have a mix of ceph-fuse 12.2.5 and kernel client 4.15.13 in our 
production cluster, and I'm looking to set `ceph osd 
set-require-min-compat-client luminous` so I can use ceph balancer with upmap 
mode.


Cheers,

Linh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Hi Guys,

The striping seems to be slightly better than non-striping write, given
that my storage is configured with 4OSS, and 48OSD. each OSD is 8+2 Raid 6
of 24TB capacity.
But still the performance is around 100MB/sec. On a single haswell core,
I'm able to get 1GB/sec with dd buffered IO.

My questions are:
1. Does librados use direct IO?
2. Is there option to leverage page cache?
3. What is the strategy to get the peak bandwidth on a rados object store?
4. Can I use MPI for parallel IO?

Best,
Jialin
NERSC

On Wed, May 30, 2018 at 11:08 AM, Jialin Liu  wrote:

> Thanks Kefu.
>
> Best,
> Jialin
> NERSC
>
> On Tue, May 29, 2018 at 11:52 PM, kefu chai  wrote:
>
>> On Wed, May 30, 2018 at 11:53 AM, Jialin Liu  wrote:
>> > Hi Brad,
>> >
>> > You are correct. the librados.so has the symbol but what I copied is a
>> wrong
>> > file.
>> > Now I can test the striper api with the previous C example and this cpp
>> > example:
>> > http://mrbojangles3.github.io/ceph/systems/striping/alignmen
>> t/2017/05/28/Ceph-Stripe/
>> > Both are working, but I haven't got any performance number yet.
>> > ps. I also found that the stripe unit must be set before object_size
>> > properly.
>> >
>> > Thanks much.
>> >
>> > Best,
>> > Jialin
>> > NERSC
>> >
>> >
>> > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard 
>> wrote:
>> >>
>> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
>> >> > Thanks Brad,
>> >> > I run nm on those .so file, it prints 'no symbol'
>> >>
>> >> OK, well you need to link to a library that exports that symbol (has
>> >> it defined in its Text section). I suspect you'll find it is defined
>> >> in libceph-common.so so try linking to that explicitly.
>>
>> please note, libceph-common does not exist in jewel (i.e. v10.2.10)
>> yet. it was introduce in luminous. and libceph-common is not supposed
>> to be linked against by rados application directly. it is an internal
>> library and is linked by librados, libradosstriper and other rados
>> tools/daemons.
>>
>> >>
>> >> >
>> >> > Then with ldd librados.so, I don't see the libceph-common.so,
>> instead:
>> >> >>
>> >> >> jialin@cori12: ldd librados.so
>> >> >> linux-vdso.so.1 (0x2aacf000)
>> >> >> libboost_thread-mt.so.1.53.0
>> >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0
>> (0x2d24d000)
>> >> >> libboost_system-mt.so.1.53.0 =>
>> >> >> /rados_install/lib/libboost_system-mt.so.1.53.0
>> (0x2d464000)
>> >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
>> >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
>> >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
>> >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
>> >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
>> >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
>> >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
>> >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
>> >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
>> >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
>> >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
>> >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
>> >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
>> >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
>> >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
>> >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
>> >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
>> >>
>> >> Things were obviously different in jewel but the concept remains the
>> >> same. For each external symbol your binary requires you need to link
>> >> to a library that exports that symbol so finding the library that
>> >> exports the symbols in the error message should resolve the error.
>> >>
>> >> >
>> >> >
>> >> > I also found this thread: http://tracker.ceph.com/issues/14788
>> >> > which looks similar to the error I run into, and that thread
>> mentioned
>> >> > the
>> >> > version between the headers file and the .so file, I think in my case
>> >> > both
>> >> > of them are v10.2.10, which should ok.
>> >> >
>> >> > Best,
>> >> > Jialin,
>> >> >
>> >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard 
>> >> > wrote:
>> >> >>
>> >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu 
>> wrote:
>> >> >> > Hi,
>> >> >> > I'm trying to use the libradosstriper api, but having some
>> trouble in
>> >> >> > linking to lradosstriper. I copied only the `required' libraries
>> from
>> >> >> > an
>> >> >> > pre-installed ceph (10.2.10), and put them under my local
>> directory
>> >> >> > /rados_install/lib and rados_install/include, on a linux machine.
>> >> >> >
>> >> >> > /rados_install/lib:
>> >> >> >>
>> >> >> >> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
>> >> >> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
>> >> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

[Yan, Zheng]

On Wed, May 30, 2018 at 5:17 PM, Oliver Freyermuth
 wrote:
> Am 30.05.2018 um 10:37 schrieb Yan, Zheng:
>> On Wed, May 30, 2018 at 3:04 PM, Oliver Freyermuth
>>  wrote:
>>> Hi,
>>>
>>> ij our case, there's only a single active MDS
>>> (+1 standby-replay + 1 standby).
>>> We also get the health warning in case it happens.
>>>
>>
>> Were there "client.xxx isn't responding to mclientcaps(revoke)"
>> warnings in cluster log.  please send them to me if there were.
>
> Yes, indeed, I almost missed them!
>
> Here you go:
>
> 
> 2018-05-29 12:16:02.491186 mon.mon003 mon.0 10.161.8.40:6789/0 11177 : 
> cluster [WRN] MDS health message (mds.0): Client XXX:XXX failing to 
> respond to capability release
> 2018-05-29 12:16:03.401014 mon.mon003 mon.0 10.161.8.40:6789/0 11178 : 
> cluster [WRN] Health check failed: 1 clients failing to respond to capability 
> release (MDS_CLIENT_LATE_RELEASE)
> 
> 2018-05-29 12:16:00.567520 mds.mon001 mds.0 10.161.8.191:6800/3068262341 
> 15745 : cluster [WRN] client.1524813 isn't responding to mclientcaps(revoke), 
> ino 0x1388ae0 pending pAsLsXsFr issued pAsLsXsFrw, sent 63.908382 seconds 
> ago
> 
>>repetition of message with increasing delays in between>
> 
> 2018-05-29 16:31:00.899416 mds.mon001 mds.0 10.161.8.191:6800/3068262341 
> 17169 : cluster [WRN] client.1524813 isn't responding to mclientcaps(revoke), 
> ino 0x1388ae0 pending pAsLsXsFr issued pAsLsXsFrw, sent 15364.240272 
> seconds ago
> 

The client failed to release Fw. When it happens again, please check
if there are hung osd requests (ceph
--admin-daemon=/var/run/ceph/ceph-client.admin.xxx.asok
objecter_requests)


>
> After evicting the client, I also get:
> 2018-05-29 17:00:00.000134 mon.mon003 mon.0 10.161.8.40:6789/0 11293 : 
> cluster [WRN] overall HEALTH_WARN 1 clients failing to respond to capability 
> release; 1 MDSs report slow requests
> 2018-05-29 17:09:50.964730 mon.mon003 mon.0 10.161.8.40:6789/0 11297 : 
> cluster [INF] MDS health message cleared (mds.0): Client XXX:XXX 
> failing to respond to capability release
> 2018-05-29 17:09:50.964767 mon.mon003 mon.0 10.161.8.40:6789/0 11298 : 
> cluster [INF] MDS health message cleared (mds.0): 123 slow requests are 
> blocked > 30 sec
> 2018-05-29 17:09:51.015071 mon.mon003 mon.0 10.161.8.40:6789/0 11299 : 
> cluster [INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients 
> failing to respond to capability release)
> 2018-05-29 17:09:51.015154 mon.mon003 mon.0 10.161.8.40:6789/0 11300 : 
> cluster [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow 
> requests)
> 2018-05-29 17:09:51.015191 mon.mon003 mon.0 10.161.8.40:6789/0 11301 : 
> cluster [INF] Cluster is now healthy
> 2018-05-29 17:14:26.178321 mds.mon002 mds.34884 10.161.8.192:6800/2102077019 
> 8 : cluster [WRN]  replayed op client.1495010:32710304,32710299 used ino 
> 0x13909d0 but session next is 0x1388af6
> 2018-05-29 17:14:26.178393 mds.mon002 mds.34884 10.161.8.192:6800/2102077019 
> 9 : cluster [WRN]  replayed op client.1495010:32710306,32710299 used ino 
> 0x13909d1 but session next is 0x1388af6
> 2018-05-29 18:00:00.000132 mon.mon003 mon.0 10.161.8.40:6789/0 11304 : 
> cluster [INF] overall HEALTH_OK
>
> Thanks for looking into it!
>
> Cheers,
> Oliver
>
>
>>
>>> Cheers,
>>> Oliver
>>>
>>> Am 30.05.2018 um 03:25 schrieb Yan, Zheng:
 I could be http://tracker.ceph.com/issues/24172


 On Wed, May 30, 2018 at 9:01 AM, Linh Vu  wrote:
> In my case, I have multiple active MDS (with directory pinning at the very
> top level), and there would be "Client xxx failing to respond to 
> capability
> release" health warning every single time that happens.
>
> 
> From: ceph-users  on behalf of Yan, 
> Zheng
> 
> Sent: Tuesday, 29 May 2018 9:53:43 PM
> To: Oliver Freyermuth
> Cc: Ceph Users; Peter Wienemann
> Subject: Re: [ceph-users] Ceph-fuse getting stuck with "currently failed 
> to
> authpin local pins"
>
> Single or multiple acitve mds? Were there "Client xxx failing to
> respond to capability release" health warning?
>
> On Mon, May 28, 2018 at 10:38 PM, Oliver Freyermuth
>  wrote:
>> Dear Cephalopodians,
>>
>> we just had a "lockup" of many MDS requests, and also trimming fell
>> behind, for over 2 days.
>> One of the clients (all ceph-fuse 12.2.5 on CentOS 7.5) was in status
>> "currently failed to authpin local pins". Metadata pool usage did grow 
>> by 10
>> GB in those 2 days.
>>
>> Rebooting the node to force a client eviction solved the issue, and now
>> metadata usage is down again, and all stuck requests were processed 
>> quickly.
>>
>> Is there any idea on what could cause something like that? On the client,
>> der was no CPU load, but many processes waiting for cephfs to respond.
>> Syslog did yield 

Re: [ceph-users] Cephfs no space on device error

[Gregory Farnum]

Given what you've shown here, it's probably one of the odder cases CephFS
is subject to, rather than an actual "there's no disk space" error. How far
is the script actually getting? Is it possible your client doesn't have
permission to write to the RADOS pool and isn't finding that out until too
late?

...actually, hrm, is your cluster running a consistent version? I see the
MDSes are different point releases, and if you've got some Jewel daemons in
the mix, there are a bunch more cases that could be applying under the old
behavior.
-Greg

On Wed, May 30, 2018 at 5:36 AM Doug Bell  wrote:

> I am new to Ceph and have built a small Ceph instance on 3 servers.  I
> realize the configuration is probably not ideal but I’d like to understand
> an error I’m getting.
>
> Ceph hosts are cm1, cm2, cm3.  Cephfs is mounted with ceph.fuse on a
> server c1.  I am attempting to perform a simple cp-rp from one directory
> tree already in cephfs to another directory also inside of cephfs.  The
> directory tree is 2740 files totaling 93G.  Approximately 3/4 of the way
> through the copy, the following error occurs:  "cp: failed to close
> ‘': No space left on device”  The odd thing is that it seems to
> finish the copy, as the final directory sizes are the same.  But scripts
> attached to the process see an error so it is causing a problem.
>
> Any idea what is happening?  I have watched all of the ceph logs on one of
> the ceph servers and haven’t seen anything.
>
> Here is some of the configuration.  The names actually aren’t obfuscated,
> they really are that generic.  IP Addresses are altered though.
>
> # ceph fs ls
> name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
>
> # ceph status
>  cluster:
>id: c14e77f1-9898-48d8-8a52-cd1f1c5bf689
>health: HEALTH_WARN
>1 MDSs behind on trimming
>
>  services:
>mon: 3 daemons, quorum cm1,cm3,cm2
>mgr: cm3(active), standbys: cm2, cm1
>mds: cephfs-1/1/1 up  {0=cm1=up:active}, 1 up:standby-replay, 1
> up:standby
>osd: 7 osds: 7 up, 7 in
>
>  data:
>pools:   2 pools, 256 pgs
>objects: 377k objects, 401 GB
>usage:   1228 GB used, 902 GB / 2131 GB avail
>pgs: 256 active+clean
>
>  io:
>client:   852 B/s rd, 2 op/s rd, 0 op/s wr
>
> # ceph osd status
>
> ++--+---+---++-++-+---+
> | id | host |  used | avail | wr ops | wr data | rd ops | rd data |
>  state   |
>
> ++--+---+---++-++-+---+
> | 0  | cm1  |  134G |  165G |0   | 0   |0   | 0   |
> exists,up |
> | 1  | cm1  |  121G |  178G |0   | 0   |0   | 0   |
> exists,up |
> | 2  | cm2  |  201G | 98.3G |0   | 0   |1   |90   |
> exists,up |
> | 3  | cm2  |  207G | 92.1G |0   | 0   |0   | 0   |
> exists,up |
> | 4  | cm3  |  217G | 82.8G |0   | 0   |0   | 0   |
> exists,up |
> | 5  | cm3  |  192G |  107G |0   | 0   |0   | 0   |
> exists,up |
> | 6  | cm1  |  153G |  177G |0   | 0   |1   |16   |
> exists,up |
> ++--+---+---++-++-+—+
>
> # ceph osd df
> ID CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL   %USE  VAR  PGS
> 0   ssd 0.29300  1.0  299G  134G165G 44.74 0.78  79
> 1   ssd 0.29300  1.0  299G  121G178G 40.64 0.70  75
> 6   ssd 0.32370  1.0  331G  153G177G 46.36 0.80 102
> 2   ssd 0.29300  1.0  299G  201G 100754M 67.20 1.17 129
> 3   ssd 0.29300  1.0  299G  207G  94366M 69.28 1.20 127
> 4   ssd 0.29300  1.0  299G  217G  84810M 72.39 1.26 131
> 5   ssd 0.29300  1.0  299G  192G107G 64.15 1.11 125
>TOTAL 2131G 1228G902G 57.65
> MIN/MAX VAR: 0.70/1.26  STDDEV: 12.36
>
> # ceph fs get cephfs
> Filesystem 'cephfs' (1)
> fs_name cephfs
> epoch   1047
> flags   c
> created 2018-03-20 13:58:51.860813
> modified2018-03-20 13:58:51.860813
> tableserver 0
> root0
> session_timeout 60
> session_autoclose   300
> max_file_size   1099511627776
> last_failure0
> last_failure_osd_epoch  98
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
> uses versioned encoding,6=dirfrag is stored in omap,8=no anchor
> table,9=file layout v2}
> max_mds 1
> in  0
> up  {0=74127}
> failed
> damaged
> stopped
> data_pools  [1]
> metadata_pool   2
> inline_data disabled
> balancer
> standby_count_wanted1
> 74127:  10.1.2.157:6800/3141645279 'cm1' mds.0.36 up:active seq 5
> (standby for rank 0)
> 64318:  10.1.2.194:6803/2623342769 'cm2' mds.0.0 up:standby-replay seq
> 497658 (standby for rank 0)
>
> # ceph fs status
> cephfs - 9 clients
> ==
> +--++-+---+---+---+
> | Rank | State  | MDS |Activity   |  dns  |  inos |
> 

[ceph-users] Ceph Call For Papers coordination pad

[Gregory Farnum]

Short version: https://pad.ceph.com/p/cfp-coordination is a space for
you to share talks you've submitted to conferences, if you want to let
other Ceph community members know what to look for and avoid
duplicating topics.

Longer version: I and a teammate almost duplicated a talk topic (for
the upcoming https://mountpoint.io — check it out!) and realized there
was no established way for us to coordinate this. Other people have
pointed out similar problems in the past. So, by the power vested in
me by the power of doing things and having Sage say "that's a good
idea", I created https://pad.ceph.com/p/cfp-coordination. Use that
space to coordinate. I've provided a template for conferences around
talk ideas and actual submissions, but please feel free to jot down
other notes around those, add new conferences you know about (even if
you aren't submitting a talk yourself), and generally use that
etherpad as a community resource.

I'll try to keep it up-to-date as conferences age out, but obviously
it's only helpful if people actually put stuff there. So go forth and
write, dear community! :)
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel/Luminous Filestore/Bluestore for a new cluster

[Simon Ironside]

On 30/05/18 20:35, Jack wrote:

Why would you deploy a Jewel cluster, which is almost 3 majors versions
away ?
Bluestore is also the good answer
It works well, have many advantages, and is simply the future of Ceph


Indeed, and normally I wouldn't even ask, but as I say there's been some 
comments/threads recently that make me doubt the obvious Luminous + 
Bluestore path. A few that stand out in my memory are:


* "Useless due to http://tracker.ceph.com/issues/22102; [1]
* OSD crash with segfault Luminous 12.2.4 [2] [3] [4]

There are others but those two stuck out for me. I realise that people 
will generally only report problems rather than "I installed ceph and 
everything went fine!" stories to this list but it was enough to 
motivate me to ask if Luminous/Bluestore was considered a good choice 
for a fresh install or if I should wait a bit.


Thanks,
Simon.

[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026339.html
[2] 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-March/025373.html

[3] http://tracker.ceph.com/issues/23431
[4] http://tracker.ceph.com/issues/23352
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel/Luminous Filestore/Bluestore for a new cluster

[Jack]

On 05/30/2018 09:20 PM, Simon Ironside wrote:
> * What's the recommendation for what to deploy?
> 
> I have a feeling the answer is going to be Luminous (as that's current
> LTS) and Bluestore (since that's the default in Luminous) but several
> recent threads and comments on this list make me doubt whether that
> would be a good choice right now.

Indeed
Why would you deploy a Jewel cluster, which is almost 3 majors versions
away ?
Bluestore is also the good answer
It works well, have many advantages, and is simply the future of Ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Jewel/Luminous Filestore/Bluestore for a new cluster

[Simon Ironside]

Hi again,

I've been happily using both Hammer and Jewel with SSD journals and 
spinning disk Filestore OSDs for several years now and, as per my other 
email, I'm about to purchase hardware to build a new (separate) 
production cluster. I intend to use the same mixture of SSD for journals 
(or DB/WAL) and spinning disks for Filestore/Bluestore data as per my 
existing cluster.


* What's the recommendation for what to deploy?

I have a feeling the answer is going to be Luminous (as that's current 
LTS) and Bluestore (since that's the default in Luminous) but several 
recent threads and comments on this list make me doubt whether that 
would be a good choice right now.


* Is using Bluestore and integrated DB/WAL (without SSDs at all) a 
reasonable option for those used to the performance of SSD Journals + 
spinning disk Filestore OSDs?


Thanks very much in advance for any advice.

Simon.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph EC profile, how are you using?

[Vasu Kulkarni]

Hello Ceph Users,

I would like to know how folks are using EC profile in the production
environment, what kind of EC configurations are you using (10+4, 5+3 ?
) with other configuration options, If you can reply to this thread or
update in the shared excel sheet below that will help design better
tests that are run on nightly basis.

https://docs.google.com/spreadsheets/d/1B7WLM3_6nV_DMf18POI7cWLWx6_vQJABVC2-bbglNEM/edit?usp=sharing

Thanks
Vasu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] SSD recommendation

[Simon Ironside]

Hi Everyone,

I'm about to purchase hardware for a new production cluster. I was going 
to use 480GB Intel DC S4600 SSDs as either Journal devices for Filestore 
and/or DB/WAL for Bluestore spinning disk OSDs until I saw David 
Herselman's "Many concurrent drive failures" thread which has given me 
the fear.


What's the current go to for Journal and/or DB/WAL SSDs if not the S4600?

I'm planning on using AMD EPYC based Supermicros for OSD nodes with 3x 
10TB SAS 7.2k to each SSD with 10gig networking. Happy to provide more 
info here if it's useful.


Thanks,
Simon.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Thanks Kefu.

Best,
Jialin
NERSC

On Tue, May 29, 2018 at 11:52 PM, kefu chai  wrote:

> On Wed, May 30, 2018 at 11:53 AM, Jialin Liu  wrote:
> > Hi Brad,
> >
> > You are correct. the librados.so has the symbol but what I copied is a
> wrong
> > file.
> > Now I can test the striper api with the previous C example and this cpp
> > example:
> > http://mrbojangles3.github.io/ceph/systems/striping/
> alignment/2017/05/28/Ceph-Stripe/
> > Both are working, but I haven't got any performance number yet.
> > ps. I also found that the stripe unit must be set before object_size
> > properly.
> >
> > Thanks much.
> >
> > Best,
> > Jialin
> > NERSC
> >
> >
> > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard 
> wrote:
> >>
> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
> >> > Thanks Brad,
> >> > I run nm on those .so file, it prints 'no symbol'
> >>
> >> OK, well you need to link to a library that exports that symbol (has
> >> it defined in its Text section). I suspect you'll find it is defined
> >> in libceph-common.so so try linking to that explicitly.
>
> please note, libceph-common does not exist in jewel (i.e. v10.2.10)
> yet. it was introduce in luminous. and libceph-common is not supposed
> to be linked against by rados application directly. it is an internal
> library and is linked by librados, libradosstriper and other rados
> tools/daemons.
>
> >>
> >> >
> >> > Then with ldd librados.so, I don't see the libceph-common.so, instead:
> >> >>
> >> >> jialin@cori12: ldd librados.so
> >> >> linux-vdso.so.1 (0x2aacf000)
> >> >> libboost_thread-mt.so.1.53.0
> >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0
> (0x2d24d000)
> >> >> libboost_system-mt.so.1.53.0 =>
> >> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000)
> >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
> >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
> >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
> >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
> >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
> >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
> >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
> >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
> >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
> >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
> >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
> >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
> >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
> >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
> >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
> >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
> >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
> >>
> >> Things were obviously different in jewel but the concept remains the
> >> same. For each external symbol your binary requires you need to link
> >> to a library that exports that symbol so finding the library that
> >> exports the symbols in the error message should resolve the error.
> >>
> >> >
> >> >
> >> > I also found this thread: http://tracker.ceph.com/issues/14788
> >> > which looks similar to the error I run into, and that thread mentioned
> >> > the
> >> > version between the headers file and the .so file, I think in my case
> >> > both
> >> > of them are v10.2.10, which should ok.
> >> >
> >> > Best,
> >> > Jialin,
> >> >
> >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard 
> >> > wrote:
> >> >>
> >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu 
> wrote:
> >> >> > Hi,
> >> >> > I'm trying to use the libradosstriper api, but having some trouble
> in
> >> >> > linking to lradosstriper. I copied only the `required' libraries
> from
> >> >> > an
> >> >> > pre-installed ceph (10.2.10), and put them under my local directory
> >> >> > /rados_install/lib and rados_install/include, on a linux machine.
> >> >> >
> >> >> > /rados_install/lib:
> >> >> >>
> >> >> >> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
> >> >> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
> >> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
> >> >> >> libboost_iostreams-mt.so.1.53.0
> >> >> >> libboost_iostreams.so.1.53.0
> >> >> >> libboost_random-mt.so.1.53.0  librados.so.2.0.0
> >> >> >> libboost_regex.so.1.53.0  librados_tp.so.2.0.0
> >> >> >> libboost_system-mt.so.1.53.0  libradosstriper.so.1.0.0
> >> >> >> libcephfs.so   librados.so  libradosstriper.so
> >> >> >
> >> >> >
> >> >> > /rados_install/include:
> >> >> >>
> >> >> >> |-- rados
> >> >> >> |   |-- buffer.h -> ../buffer.h
> >> >> >> |   |-- buffer_fwd.h -> ../buffer_fwd.h
> >> >> >> |   |-- crc32c.h -> ../crc32c.h
> >> >> >> |   |-- inline_memory.h -> ../inline_memory.h
> >> >> >> |   |-- librados.h
> >> >> >> |   |-- librados.hpp
> >> >> >> |   |-- 

[ceph-users] RGW unable to start gateway for 2nd realm

[Brett Chancellor]

Hi All,
  I'm having issues trying to get a 2nd Rados GW realm/zone up and
running.  The configuration seemed to go well, but I'm unable to start the
gateway.

2018-05-29 21:21:27.119192 7fd26cfdd9c0  0 ERROR: failed to decode obj from
.rgw.root:zone_info.fe2e0680-d7e8-415f-bf91-501dda96d075
2018-05-29 21:21:27.119198 7fd26cfdd9c0  0 replace_region_with_zonegroup:
error initializing default zone params: (5) Input/output error
2018-05-29 21:21:27.119207 7fd26cfdd9c0 -1 failed converting region to
zonegroup : ret -5 (5) Input/output error
2018-05-29 21:21:27.120479 7fd26cfdd9c0  1 --
10.252.174.9:0/3447328109 mark_down
0x55dddc157a30 -- 0x55dddc153630
2018-05-29 21:21:27.11 7fd26cfdd9c0  1 -- 10.252.174.9:0/3447328109
 mark_down_all
2018-05-29 21:21:27.122393 7fd26cfdd9c0  1 --
10.252.174.9:0/3447328109 shutdown
complete.
2018-05-29 21:21:27.122800 7fd26cfdd9c0 -1 Couldn't init storage provider
(RADOS)

Existing RadosGW .. this one works fine
Realm : realm01 (default)
zonegroup: us (master, default)
zone: us-prd-1 (master, default)

The problem comes when I'm attempting to add a new realm.
New Realm: realmtest
new ZG: maintest (master)
new zone: lumtest (master)

Steps taken:
=
* Created new rgw pools lumtest.rgw.* (14 pools) on dedicated root
*  radosgw-admin realm create --rgw-realm=realmtest
*  radosgw-admin zonegroup create --rgw-zonegroup=maintest
--rgw-realm=realmtest --master
* radosgw-admin zone create --rgw-realm=realmtest --rgw-zonegroup=maintest
--rgw-zone=lumtest --master
* radosgw-admin user create --rgw-realm realmtest --rgw-zonegroup maintest
--rgw-zone lumtest --uid="REMOVED" --display-name="System User" --system
* radosgw-admin zone modify -rgw-realm realmtest --rgw-zonegroup maintest
--rgw-zone lumtest  [added the access key and secret of system user]
*  radosgw-admin user create --rgw-realm realmtest --rgw-zonegroup maintest
--rgw-zone lumtest --uid="test" --display-name="test User"
* radosgw-admin period update --rgw-realm realmtest

ceph.conf

[client.radosgw.rgw-test]
host = rgw-test
keyring = /etc/ceph/ceph.client.radosgw.rgw-test
log file = /var/log/ceph/radosgw.rgw-test
rgw frontends = civetweb port=80
rgw realm=realmtest
rgw zonegroup=maintest
rgw zone=lumtest
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS-ganesha with RGW

[Matt Benjamin]

Hi Josef,

The main thing to make sure is that you have set up the host/vm
running nfs-ganesha exactly as if it were going to run radosgw.  For
example, you need an appropriate keyring and ceph config.  If radosgw
starts and services requests, nfs-ganesha should too.

With the debug settings you've described, you should be able to see a
bunch of output when you run ganesha.nfsd with -F.  You should see the
FSAL starting up with lots of debug output.

Matt

On Wed, May 30, 2018 at 8:19 AM, Josef Zelenka
 wrote:
> Hi, thanks for the quick reply. As for 1. I mentioned that i'm running
> ubuntu 16.04, kernel 4.4.0-121 - as it seems the platform
> package(nfs-ganesha-ceph) does not include the rgw fsal.
>
> 2. Nfsd was running - after rebooting i managed to get ganesha to bind,
> rpcbind is running, though i still can't mount the rgw due to timeouts. I
> suspect my conf might be wrong, but i'm not sure how to make sure it is.
> I've set up my ganesha.conf with the FSAL and RGW block - do i need anything
> else?
>
> EXPORT
> {
>  Export_ID=1;
>  Path = "/";
>  Pseudo = "/";
>  Access_Type = RW;
>  SecType = "sys";
>  NFS_Protocols = 4;
>  Transport_Protocols = TCP;
>
>  # optional, permit unsquashed access by client "root" user
>  #Squash = No_Root_Squash;
>
> FSAL {
>  Name = RGW;
>  User_Id =  key/secret>;
>  Access_Key_Id = "";
>  Secret_Access_Key = "";
>  }
>
> RGW {
> cluster = "ceph";
> name = "client.radosgw.radosgw-s2";
> ceph_conf = "/etc/ceph/ceph.conf";
> init_args = "-d --debug-rgw=16";
> }
> }
> Josef
>
>
>
>
>
> On 30/05/18 13:18, Matt Benjamin wrote:
>>
>> Hi Josef,
>>
>> 1. You do need the Ganesha fsal driver to be present;  I don't know
>> your platform and os version, so I couldn't look up what packages you
>> might need to install (or if the platform package does not build the
>> RGW fsal)
>> 2. The most common reason for ganesha.nfsd to fail to bind to a port
>> is that a Linux kernel nfsd is already running--can you make sure
>> that's not the case;  meanwhile you -do- need rpcbind to be running
>>
>> Matt
>>
>> On Wed, May 30, 2018 at 6:03 AM, Josef Zelenka
>>  wrote:
>>>
>>> Hi everyone, i'm currently trying to set up a NFS-ganesha instance that
>>> mounts a RGW storage, however i'm not succesful in this. I'm running Ceph
>>> Luminous 12.2.4 and ubuntu 16.04. I tried compiling ganesha from
>>> source(latest version), however i didn't manage to get the mount running
>>> with that, as ganesha refused to bind to the ipv6 interface - i assume
>>> this
>>> is a ganesha issue, but i didn't find any relevant info on what might
>>> cause
>>> this - my network setup should allow for that. Then i installed
>>> ganesha-2.6
>>> from the official repos, set up the config for RGW as per the official
>>> howto
>>> http://docs.ceph.com/docs/master/radosgw/nfs/, but i'm getting:
>>> Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so
>>> Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so: cannot open shared
>>> object file: No such file or directory
>>> and lo and behold, the libfsalrgw.so isn't present in the folder. I
>>> installed the nfs-ganesha and nfs-ganesha-fsal packages. I tried googling
>>> around, but i didn't find any relevant info or walkthroughs for this
>>> setup,
>>> so i'm asking - was anyone succesful in setting this up? I can see that
>>> even
>>> the redhat solution is still in progress, so i'm not sure if this even
>>> works. Thanks for any help,
>>>
>>> Josef
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cephfs no space on device error

[Doug Bell]

I am new to Ceph and have built a small Ceph instance on 3 servers.  I realize 
the configuration is probably not ideal but I’d like to understand an error I’m 
getting.

Ceph hosts are cm1, cm2, cm3.  Cephfs is mounted with ceph.fuse on a server c1. 
 I am attempting to perform a simple cp-rp from one directory tree already in 
cephfs to another directory also inside of cephfs.  The directory tree is 2740 
files totaling 93G.  Approximately 3/4 of the way through the copy, the 
following error occurs:  "cp: failed to close ‘': No space left on 
device”  The odd thing is that it seems to finish the copy, as the final 
directory sizes are the same.  But scripts attached to the process see an error 
so it is causing a problem.

Any idea what is happening?  I have watched all of the ceph logs on one of the 
ceph servers and haven’t seen anything.

Here is some of the configuration.  The names actually aren’t obfuscated, they 
really are that generic.  IP Addresses are altered though.

# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

# ceph status
 cluster:
   id: c14e77f1-9898-48d8-8a52-cd1f1c5bf689
   health: HEALTH_WARN
   1 MDSs behind on trimming

 services:
   mon: 3 daemons, quorum cm1,cm3,cm2
   mgr: cm3(active), standbys: cm2, cm1
   mds: cephfs-1/1/1 up  {0=cm1=up:active}, 1 up:standby-replay, 1 up:standby
   osd: 7 osds: 7 up, 7 in

 data:
   pools:   2 pools, 256 pgs
   objects: 377k objects, 401 GB
   usage:   1228 GB used, 902 GB / 2131 GB avail
   pgs: 256 active+clean

 io:
   client:   852 B/s rd, 2 op/s rd, 0 op/s wr

# ceph osd status
++--+---+---++-++-+---+
| id | host |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
++--+---+---++-++-+---+
| 0  | cm1  |  134G |  165G |0   | 0   |0   | 0   | exists,up |
| 1  | cm1  |  121G |  178G |0   | 0   |0   | 0   | exists,up |
| 2  | cm2  |  201G | 98.3G |0   | 0   |1   |90   | exists,up |
| 3  | cm2  |  207G | 92.1G |0   | 0   |0   | 0   | exists,up |
| 4  | cm3  |  217G | 82.8G |0   | 0   |0   | 0   | exists,up |
| 5  | cm3  |  192G |  107G |0   | 0   |0   | 0   | exists,up |
| 6  | cm1  |  153G |  177G |0   | 0   |1   |16   | exists,up |
++--+---+---++-++-+—+

# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL   %USE  VAR  PGS
0   ssd 0.29300  1.0  299G  134G165G 44.74 0.78  79
1   ssd 0.29300  1.0  299G  121G178G 40.64 0.70  75
6   ssd 0.32370  1.0  331G  153G177G 46.36 0.80 102
2   ssd 0.29300  1.0  299G  201G 100754M 67.20 1.17 129
3   ssd 0.29300  1.0  299G  207G  94366M 69.28 1.20 127
4   ssd 0.29300  1.0  299G  217G  84810M 72.39 1.26 131
5   ssd 0.29300  1.0  299G  192G107G 64.15 1.11 125
   TOTAL 2131G 1228G902G 57.65
MIN/MAX VAR: 0.70/1.26  STDDEV: 12.36

# ceph fs get cephfs
Filesystem 'cephfs' (1)
fs_name cephfs
epoch   1047
flags   c
created 2018-03-20 13:58:51.860813
modified2018-03-20 13:58:51.860813
tableserver 0
root0
session_timeout 60
session_autoclose   300
max_file_size   1099511627776
last_failure0
last_failure_osd_epoch  98
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses 
versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout 
v2}
max_mds 1
in  0
up  {0=74127}
failed
damaged
stopped
data_pools  [1]
metadata_pool   2
inline_data disabled
balancer
standby_count_wanted1
74127:  10.1.2.157:6800/3141645279 'cm1' mds.0.36 up:active seq 5 (standby for 
rank 0)
64318:  10.1.2.194:6803/2623342769 'cm2' mds.0.0 up:standby-replay seq 497658 
(standby for rank 0)

# ceph fs status
cephfs - 9 clients
==
+--++-+---+---+---+
| Rank | State  | MDS |Activity   |  dns  |  inos |
+--++-+---+---+---+
|  0   | active | cm1 | Reqs:0 /s |  295k |  292k |
| 0-s  | standby-replay | cm2 | Evts:0 /s |0  |0  |
+--++-+---+---+---+
+-+--+---+---+
|   Pool  |   type   |  used | avail |
+-+--+---+---+
| cephfs_metadata | metadata |  167M |  160G |
|   cephfs_data   |   data   |  401G |  160G |
+-+--+---+---+

+-+
| Standby MDS |
+-+
| cm3 |
+-+
+--+-+
| version   
   | daemons |

Re: [ceph-users] ceph-disk is getting removed from master

[Alfredo Deza]

On Wed, May 30, 2018 at 8:13 AM, Konstantin Shalygin  wrote:
> On 05/30/2018 07:08 PM, Alfredo Deza wrote:
>>
>> ceph-volume accepts a bare block device as input, but it will create
>> an LV behind the scenes
>
>
> I think this is regression. What if I don't need LV?

ceph-volume has always used LVM behind the scenes, this is the
default, and there is no other support for creating new OSDs that is
not LVM-based.

If you don't want LVM, you can continue to use ceph-disk. There is
room for developing non-LVM support, but we haven't seen any issues
with LVM so far, so there is no
incentive to try a different route at this point.


>
>
>
>
> k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS-ganesha with RGW

[Josef Zelenka]

Hi, thanks for the quick reply. As for 1. I mentioned that i'm running 
ubuntu 16.04, kernel 4.4.0-121 - as it seems the platform 
package(nfs-ganesha-ceph) does not include the rgw fsal.


2. Nfsd was running - after rebooting i managed to get ganesha to bind, 
rpcbind is running, though i still can't mount the rgw due to timeouts. 
I suspect my conf might be wrong, but i'm not sure how to make sure it 
is. I've set up my ganesha.conf with the FSAL and RGW block - do i need 
anything else?


EXPORT
{
 Export_ID=1;
 Path = "/";
 Pseudo = "/";
 Access_Type = RW;
 SecType = "sys";
 NFS_Protocols = 4;
 Transport_Protocols = TCP;

 # optional, permit unsquashed access by client "root" user
 #Squash = No_Root_Squash;

    FSAL {
 Name = RGW;
 User_Id = access key/secret>;

 Access_Key_Id = "";
 Secret_Access_Key = "";
 }

    RGW {
    cluster = "ceph";
    name = "client.radosgw.radosgw-s2";
    ceph_conf = "/etc/ceph/ceph.conf";
    init_args = "-d --debug-rgw=16";
    }
}
Josef




On 30/05/18 13:18, Matt Benjamin wrote:

Hi Josef,

1. You do need the Ganesha fsal driver to be present;  I don't know
your platform and os version, so I couldn't look up what packages you
might need to install (or if the platform package does not build the
RGW fsal)
2. The most common reason for ganesha.nfsd to fail to bind to a port
is that a Linux kernel nfsd is already running--can you make sure
that's not the case;  meanwhile you -do- need rpcbind to be running

Matt

On Wed, May 30, 2018 at 6:03 AM, Josef Zelenka
 wrote:

Hi everyone, i'm currently trying to set up a NFS-ganesha instance that
mounts a RGW storage, however i'm not succesful in this. I'm running Ceph
Luminous 12.2.4 and ubuntu 16.04. I tried compiling ganesha from
source(latest version), however i didn't manage to get the mount running
with that, as ganesha refused to bind to the ipv6 interface - i assume this
is a ganesha issue, but i didn't find any relevant info on what might cause
this - my network setup should allow for that. Then i installed ganesha-2.6
from the official repos, set up the config for RGW as per the official howto
http://docs.ceph.com/docs/master/radosgw/nfs/, but i'm getting:
Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so
Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so: cannot open shared
object file: No such file or directory
and lo and behold, the libfsalrgw.so isn't present in the folder. I
installed the nfs-ganesha and nfs-ganesha-fsal packages. I tried googling
around, but i didn't find any relevant info or walkthroughs for this setup,
so i'm asking - was anyone succesful in setting this up? I can see that even
the redhat solution is still in progress, so i'm not sure if this even
works. Thanks for any help,

Josef

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume created filestore journal bad header magic

[Alfredo Deza]

On Tue, May 29, 2018 at 11:44 PM, Zhang Qiang  wrote:
> Hi all,
>
> I'm new to Luminous, when I use ceph-volume create to add a new
> filestore OSD, it will tell me that the journal's header magic is not
> good. But the journal device is a new LV. How to make it write the new
> OSD's header to the journal?

I think this is normal for the first/initial start of the OSD and will
probably not show up again.


>
> And it seems this error message will not affect the creation and start
> of the OSD, but it complains the bad header magic  in the log every
> time it boots.
>
> journal _open /var/lib/ceph/osd/ceph-1/journal fd 30: 21474836480
> bytes, block size 4096 bytes, directio = 1, aio = 1
> journal do_read_entry(3922624512): bad header magic
> journal do_read_entry(3922624512): bad header magic
> journal _open /var/lib/ceph/osd/ceph-1/journal fd 30: 21474836480
> bytes, block size 4096 bytes, directio = 1, aio = 1
>
> Should I care about this? Is the OSD using the journal with bad magic
> header normally?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-disk is getting removed from master

[Konstantin Shalygin]

On 05/30/2018 07:08 PM, Alfredo Deza wrote:

ceph-volume accepts a bare block device as input, but it will create
an LV behind the scenes


I think this is regression. What if I don't need LV?




k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-disk is getting removed from master

[Alfredo Deza]

On Fri, May 25, 2018 at 3:22 AM, Konstantin Shalygin  wrote:
> ceph-disk should be considered as "frozen" and deprecated for Mimic,
> in favor of ceph-volume.
>
>
> ceph-volume will continue to support bare block device, i.e. without lvm'ish
> stuff?

Not sure I follow, ceph-volume has two ways of provisioning OSDs, one
is by taking over currently deployed OSDs (ceph-disk or manual)
and the other one is with LVM.

ceph-volume accepts a bare block device as input, but it will create
an LV behind the scenes
>
>
>
>
>
> k
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS-ganesha with RGW

[Matt Benjamin]

Hi Josef,

1. You do need the Ganesha fsal driver to be present;  I don't know
your platform and os version, so I couldn't look up what packages you
might need to install (or if the platform package does not build the
RGW fsal)
2. The most common reason for ganesha.nfsd to fail to bind to a port
is that a Linux kernel nfsd is already running--can you make sure
that's not the case;  meanwhile you -do- need rpcbind to be running

Matt

On Wed, May 30, 2018 at 6:03 AM, Josef Zelenka
 wrote:
> Hi everyone, i'm currently trying to set up a NFS-ganesha instance that
> mounts a RGW storage, however i'm not succesful in this. I'm running Ceph
> Luminous 12.2.4 and ubuntu 16.04. I tried compiling ganesha from
> source(latest version), however i didn't manage to get the mount running
> with that, as ganesha refused to bind to the ipv6 interface - i assume this
> is a ganesha issue, but i didn't find any relevant info on what might cause
> this - my network setup should allow for that. Then i installed ganesha-2.6
> from the official repos, set up the config for RGW as per the official howto
> http://docs.ceph.com/docs/master/radosgw/nfs/, but i'm getting:
> Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so
> Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so: cannot open shared
> object file: No such file or directory
> and lo and behold, the libfsalrgw.so isn't present in the folder. I
> installed the nfs-ganesha and nfs-ganesha-fsal packages. I tried googling
> around, but i didn't find any relevant info or walkthroughs for this setup,
> so i'm asking - was anyone succesful in setting this up? I can see that even
> the redhat solution is still in progress, so i'm not sure if this even
> works. Thanks for any help,
>
> Josef
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS-ganesha with RGW

[Marc Roos]

I think it is not working, I'am having the same problem. I'am on the 
ganesha mailing list and they have given me a patch for detailed logging 
on this issue, so they can determine what is going on. (Didn't have time 
to this though)
 



-Original Message-
From: Josef Zelenka [mailto:josef.zele...@cloudevelops.com] 
Sent: woensdag 30 mei 2018 12:03
To: ceph-users@lists.ceph.com
Subject: [ceph-users] NFS-ganesha with RGW

Hi everyone, i'm currently trying to set up a NFS-ganesha instance that 
mounts a RGW storage, however i'm not succesful in this. I'm running 
Ceph Luminous 12.2.4 and ubuntu 16.04. I tried compiling ganesha from 
source(latest version), however i didn't manage to get the mount running 
with that, as ganesha refused to bind to the ipv6 interface - i assume 
this is a ganesha issue, but i didn't find any relevant info on what 
might cause this - my network setup should allow for that. Then i 
installed ganesha-2.6 from the official repos, set up the config for RGW 
as per the official howto http://docs.ceph.com/docs/master/radosgw/nfs/,
but i'm getting:
Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so
Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so: cannot open 
shared object file: No such file or directory and lo and behold, the 
libfsalrgw.so isn't present in the folder. I installed the nfs-ganesha 
and nfs-ganesha-fsal packages. I tried googling around, but i didn't 
find any relevant info or walkthroughs for this setup, so i'm asking - 
was anyone succesful in setting this up? I can see that even the redhat 
solution is still in progress, so i'm not sure if this even works. 
Thanks for any help,

Josef

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] NFS-ganesha with RGW

[Josef Zelenka]

Hi everyone, i'm currently trying to set up a NFS-ganesha instance that 
mounts a RGW storage, however i'm not succesful in this. I'm running 
Ceph Luminous 12.2.4 and ubuntu 16.04. I tried compiling ganesha from 
source(latest version), however i didn't manage to get the mount running 
with that, as ganesha refused to bind to the ipv6 interface - i assume 
this is a ganesha issue, but i didn't find any relevant info on what 
might cause this - my network setup should allow for that. Then i 
installed ganesha-2.6 from the official repos, set up the config for RGW 
as per the official howto http://docs.ceph.com/docs/master/radosgw/nfs/, 
but i'm getting:
Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so 
Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalrgw.so: cannot open 
shared object file: No such file or directory
and lo and behold, the libfsalrgw.so isn't present in the folder. I 
installed the nfs-ganesha and nfs-ganesha-fsal packages. I tried 
googling around, but i didn't find any relevant info or walkthroughs for 
this setup, so i'm asking - was anyone succesful in setting this up? I 
can see that even the redhat solution is still in progress, so i'm not 
sure if this even works. Thanks for any help,


Josef

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

[Oliver Freyermuth]

Am 30.05.2018 um 10:37 schrieb Yan, Zheng:
> On Wed, May 30, 2018 at 3:04 PM, Oliver Freyermuth
>  wrote:
>> Hi,
>>
>> ij our case, there's only a single active MDS
>> (+1 standby-replay + 1 standby).
>> We also get the health warning in case it happens.
>>
> 
> Were there "client.xxx isn't responding to mclientcaps(revoke)"
> warnings in cluster log.  please send them to me if there were.

Yes, indeed, I almost missed them!

Here you go:


2018-05-29 12:16:02.491186 mon.mon003 mon.0 10.161.8.40:6789/0 11177 : cluster 
[WRN] MDS health message (mds.0): Client XXX:XXX failing to respond to 
capability release
2018-05-29 12:16:03.401014 mon.mon003 mon.0 10.161.8.40:6789/0 11178 : cluster 
[WRN] Health check failed: 1 clients failing to respond to capability release 
(MDS_CLIENT_LATE_RELEASE)

2018-05-29 12:16:00.567520 mds.mon001 mds.0 10.161.8.191:6800/3068262341 15745 
: cluster [WRN] client.1524813 isn't responding to mclientcaps(revoke), ino 
0x1388ae0 pending pAsLsXsFr issued pAsLsXsFrw, sent 63.908382 seconds ago

>repetition of message with increasing delays in between>

2018-05-29 16:31:00.899416 mds.mon001 mds.0 10.161.8.191:6800/3068262341 17169 
: cluster [WRN] client.1524813 isn't responding to mclientcaps(revoke), ino 
0x1388ae0 pending pAsLsXsFr issued pAsLsXsFrw, sent 15364.240272 seconds ago


After evicting the client, I also get:
2018-05-29 17:00:00.000134 mon.mon003 mon.0 10.161.8.40:6789/0 11293 : cluster 
[WRN] overall HEALTH_WARN 1 clients failing to respond to capability release; 1 
MDSs report slow requests
2018-05-29 17:09:50.964730 mon.mon003 mon.0 10.161.8.40:6789/0 11297 : cluster 
[INF] MDS health message cleared (mds.0): Client XXX:XXX failing to 
respond to capability release
2018-05-29 17:09:50.964767 mon.mon003 mon.0 10.161.8.40:6789/0 11298 : cluster 
[INF] MDS health message cleared (mds.0): 123 slow requests are blocked > 30 sec
2018-05-29 17:09:51.015071 mon.mon003 mon.0 10.161.8.40:6789/0 11299 : cluster 
[INF] Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to 
respond to capability release)
2018-05-29 17:09:51.015154 mon.mon003 mon.0 10.161.8.40:6789/0 11300 : cluster 
[INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests)
2018-05-29 17:09:51.015191 mon.mon003 mon.0 10.161.8.40:6789/0 11301 : cluster 
[INF] Cluster is now healthy
2018-05-29 17:14:26.178321 mds.mon002 mds.34884 10.161.8.192:6800/2102077019 8 
: cluster [WRN]  replayed op client.1495010:32710304,32710299 used ino 
0x13909d0 but session next is 0x1388af6
2018-05-29 17:14:26.178393 mds.mon002 mds.34884 10.161.8.192:6800/2102077019 9 
: cluster [WRN]  replayed op client.1495010:32710306,32710299 used ino 
0x13909d1 but session next is 0x1388af6
2018-05-29 18:00:00.000132 mon.mon003 mon.0 10.161.8.40:6789/0 11304 : cluster 
[INF] overall HEALTH_OK

Thanks for looking into it!

Cheers,
Oliver


> 
>> Cheers,
>> Oliver
>>
>> Am 30.05.2018 um 03:25 schrieb Yan, Zheng:
>>> I could be http://tracker.ceph.com/issues/24172
>>>
>>>
>>> On Wed, May 30, 2018 at 9:01 AM, Linh Vu  wrote:
 In my case, I have multiple active MDS (with directory pinning at the very
 top level), and there would be "Client xxx failing to respond to capability
 release" health warning every single time that happens.

 
 From: ceph-users  on behalf of Yan, 
 Zheng
 
 Sent: Tuesday, 29 May 2018 9:53:43 PM
 To: Oliver Freyermuth
 Cc: Ceph Users; Peter Wienemann
 Subject: Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to
 authpin local pins"

 Single or multiple acitve mds? Were there "Client xxx failing to
 respond to capability release" health warning?

 On Mon, May 28, 2018 at 10:38 PM, Oliver Freyermuth
  wrote:
> Dear Cephalopodians,
>
> we just had a "lockup" of many MDS requests, and also trimming fell
> behind, for over 2 days.
> One of the clients (all ceph-fuse 12.2.5 on CentOS 7.5) was in status
> "currently failed to authpin local pins". Metadata pool usage did grow by 
> 10
> GB in those 2 days.
>
> Rebooting the node to force a client eviction solved the issue, and now
> metadata usage is down again, and all stuck requests were processed 
> quickly.
>
> Is there any idea on what could cause something like that? On the client,
> der was no CPU load, but many processes waiting for cephfs to respond.
> Syslog did yield anything. It only affected one user and his user
> directory.
>
> If there are no ideas: How can I collect good debug information in case
> this happens again?
>
> Cheers,
> Oliver
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
>
> 

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

[Yan, Zheng]

On Wed, May 30, 2018 at 3:04 PM, Oliver Freyermuth
 wrote:
> Hi,
>
> ij our case, there's only a single active MDS
> (+1 standby-replay + 1 standby).
> We also get the health warning in case it happens.
>

Were there "client.xxx isn't responding to mclientcaps(revoke)"
warnings in cluster log.  please send them to me if there were.

> Cheers,
> Oliver
>
> Am 30.05.2018 um 03:25 schrieb Yan, Zheng:
>> I could be http://tracker.ceph.com/issues/24172
>>
>>
>> On Wed, May 30, 2018 at 9:01 AM, Linh Vu  wrote:
>>> In my case, I have multiple active MDS (with directory pinning at the very
>>> top level), and there would be "Client xxx failing to respond to capability
>>> release" health warning every single time that happens.
>>>
>>> 
>>> From: ceph-users  on behalf of Yan, Zheng
>>> 
>>> Sent: Tuesday, 29 May 2018 9:53:43 PM
>>> To: Oliver Freyermuth
>>> Cc: Ceph Users; Peter Wienemann
>>> Subject: Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to
>>> authpin local pins"
>>>
>>> Single or multiple acitve mds? Were there "Client xxx failing to
>>> respond to capability release" health warning?
>>>
>>> On Mon, May 28, 2018 at 10:38 PM, Oliver Freyermuth
>>>  wrote:
 Dear Cephalopodians,

 we just had a "lockup" of many MDS requests, and also trimming fell
 behind, for over 2 days.
 One of the clients (all ceph-fuse 12.2.5 on CentOS 7.5) was in status
 "currently failed to authpin local pins". Metadata pool usage did grow by 
 10
 GB in those 2 days.

 Rebooting the node to force a client eviction solved the issue, and now
 metadata usage is down again, and all stuck requests were processed 
 quickly.

 Is there any idea on what could cause something like that? On the client,
 der was no CPU load, but many processes waiting for cephfs to respond.
 Syslog did yield anything. It only affected one user and his user
 directory.

 If there are no ideas: How can I collect good debug information in case
 this happens again?

 Cheers,
 Oliver


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com

 https://protect-au.mimecast.com/s/Zl9aCXLKNwFxY9nNc6jQJC?domain=lists.ceph.com

>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

[Oliver Freyermuth]

Hi,

ij our case, there's only a single active MDS
(+1 standby-replay + 1 standby). 
We also get the health warning in case it happens. 

Cheers,
Oliver

Am 30.05.2018 um 03:25 schrieb Yan, Zheng:
> I could be http://tracker.ceph.com/issues/24172
> 
> 
> On Wed, May 30, 2018 at 9:01 AM, Linh Vu  wrote:
>> In my case, I have multiple active MDS (with directory pinning at the very
>> top level), and there would be "Client xxx failing to respond to capability
>> release" health warning every single time that happens.
>>
>> 
>> From: ceph-users  on behalf of Yan, Zheng
>> 
>> Sent: Tuesday, 29 May 2018 9:53:43 PM
>> To: Oliver Freyermuth
>> Cc: Ceph Users; Peter Wienemann
>> Subject: Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to
>> authpin local pins"
>>
>> Single or multiple acitve mds? Were there "Client xxx failing to
>> respond to capability release" health warning?
>>
>> On Mon, May 28, 2018 at 10:38 PM, Oliver Freyermuth
>>  wrote:
>>> Dear Cephalopodians,
>>>
>>> we just had a "lockup" of many MDS requests, and also trimming fell
>>> behind, for over 2 days.
>>> One of the clients (all ceph-fuse 12.2.5 on CentOS 7.5) was in status
>>> "currently failed to authpin local pins". Metadata pool usage did grow by 10
>>> GB in those 2 days.
>>>
>>> Rebooting the node to force a client eviction solved the issue, and now
>>> metadata usage is down again, and all stuck requests were processed quickly.
>>>
>>> Is there any idea on what could cause something like that? On the client,
>>> der was no CPU load, but many processes waiting for cephfs to respond.
>>> Syslog did yield anything. It only affected one user and his user
>>> directory.
>>>
>>> If there are no ideas: How can I collect good debug information in case
>>> this happens again?
>>>
>>> Cheers,
>>> Oliver
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>>
>>> https://protect-au.mimecast.com/s/Zl9aCXLKNwFxY9nNc6jQJC?domain=lists.ceph.com
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to build libradosstriper

[kefu chai]

On Wed, May 30, 2018 at 11:53 AM, Jialin Liu  wrote:
> Hi Brad,
>
> You are correct. the librados.so has the symbol but what I copied is a wrong
> file.
> Now I can test the striper api with the previous C example and this cpp
> example:
> http://mrbojangles3.github.io/ceph/systems/striping/alignment/2017/05/28/Ceph-Stripe/
> Both are working, but I haven't got any performance number yet.
> ps. I also found that the stripe unit must be set before object_size
> properly.
>
> Thanks much.
>
> Best,
> Jialin
> NERSC
>
>
> On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard  wrote:
>>
>> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
>> > Thanks Brad,
>> > I run nm on those .so file, it prints 'no symbol'
>>
>> OK, well you need to link to a library that exports that symbol (has
>> it defined in its Text section). I suspect you'll find it is defined
>> in libceph-common.so so try linking to that explicitly.

please note, libceph-common does not exist in jewel (i.e. v10.2.10)
yet. it was introduce in luminous. and libceph-common is not supposed
to be linked against by rados application directly. it is an internal
library and is linked by librados, libradosstriper and other rados
tools/daemons.

>>
>> >
>> > Then with ldd librados.so, I don't see the libceph-common.so, instead:
>> >>
>> >> jialin@cori12: ldd librados.so
>> >> linux-vdso.so.1 (0x2aacf000)
>> >> libboost_thread-mt.so.1.53.0
>> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000)
>> >> libboost_system-mt.so.1.53.0 =>
>> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000)
>> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
>> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
>> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
>> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
>> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
>> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
>> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
>> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
>> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
>> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
>> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
>> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
>> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
>> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
>> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
>> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
>> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
>>
>> Things were obviously different in jewel but the concept remains the
>> same. For each external symbol your binary requires you need to link
>> to a library that exports that symbol so finding the library that
>> exports the symbols in the error message should resolve the error.
>>
>> >
>> >
>> > I also found this thread: http://tracker.ceph.com/issues/14788
>> > which looks similar to the error I run into, and that thread mentioned
>> > the
>> > version between the headers file and the .so file, I think in my case
>> > both
>> > of them are v10.2.10, which should ok.
>> >
>> > Best,
>> > Jialin,
>> >
>> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard 
>> > wrote:
>> >>
>> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu  wrote:
>> >> > Hi,
>> >> > I'm trying to use the libradosstriper api, but having some trouble in
>> >> > linking to lradosstriper. I copied only the `required' libraries from
>> >> > an
>> >> > pre-installed ceph (10.2.10), and put them under my local directory
>> >> > /rados_install/lib and rados_install/include, on a linux machine.
>> >> >
>> >> > /rados_install/lib:
>> >> >>
>> >> >> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
>> >> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
>> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
>> >> >> libboost_iostreams-mt.so.1.53.0
>> >> >> libboost_iostreams.so.1.53.0
>> >> >> libboost_random-mt.so.1.53.0  librados.so.2.0.0
>> >> >> libboost_regex.so.1.53.0  librados_tp.so.2.0.0
>> >> >> libboost_system-mt.so.1.53.0  libradosstriper.so.1.0.0
>> >> >> libcephfs.so   librados.so  libradosstriper.so
>> >> >
>> >> >
>> >> > /rados_install/include:
>> >> >>
>> >> >> |-- rados
>> >> >> |   |-- buffer.h -> ../buffer.h
>> >> >> |   |-- buffer_fwd.h -> ../buffer_fwd.h
>> >> >> |   |-- crc32c.h -> ../crc32c.h
>> >> >> |   |-- inline_memory.h -> ../inline_memory.h
>> >> >> |   |-- librados.h
>> >> >> |   |-- librados.hpp
>> >> >> |   |-- librgw.h
>> >> >> |   |-- memory.h -> ../memory.h
>> >> >> |   |-- objclass.h
>> >> >> |   |-- page.h -> ../page.h
>> >> >> |   |-- rados_types.h
>> >> >> |   |-- rados_types.hpp
>> >> >> |   `-- rgw_file.h
>> >> >> `-- radosstriper
>> >> >> |-- libradosstriper.h
>> >> >> `-- libradosstriper.hpp
>> >> >
>> >> >
>> >> 

Re: [ceph-users] Move data from Hammer to Mimic

[Steffen Winther Sørensen]

> On 29 May 2018, at 23.29, Gregory Farnum  wrote:
> 
> On Tue, May 29, 2018 at 3:59 AM Steffen Winther Sørensen  > wrote:
> (ie. would Jewel be able to connect to both clusters)?
> 
> Yes; that should work without any issues
Great, that then leads to where I might possible find Debian Wheezy Jewel 
Packages as they EoL by now :/

apt-get update against:

deb http://ceph.com/debian-jewel wheezy main

=>

W: Failed to fetch 
http://ceph.com/debian-jewel/dists/wheezy/main/binary-amd64/Packages  301  
Moved Permanently

> 
> You could also update the Hammer cluster, although you'd need to go through a 
> few intermediate upgrades.
I thought of this too, but prefers to keep original cluster and vm images 
intact and migrate to a new cluster.

/Steffen___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com