Re: [ceph-users] Possible improvements for a slow write speed (excluding independent SSD journals)

[Christian Balzer]

On Thu, 30 Apr 2015 18:01:44 -0400 Anthony Levesque wrote:

> Im planning to setup 4-6 POC in the next 2 week to test various
> scenarios here.
> 
> Im checking to get POC with s3610, s3710, p3500(seem to be knew.  I know
> the lifespam is lower) and maybe P3700
> 
Don't ignore the S3700, it is faster in sequential writes than the 3710
because it uses older, less dense flash modules, thus more parallelism.

And with Ceph, especially when looking at the journals, you will hit the
max sequential write speed limit of the SSD long, lng before you'll
hit the IOPS limit. 
Both due to the nature of journal write and the little detail that you'll
hit the CPU performance wall before that.

> The speed  of the 400GB p3500 seem very nice and the price is alright.
> The major difference will be the durability between the P3700 and P3500
> and the IOPS.
> 
Read the link below about write amplification, but that is something that
happens mostly on the OSD part, which in your case of 1TB EVOs is already
a scary prospect in my book.

> in both option, they are the model with the lowest price per MB/s when
> compare to the S series.
> 
Price per MB/s is a good start, don't forget to factor in TBW/$ and try to
estimate what write loads your cluster will see.

But all of this is irrelevant if your typical write patterns will exceed
your CPU resources while your SSDs are bored.
For example this fio in a VM here:
---
# fio --size=4G --ioengine=libaio --invalidate=1 --direct=0 --numjobs=1 
--rw=randwrite --name=fiojob --blocksize=4K --iodepth=32

  write: io=1381.4MB, bw=16364KB/s, iops=4091 , runt= 86419msec
---

Will utilize all 8 3.1 GHz cores here, on a 3 node firefly cluster with 8
HDD OSDs and 4 journal SSDs (100GB S3700) per node. 
While the journal SSDs are at 11% and the OSD HDDs at 30-40% utilization. 

When changing that fio to direct=1, the IOPS drop to half of that.

With a block size of 4MB things of course change to the OSDs being 100%
busy, the SSDs about 60% (they can only do 200MB/s) and with 3-4 cores
worth being idle or in IOwait.

> Model
> Price per MB/s
> DC S3500
> 
> 120GB
> $1.10
> 240GB
> $1.01
> 300GB
> $1.03
> 480GB
> $1.28
> 
> 
> 
> 
> DC S3610
> 
> 200GB
> $0.99
> 400GB
> $1.14
> 480GB
> $1.24
> 
> 
> DC S3710
> 
> 200GB
> $1.17
> 
> 
> DC P3500
> 
> 400GB
> $0.64
> 
> 
> DC P3700
> 
> 400GB
> $0.96
> 
> As a side note,  the expense doesn’t scare me directly,  Its more that
> we are going blind here since it seem not a lot of people do full SSD
> setup.(Or share there experiences)
> 
See this:
http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2014-October/043949.html

I'd suggest you try the above tests yourself, you seem to have a
significant amount of hardware already.

There are many SSD threads, but so far there's at best one example of a
setup going from Firefly to Giant and Hammer.
So for me it's hard to qualify and quantify the improvements Hammer brings
to SSD based clusters other than "better", maybe about 50%.
Which while significant, is obviously nowhere near the raw performance the
hardware would be capable of.

But then again, my guestimate is that aside from the significant code that
gets executed per Ceph IOP, any such Ceph IOP results in 5-10 real IOPs
down the line.

Christian

> Anyway still brainstorm this so we can work on some POC. Will you guys
> posted here. ---
> Anthony Lévesque
> 
> 
> > On Apr 29, 2015, at 11:27 PM, Christian Balzer  wrote:
> > 
> > 
> > 
> > Hello,
> > 
> > On Wed, 29 Apr 2015 15:01:49 -0400 Anthony Levesque wrote:
> > 
> >> We redid the test with 4MB Block Size (using the same command as
> >> before but with 4MB for the BS) and we are getting better result from
> >> all devices:
> >> 
> > That's to be expected of course.
> > 
> >> Intel DC S3500 120GB = 148 MB/s
> >> Samsung Pro 128GB =187 MB/s
> >> Intel 520 120GB =  154 MB/s
> >> Samsung EVO 1TB =  186 MB/s
> >> Intel DC S3500 300GB = 250 MB/s
> > 
> > You will need to keep _both_ of these results in mind, the 4KB and 4MB
> > ones. For worst and best case scenarios.
> > And those dd tests are indicators, not a perfect replication of what
> > Ceph actually does. 
> > Looking at your original results of 920MB/s over 96 1TB EVOs those SSDs
> > are thus capable of handling about 20MB/s combined journal/data
> > traffic. Ignoring any CPU limitations of course.
> > 
> > Unsurprisingly the best of both worlds in the SSDs you compared is the
> > Intel DC S3500. 
> > And that is the slowest write and least endurable of the Intel DC SSDs.
> > 
> > To make a (rough) comparison between Intel and Samsung, the Intel DC
> > S3500 are comparable to the Samsung DC Evo ones and the Intel S3700 to
> > the Samsung DC Pro ones.
> > 
> >> I have not tested the DC S3610 yet but I will be ordering some soon
> > 
> > Those will be (for journal purposes) the worst choice when it comes to
> > bandwidth, as they use higher density FLASH, thus less speed at the

Re: [ceph-users] Possible improvements for a slow write speed (excluding independent SSD journals)

[Anthony Levesque]

Im planning to setup 4-6 POC in the next 2 week to test various scenarios here.

Im checking to get POC with s3610, s3710, p3500(seem to be knew.  I know the 
lifespam is lower) and maybe P3700

The speed  of the 400GB p3500 seem very nice and the price is alright.  The 
major difference will be the durability between the P3700 and P3500 and the 
IOPS.

in both option, they are the model with the lowest price per MB/s when compare 
to the S series.

Model
Price per MB/s
DC S3500

120GB
$1.10
240GB
$1.01
300GB
$1.03
480GB
$1.28




DC S3610

200GB
$0.99
400GB
$1.14
480GB
$1.24


DC S3710

200GB
$1.17


DC P3500

400GB
$0.64


DC P3700

400GB
$0.96

As a side note,  the expense doesn’t scare me directly,  Its more that we are 
going blind here since it seem not a lot of people do full SSD setup.(Or share 
there experiences)

Anyway still brainstorm this so we can work on some POC. Will you guys posted 
here.
---
Anthony Lévesque


> On Apr 29, 2015, at 11:27 PM, Christian Balzer  wrote:
> 
> 
> 
> Hello,
> 
> On Wed, 29 Apr 2015 15:01:49 -0400 Anthony Levesque wrote:
> 
>> We redid the test with 4MB Block Size (using the same command as before
>> but with 4MB for the BS) and we are getting better result from all
>> devices:
>> 
> That's to be expected of course.
> 
>> Intel DC S3500 120GB =   148 MB/s
>> Samsung Pro 128GB =  187 MB/s
>> Intel 520 120GB =154 MB/s
>> Samsung EVO 1TB =186 MB/s
>> Intel DC S3500 300GB =   250 MB/s
> 
> You will need to keep _both_ of these results in mind, the 4KB and 4MB
> ones. For worst and best case scenarios.
> And those dd tests are indicators, not a perfect replication of what Ceph
> actually does. 
> Looking at your original results of 920MB/s over 96 1TB EVOs those SSDs
> are thus capable of handling about 20MB/s combined journal/data traffic.
> Ignoring any CPU limitations of course.
> 
> Unsurprisingly the best of both worlds in the SSDs you compared is the
> Intel DC S3500. 
> And that is the slowest write and least endurable of the Intel DC SSDs.
> 
> To make a (rough) comparison between Intel and Samsung, the Intel DC S3500
> are comparable to the Samsung DC Evo ones and the Intel S3700 to the
> Samsung DC Pro ones.
> 
>> I have not tested the DC S3610 yet but I will be ordering some soon
> 
> Those will be (for journal purposes) the worst choice when it comes to
> bandwidth, as they use higher density FLASH, thus less speed at the same
> size.
> They are however significantly more durable than the S3500 ones at only a
> slightly higher price, thus making them good candidates for a combined
> journal/data SSD. IF your expected write load fits the endurance limits.
> 
> See:
> http://ark.intel.com/compare/75682,86640,71914
> 
>> Since previously we had the journal and OSD on the same SSD Im still
>> wondering if having the journal separate from the SSD (with a ration of
>> 1:3 or 1:4) will actually bring more Write speed. This is the
>> configuration I was thinking of if we separate the Journal from the OSD:
>> 
> Your speed will go up of course. 
> However it will not reach the fullest potential unless you put really fast
> SSDs (or a PCIe, NVMe unit like this:
> http://ark.intel.com/products/79624/Intel-SSD-DC-P3700-Series-400GB-12-Height-PCIe-3_0-20nm-MLC)
>  in there, see below.
> 
>> ÷Each OSD_Node÷
>> Dual E5-2620v2 with 64GB of RAM
> Underpowered CPU when dealing with small write IOPS.
> If all/most your writes are nicely coalesced by the RBD cache this may not
> be a problem, but without knowing what your client VMs will do it's
> impossible to predict.
> 
>> ---
>> HBA 9207-8i #1
>> 3x1TB Samsung 1TB for the Storage layer + 1 Intel S3610 200GB for the
>> Journal 3x1TB Samsung 1TB for the Storage layer + 1 Intel S3610 200GB
>> for the Journal ---
>> HBA 9207-8i #2
>> 3x1TB Samsung 1TB for the Storage layer + 1 Intel S3610 200GB for the
>> Journal 3x1TB Samsung 1TB for the Storage layer + 1 Intel S3610 200GB
>> for the Journal ---
>> 1x LSI RAID Card + 2x 120GB SSD (For OS)
>> 2x 10GbE dual port
>> 
> I suppose you already have that hardware except for the journal SSDs,
> right? 
> I would have forgone extra OS SSDs and controllers and put the OS on the
> journal SSDs in a nice RAID10.
> 
>> There would be between 6-8 OSD Node like this to start the cluster.
>> 
>> My goal would be to max out at least 20 Gbps switch ports in writes to a
>> single OpenStack Compute node. (Im still not sure about the CPU capacity)
>> 
> Maxing out the port can come in many flavors/ways. Most of which are not
> realistic scenarios, meaning that your VMs are more likely to run out of
> IOPS before they run out of bandwidth. 
> But to achieve something like 2GB/s writes your SSDs and/or journal SSDs
> will have to handle that speed. 
> 4x 200GB DC S3610s at 230MB/s each for journal SSDs are clearly not going
> to do that.
> 4x 400GB DC S3700s at 460MB/s each will come pretty clo

Re: [ceph-users] Shadow Files

[Ben Hines]

Going to hold off on our 94.1 update for this issue

Hopefully this can make it into a 94.2 or a v95 git release.

-Ben

On Mon, Apr 27, 2015 at 2:32 PM, Ben  wrote:

> How long are you thinking here?
>
> We added more storage to our cluster to overcome these issues, and we
> can't keep throwing storage at it until the issues are fixed.
>
>
> On 28/04/15 01:49, Yehuda Sadeh-Weinraub wrote:
>
>> It will get to the ceph mainline eventually. We're still reviewing and
>> testing the fix, and there's more work to be done on the cleanup tool.
>>
>> Yehuda
>>
>> - Original Message -
>>
>>> From: "Ben" 
>>> To: "Yehuda Sadeh-Weinraub" 
>>> Cc: "ceph-users" 
>>> Sent: Sunday, April 26, 2015 11:02:23 PM
>>> Subject: Re: [ceph-users] Shadow Files
>>>
>>> Are these fixes going to make it into the repository versions of ceph,
>>> or will we be required to compile and install manually?
>>>
>>> On 2015-04-26 02:29, Yehuda Sadeh-Weinraub wrote:
>>>
 Yeah, that's definitely something that we'd address soon.

 Yehuda

 - Original Message -

> From: "Ben" 
> To: "Ben Hines" , "Yehuda Sadeh-Weinraub"
> 
> Cc: "ceph-users" 
> Sent: Friday, April 24, 2015 5:14:11 PM
> Subject: Re: [ceph-users] Shadow Files
>
> Definitely need something to help clear out these old shadow files.
>
> I'm sure our cluster has around 100TB of these shadow files.
>
> I've written a script to go through known objects to get prefixes of
> objects
> that should exist to compare to ones that shouldn't, but the time it
> takes
> to do this over millions and millions of objects is just too long.
>
> On 25/04/15 09:53, Ben Hines wrote:
>
>
>
> When these are fixed it would be great to get good steps for listing /
> cleaning up any orphaned objects. I have suspicions this is affecting
> us.
>
> thanks-
>
> -Ben
>
> On Fri, Apr 24, 2015 at 3:10 PM, Yehuda Sadeh-Weinraub <
> yeh...@redhat.com >
> wrote:
>
>
> These ones:
>
> http://tracker.ceph.com/issues/10295
> http://tracker.ceph.com/issues/11447
>
> - Original Message -
>
>> From: "Ben Jackson" 
>> To: "Yehuda Sadeh-Weinraub" < yeh...@redhat.com >
>> Cc: "ceph-users" < ceph-us...@ceph.com >
>> Sent: Friday, April 24, 2015 3:06:02 PM
>> Subject: Re: [ceph-users] Shadow Files
>>
>> We were firefly, then we upgraded to giant, now we are on hammer.
>>
>> What issues?
>>
>> On 25 Apr 2015 2:12 am, Yehuda Sadeh-Weinraub < yeh...@redhat.com >
>> wrote:
>>
>>> What version are you running? There are two different issues that we
>>> were
>>> fixing this week, and we should have that upstream pretty soon.
>>>
>>> Yehuda
>>>
>>> - Original Message -
>>>
 From: "Ben" 
 To: "ceph-users" < ceph-us...@ceph.com >
 Cc: "Yehuda Sadeh-Weinraub" < yeh...@redhat.com >
 Sent: Thursday, April 23, 2015 7:42:06 PM
 Subject: [ceph-users] Shadow Files

 We are still experiencing a problem with out gateway not properly
 clearing out shadow files.

 I have done numerous tests where I have:
 -Uploaded a file of 1.5GB in size using s3browser application
 -Done an object stat on the file to get its prefix
 -Done rados ls -p .rgw.buckets | grep  to count the number
 of
 shadow files associated (in this case it is around 290 shadow files)
 -Deleted said file with s3browser
 -Performed a gc list, which shows the ~290 files listed
 -Waited 24 hours to redo the rados ls -p .rgw.buckets | grep
 
 to
 recount the shadow files only to be left with 290 files still there

  From log output /var/log/ceph/radosgw.log, I can see the following
 when
 clicking DELETE (this appears 290 times)
 2015-04-24 10:43:29.996523 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
 2015-04-24 10:43:29.996557 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
 2015-04-24 10:43:29.996564 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=13107200 stripe_ofs=13107200 part_ofs=0
 rule->part_size=0
 2015-04-24 10:43:29.996570 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=17301504 stripe_ofs=17301504 part_ofs=0
 rule->part_size=0
 2015-04-24 10:43:29.996576 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=21495808 stripe_ofs=21495808 part_ofs=0
 rule->part_size=0
 2015-04-24 10:43:29.996581 7f0b0afb5700 0
 RGWObjManifest::operator++():
 result: ofs=256901

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Gregory Farnum]

On Thu, Apr 30, 2015 at 10:57 AM, Nick Fisk  wrote:
> I'm using Inkscope to monitor my cluster and looking at the pool details I
> saw that mode was set to none. I'm pretty sure there must be a ceph cmd line
> to get the option state but I couldn't find anything obvious when I was
> looking for it.

If you run "ceph osd dump" then any cache parameters that are set will
be output. I just set one up on a vstart cluster and the output
includes this line:
> pool 3 'ctp' replicated size 3 min_size 1 crush_ruleset 0 object_hash 
> rjenkins pg_num 16 pgp_num 16 last_change 10 flags 
> hashpspool,incomplete_clones tier_of 1 cache_mode writeback stripe_width 0


If you dump it with --format=json[-pretty] it will include everything
unconditionally and you can inspect that way as well.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Don Doerner]

Mohamed,

Well, that’s interesting… and in direct conflict with what is written in the 
documentation 
(wherein it describes relative sizing as proportional to the cache pool’s 
capacity).  I am presently reinstalling, so I’ll give that a try.  Thanks very 
much.

-don-

From: Mohamed Pakkeer [mailto:mdfakk...@gmail.com]
Sent: 30 April, 2015 11:45
To: Don Doerner
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

Hi Don,

You have to provide the target size through target_max_bytes. 
target_dirty_ratio and target_full_ratio values are based upon 
target_max_bytes. If you provide target_max bytes as 200 GB and 
target_dirty_ratio as 0.4, ceph will start the fulshing, when the cache tier 
has 60GB of dirty objects.

Mohamed

On Thu, Apr 30, 2015 at 11:56 PM, Don Doerner 
mailto:don.doer...@quantum.com>> wrote:
Hi Mohamed,

I did not.  But:

•I confirmed that (by default) cache_target_dirty_ratio was set to 0.4 
(40%) and cache_target_full_ratio was set to 0.8 (80%).

•I did not set target_max_bytes (I felt that the simple, pure relative 
sizing controls were sufficient for my experiment).

•I confirmed that (by default) cache_min_flush_age and 
cache_min_evict_age were set to 0 (so no required delay for either flushing or 
eviction).

Given these settings, I expected to see:

•Flushing begin to happen at 40% of my cache tier size (~200 GB, as it 
happened), or about 80 GB.  Or earlier.

•Eviction begin to happen at 80% of my cache tier size, or about 160 
GB.  Or earlier.

•Cache tier capacity would only exceed 80% only if the flushing process 
couldn’t keep up with the ingest process for fairly long periods of time (at 
the observed ingest rate of ~400 MB/sec, a few hundred seconds).

Am I misunderstanding something?

Thank you very much for your assistance!

-don-

From: Mohamed Pakkeer [mailto:mdfakk...@gmail.com]
Sent: 30 April, 2015 10:52
To: Don Doerner
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

Hi Don,

Did you configure target_ dirty_ratio, target_full_ratio and target_max_bytes?


K.Mohamed Pakkeer

On Thu, Apr 30, 2015 at 10:26 PM, Don Doerner 
mailto:don.doer...@quantum.com>> wrote:
All,

Synopsis: I can’t get cache tiering to work in HAMMER on RHEL7.

Process:

1.  Fresh install of HAMMER on RHEL7 went well.

2.  Crush map adapted to provide two “root” level resources

a.   “ctstorage”, to use as a cache tier based on very high-performance, 
high IOPS SSD (intrinsic journal).  2 OSDs.

b.  “ecstorage”, to use as an erasure-coded poolbased on low-performance, 
cost effective storage (extrinsic journal).  12 OSDs.

3.  Established a pool “ctpool”, 32 PGs on ctstorage (pool size = min_size 
= 1).  Ran a quick RADOS bench test, all worked as expected.  Cleaned up.

4.  Established a pool “ecpool”, 256 PGs on ecstorage (5+3 profile).  Ran a 
quick RADOS bench test, all worked as expected.  Cleaned up.

5.  Ensured that both pools were empty (i.e., “rados ls” shows no objects)

6.  Put the cache tier on the erasure coded storage (one Bloom hit set, 
interval 900 seconds), set up the overlay.  Used defaults for flushing and 
eviction.  No errors.

7.  Started a 3600-second write test to ecpool.

Objects piled up in ctpool (as expected) – went past the 40% mark (as 
expected), then past the 80% mark (unexpected), then ran into the wall (95% 
full – very unexpected).  Using “rados df”, I can see that the cache tier is 
full (duh!) but not one single object lives in the ecpool.  Nothing was ever 
flushed, nothing was ever evicted.  Thought I might be misreading that, so I 
went back to SAR data that I captured during the test: the SSDs were the only 
[ceph] devices that sustained I/O.

I based this experiment on another (much more successful) experiment that I 
performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I used RAM 
as a cache tier); that all worked.  It seems there are at least three 
possibilities…

•I forgot a critical step this time around.

•The steps needed for a cache tier in HAMMER are different than the 
steps needed in GIANT (and different than the documentation online).

•There is a problem with HAMMER in the area of cache tier.

Has anyone successfully deployed cache-tiering in HAMMER?  Did you have to do 
anything unusual?  Do you see any steps that I missed?

Regards,

-don-


The information contained in this transmission may be confidential. Any 
disclosure, copying, or further distribution of confidential information is not 
permitted unless such privilege is explicitly granted in writing by Quantum. 
Quantum reserves the right to have electronic communications, including email 
and attachments,

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Mohamed Pakkeer]

Hi Don,

You have to provide the target size through target_max_bytes.
target_dirty_ratio and target_full_ratio values are based upon
target_max_bytes. If you provide target_max bytes as 200 GB and
target_dirty_ratio as 0.4, ceph will start the fulshing, when the cache
tier has 60GB of dirty objects.

Mohamed

On Thu, Apr 30, 2015 at 11:56 PM, Don Doerner 
wrote:

>  Hi Mohamed,
>
>
>
> I did not.  But:
>
> ·I confirmed that (by default) cache_target_dirty_ratio was set
> to 0.4 (40%) and cache_target_full_ratio was set to 0.8 (80%).
>
> ·I did not set target_max_bytes (I felt that the simple, pure
> relative sizing controls were sufficient for my experiment).
>
> ·I confirmed that (by default) cache_min_flush_age and
> cache_min_evict_age were set to 0 (so no required delay for either
> flushing or eviction).
>
>
>
> Given these settings, I expected to see:
>
> ·Flushing begin to happen at 40% of my cache tier size (~200 GB,
> as it happened), or about 80 GB.  Or earlier.
>
> ·Eviction begin to happen at 80% of my cache tier size, or about
> 160 GB.  Or earlier.
>
> ·Cache tier capacity would only exceed 80% only if the flushing
> process couldn’t keep up with the ingest process for fairly long periods of
> time (at the observed ingest rate of ~400 MB/sec, a few hundred seconds).
>
>
>
> Am I misunderstanding something?
>
>
>
> Thank you very much for your assistance!
>
>
>
> -don-
>
>
>
> *From:* Mohamed Pakkeer [mailto:mdfakk...@gmail.com]
> *Sent:* 30 April, 2015 10:52
> *To:* Don Doerner
> *Cc:* ceph-users@lists.ceph.com
> *Subject:* Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or
> evict?
>
>
>
> Hi Don,
>
>
>
> Did you configure target_ dirty_ratio, target_full_ratio and
> target_max_bytes?
>
>
>
>
>
> K.Mohamed Pakkeer
>
>
>
> On Thu, Apr 30, 2015 at 10:26 PM, Don Doerner 
> wrote:
>
> All,
>
>
>
> Synopsis: I can’t get cache tiering to work in HAMMER on RHEL7.
>
>
>
> Process:
>
> 1.  Fresh install of HAMMER on RHEL7 went well.
>
> 2.  Crush map adapted to provide two “root” level resources
>
> a.   “ctstorage”, to use as a cache tier based on very
> high-performance, high IOPS SSD (intrinsic journal).  2 OSDs.
>
> b.  “ecstorage”, to use as an erasure-coded poolbased on
> low-performance, cost effective storage (extrinsic journal).  12 OSDs.
>
> 3.  Established a pool “ctpool”, 32 PGs on ctstorage (pool size =
> min_size = 1).  Ran a quick RADOS bench test, all worked as expected.
> Cleaned up.
>
> 4.  Established a pool “ecpool”, 256 PGs on ecstorage (5+3 profile).
> Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
>
> 5.  Ensured that both pools were empty (i.e., “rados ls” shows no
> objects)
>
> 6.  Put the cache tier on the erasure coded storage (one Bloom hit
> set, interval 900 seconds), set up the overlay.  Used defaults for flushing
> and eviction.  No errors.
>
> 7.  Started a 3600-second write test to ecpool.
>
>
>
> Objects piled up in ctpool (as expected) – went past the 40% mark (as
> expected), then past the 80% mark (unexpected), then ran into the wall (95%
> full – *very* unexpected).  Using “rados df”, I can see that the cache
> tier is full (duh!) but not one single object lives in the ecpool.  Nothing
> was ever flushed, nothing was ever evicted.  Thought I might be misreading
> that, so I went back to SAR data that I captured during the test: the SSDs
> were the only [ceph] devices that sustained I/O.
>
>
>
> I based this experiment on another (much more successful) experiment that
> I performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I used
> RAM as a cache tier); that all worked.  It seems there are at least three
> possibilities…
>
> ·I forgot a critical step this time around.
>
> ·The steps needed for a cache tier in HAMMER are different than
> the steps needed in GIANT (and different than the documentation online).
>
> ·There is a problem with HAMMER in the area of cache tier.
>
>
>
> Has anyone successfully deployed cache-tiering in HAMMER?  Did you have to
> do anything unusual?  Do you see any steps that I missed?
>
>
>
> Regards,
>
>
>
> -don-
>
>
>  --
>
> The information contained in this transmission may be confidential. Any
> disclosure, copying, or further distribution of confidential information is
> not permitted unless such privilege is explicitly granted in writing by
> Quantum. Quantum reserves the right to have electronic communications,
> including email and attachments, sent across its networks filtered through
> anti virus and spam software programs and retain such messages in order to
> comply with applicable data security and retention requirements. Quantum is
> not responsible for the proper and complete transmission of the substance
> of this communication or for any delay in its receipt.
>
>
> ___
> ceph-users mailing list
> ce

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Don Doerner]

OK...  When I hit the wall, Ceph got pretty unusable; I haven't figured out how 
to restore it to health.  So to do as you suggest, I am going to have to scrape 
everything into the trash and try again (3rd time's the charm, right?) - so let 
me get started on that.  I will pause before I run the big test that can 
overflow the cache and consult with you on what specific steps you might 
recommend.

-don-


-Original Message-
From: Nick Fisk [mailto:n...@fisk.me.uk] 
Sent: 30 April, 2015 10:58
To: Don Doerner; ceph-users@lists.ceph.com
Subject: RE: RHEL7/HAMMER cache tier doesn't flush or evict?
Sensitivity: Personal

I'm using Inkscope to monitor my cluster and looking at the pool details I saw 
that mode was set to none. I'm pretty sure there must be a ceph cmd line to get 
the option state but I couldn't find anything obvious when I was looking for it.

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf 
> Of Don Doerner
> Sent: 30 April 2015 18:47
> To: Nick Fisk; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> Hi Nick,
> 
> For brevity, I didn't detail all of the commands I issued.  Looking 
> back
through
> my command history, I can confirm that I did explicitly set cache-mode 
> to writeback (and later reset it to forward to try flush-and-evict).
Question:
> how did you determine that your cache-mode was not writeback?  I'll do 
> that, just to  confirm that this is the problem, then reestablish the
cache-
> mode.
> 
> Thank you very much for your assistance!
> 
> -don-
> 
> -Original Message-
> From: Nick Fisk [mailto:n...@fisk.me.uk]
> Sent: 30 April, 2015 10:38
> To: Don Doerner; ceph-users@lists.ceph.com
> Subject: RE: RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> Hi Don,
> 
> I experienced the same thing a couple of days ago on Hammer. On 
> investigation the cache mode wasn't set to writeback even though I'm 
> sure
it
> accepted the command successfully when I set the pool up.
> 
> Could you reapply the cache mode writeback command and see if that 
> makes a difference?
> 
> Nick
> 
> > -Original Message-
> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On 
> > Behalf Of Don Doerner
> > Sent: 30 April 2015 17:57
> > To: ceph-users@lists.ceph.com
> > Subject: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> > Sensitivity: Personal
> >
> > All,
> >
> > Synopsis: I can't get cache tiering to work in HAMMER on RHEL7.
> >
> > Process:
> > 1. Fresh install of HAMMER on RHEL7 went well.
> > 2. Crush map adapted to provide two "root" level resources a.
> > "ctstorage", to use as a cache tier based on very high-performance,
> high
> > IOPS SSD (intrinsic journal).  2 OSDs.
> > b. "ecstorage", to use as an erasure-coded poolbased on 
> > low-performance, cost effective storage (extrinsic journal).  12 OSDs.
> > 3. Established a pool "ctpool", 32 PGs on ctstorage (pool size = 
> > min_size
> = 1).
> > Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> > 4. Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile).
> > Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> > 5. Ensured that both pools were empty (i.e., "rados ls" shows no
> > objects) 6. Put the cache tier on the erasure coded storage (one 
> > Bloom hit set, interval 900 seconds), set up the overlay.  Used 
> > defaults for flushing and eviction.  No errors.
> > 7. Started a 3600-second write test to ecpool.
> >
> > Objects piled up in ctpool (as expected) - went past the 40% mark 
> > (as expected), then past the 80% mark (unexpected), then ran into 
> > the wall (95% full - very unexpected).  Using "rados df", I can see 
> > that the cache
> tier is
> > full (duh!) but not one single object lives in the ecpool.  Nothing 
> > was
> ever
> > flushed, nothing was ever evicted.  Thought I might be misreading 
> > that, so
> I
> > went back to SAR data that I captured during the test: the SSDs were 
> > the
> only
> > [ceph] devices that sustained I/O.
> >
> > I based this experiment on another (much more successful) experiment 
> > that I performed using GIANT (.1) on RHEL7 a couple of weeks ago 
> > (wherein I used RAM as a cache tier); that all worked.  It seems 
> > there are at least
> three
> > possibilities.
> > . I forgot a critical step this time around.
> > . The steps needed for a cache tier in HAMMER are different than the 
> > steps needed in GIANT (and different than the documentation online).
> > . There is a problem with HAMMER in the area of cache tier.
> >
> > Has anyone successfully deployed cache-tiering in HAMMER?  Did you 
> > have to do anything unusual?  Do you see any steps that I missed?
> >
> > Regards,
> >
> > -don-
> >
> > 
> > The information contained in this transmission may be confidential.
> > Any disclosure, copying

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Don Doerner]

Hi Mohamed,

I did not.  But:

·I confirmed that (by default) cache_target_dirty_ratio was set to 0.4 
(40%) and cache_target_full_ratio was set to 0.8 (80%).

·I did not set target_max_bytes (I felt that the simple, pure relative 
sizing controls were sufficient for my experiment).

·I confirmed that (by default) cache_min_flush_age and 
cache_min_evict_age were set to 0 (so no required delay for either flushing or 
eviction).

Given these settings, I expected to see:

·Flushing begin to happen at 40% of my cache tier size (~200 GB, as it 
happened), or about 80 GB.  Or earlier.

·Eviction begin to happen at 80% of my cache tier size, or about 160 
GB.  Or earlier.

·Cache tier capacity would only exceed 80% only if the flushing process 
couldn’t keep up with the ingest process for fairly long periods of time (at 
the observed ingest rate of ~400 MB/sec, a few hundred seconds).

Am I misunderstanding something?

Thank you very much for your assistance!

-don-

From: Mohamed Pakkeer [mailto:mdfakk...@gmail.com]
Sent: 30 April, 2015 10:52
To: Don Doerner
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

Hi Don,

Did you configure target_ dirty_ratio, target_full_ratio and target_max_bytes?


K.Mohamed Pakkeer

On Thu, Apr 30, 2015 at 10:26 PM, Don Doerner 
mailto:don.doer...@quantum.com>> wrote:
All,

Synopsis: I can’t get cache tiering to work in HAMMER on RHEL7.

Process:

1.  Fresh install of HAMMER on RHEL7 went well.

2.  Crush map adapted to provide two “root” level resources

a.   “ctstorage”, to use as a cache tier based on very high-performance, 
high IOPS SSD (intrinsic journal).  2 OSDs.

b.  “ecstorage”, to use as an erasure-coded poolbased on low-performance, 
cost effective storage (extrinsic journal).  12 OSDs.

3.  Established a pool “ctpool”, 32 PGs on ctstorage (pool size = min_size 
= 1).  Ran a quick RADOS bench test, all worked as expected.  Cleaned up.

4.  Established a pool “ecpool”, 256 PGs on ecstorage (5+3 profile).  Ran a 
quick RADOS bench test, all worked as expected.  Cleaned up.

5.  Ensured that both pools were empty (i.e., “rados ls” shows no objects)

6.  Put the cache tier on the erasure coded storage (one Bloom hit set, 
interval 900 seconds), set up the overlay.  Used defaults for flushing and 
eviction.  No errors.

7.  Started a 3600-second write test to ecpool.

Objects piled up in ctpool (as expected) – went past the 40% mark (as 
expected), then past the 80% mark (unexpected), then ran into the wall (95% 
full – very unexpected).  Using “rados df”, I can see that the cache tier is 
full (duh!) but not one single object lives in the ecpool.  Nothing was ever 
flushed, nothing was ever evicted.  Thought I might be misreading that, so I 
went back to SAR data that I captured during the test: the SSDs were the only 
[ceph] devices that sustained I/O.

I based this experiment on another (much more successful) experiment that I 
performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I used RAM 
as a cache tier); that all worked.  It seems there are at least three 
possibilities…

•I forgot a critical step this time around.

•The steps needed for a cache tier in HAMMER are different than the 
steps needed in GIANT (and different than the documentation online).

•There is a problem with HAMMER in the area of cache tier.

Has anyone successfully deployed cache-tiering in HAMMER?  Did you have to do 
anything unusual?  Do you see any steps that I missed?

Regards,

-don-


The information contained in this transmission may be confidential. Any 
disclosure, copying, or further distribution of confidential information is not 
permitted unless such privilege is explicitly granted in writing by Quantum. 
Quantum reserves the right to have electronic communications, including email 
and attachments, sent across its networks filtered through anti virus and spam 
software programs and retain such messages in order to comply with applicable 
data security and retention requirements. Quantum is not responsible for the 
proper and complete transmission of the substance of this communication or for 
any delay in its receipt.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Thanks & Regards
K.Mohamed Pakkeer
Mobile- 0091-8754410114
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Nick Fisk]

I'm using Inkscope to monitor my cluster and looking at the pool details I
saw that mode was set to none. I'm pretty sure there must be a ceph cmd line
to get the option state but I couldn't find anything obvious when I was
looking for it.

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Don Doerner
> Sent: 30 April 2015 18:47
> To: Nick Fisk; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> Hi Nick,
> 
> For brevity, I didn't detail all of the commands I issued.  Looking back
through
> my command history, I can confirm that I did explicitly set cache-mode to
> writeback (and later reset it to forward to try flush-and-evict).
Question:
> how did you determine that your cache-mode was not writeback?  I'll do
> that, just to  confirm that this is the problem, then reestablish the
cache-
> mode.
> 
> Thank you very much for your assistance!
> 
> -don-
> 
> -Original Message-
> From: Nick Fisk [mailto:n...@fisk.me.uk]
> Sent: 30 April, 2015 10:38
> To: Don Doerner; ceph-users@lists.ceph.com
> Subject: RE: RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> Hi Don,
> 
> I experienced the same thing a couple of days ago on Hammer. On
> investigation the cache mode wasn't set to writeback even though I'm sure
it
> accepted the command successfully when I set the pool up.
> 
> Could you reapply the cache mode writeback command and see if that
> makes a difference?
> 
> Nick
> 
> > -Original Message-
> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> > Of Don Doerner
> > Sent: 30 April 2015 17:57
> > To: ceph-users@lists.ceph.com
> > Subject: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> > Sensitivity: Personal
> >
> > All,
> >
> > Synopsis: I can't get cache tiering to work in HAMMER on RHEL7.
> >
> > Process:
> > 1. Fresh install of HAMMER on RHEL7 went well.
> > 2. Crush map adapted to provide two "root" level resources a.
> > "ctstorage", to use as a cache tier based on very high-performance,
> high
> > IOPS SSD (intrinsic journal).  2 OSDs.
> > b. "ecstorage", to use as an erasure-coded poolbased on
> > low-performance, cost effective storage (extrinsic journal).  12 OSDs.
> > 3. Established a pool "ctpool", 32 PGs on ctstorage (pool size =
> > min_size
> = 1).
> > Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> > 4. Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile).
> > Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> > 5. Ensured that both pools were empty (i.e., "rados ls" shows no
> > objects) 6. Put the cache tier on the erasure coded storage (one Bloom
> > hit set, interval 900 seconds), set up the overlay.  Used defaults for
> > flushing and eviction.  No errors.
> > 7. Started a 3600-second write test to ecpool.
> >
> > Objects piled up in ctpool (as expected) - went past the 40% mark (as
> > expected), then past the 80% mark (unexpected), then ran into the wall
> > (95% full - very unexpected).  Using "rados df", I can see that the
> > cache
> tier is
> > full (duh!) but not one single object lives in the ecpool.  Nothing
> > was
> ever
> > flushed, nothing was ever evicted.  Thought I might be misreading
> > that, so
> I
> > went back to SAR data that I captured during the test: the SSDs were
> > the
> only
> > [ceph] devices that sustained I/O.
> >
> > I based this experiment on another (much more successful) experiment
> > that I performed using GIANT (.1) on RHEL7 a couple of weeks ago
> > (wherein I used RAM as a cache tier); that all worked.  It seems there
> > are at least
> three
> > possibilities.
> > . I forgot a critical step this time around.
> > . The steps needed for a cache tier in HAMMER are different than the
> > steps needed in GIANT (and different than the documentation online).
> > . There is a problem with HAMMER in the area of cache tier.
> >
> > Has anyone successfully deployed cache-tiering in HAMMER?  Did you
> > have to do anything unusual?  Do you see any steps that I missed?
> >
> > Regards,
> >
> > -don-
> >
> > 
> > The information contained in this transmission may be confidential.
> > Any disclosure, copying, or further distribution of confidential
> > information
> is not
> > permitted unless such privilege is explicitly granted in writing by
> Quantum.
> > Quantum reserves the right to have electronic communications,
> > including email and attachments, sent across its networks filtered
> > through anti
> virus
> > and spam software programs and retain such messages in order to comply
> > with applicable data security and retention requirements. Quantum is
> > not responsible for the proper and complete transmission of the
> > substance of this communication or for any delay in its receipt.
> 
> 
> 
> 
> ___
> ce

Re: [ceph-users] Ceph Fuse Crashed when Reading and How to Backup the data

[Gregory Farnum]

The not permitted bit usually means that your client doesn't have
access permissions to the data pool in use.

I'm not sure why it would be getting aborted without any output though
— is there any traceback at all in the logs? A message about the
OOM-killer zapping it or something?
-Greg


On Thu, Apr 30, 2015 at 1:45 AM, flisky  wrote:
> Sorry，I cannot reproduce the "Operation not permitted" log
>
> Here is a small portion of log with "debug_client = 20/20"
> ==
>-22> 2015-04-30 16:29:12.858309 7fe9757f2700 10 client.58272 check_caps
> on 115.head(ref=2 ll_ref=10 cap_refs={} open={1=1} mode=100664
> size=119/0 mtime=2015-04-20 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr)
> objectset[115 ts 0/0 objects 0 dirty_or_tx 0] parents=0x7fe968022980
> 0x7fe968021c30) wanted pFscr used - is_delayed=1
>-21> 2015-04-30 16:29:12.858326 7fe9757f2700 10 client.58272  cap mds.0
> issued pAsLsXsFscr implemented pAsLsXsFscr revoking -
>-20> 2015-04-30 16:29:12.858333 7fe9757f2700 10 client.58272 send_cap
> 115.head(ref=2 ll_ref=10 cap_refs={} open={1=1} mode=100664
> size=119/0 mtime=2015-04-20 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr)
> objectset[115 ts 0/0 objects 0 dirty_or_tx 0] parents=0x7fe968022980
> 0x7fe968021c30) mds.0 seq 1 used - want pFscr flush - retain
> pAsxLsxXsxFsxcrwbl held pAsLsXsFscr revoking - dropping -
>-19> 2015-04-30 16:29:12.858358 7fe9757f2700 15 client.58272 auth cap,
> setting max_size = 0
>-18> 2015-04-30 16:29:12.858368 7fe9757f2700 10 client.58272 _create_fh
> 115 mode 1
>-17> 2015-04-30 16:29:12.858376 7fe9757f2700 20 client.58272 trim_cache
> size 14 max 16384
>-16> 2015-04-30 16:29:12.858378 7fe9757f2700  3 client.58272 ll_open
> 115.head 32768 = 0 (0x7fe95c0052c0)
>-15> 2015-04-30 16:29:12.858385 7fe9757f2700  3 client.58272 ll_forget
> 115 1
>-14> 2015-04-30 16:29:12.858386 7fe9757f2700 20 client.58272 _ll_put
> 0x7fe968021c30 115 1 -> 9
>-13> 2015-04-30 16:29:12.858500 7fe974ff1700 20 client.58272 _ll_get
> 0x7fe968021c30 115 -> 10
>-12> 2015-04-30 16:29:12.858503 7fe974ff1700  3 client.58272 ll_getattr
> 115.head
>-11> 2015-04-30 16:29:12.858505 7fe974ff1700 10 client.58272 _getattr
> mask pAsLsXsFs issued=1
>-10> 2015-04-30 16:29:12.858509 7fe974ff1700 10 client.58272 fill_stat on
> 115 snap/devhead mode 0100664 mtime 2015-04-20 14:14:57.961482 ctime
> 2015-04-20 14:14:57.960359
> -9> 2015-04-30 16:29:12.858518 7fe974ff1700  3 client.58272 ll_getattr
> 115.head = 0
> -8> 2015-04-30 16:29:12.858525 7fe974ff1700  3 client.58272 ll_forget
> 115 1
> -7> 2015-04-30 16:29:12.858526 7fe974ff1700 20 client.58272 _ll_put
> 0x7fe968021c30 115 1 -> 9
> -6> 2015-04-30 16:29:12.858536 7fe9577fe700  3 client.58272 ll_read
> 0x7fe95c0052c0 115  0~4096
> -5> 2015-04-30 16:29:12.858539 7fe9577fe700 10 client.58272 get_caps
> 115.head(ref=3 ll_ref=9 cap_refs={} open={1=1} mode=100664
> size=119/0 mtime=2015-04-20 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr)
> objectset[115 ts 0/0 objects 0 dirty_or_tx 0] parents=0x7fe968022980
> 0x7fe968021c30) have pAsLsXsFscr need Fr want Fc but not Fc revoking -
> -4> 2015-04-30 16:29:12.858561 7fe9577fe700 10 client.58272 _read_async
> 115.head(ref=3 ll_ref=9 cap_refs={2048=1} open={1=1} mode=100664
> size=119/0 mtime=2015-04-20 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr)
> objectset[115 ts 0/0 objects 0 dirty_or_tx 0] parents=0x7fe968022980
> 0x7fe968021c30) 0~4096
> -3> 2015-04-30 16:29:12.858575 7fe9577fe700 10 client.58272 max_byes=0
> max_periods=4
> -2> 2015-04-30 16:29:12.858692 7fe9577fe700  5 client.58272 get_cap_ref
> got first FILE_CACHE ref on 115.head(ref=3 ll_ref=9
> cap_refs={1024=0,2048=1} open={1=1} mode=100664 size=119/0 mtime=2015-04-20
> 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[115 ts 0/0
> objects 1 dirty_or_tx 0] parents=0x7fe968022980 0x7fe968021c30)
> -1> 2015-04-30 16:29:12.867657 7fe9797fa700 10 client.58272
> ms_handle_connect on 172.16.3.149:6823/982446
>  0> 2015-04-30 16:29:12.872787 7fe97bfff700 -1 *** Caught signal
> (Aborted) **
>
>
>
> On 2015年04月30日 16:21, flisky wrote:
>>
>> When I read the file through the ceph-fuse, the process crashed.
>>
>> Here is the log -
>> 
>> terminate called after throwing an instance of
>> 'ceph::buffer::end_of_buffer'
>>what():  buffer::end_of_buffer
>> *** Caught signal (Aborted) **
>>   in thread 7fe0814d3700
>>   ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
>>   1: (()+0x249805) [0x7fe08670b805]
>>   2: (()+0x10d10) [0x7fe085c39d10]
>>   3: (gsignal()+0x37) [0x7fe0844d3267]
>>   4: (abort()+0x16a) [0x7fe0844d4eca]
>>   5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7fe084de706d]
>>   6: (()+0x5eee6) [0x7

Re: [ceph-users] ceph-dokan mount error

[James Devine]

Yup I think you are correct, I see this listed under issues
https://github.com/ketor/ceph-dokan/issues/5

On Thu, Apr 30, 2015 at 12:58 PM, Gregory Farnum  wrote:
> On Thu, Apr 30, 2015 at 9:49 AM, James Devine  wrote:
>> So I am trying to get ceph-dokan to work.  Upon running it with
>> ./ceph-dokan.exe -c ceph.conf -l e it indicates there was a mount
>> error and the monitor it connects to logs cephx server client.admin:
>> unexpected key: req.key=0 expected_key=d7901d515f6b0c61
>>
>> According to the debug output attached ceph-dokan reads the config
>> file and keyring fine.  Any ideas where I might be going wrong?
>
> I haven't used or looked at dokan at all, but if the received key is 0
> then I'm guessing dokan isn't using the cephx security that the server
> is expecting. And there was a note from the guys who did rados_dll
> that they had to disable cephx — so perhaps dokan doesn't support
> cephx at all and you need to disable it? I'm not sure, just spitting
> into the wind here... ;)
> -Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-dokan mount error

[Gregory Farnum]

On Thu, Apr 30, 2015 at 9:49 AM, James Devine  wrote:
> So I am trying to get ceph-dokan to work.  Upon running it with
> ./ceph-dokan.exe -c ceph.conf -l e it indicates there was a mount
> error and the monitor it connects to logs cephx server client.admin:
> unexpected key: req.key=0 expected_key=d7901d515f6b0c61
>
> According to the debug output attached ceph-dokan reads the config
> file and keyring fine.  Any ideas where I might be going wrong?

I haven't used or looked at dokan at all, but if the received key is 0
then I'm guessing dokan isn't using the cephx security that the server
is expecting. And there was a note from the guys who did rados_dll
that they had to disable cephx — so perhaps dokan doesn't support
cephx at all and you need to disable it? I'm not sure, just spitting
into the wind here... ;)
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cache pool parameters and pressure

[Gregory Farnum]

On Thu, Apr 30, 2015 at 2:03 AM, Kenneth Waegeman
 wrote:
> So the cache is empty, but I get warning when I check the health:
>  health HEALTH_WARN
> mds0: Client cephtst.cubone.os failing to respond to cache
> pressure
>
> Someone an idea what is happening here?

This means that the MDS sent the client a request to drop some cached
inodes/dentries and it isn't. This could mean that the client is too
old to respond correctly, or that you've actually got so many files
open that the client can't drop any of the requested items.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Mohamed Pakkeer]

Hi Don,

Did you configure target_ dirty_ratio, target_full_ratio and
target_max_bytes?


K.Mohamed Pakkeer

On Thu, Apr 30, 2015 at 10:26 PM, Don Doerner 
wrote:

>  All,
>
>
>
> Synopsis: I can’t get cache tiering to work in HAMMER on RHEL7.
>
>
>
> Process:
>
> 1.  Fresh install of HAMMER on RHEL7 went well.
>
> 2.  Crush map adapted to provide two “root” level resources
>
> a.   “ctstorage”, to use as a cache tier based on very
> high-performance, high IOPS SSD (intrinsic journal).  2 OSDs.
>
> b.  “ecstorage”, to use as an erasure-coded poolbased on
> low-performance, cost effective storage (extrinsic journal).  12 OSDs.
>
> 3.  Established a pool “ctpool”, 32 PGs on ctstorage (pool size =
> min_size = 1).  Ran a quick RADOS bench test, all worked as expected.  Cleaned
> up.
>
> 4.  Established a pool “ecpool”, 256 PGs on ecstorage (5+3 profile).  Ran
> a quick RADOS bench test, all worked as expected.  Cleaned up.
>
> 5.  Ensured that both pools were empty (i.e., “rados ls” shows no
> objects)
>
> 6.  Put the cache tier on the erasure coded storage (one Bloom hit
> set, interval 900 seconds), set up the overlay.  Used defaults for
> flushing and eviction.  No errors.
>
> 7.  Started a 3600-second write test to ecpool.
>
>
>
> Objects piled up in ctpool (as expected) – went past the 40% mark (as
> expected), then past the 80% mark (unexpected), then ran into the wall (95%
> full – *very* unexpected).  Using “rados df”, I can see that the cache
> tier is full (duh!) but not one single object lives in the ecpool.  Nothing
> was ever flushed, nothing was ever evicted.  Thought I might be
> misreading that, so I went back to SAR data that I captured during the
> test: the SSDs were the only [ceph] devices that sustained I/O.
>
>
>
> I based this experiment on another (much more successful) experiment that
> I performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I used
> RAM as a cache tier); that all worked.  It seems there are at least three
> possibilities…
>
> ·I forgot a critical step this time around.
>
> ·The steps needed for a cache tier in HAMMER are different than
> the steps needed in GIANT (and different than the documentation online).
>
> ·There is a problem with HAMMER in the area of cache tier.
>
>
>
> Has anyone successfully deployed cache-tiering in HAMMER?  Did you have
> to do anything unusual?  Do you see any steps that I missed?
>
>
>
> Regards,
>
>
>
> -don-
>
>
>  --
> The information contained in this transmission may be confidential. Any
> disclosure, copying, or further distribution of confidential information is
> not permitted unless such privilege is explicitly granted in writing by
> Quantum. Quantum reserves the right to have electronic communications,
> including email and attachments, sent across its networks filtered through
> anti virus and spam software programs and retain such messages in order to
> comply with applicable data security and retention requirements. Quantum is
> not responsible for the proper and complete transmission of the substance
> of this communication or for any delay in its receipt.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Thanks & Regards
K.Mohamed Pakkeer
Mobile- 0091-8754410114
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Don Doerner]

Hi Nick,

For brevity, I didn't detail all of the commands I issued.  Looking back 
through my command history, I can confirm that I did explicitly set cache-mode 
to writeback (and later reset it to forward to try flush-and-evict).  Question: 
how did you determine that your cache-mode was not writeback?  I'll do that, 
just to  confirm that this is the problem, then reestablish the cache-mode.

Thank you very much for your assistance!

-don-

-Original Message-
From: Nick Fisk [mailto:n...@fisk.me.uk] 
Sent: 30 April, 2015 10:38
To: Don Doerner; ceph-users@lists.ceph.com
Subject: RE: RHEL7/HAMMER cache tier doesn't flush or evict?
Sensitivity: Personal

Hi Don,

I experienced the same thing a couple of days ago on Hammer. On investigation 
the cache mode wasn't set to writeback even though I'm sure it accepted the 
command successfully when I set the pool up.

Could you reapply the cache mode writeback command and see if that makes a 
difference?

Nick

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf 
> Of Don Doerner
> Sent: 30 April 2015 17:57
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> All,
> 
> Synopsis: I can't get cache tiering to work in HAMMER on RHEL7.
> 
> Process:
> 1. Fresh install of HAMMER on RHEL7 went well.
> 2. Crush map adapted to provide two "root" level resources a.  
> "ctstorage", to use as a cache tier based on very high-performance,
high
> IOPS SSD (intrinsic journal).  2 OSDs.
> b. "ecstorage", to use as an erasure-coded poolbased on 
> low-performance, cost effective storage (extrinsic journal).  12 OSDs.
> 3. Established a pool "ctpool", 32 PGs on ctstorage (pool size = 
> min_size
= 1).
> Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> 4. Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile).  
> Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> 5. Ensured that both pools were empty (i.e., "rados ls" shows no 
> objects) 6. Put the cache tier on the erasure coded storage (one Bloom 
> hit set, interval 900 seconds), set up the overlay.  Used defaults for 
> flushing and eviction.  No errors.
> 7. Started a 3600-second write test to ecpool.
> 
> Objects piled up in ctpool (as expected) - went past the 40% mark (as 
> expected), then past the 80% mark (unexpected), then ran into the wall 
> (95% full - very unexpected).  Using "rados df", I can see that the 
> cache
tier is
> full (duh!) but not one single object lives in the ecpool.  Nothing 
> was
ever
> flushed, nothing was ever evicted.  Thought I might be misreading 
> that, so
I
> went back to SAR data that I captured during the test: the SSDs were 
> the
only
> [ceph] devices that sustained I/O.
> 
> I based this experiment on another (much more successful) experiment 
> that I performed using GIANT (.1) on RHEL7 a couple of weeks ago 
> (wherein I used RAM as a cache tier); that all worked.  It seems there 
> are at least
three
> possibilities.
> . I forgot a critical step this time around.
> . The steps needed for a cache tier in HAMMER are different than the 
> steps needed in GIANT (and different than the documentation online).
> . There is a problem with HAMMER in the area of cache tier.
> 
> Has anyone successfully deployed cache-tiering in HAMMER?  Did you 
> have to do anything unusual?  Do you see any steps that I missed?
> 
> Regards,
> 
> -don-
> 
> 
> The information contained in this transmission may be confidential. 
> Any disclosure, copying, or further distribution of confidential 
> information
is not
> permitted unless such privilege is explicitly granted in writing by
Quantum.
> Quantum reserves the right to have electronic communications, 
> including email and attachments, sent across its networks filtered 
> through anti
virus
> and spam software programs and retain such messages in order to comply 
> with applicable data security and retention requirements. Quantum is 
> not responsible for the proper and complete transmission of the 
> substance of this communication or for any delay in its receipt.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Nick Fisk]

Hi Don,

I experienced the same thing a couple of days ago on Hammer. On
investigation the cache mode wasn't set to writeback even though I'm sure it
accepted the command successfully when I set the pool up.

Could you reapply the cache mode writeback command and see if that makes a
difference?

Nick

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Don Doerner
> Sent: 30 April 2015 17:57
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?
> Sensitivity: Personal
> 
> All,
> 
> Synopsis: I can't get cache tiering to work in HAMMER on RHEL7.
> 
> Process:
> 1. Fresh install of HAMMER on RHEL7 went well.
> 2. Crush map adapted to provide two "root" level resources
> a.  "ctstorage", to use as a cache tier based on very high-performance,
high
> IOPS SSD (intrinsic journal).  2 OSDs.
> b. "ecstorage", to use as an erasure-coded poolbased on low-performance,
> cost effective storage (extrinsic journal).  12 OSDs.
> 3. Established a pool "ctpool", 32 PGs on ctstorage (pool size = min_size
= 1).
> Ran a quick RADOS bench test, all worked as expected.  Cleaned up.
> 4. Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile).  Ran a
> quick RADOS bench test, all worked as expected.  Cleaned up.
> 5. Ensured that both pools were empty (i.e., "rados ls" shows no objects)
> 6. Put the cache tier on the erasure coded storage (one Bloom hit set,
> interval 900 seconds), set up the overlay.  Used defaults for flushing and
> eviction.  No errors.
> 7. Started a 3600-second write test to ecpool.
> 
> Objects piled up in ctpool (as expected) - went past the 40% mark (as
> expected), then past the 80% mark (unexpected), then ran into the wall
> (95% full - very unexpected).  Using "rados df", I can see that the cache
tier is
> full (duh!) but not one single object lives in the ecpool.  Nothing was
ever
> flushed, nothing was ever evicted.  Thought I might be misreading that, so
I
> went back to SAR data that I captured during the test: the SSDs were the
only
> [ceph] devices that sustained I/O.
> 
> I based this experiment on another (much more successful) experiment that
> I performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I
> used RAM as a cache tier); that all worked.  It seems there are at least
three
> possibilities.
> . I forgot a critical step this time around.
> . The steps needed for a cache tier in HAMMER are different than the steps
> needed in GIANT (and different than the documentation online).
> . There is a problem with HAMMER in the area of cache tier.
> 
> Has anyone successfully deployed cache-tiering in HAMMER?  Did you have
> to do anything unusual?  Do you see any steps that I missed?
> 
> Regards,
> 
> -don-
> 
> 
> The information contained in this transmission may be confidential. Any
> disclosure, copying, or further distribution of confidential information
is not
> permitted unless such privilege is explicitly granted in writing by
Quantum.
> Quantum reserves the right to have electronic communications, including
> email and attachments, sent across its networks filtered through anti
virus
> and spam software programs and retain such messages in order to comply
> with applicable data security and retention requirements. Quantum is not
> responsible for the proper and complete transmission of the substance of
> this communication or for any delay in its receipt.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kicking 'Remapped' PGs

[Gregory Farnum]

On Wed, Apr 29, 2015 at 6:06 PM, Paul Evans  wrote:
> In one of our clusters we sometimes end up with PGs that are mapped
> incorrectly and settle into a ‘remapped’ state (forever).  Is there a way to
> nudge a specific PG to recalculate placement and relocate the data?  One
> option that we’re *dangerously* unclear about is the use of ceph pg
> force_create_pg . Is this a viable command to use in a ‘remapped’
> situation?

Remapped PGs that are stuck that way mean that CRUSH is failing to map
them appropriately — I think we talked about the circumstances around
that previously. :) So nudging CRUSH can't do anything; it will just
fail to map them appropriately again. (And indeed this is what happens
whenever anyone does something to that PG or the OSD Map gets
changed.)

The force_create_pg command does exactly what it sounds like: it tells
the OSDs which should currently host the named PG to create it. You
shouldn't need to run it and I don't remember exactly what checks it
goes through, but it's generally for when you've given up on
retrieving any data out of a PG whose OSDs died and want to just start
over with a completely blank one.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] journal raw partition

[Deneau, Tom]

I am experimenting with different external journal partitions as raw partitions 
(no file system).

using 

ceph-deploy osd prepare foo:/mount-point-for-data-partition:journal-partition

followed by 
ceph-deploy osd activate (same arguments)

When the specified journal-partition is on an ssd drive I notice that
osd prepare reports
[WARNIN] DEBUG:ceph-disk:Journal /dev/sdb2 is a partition
[WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the 
same device as the osd data
WARNIN] INFO:ceph-disk:Running command: /usr/sbin/blkid -p -o udev /dev/sdb2
WARNIN] WARNING:ceph-disk:Journal /dev/sdb2 was not prepared with ceph-disk. 
Symlinking directly.
[WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/local//dev/sdc1
[WARNIN] DEBUG:ceph-disk:Creating symlink /var/local//dev/sdc1/journal -> 
/dev/sdb2

and the later ceph-deploy osd activate works fine

But when the journal-partition is a small partition on the beginning of the 
data drive
osd prepare reports
[WARNIN] DEBUG:ceph-disk:Journal is file /dev/sdc2   
<<=
[WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the 
same device as the osd data
[WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/local//dev/sdc1
[WARNIN] DEBUG:ceph-disk:Creating symlink /var/local//dev/sdc1/journal -> 
/dev/sdc2

and then the later ceph-deploy osd activate fails with

[WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster ceph 
--mkfs --mkkey -i 0 --monmap /var/local//dev/sd\
c1/activate.monmap --osd-data /var/local//dev/sdc1 --osd-journal 
/var/local//dev/sdc1/journal --osd-uuid 05b3933e-9ac4-453a-9072-c7ebf242ba7\
0 --keyring /var/local//dev/sdc1/keyring
[WARNIN] 2015-04-30 13:09:30.869745 3ff9df5cec0 -1 journal FileJournal::_open: 
disabling aio for non-block journal.  Use journal_force_aio to force use of aio 
anyway
[WARNIN] 2015-04-30 13:09:30.869810 3ff9df5cec0 -1 journal check: ondisk fsid 
---- doesn't match expected 
05b3933e-9ac4-453a-9072-c7ebf242ba70, invalid (someone else's?) journal
[WARNIN] 2015-04-30 13:09:30.869863 3ff9df5cec0 -1 
filestore(/var/local//dev/sdc1) mkjournal error creating journal on 
/var/local//dev/sdc1/journal: (22) Invalid argument
[WARNIN] 2015-04-30 13:09:30.869883 3ff9df5cec0 -1 OSD::mkfs: ObjectStore::mkfs 
failed with error -22
[WARNIN] 2015-04-30 13:09:30.869934 3ff9df5cec0 -1  ** ERROR: error creating 
empty object store in /var/local//dev/sdc1: (22) Invalid argument

I'm assuming the problem started with osd prepare thinking that /dev/sdc2 was a 
file rather than a partition.
Is there some partition table thing I am missing here?

parted /dev/sdc print gives

Disk /dev/sdc: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End SizeFile system  Name Flags
 2  1049kB  5000MB  4999MB   primary
 1  5000MB  2000GB  1995GB  xfs  primary

Not sure if it is related but I do know that in the past I had created a
single partition on /dev/sdc and used that as an xfs data partition.

-- Tom Deneau, AMD



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade from Giant to Hammer and after some basic operations most of the OSD's went down

[tuomas.juntunen]

Thanks man. I'll try it tomorrow. Have a good one.

Br,T

 Original message 
From: Sage Weil  
Date: 30/04/2015  18:23  (GMT+02:00) 
To: Tuomas Juntunen  
Cc: ceph-users@lists.ceph.com, ceph-de...@vger.kernel.org 
Subject: RE: [ceph-users] Upgrade from Giant to Hammer and after some basic
   operations most of the OSD's went down 

On Thu, 30 Apr 2015, tuomas.juntu...@databasement.fi wrote:
> Hey
> 
> Yes I can drop the images data, you think this will fix it?

It's a slightly different assert that (I believe) should not trigger once 
the pool is deleted.  Please give that a try and if you still hit it I'll 
whip up a workaround.

Thanks!
sage

 > 
> 
> Br,
> 
> Tuomas
> 
> > On Wed, 29 Apr 2015, Tuomas Juntunen wrote:
> >> Hi
> >>
> >> I updated that version and it seems that something did happen, the osd's
> >> stayed up for a while and 'ceph status' got updated. But then in couple of
> >> minutes, they all went down the same way.
> >>
> >> I have attached new 'ceph osd dump -f json-pretty' and got a new log from
> >> one of the osd's with osd debug = 20,
> >> http://beta.xaasbox.com/ceph/ceph-osd.15.log
> >
> > Sam mentioned that you had said earlier that this was not critical data?
> > If not, I think the simplest thing is to just drop those pools.  The
> > important thing (from my perspective at least :) is that we understand the
> > root cause and can prevent this in the future.
> >
> > sage
> >
> >
> >>
> >> Thank you!
> >>
> >> Br,
> >> Tuomas
> >>
> >>
> >>
> >> -Original Message-
> >> From: Sage Weil [mailto:s...@newdream.net]
> >> Sent: 28. huhtikuuta 2015 23:57
> >> To: Tuomas Juntunen
> >> Cc: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org
> >> Subject: Re: [ceph-users] Upgrade from Giant to Hammer and after some basic
> >> operations most of the OSD's went down
> >>
> >> Hi Tuomas,
> >>
> >> I've pushed an updated wip-hammer-snaps branch.  Can you please try it?
> >> The build will appear here
> >>
> >>
> >> http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/08bf531331afd5e
> >> 2eb514067f72afda11bcde286
> >>
> >> (or a similar url; adjust for your distro).
> >>
> >> Thanks!
> >> sage
> >>
> >>
> >> On Tue, 28 Apr 2015, Sage Weil wrote:
> >>
> >> > [adding ceph-devel]
> >> >
> >> > Okay, I see the problem.  This seems to be unrelated ot the giant ->
> >> > hammer move... it's a result of the tiering changes you made:
> >> >
> >> > > > > > > > The following:
> >> > > > > > > >
> >> > > > > > > > ceph osd tier add img images --force-nonempty ceph osd
> >> > > > > > > > tier cache-mode images forward ceph osd tier set-overlay
> >> > > > > > > > img images
> >> >
> >> > Specifically, --force-nonempty bypassed important safety checks.
> >> >
> >> > 1. images had snapshots (and removed_snaps)
> >> >
> >> > 2. images was added as a tier *of* img, and img's removed_snaps was
> >> > copied to images, clobbering the removed_snaps value (see
> >> > OSDMap::Incremental::propagate_snaps_to_tiers)
> >> >
> >> > 3. tiering relation was undone, but removed_snaps was still gone
> >> >
> >> > 4. on OSD startup, when we load the PG, removed_snaps is initialized
> >> > with the older map.  later, in PGPool::update(), we assume that
> >> > removed_snaps alwasy grows (never shrinks) and we trigger an assert.
> >> >
> >> > To fix this I think we need to do 2 things:
> >> >
> >> > 1. make the OSD forgiving out removed_snaps getting smaller.  This is
> >> > probably a good thing anyway: once we know snaps are removed on all
> >> > OSDs we can prune the interval_set in the OSDMap.  Maybe.
> >> >
> >> > 2. Fix the mon to prevent this from happening, *even* when
> >> > --force-nonempty is specified.  (This is the root cause.)
> >> >
> >> > I've opened http://tracker.ceph.com/issues/11493 to track this.
> >> >
> >> > sage
> >> >
> >> >
> >> >
> >> > > > > > > >
> >> > > > > > > > Idea was to make images as a tier to img, move data to img
> >> > > > > > > > then change
> >> > > > > > > clients to use the new img pool.
> >> > > > > > > >
> >> > > > > > > > Br,
> >> > > > > > > > Tuomas
> >> > > > > > > >
> >> > > > > > > > > Can you explain exactly what you mean by:
> >> > > > > > > > >
> >> > > > > > > > > "Also I created one pool for tier to be able to move
> >> > > > > > > > > data without
> >> > > > > > > outage."
> >> > > > > > > > >
> >> > > > > > > > > -Sam
> >> > > > > > > > > - Original Message -
> >> > > > > > > > > From: "tuomas juntunen"
> >> > > > > > > > > 
> >> > > > > > > > > To: "Ian Colle" 
> >> > > > > > > > > Cc: ceph-users@lists.ceph.com
> >> > > > > > > > > Sent: Monday, April 27, 2015 4:23:44 AM
> >> > > > > > > > > Subject: Re: [ceph-users] Upgrade from Giant to Hammer
> >> > > > > > > > > and after some basic operations most of the OSD's went
> >> > > > > > > > > down
> >> > > > > > > > >
> >> > > > > > > > > Hi
> >> > > > > > > > >
> >> > > > > > > > > Any solution for this yet?
> >> > > > > > > > >
> >> > > > > > > > > Br,
> >> > > > > > > > > Tuom

[ceph-users] RHEL7/HAMMER cache tier doesn't flush or evict?

[Don Doerner]

All,

Synopsis: I can't get cache tiering to work in HAMMER on RHEL7.

Process:

1.  Fresh install of HAMMER on RHEL7 went well.

2.  Crush map adapted to provide two "root" level resources

a.   "ctstorage", to use as a cache tier based on very high-performance, 
high IOPS SSD (intrinsic journal).  2 OSDs.

b.  "ecstorage", to use as an erasure-coded poolbased on low-performance, 
cost effective storage (extrinsic journal).  12 OSDs.

3.  Established a pool "ctpool", 32 PGs on ctstorage (pool size = min_size 
= 1).  Ran a quick RADOS bench test, all worked as expected.  Cleaned up.

4.  Established a pool "ecpool", 256 PGs on ecstorage (5+3 profile).  Ran a 
quick RADOS bench test, all worked as expected.  Cleaned up.

5.  Ensured that both pools were empty (i.e., "rados ls" shows no objects)

6.  Put the cache tier on the erasure coded storage (one Bloom hit set, 
interval 900 seconds), set up the overlay.  Used defaults for flushing and 
eviction.  No errors.

7.  Started a 3600-second write test to ecpool.

Objects piled up in ctpool (as expected) - went past the 40% mark (as 
expected), then past the 80% mark (unexpected), then ran into the wall (95% 
full - very unexpected).  Using "rados df", I can see that the cache tier is 
full (duh!) but not one single object lives in the ecpool.  Nothing was ever 
flushed, nothing was ever evicted.  Thought I might be misreading that, so I 
went back to SAR data that I captured during the test: the SSDs were the only 
[ceph] devices that sustained I/O.

I based this experiment on another (much more successful) experiment that I 
performed using GIANT (.1) on RHEL7 a couple of weeks ago (wherein I used RAM 
as a cache tier); that all worked.  It seems there are at least three 
possibilities...

*I forgot a critical step this time around.

*The steps needed for a cache tier in HAMMER are different than the 
steps needed in GIANT (and different than the documentation online).

*There is a problem with HAMMER in the area of cache tier.

Has anyone successfully deployed cache-tiering in HAMMER?  Did you have to do 
anything unusual?  Do you see any steps that I missed?

Regards,

-don-

--
The information contained in this transmission may be confidential. Any 
disclosure, copying, or further distribution of confidential information is not 
permitted unless such privilege is explicitly granted in writing by Quantum. 
Quantum reserves the right to have electronic communications, including email 
and attachments, sent across its networks filtered through anti virus and spam 
software programs and retain such messages in order to comply with applicable 
data security and retention requirements. Quantum is not responsible for the 
proper and complete transmission of the substance of this communication or for 
any delay in its receipt.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-dokan mount error

[James Devine]

So I am trying to get ceph-dokan to work.  Upon running it with
./ceph-dokan.exe -c ceph.conf -l e it indicates there was a mount
error and the monitor it connects to logs cephx server client.admin:
unexpected key: req.key=0 expected_key=d7901d515f6b0c61

According to the debug output attached ceph-dokan reads the config
file and keyring fine.  Any ideas where I might be going wrong?


ceph-dokan.log
Description: Binary data
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[黄文俊]

Oh, I'm sorry, I found that it is the China GWF's problem, because I can
access the site from my new proxy. So sad for the truth!

Thank you!
Wenjun Huang

2015-04-30 18:30 GMT+08:00 Milton Suen 孫文東 :

>   Yes, I think it’s China Great Firewall issue. It works well in Hong
> Kong.
>
>   Thanks & Best Regards,
>
>
>   From: Ray Sun
> Date: Thursday, 30 April, 2015 3:08 pm
> To: Karan Singh
> Cc: 黄文俊, "ceph-users@lists.ceph.com"
> Subject: Re: [ceph-users] Can not access the Ceph's main page ceph.com
> intermittently
>
>I think it's GFW's problem. I think it's ok for me to use a VPN inside
> a wall.
>
>  Best Regards
> -- Ray
>
> On Thu, Apr 30, 2015 at 3:00 PM, Karan Singh  wrote:
>
>> Hi Wenjun
>>
>>  Are you sure , is everything all right at your end.
>>
>>  ceph.com seems to be working fine for me from Finland
>>
>> 
>>  Karan Singh
>>  Systems Specialist , Storage Platforms
>>  CSC - IT Center for Science,
>>  Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
>> mobile: +358 503 812758
>> tel. +358 9 4572001
>> fax +358 9 4572302
>> http://www.csc.fi/
>>  
>>
>>   On 30 Apr 2015, at 09:56, 黄文俊  wrote:
>>
>>Hi, Sage
>>
>>  This is Wenjun Huang from Beijing China, I found that I can not access
>> ceph's main site *ceph.com * intermittently. The issue
>> looks so strange, always I can access the site normally at a time, but I
>> can not access it some seconds later(the site does not response for a very
>> long time and then timeout). The issue has existed some days, so I beg for
>> your help. Could you please help to check the problem. I do not think it is
>> the GFW's problem of China, because I can access it normally sometimes,
>>  but seems like that there is some problem with the site server.
>>
>>  Thanks
>>
>>  Wenjun
>>   ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [cephfs][ceph-fuse] cache size or memory leak?

[Yan, Zheng]

On Thu, Apr 30, 2015 at 4:37 PM, Dexter Xiong  wrote:
> Hi,
> I got these message when I remount:
> 2015-04-30 15:47:58.199837 7f9ad30a27c0 -1 asok(0x3c83480)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17)
> File exists
> fuse: bad mount point `ceph-fuse': No such file or directory
> ceph-fuse[2576]: fuse failed to initialize
> 2015-04-30 15:47:58.199980 7f9ad30a27c0 -1 init, newargv = 0x3ca9b00
> newargc=14
> 2015-04-30 15:47:58.200020 7f9ad30a27c0 -1 fuse_parse_cmdline failed.
> ceph-fuse[2574]: mount failed: (22) Invalid argument.
>
> It seems that FUSE doesn't support remount? This link is google result.
>

please try "echo 3 > /proc/sys/vm/drop_caches". check if the pinned
dentries count drops after executing the command.

Regards
Yan, Zheng

> I am using ceph-dokan too. And I got the similar memory problem. I don't
> know if it is the same problem. I switched to use kernel module and Samba to
> replace previous solution temporarily.
> I'm trying to read and track the ceph & ceph-dokan source code to find
> more useful information.
>
>
> I don't know if my previous email arrived the list(Maybe the attachment
> is too large). Here is its content:
>
> I wrote a test case with Python:
> '''
> import os
> for i in range(200):
> dir_name = '/srv/ceph_fs/test/d%s'%i
> os.mkdir(dir_name)
> for j in range(3):
> with open('%s/%s'%(dir_name, j), 'w') as f:
> f.write('0')
> '''
>
> The output of status command after test on a fresh mount:
> {
> "metadata": {
> "ceph_sha1": "e4bfad3a3c51054df7e537a724c8d0bf9be972ff",
> "ceph_version": "ceph version 0.94.1
> (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)",
> "entity_id": "admin",
> "hostname": "local-share-server",
> "mount_point": "\/srv\/ceph_fs"
> },
> "dentry_count": 204,
> "dentry_pinned_count": 201,
> "inode_count": 802,
> "mds_epoch": 25,
> "osd_epoch": 177,
> "osd_epoch_barrier": 176
> }
> It seems that all pinned dentrys are directories from dump cache command
> output.
>
> Attachment is a package of debug log and dump cache content.
>
>
> On Thu, Apr 30, 2015 at 2:55 PM Yan, Zheng  wrote:
>>
>> On Wed, Apr 29, 2015 at 4:33 PM, Dexter Xiong  wrote:
>> > The output of status command of fuse daemon:
>> > "dentry_count": 128966,
>> > "dentry_pinned_count": 128965,
>> > "inode_count": 409696,
>> > I saw the pinned dentry is nearly the same as dentry.
>> > So I enabled debug log(debug client = 20/20) and  read  Client.cc source
>> > code in general. I found that an entry will not be trimed if it is
>> > pinned.
>> > But how can I unpin dentrys?
>>
>> Maybe these dentries are pinned by fuse kernel module (ceph-fuse does
>> not try trimming kernel cache when its cache size >
>> client_cache_size). Could you please run "mount -o remount > point>", then run the status command again. check if number of pinned
>> dentries drops.
>>
>> Regards
>> Yan, Zheng
>>
>>
>> >
>> > On Wed, Apr 29, 2015 at 12:19 PM Dexter Xiong 
>> > wrote:
>> >>
>> >> I tried set client cache size = 100, but it doesn't solve the problem.
>> >> I tested ceph-fuse with kernel version 3.13.0-24 3.13.0-49 and
>> >> 3.16.0-34.
>> >>
>> >>
>> >>
>> >> On Tue, Apr 28, 2015 at 7:39 PM John Spray 
>> >> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On 28/04/2015 06:55, Dexter Xiong wrote:
>> >>> > Hi,
>> >>> > I've deployed a small hammer cluster 0.94.1. And I mount it via
>> >>> > ceph-fuse on Ubuntu 14.04. After several hours I found that the
>> >>> > ceph-fuse process crashed. The end is the crash log from
>> >>> > /var/log/ceph/ceph-client.admin.log. The memory cost of ceph-fuse
>> >>> > process was huge(more than 4GB) when it crashed.
>> >>> > Then I did some test and found these actions will increase
>> >>> > memory
>> >>> > cost of ceph-fuse rapidly and the memory cost never seem to
>> >>> > decrease:
>> >>> >
>> >>> >   * rsync command to sync small files(rsync -a /mnt/some_small
>> >>> > /srv/ceph)
>> >>> >   * chown command/ chmod command(chmod 775 /srv/ceph -R)
>> >>> >
>> >>> > But chown/chmod command on accessed files will not increase the
>> >>> > memory
>> >>> > cost.
>> >>> > It seems that ceph-fuse caches the file nodes but never releases
>> >>> > them.
>> >>> > I don't know if there is an option to control the cache size. I
>> >>> > set mds cache size = 2147483647 option to improve the performance of
>> >>> > mds, and I tried to set mds cache size = 1000 at client side but it
>> >>> > doesn't effect the result.
>> >>>
>> >>> The setting for client-side cache limit is "client cache size",
>> >>> default
>> >>> is 16384
>> >>>
>> >>> What kernel version are you using on the client?  There have been some
>> >>> issues with cache trimming vs. fuse in recent kernels, but we thought
>> >>> we
>> >>> had workarounds in place...
>> >>>
>>

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[Rafael Coninck Teigão]

  
  
I've got this problem sometimes as well (accessing from Brazil,
BTW).

The site takes a long time to load and I get some timeouts. But when
I try accessing it during the day, it will eventually load. I don't
see this problem when I'm using the EU mirror.


On 30/04/2015 07:30, Milton Suen 孫文東 wrote:

  
  

  Yes, I think it’s China Great Firewall issue. It works
well in Hong Kong.
  
  
  

  Thanks & Best Regards,
  
  
  

  

  
  
  
  

  From: Ray Sun
  Date: Thursday, 30
  April, 2015 3:08 pm
  To: Karan Singh
  Cc: 黄文俊, "ceph-users@lists.ceph.com"
  Subject: Re:
  [ceph-users] Can not access the Ceph's main page ceph.com
  intermittently




  

  
I think it's GFW's problem. I think it's ok for me to
use a VPN inside a wall.


  
Best Regards
-- Ray
  
  
  On Thu, Apr 30, 2015 at 3:00 PM,
Karan Singh 
  
wrote:

  Hi Wenjun


Are you sure , is everything all right at your
  end. 


ceph.com seems
  to be working fine for me from Finland
  

  


Karan
  Singh 
Systems
  Specialist , Storage Platforms
CSC
  - IT Center for Science,
Keilaranta
  14, P. O. Box 405, FIN-02101 Espoo,
  Finland
  mobile: +358 503 812758
  tel. +358
9 4572001
  fax +358
9 4572302
  http://www.csc.fi/

  

  
  
  

  

  On 30 Apr 2015, at 09:56, 黄文俊 
wrote:
  

  
  

  
Hi, Sage
  

  This is Wenjun
  Huang from Beijing China, I found
  that I can not access ceph's main
  site
  ceph.com
  intermittently. The issue looks so
  strange, always I can access the
  site normally at a time, but I can
  not access it some seconds
  later(the site does not response
  for a very long time and then
  timeout). The issue has existed
  some days, so I beg for your help.
  Could you please help to check the
  problem. I do not think it is the
  GFW's problem of China, because I
  can access it normally sometimes,
   but seems like that there is some
  problem with the site server.
  

  Thanks
  

  Wenjun

  

___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] RBD storage pool support in Libvirt not enabled on CentOS

[Jason Dillaman]

The issue appears to be tracked with the following BZ for RHEL 7:  
https://bugzilla.redhat.com/show_bug.cgi?id=1187533

-- 

Jason Dillaman 
Red Hat 
dilla...@redhat.com 
http://www.redhat.com 


- Original Message -
From: "Wido den Hollander" 
To: "Somnath Roy" , ceph-users@lists.ceph.com
Sent: Wednesday, April 29, 2015 3:33:18 PM
Subject: Re: [ceph-users] RBD storage pool support in Libvirt not enabled on 
CentOS

On 04/29/2015 09:24 PM, Somnath Roy wrote:
> Wido,
> Is this true for RHEL as well then ?
> 

I think so. The spec file says FC only for RBD:

%if 0%{?fedora} >= 16
%define with_storage_rbd

While for gluster for example:

%if 0%{?fedora} >= 19 || 0%{?rhel} >= 6
%define with_storage_gluster

I would like to see this enabled in both RHEL and CentOS.

Wido

> Thanks & Regards
> Somnath
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido 
> den Hollander
> Sent: Wednesday, April 29, 2015 12:22 PM
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RBD storage pool support in Libvirt not enabled on 
> CentOS
> 
> On 04/29/2015 12:34 PM, Wido den Hollander wrote:
>> Hi,
>>
>> While working with some CentOS machines I found out that Libvirt
>> currently is not build with RBD storage pool support.
>>
>> While that support has been upstream for a very long time and enabled
>> in Ubuntu as well I was wondering if anybody knew why it isn't enabled
>> on CentOS?
>>
>> Under CentOS 7.1 my libvirt (1.2.8-16.el7) isn't build with RBD
>> storage pool support, which is required by CloudStack.
>>
> 
> I filed a bug in the CentOS tracker for this btw:
> http://bugs.centos.org/view.php?id=8585
> 
>> Any ideas? Would be great to have. Librados and librbd are available,
>> so that's not the issue.
>>
> 
> 
> --
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> PLEASE NOTE: The information contained in this electronic mail message is 
> intended only for the use of the designated recipient(s) named above. If the 
> reader of this message is not the intended recipient, you are hereby notified 
> that you have received this message in error and that any review, 
> dissemination, distribution, or copying of this message is strictly 
> prohibited. If you have received this communication in error, please notify 
> the sender by telephone or e-mail (as shown above) immediately and destroy 
> any and all copies of this message in your possession (whether hard copies or 
> electronically stored copies).
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy with multipath devices

[Dhiraj Kamble]

Hi,

Tried ceph-deploy with multipath devices; seems like it does not recognize such 
devices due to certain hardcoded checks in ceph-disk, etc.
Has anybody got ceph-deploy to work with multipath devices (/dev/mapper/mpath0, 
etc). If so, can you provide the steps.

Regards,
Dhiraj




PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cache Pool Flush/Eviction Limits - Hard of Soft?

[Nick Fisk]

Does anyone know if the Flush and Eviction limits are hard limits, ie as
soon as they are exceeded writes will block, or will the pool only block
when it reaches Target_max_bytes?

 

I'm see really poor performance and frequent requests are blocked messages
once data starts having to be evicted/flushed and I was just wondering if
the above was true.

 

If the limits are soft, I would imagine making high and low target limits
would help:-

 

Target_dirty_bytes_low=.3

Target_dirty_bytes_high=.4

 

Once the amount of dirty bytes passes the low limit a very low priority
flush occurs, if the high limit is reached data is flushed much more
aggressively. The same could also exist for eviction. This will allow burst
of write activity to occur before flushing starts heavily impacting
performance.

 

Nick




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Hammer

[flisky]

I think you can take a look at [ceph-ansible][1], and the 
[rolling_update][2] particularly.


And in the upgrade, every thing goes smoothly, except the [dirty data 
issue][3] bugged me a lot.


[1]: https://github.com/ceph/ceph-ansible
[2]: https://github.com/ceph/ceph-ansible/blob/master/rolling_update.yml
[3]: http://tracker.ceph.com/issues/11373

On 2015年04月30日 17:52, Patrik Plank wrote:

Hi cephler,


how critical is the Upgrade from 0.87.1 to 0.94.1!

What is the "best practice method" to upgrade the cluster??


thx

best regards




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[Milton Suen 孫文東]

Yes, I think it’s China Great Firewall issue. It works well in Hong Kong.

Thanks & Best Regards,

[cid:4EABB7FB-72B4-47CB-AF26-E521060C19B8]

From: Ray Sun
Date: Thursday, 30 April, 2015 3:08 pm
To: Karan Singh
Cc: 黄文俊, "ceph-users@lists.ceph.com"
Subject: Re: [ceph-users] Can not access the Ceph's main page ceph.com 
intermittently

I think it's GFW's problem. I think it's ok for me to use a VPN inside a wall.

Best Regards
-- Ray

On Thu, Apr 30, 2015 at 3:00 PM, Karan Singh 
mailto:karan.si...@csc.fi>> wrote:
Hi Wenjun

Are you sure , is everything all right at your end.

ceph.com seems to be working fine for me from Finland


Karan Singh
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 30 Apr 2015, at 09:56, 黄文俊 
mailto:huangwenjun...@gmail.com>> wrote:

Hi, Sage

This is Wenjun Huang from Beijing China, I found that I can not access ceph's 
main site ceph.com intermittently. The issue looks so 
strange, always I can access the site normally at a time, but I can not access 
it some seconds later(the site does not response for a very long time and then 
timeout). The issue has existed some days, so I beg for your help. Could you 
please help to check the problem. I do not think it is the GFW's problem of 
China, because I can access it normally sometimes,  but seems like that there 
is some problem with the site server.

Thanks

Wenjun
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache Pool PG Split

[Nick Fisk]

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Sage Weil
> Sent: 29 April 2015 17:46
> To: Nick Fisk
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Cache Pool PG Split
>
> On Wed, 29 Apr 2015, Nick Fisk wrote:
> > Hi All,
> >
> > When trying to increase the number of PG's of a cache pool I get the
> > warning message about running a scrub afterwards and being careful
> > about not overfilling the pool. I've also looked at this issue to
> > better understand the underlying cause:
> > http://tracker.ceph.com/issues/8043
> >
> > Am I best to lower the target_max_bytes and minimising IO prior to
> > doing the PG split to avoid filling the pool? Is there anything else I
> > should be aware of?
>
> It depends on how quickly to expect it to fill up.  If your disks are close 
> to full,
> then yeah, give yourself some extra space.

Thanks for your reply Sage, I have started expanding in small increments and am 
not seeing any adverse effects.

>
> The OSDs will automatically trigger an immediate scrub of the split PGs, so
> the window shouldn't be too long.  Try doing a small split (increase pg_num
> by 1) and see how long it takes the PG to split and scrub.
>
> sage
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Nick Fisk
Technical Support Engineer

System Professional Ltd
tel: 01825 83
mob: 07711377522
fax: 01825 830001
mail: nick.f...@sys-pro.co.uk
web: www.sys-pro.co.uk

IT SUPPORT SERVICES | VIRTUALISATION | STORAGE | BACKUP AND DR | IT CONSULTING

Registered Office:
Wilderness Barns, Wilderness Lane, Hadlow Down, East Sussex, TN22 4HU
Registered in England and Wales.
Company Number: 04754200


Confidentiality: This e-mail and its attachments are intended for the above 
named only and may be confidential. If they have come to you in error you must 
take no action based on them, nor must you copy or show them to anyone; please 
reply to this e-mail and highlight the error.

Security Warning: Please note that this e-mail has been created in the 
knowledge that Internet e-mail is not a 100% secure communications medium. We 
advise that you understand and observe this lack of security when e-mailing us.

Viruses: Although we have taken steps to ensure that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free. 
Any views expressed in this e-mail message are those of the individual and not 
necessarily those of the company or any of its subsidiaries.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cache pool parameters and pressure

[Nick Fisk]

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Kenneth Waegeman
> Sent: 30 April 2015 10:03
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] cache pool parameters and pressure
> 
> Hi all,
> 
> I have some question related to the caching layer. I am using the latest
> version of ceph: 0.94.1. I created the ceph pool with this options:
> 
> ceph osd tier add ecdata cache
> ceph osd tier cache-mode cache writeback ceph osd tier set-overlay ecdata
> cache ceph osd pool set cache hit_set_type bloom ceph osd pool set cache
> hit_set_count 1 ceph osd pool set cache hit_set_period 3600 ceph osd pool
> set cache target_max_bytes $((300*1024*1024*1024))
> 
> I checked these values were set correctly
> 
> I also checked some other parameters:
> 
> ceph osd pool get cache cache_target_dirty_ratio
> cache_target_dirty_ratio: 0
> ceph osd pool get cache cache_target_full_ratio
> cache_target_full_ratio: 0
> get cache cache_min_flush_age
> cache_min_flush_age: 0
> ceph osd pool get cache cache_min_evict_age
> cache_min_evict_age: 0

These control when data is flushed/evicted out of the pool. By having them
set at 0, you will be getting what you are seeing where data is immediately
flushed.

Try setting the dirty ratio to 0.4 and the target to 0.8 for starters. This
will flush dirty objects when 40% full and all objects when 80% full.

The min age parameters control how long a object must remain in the tier
before it is eligble for being flushed/evicted


> 
> My first question: What do these zero values mean? Is this really zero, or
> some default value. I don't find defaults in the documentation.
> 
> Now a strange thing is: at first the cache pool was being filled until it
was near
> the target_max_bytes value of 300G. But now (about 20hours
> later) I check again and there is constantly only about 2G of data in the
cache
> pool, so it flushes everything immediately..
> 
> Is this the result of a parameter? Can the strange behaviour of an initial
filling
> cache that after it started flushing the first time keeps on flushing
everything
> be explained?
> 
> So the cache is empty, but I get warning when I check the health:
>   health HEALTH_WARN
>  mds0: Client cephtst.cubone.os failing to respond to cache
pressure
> 
> Someone an idea what is happening here?
> 
> Thank you very much!
> 
> Kenneth
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Upgrade to Hammer

[Patrik Plank]

Hi cephler, 



how critical is the Upgrade from 0.87.1 to 0.94.1!

What is the "best practice method" to upgrade the cluster??



thx

best regards



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cost- and Powerefficient OSD-Nodes

[Frank Brendel]

If you don't need LACP you could use round-robin bonding mode.
With 4x1Gbit NICs you can get a bandwidth of 4Gbit per TCP connection.

Either create trunks on stacked switches (e.g. Avaya) or use single 
switches (e.g. HP 1810-24) and a locally managed MAC address per node/bond.
The latter is somewhat sophisticated because you have to monitor the 
NICs. If the link or the NIC itself on one node goes down you have to 
remove the corresponding slaves on the other nodes.


To make it really cheap you could configure VLANs to get 4Gbit or more 
per connection with two switches, for testing of course ;-)




Am 28.04.2015 um 23:01 schrieb Patrick Hahn:
I haven't used them myself but switching silicon is getting pretty 
cheap nowadays:


http://whiteboxswitch.com/products/edge-core-as5610-52x

There's similar products (basically the same Broadcom ASIC) from 
Quanta and I think Supermicro announced one recently as well.


They're not as plug and play since they run linux but you have a lot 
more options for tweaking.


Thanks,
Patrick

On Tue, Apr 28, 2015 at 4:35 PM, Dominik Hannen > wrote:


> FYI, most Juniper switches hash LAGs on IP+port, so you'd get somewhat
> better performance than you would with simple MAC or IP
hashing.  10G is
> better if you can afford it, though.

interesting, I just read up about the topic, those
Juniper-Switches seem to
be a nice pick then.
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Patrick Hahn


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cache pool parameters and pressure

[Kenneth Waegeman]

Hi all,

I have some question related to the caching layer. I am using the latest 
version of ceph: 0.94.1. I created the ceph pool with this options:


ceph osd tier add ecdata cache
ceph osd tier cache-mode cache writeback
ceph osd tier set-overlay ecdata cache
ceph osd pool set cache hit_set_type bloom
ceph osd pool set cache hit_set_count 1
ceph osd pool set cache hit_set_period 3600
ceph osd pool set cache target_max_bytes $((300*1024*1024*1024))

I checked these values were set correctly

I also checked some other parameters:

ceph osd pool get cache cache_target_dirty_ratio
cache_target_dirty_ratio: 0
ceph osd pool get cache cache_target_full_ratio
cache_target_full_ratio: 0
get cache cache_min_flush_age
cache_min_flush_age: 0
ceph osd pool get cache cache_min_evict_age
cache_min_evict_age: 0

My first question: What do these zero values mean? Is this really zero, 
or some default value. I don't find defaults in the documentation.


Now a strange thing is: at first the cache pool was being filled until 
it was near the target_max_bytes value of 300G. But now (about 20hours 
later) I check again and there is constantly only about 2G of data in 
the cache pool, so it flushes everything immediately..


Is this the result of a parameter? Can the strange behaviour of an 
initial filling cache that after it started flushing the first time 
keeps on flushing everything be explained?


So the cache is empty, but I get warning when I check the health:
 health HEALTH_WARN
mds0: Client cephtst.cubone.os failing to respond to cache 
pressure


Someone an idea what is happening here?

Thank you very much!

Kenneth

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[flisky]

Second this. An VPN works when you cannot open ceph.com.

And FYI, you can use 'eu.ceph.com' to download packages when GFW sucks.

On 2015年04月30日 15:08, Ray Sun wrote:

I think it's GFW's problem. I think it's ok for me to use a VPN inside a
wall.

Best Regards
-- Ray

On Thu, Apr 30, 2015 at 3:00 PM, Karan Singh
mailto:karan.si...@csc.fi>> wrote:

Hi Wenjun

Are you sure , is everything all right at your end.

ceph.com  seems to be working fine for me from Finland


Karan Singh
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001 
fax +358 9 4572302 
http://www.csc.fi/



On 30 Apr 2015, at 09:56, 黄文俊
mailto:huangwenjun...@gmail.com>>
wrote:

Hi, Sage

This is Wenjun Huang from Beijing China, I found that I can not
access ceph's main site *ceph.com *
intermittently. The issue looks so strange, always I can access
the site normally at a time, but I can not access it some seconds
later(the site does not response for a very long time and then
timeout). The issue has existed some days, so I beg for your help.
Could you please help to check the problem. I do not think it is
the GFW's problem of China, because I can access it normally
sometimes,  but seems like that there is some problem with the
site server.

Thanks

Wenjun
___
ceph-users mailing list
ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Fuse Crashed when Reading and How to Backup the data

[flisky]

Sorry，I cannot reproduce the "Operation not permitted" log

Here is a small portion of log with "debug_client = 20/20"
==
   -22> 2015-04-30 16:29:12.858309 7fe9757f2700 10 client.58272 
check_caps on 115.head(ref=2 ll_ref=10 cap_refs={} open={1=1} 
mode=100664 size=119/0 mtime=2015-04-20 14:14:57.961482 
caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[115 ts 0/0 objects 0 
dirty_or_tx 0] parents=0x7fe968022980 0x7fe968021c30) wanted pFscr used 
- is_delayed=1
   -21> 2015-04-30 16:29:12.858326 7fe9757f2700 10 client.58272  cap 
mds.0 issued pAsLsXsFscr implemented pAsLsXsFscr revoking -
   -20> 2015-04-30 16:29:12.858333 7fe9757f2700 10 client.58272 
send_cap 115.head(ref=2 ll_ref=10 cap_refs={} open={1=1} 
mode=100664 size=119/0 mtime=2015-04-20 14:14:57.961482 
caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[115 ts 0/0 objects 0 
dirty_or_tx 0] parents=0x7fe968022980 0x7fe968021c30) mds.0 seq 1 used - 
want pFscr flush - retain pAsxLsxXsxFsxcrwbl held pAsLsXsFscr revoking - 
dropping -
   -19> 2015-04-30 16:29:12.858358 7fe9757f2700 15 client.58272 auth 
cap, setting max_size = 0
   -18> 2015-04-30 16:29:12.858368 7fe9757f2700 10 client.58272 
_create_fh 115 mode 1
   -17> 2015-04-30 16:29:12.858376 7fe9757f2700 20 client.58272 
trim_cache size 14 max 16384
   -16> 2015-04-30 16:29:12.858378 7fe9757f2700  3 client.58272 ll_open 
115.head 32768 = 0 (0x7fe95c0052c0)
   -15> 2015-04-30 16:29:12.858385 7fe9757f2700  3 client.58272 
ll_forget 115 1
   -14> 2015-04-30 16:29:12.858386 7fe9757f2700 20 client.58272 _ll_put 
0x7fe968021c30 115 1 -> 9
   -13> 2015-04-30 16:29:12.858500 7fe974ff1700 20 client.58272 _ll_get 
0x7fe968021c30 115 -> 10
   -12> 2015-04-30 16:29:12.858503 7fe974ff1700  3 client.58272 
ll_getattr 115.head
   -11> 2015-04-30 16:29:12.858505 7fe974ff1700 10 client.58272 
_getattr mask pAsLsXsFs issued=1
   -10> 2015-04-30 16:29:12.858509 7fe974ff1700 10 client.58272 
fill_stat on 115 snap/devhead mode 0100664 mtime 2015-04-20 
14:14:57.961482 ctime 2015-04-20 14:14:57.960359
-9> 2015-04-30 16:29:12.858518 7fe974ff1700  3 client.58272 
ll_getattr 115.head = 0
-8> 2015-04-30 16:29:12.858525 7fe974ff1700  3 client.58272 
ll_forget 115 1
-7> 2015-04-30 16:29:12.858526 7fe974ff1700 20 client.58272 _ll_put 
0x7fe968021c30 115 1 -> 9
-6> 2015-04-30 16:29:12.858536 7fe9577fe700  3 client.58272 ll_read 
0x7fe95c0052c0 115  0~4096
-5> 2015-04-30 16:29:12.858539 7fe9577fe700 10 client.58272 
get_caps 115.head(ref=3 ll_ref=9 cap_refs={} open={1=1} 
mode=100664 size=119/0 mtime=2015-04-20 14:14:57.961482 
caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[115 ts 0/0 objects 0 
dirty_or_tx 0] parents=0x7fe968022980 0x7fe968021c30) have pAsLsXsFscr 
need Fr want Fc but not Fc revoking -
-4> 2015-04-30 16:29:12.858561 7fe9577fe700 10 client.58272 
_read_async 115.head(ref=3 ll_ref=9 cap_refs={2048=1} open={1=1} 
mode=100664 size=119/0 mtime=2015-04-20 14:14:57.961482 
caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[115 ts 0/0 objects 0 
dirty_or_tx 0] parents=0x7fe968022980 0x7fe968021c30) 0~4096
-3> 2015-04-30 16:29:12.858575 7fe9577fe700 10 client.58272 
max_byes=0 max_periods=4
-2> 2015-04-30 16:29:12.858692 7fe9577fe700  5 client.58272 
get_cap_ref got first FILE_CACHE ref on 115.head(ref=3 ll_ref=9 
cap_refs={1024=0,2048=1} open={1=1} mode=100664 size=119/0 
mtime=2015-04-20 14:14:57.961482 caps=pAsLsXsFscr(0=pAsLsXsFscr) 
objectset[115 ts 0/0 objects 1 dirty_or_tx 0] 
parents=0x7fe968022980 0x7fe968021c30)
-1> 2015-04-30 16:29:12.867657 7fe9797fa700 10 client.58272 
ms_handle_connect on 172.16.3.149:6823/982446
 0> 2015-04-30 16:29:12.872787 7fe97bfff700 -1 *** Caught signal 
(Aborted) **



On 2015年04月30日 16:21, flisky wrote:

When I read the file through the ceph-fuse, the process crashed.

Here is the log -

terminate called after throwing an instance of
'ceph::buffer::end_of_buffer'
   what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
  in thread 7fe0814d3700
  ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
  1: (()+0x249805) [0x7fe08670b805]
  2: (()+0x10d10) [0x7fe085c39d10]
  3: (gsignal()+0x37) [0x7fe0844d3267]
  4: (abort()+0x16a) [0x7fe0844d4eca]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7fe084de706d]
  6: (()+0x5eee6) [0x7fe084de4ee6]
  7: (()+0x5ef31) [0x7fe084de4f31]
  8: (()+0x5f149) [0x7fe084de5149]
  9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned
int, unsigned int)+0x24b) [0x7fe08688993b]
  10: (ObjectCacher::_readx(ObjectCacher::OSDRead*,
ObjectCacher::ObjectSet*, Context*, bool)+0x1423) [0x7fe0866c6b73]
  11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7fe0866cd870]
  12: (Context::complete(int)+0x9) [0x7fe086687eb9]
  13: (void finish_contexts(CephContext*, s

Re: [ceph-users] [cephfs][ceph-fuse] cache size or memory leak?

[Dexter Xiong]

Hi,
I got these message when I remount:
2015-04-30 15:47:58.199837 7f9ad30a27c0 -1 asok(0x3c83480)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17)
File exists
fuse: bad mount point `ceph-fuse': No such file or directory
ceph-fuse[2576]: fuse failed to initialize
2015-04-30 15:47:58.199980 7f9ad30a27c0 -1 init, newargv = 0x3ca9b00
newargc=14
2015-04-30 15:47:58.200020 7f9ad30a27c0 -1 fuse_parse_cmdline failed.
ceph-fuse[2574]: mount failed: (22) Invalid argument.

It seems that FUSE doesn't support remount? This link
 is google
result.

I am using ceph-dokan too. And I got the similar memory problem. I
don't know if it is the same problem. I switched to use kernel module and
Samba to replace previous solution temporarily.
I'm trying to read and track the ceph & ceph-dokan source code to find
more useful information.


I don't know if my previous email arrived the list(Maybe the attachment
is too large). Here is its content:

I wrote a test case with Python:
'''
import os
for i in range(200):
dir_name = '/srv/ceph_fs/test/d%s'%i
os.mkdir(dir_name)
for j in range(3):
with open('%s/%s'%(dir_name, j), 'w') as f:
f.write('0')
'''

The output of status command after test on a fresh mount:
{
"metadata": {
"ceph_sha1": "e4bfad3a3c51054df7e537a724c8d0bf9be972ff",
"ceph_version": "ceph version 0.94.1
(e4bfad3a3c51054df7e537a724c8d0bf9be972ff)",
"entity_id": "admin",
"hostname": "local-share-server",
"mount_point": "\/srv\/ceph_fs"
},
"dentry_count": 204,
"dentry_pinned_count": 201,
"inode_count": 802,
"mds_epoch": 25,
"osd_epoch": 177,
"osd_epoch_barrier": 176
}
It seems that all pinned dentrys are directories from dump cache command
output.

Attachment
 is a
package of debug log and dump cache content.


On Thu, Apr 30, 2015 at 2:55 PM Yan, Zheng  wrote:

> On Wed, Apr 29, 2015 at 4:33 PM, Dexter Xiong  wrote:
> > The output of status command of fuse daemon:
> > "dentry_count": 128966,
> > "dentry_pinned_count": 128965,
> > "inode_count": 409696,
> > I saw the pinned dentry is nearly the same as dentry.
> > So I enabled debug log(debug client = 20/20) and  read  Client.cc source
> > code in general. I found that an entry will not be trimed if it is
> pinned.
> > But how can I unpin dentrys?
>
> Maybe these dentries are pinned by fuse kernel module (ceph-fuse does
> not try trimming kernel cache when its cache size >
> client_cache_size). Could you please run "mount -o remount  point>", then run the status command again. check if number of pinned
> dentries drops.
>
> Regards
> Yan, Zheng
>
>
> >
> > On Wed, Apr 29, 2015 at 12:19 PM Dexter Xiong 
> wrote:
> >>
> >> I tried set client cache size = 100, but it doesn't solve the problem.
> >> I tested ceph-fuse with kernel version 3.13.0-24 3.13.0-49 and
> 3.16.0-34.
> >>
> >>
> >>
> >> On Tue, Apr 28, 2015 at 7:39 PM John Spray 
> wrote:
> >>>
> >>>
> >>>
> >>> On 28/04/2015 06:55, Dexter Xiong wrote:
> >>> > Hi,
> >>> > I've deployed a small hammer cluster 0.94.1. And I mount it via
> >>> > ceph-fuse on Ubuntu 14.04. After several hours I found that the
> >>> > ceph-fuse process crashed. The end is the crash log from
> >>> > /var/log/ceph/ceph-client.admin.log. The memory cost of ceph-fuse
> >>> > process was huge(more than 4GB) when it crashed.
> >>> > Then I did some test and found these actions will increase memory
> >>> > cost of ceph-fuse rapidly and the memory cost never seem to decrease:
> >>> >
> >>> >   * rsync command to sync small files(rsync -a /mnt/some_small
> >>> > /srv/ceph)
> >>> >   * chown command/ chmod command(chmod 775 /srv/ceph -R)
> >>> >
> >>> > But chown/chmod command on accessed files will not increase the
> memory
> >>> > cost.
> >>> > It seems that ceph-fuse caches the file nodes but never releases
> them.
> >>> > I don't know if there is an option to control the cache size. I
> >>> > set mds cache size = 2147483647 option to improve the performance of
> >>> > mds, and I tried to set mds cache size = 1000 at client side but it
> >>> > doesn't effect the result.
> >>>
> >>> The setting for client-side cache limit is "client cache size", default
> >>> is 16384
> >>>
> >>> What kernel version are you using on the client?  There have been some
> >>> issues with cache trimming vs. fuse in recent kernels, but we thought
> we
> >>> had workarounds in place...
> >>>
> >>> Cheers,
> >>> John
> >>>
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@

[ceph-users] Ceph Fuse Crashed when Reading and How to Backup the data

[flisky]

When I read the file through the ceph-fuse, the process crashed.

Here is the log -

terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7fe0814d3700
 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: (()+0x249805) [0x7fe08670b805]
 2: (()+0x10d10) [0x7fe085c39d10]
 3: (gsignal()+0x37) [0x7fe0844d3267]
 4: (abort()+0x16a) [0x7fe0844d4eca]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7fe084de706d]
 6: (()+0x5eee6) [0x7fe084de4ee6]
 7: (()+0x5ef31) [0x7fe084de4f31]
 8: (()+0x5f149) [0x7fe084de5149]
 9: (ceph::buffer::list::substr_of(ceph::buffer::list const&, unsigned 
int, unsigned int)+0x24b) [0x7fe08688993b]
 10: (ObjectCacher::_readx(ObjectCacher::OSDRead*, 
ObjectCacher::ObjectSet*, Context*, bool)+0x1423) [0x7fe0866c6b73]

 11: (ObjectCacher::C_RetryRead::finish(int)+0x20) [0x7fe0866cd870]
 12: (Context::complete(int)+0x9) [0x7fe086687eb9]
 13: (void finish_contexts(CephContext*, std::liststd::allocator >&, int)+0xac) [0x7fe0866ca73c]
 14: (ObjectCacher::bh_read_finish(long, sobject_t, unsigned long, 
long, unsigned long, ceph::buffer::list&, int, bool)+0x29e) [0x7fe0866bfd2e]

 15: (ObjectCacher::C_ReadFinish::finish(int)+0x7f) [0x7fe0866cc85f]
 16: (Context::complete(int)+0x9) [0x7fe086687eb9]
 17: (C_Lock::finish(int)+0x29) [0x7fe086688269]
 18: (Context::complete(int)+0x9) [0x7fe086687eb9]
 19: (Finisher::finisher_thread_entry()+0x1b4) [0x7fe08671f184]
 20: (()+0x76aa) [0x7fe085c306aa]
 21: (clone()+0x6d) [0x7fe0845a4eed]
=
Some part may be interesting -
   -11> 2015-04-30 15:55:59.063828 7fd6a816c700 10 -- 
172.30.11.188:0/10443 >> 172.16.3.153:6820/1532355 pipe(0x7fd6740344c0 
sd=8 :58596 s=2 pgs=3721 cs=1 l=1 c=0x7fd674038760).reader got message 1 
0x7fd65c001940 osd_op_reply(1 119. [read 0~4390] v0'0 
uv0 ack = -1 ((1) Operation not permitted)) v6
   -10> 2015-04-30 15:55:59.063848 7fd6a816c700  1 -- 
172.30.11.188:0/10443 <== osd.9 172.16.3.153:6820/1532355 1  
osd_op_reply(1 119. [read 0~4390] v0'0 uv0 ack = -1 ((1) 
Operation not permitted)) v6  187+0+0 (689339676 0 0) 0x7fd65c001940 
con 0x7fd674038760



And the cephfs-journal seems okay.

Could anyone tell me why it is happening?

And more important, does Ceph offer any tool to export a cephfs data 
from underlaid pools?


Thanks very much!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw : Cannot set a new region as default

[10 minus]

Hi,

I am in the process of setting up radosgw for a firefly ceph-cluster.

I have followed the docs for creating alternate region region map and
zones.

Now I want to delete the default region .

Is it possible to do that ?

Also I'm not able to promote my new region regeion1 as default region.


--snip--
 radosgw-admin region-map set --infile region-map.json
{ "regions": [
{ "key": "default",
  "val": { "name": "region1",
  "api_name": "",
  "is_master": "true",
  "endpoints": [],
  "master_zone": "",
  "zones": [
{ "name": "zn1",
  "endpoints": [],
  "log_meta": "false",
  "log_data": "false"}],
  "placement_targets": [
{ "name": "default-placement",
  "tags": []}],
  "default_placement": "default-placement"}}],
  "master_region": "region1",
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "user_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -radosgw-admin regionmap updaten-map.json
{ "regions": [
{ "key": "default",
  "val": { "name": "default",
  "api_name": "",
  "is_master": "false",
  "endpoints": [],
  "master_zone": "",
  "zones": [
{ "name": "default",
  "endpoints": [],
  "log_meta": "false",
  "log_data": "false"}],
  "placement_targets": [
{ "name": "default-placement",
  "tags": []}],
  "default_placement": "default-placement"}},
{ "key": "region1",
  "val": { "name": "region1",
  "api_name": "",
  "is_master": "true",
  "endpoints": [],
  "master_zone": "",
  "zones": [
{ "name": "zn1",
  "endpoints": [],
  "log_meta": "false",
  "log_data": "false"}],
  "placement_targets": [
{ "name": "default-placement",
  "tags": []}],
  "default_placement": "default-placement"}}],
  "master_region": "region1",
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "user_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1}}

## region list

radosgw-admin regions list
{ "default_info": { "default_region": "default"},
  "regions": [
"default",
"region1"]}

## set region1 as default

radosgw-admin region default region1
[root@cc03 ceph-rgw]# radosgw-admin regions list
{ "default_info": { "default_region": "default"},
  "regions": [
"default",
"region1"]}


--snip--

any pointers to fix will be great
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] can't delete buckets in radosgw after i recreated the radosgw pools

[Makkelie, R (ITCDCC) - KLM]

okay i used the following python client
https://github.com/AlexYangYu/s3-cli

and created a bucket with the command
s3-cli create_bucket MYBUCKET

then with radosgw-admin i tried a fix with the following command



radosgw-admin --uid=chef bucket check --bucket=MYBUCKET--fix

and did the trick i can now add objects to my buckets again and can delete the 
buckets again


-Original Message-
From: Colin Corr 
mailto:colin%20corr%20%3cco...@pc-doctor.com%3e>>
To: ceph-users@lists.ceph.com 
mailto:%22ceph-us...@lists.ceph.com%22%20%3cceph-us...@lists.ceph.com%3e>>
Subject: Re: [ceph-users] can't delete buckets in radosgw after i recreated the 
radosgw pools
Date: Wed, 29 Apr 2015 12:35:38 -0700




On 04/29/2015 07:55 AM, Makkelie, R (ITCDCC) - KLM wrote:
> i first had some major disaster i had 12 incomplete pgs that couldn't be 
> fixed. (due to several harddisk failures at once)
> alls these incomplete pgs where all in the ".rgw" and ".rgw.buckets" pools
>
> so the only option i could think of is to take my loses and delete and 
> recreate those pools.
>
> the only issue now is that radosgw still think there is data because its 
> showing my old directories
>
> i tried:
> - deleting bucket with recursive option
> - scan and fix buckets
> - delete user
>
> and i can't do any of these things because he wants to remove them but it 
> can't because it receives a 404
> i even upgraded to the latest giant release but no luck so far
>
> does any one have a idea?
>
> greetz
> Ramonskie
>
>

I encountered the same problem while testing different recovery scenarios. What 
worked for me:

Create a bucket (directory) with the same name (as an existing deleted bucket)
Then, Delete the bucket

I used the Python S3 API to do it because the non-existent buckets were only 
seen via the S3 API and the radosgw-admin could not see them anymore.

hope that helps,
Colin

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[Ray Sun]

I think it's GFW's problem. I think it's ok for me to use a VPN inside a
wall.

Best Regards
-- Ray

On Thu, Apr 30, 2015 at 3:00 PM, Karan Singh  wrote:

> Hi Wenjun
>
> Are you sure , is everything all right at your end.
>
> ceph.com seems to be working fine for me from Finland
>
> 
> Karan Singh
> Systems Specialist , Storage Platforms
> CSC - IT Center for Science,
> Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
> mobile: +358 503 812758
> tel. +358 9 4572001
> fax +358 9 4572302
> http://www.csc.fi/
> 
>
> On 30 Apr 2015, at 09:56, 黄文俊  wrote:
>
> Hi, Sage
>
> This is Wenjun Huang from Beijing China, I found that I can not access
> ceph's main site *ceph.com * intermittently. The issue
> looks so strange, always I can access the site normally at a time, but I
> can not access it some seconds later(the site does not response for a very
> long time and then timeout). The issue has existed some days, so I beg for
> your help. Could you please help to check the problem. I do not think it is
> the GFW's problem of China, because I can access it normally sometimes,
>  but seems like that there is some problem with the site server.
>
> Thanks
>
> Wenjun
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[Karan Singh]

Hi Wenjun

Are you sure , is everything all right at your end. 

ceph.com  seems to be working fine for me from Finland


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


> On 30 Apr 2015, at 09:56, 黄文俊  wrote:
> 
> Hi, Sage
> 
> This is Wenjun Huang from Beijing China, I found that I can not access ceph's 
> main site ceph.com  intermittently. The issue looks so 
> strange, always I can access the site normally at a time, but I can not 
> access it some seconds later(the site does not response for a very long time 
> and then timeout). The issue has existed some days, so I beg for your help. 
> Could you please help to check the problem. I do not think it is the GFW's 
> problem of China, because I can access it normally sometimes,  but seems like 
> that there is some problem with the site server.
> 
> Thanks
> 
> Wenjun
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com