Hi,
We’ve been operating a ceph storage system storing files using librados (using
a replicated pool on rust disks). We implemented a cache over the top of this
with SSDs, however we now want to turn this off.
The documentation suggests setting the cache mode to forward before draining
the poo
Hello,
I'm running preliminary test on cache tier removal on a live cluster,
before I try to do that on a production one. I'm trying to avoid
downtime, but from what I noticed it's either impossible or I'm doing
something wrong. My cluster is running Centos 7.2 and 0.94.9 ceph.
Example 1:
I'm s
Hi,
we have a 6 node ceph 10.2.3 cluster on centos 7.2 servers, currently no
hosting any rbds or anything else. MONs are on the OSD nodes.
My question is as centos 7.3 is out now for some time and there is a
ceph update to 10.2.5 available, what would be a good or the best path
to update everythi
Hello Nick,
Thank you for your help. We have contacted RedHat for additional help
and they think this bug is related to gmt bug in Version 94.7 of ceph.
I'm not really sure how can this be as the cluster was using 94.6/94.9
versions. After a month + of slowly moving data I'm up with all the
same
Hi,
I see the following on my Ubuntu / Jewel ceph cluster:
root@sto-1-2:~# radosgw-admin -n client.rgw.sto-1-2 pools list
2017-01-09 16:27:09.645018 7f365bc2ba00 0 RGWZoneParams::create():
error creating default zone params: (17) File exists
could not list placement set: (2) No such file or d
> Why would you still be using journals when running fully OSDs on SSDs?
In our case, we use cheaper large SSDs for the data (Samsung 850 Pro 2TB),
whose performance is excellent in the cluster, but as has been pointed out in
this thread can lose data if power is suddenly removed.
We therefore
Hi all,
Recently I added a new node with new osds to my cluster, which, of course
resulted in backfilling. At the end, there are 4 pgs left in the state 4
active+remapped and I don’t know what to do.
Here is how my cluster looks like currently:
ceph -s
health HEALTH_WARN
4 p
Yes, it doesn't cause issues, but I don't see any way to "repair" the
problem. One possible idea that I might do eventually if no solution is
found is to copy the CephFS files in question and remove the ones with
inconsistencies (which should remove the underlying rados objects). But
it'd be
On Tue, Jan 10, 2017 at 8:23 AM, Marcus Müller
wrote:
> Hi all,
>
> Recently I added a new node with new osds to my cluster, which, of course
> resulted in backfilling. At the end, there are 4 pgs left in the state 4
> active+remapped and I don’t know what to do.
>
> Here is how my cluster looks
> Trying google with "ceph pg stuck in active and remapped" points to a couple
> of post on this ML typically indicating that it's a problem with the CRUSH
> map and ceph being unable to satisfy the mapping rules. Your ceph -s output
> indicates that your using replication of size 3 in your pool
There is currently a thread about this very issue on the ceph-devel
mailing list (check archives for "PG stuck unclean after
rebalance-by-weight" in the last few days.
Have a read of
http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
and try bumping choose_total_tries up t
On Tue, Jan 10, 2017 at 10:22 AM, Marcus Müller
wrote:
> Trying google with "ceph pg stuck in active and remapped" points to a
> couple of post on this ML typically indicating that it's a problem with the
> CRUSH map and ceph being unable to satisfy the mapping rules. Your ceph -s
> output indica
On 9-1-2017 18:46, Oliver Humpage wrote:
>
>> Why would you still be using journals when running fully OSDs on
>> SSDs?
>
> In our case, we use cheaper large SSDs for the data (Samsung 850 Pro
> 2TB), whose performance is excellent in the cluster, but as has been
> pointed out in this thread can
According to the output you provided previously, OSD.51/90 might have
unfound object. To shuffle object again, you could do:
# ceph osd set noout; ceph osd set nodown
# systemctl restart ceph-osd@51
* wait for OSD.51's process to be up
# systemctl restart ceph-osd@90
* wait for OSD.90's p
Hi Willem, the SSDs are probably fine for backing OSDs, it's the O_DSYNC
writes they tend to lie about.
They may have a failure rate higher than enterprise-grade SSDs, but are
otherwise suitable for use as OSDs if journals are placed elsewhere.
On Mon, Jan 9, 2017 at 2:39 PM, Willem Jan Withagen
Sorry for spam... I meant D_SYNC.
On Mon, Jan 9, 2017 at 2:56 PM, Brian Andrus
wrote:
> Hi Willem, the SSDs are probably fine for backing OSDs, it's the O_DSYNC
> writes they tend to lie about.
>
> They may have a failure rate higher than enterprise-grade SSDs, but are
> otherwise suitable for u
On Wed, Jan 4, 2017 at 11:59 PM, Craig Chi wrote:
> Hi ,
>
> I'm glad to know that it happened not only to me.
> Though it is unharmful, it seems like kind of bug...
> Are there any Ceph developers who know how exactly is the implementation of
> "ceph osd perf" command?
> Is the leap second really
> pg 9.7 is stuck unclean for 512936.160212, current state active+remapped,
> last acting [7,3,0]
> pg 7.84 is stuck unclean for 512623.894574, current state active+remapped,
> last acting [4,8,1]
> pg 8.1b is stuck unclean for 513164.616377, current state active+remapped,
> last acting [4,7,2]
On 9-1-2017 23:58, Brian Andrus wrote:
> Sorry for spam... I meant D_SYNC.
That term does not run any lights in Google...
So I would expect it has to O_DSYNC.
(https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/)
Now you tell me there is a SS
Last week I decided to play around with Kraken (11.1.1-1xenial) on a
single node, two OSD cluster, and after a while I noticed that the new
ceph-mgr daemon is frequently using a lot of the CPU:
17519 ceph 20 0 850044 168104208 S 102.7 4.3 1278:27
ceph-mgr
Restarting it with 'system
Looking at ``ceph -s`` you originally provided, all OSDs are up.
> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
like related to ?:
> Ceph1, ceph2 and ceph3 are vms on one physical host
Are those OSDs running on vm instanc
All osds are currently up:
health HEALTH_WARN
4 pgs stuck unclean
recovery 4482/58798254 objects degraded (0.008%)
recovery 420522/58798254 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set
monmap e9: 5 mons at
{ceph1=192.168.10.
On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller wrote:
> All osds are currently up:
>
> health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 4482/58798254 objects degraded (0.008%)
> recovery 420522/58798254 objects misplaced (0.715%)
> noscrub,no
>
> That's not perfectly correct.
>
> OSD.0/1/2 seem to be down.
Sorry but where do you see this? I think this indicates that they are up:
osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller
e.g.,
OSD7 / 3 / 0 are in the same acting set. They should be up, if they
are properly running.
# 9.7
>"up": [
>7,
>3
>],
>"acting": [
>7,
>3,
>0
>],
Here is an example:
"up": [
1,
0,
2
],
"acting": [
1,
0,
Ok, i understand but how can I debug why they are not running as they should?
For me I thought everything is fine because ceph -s said they are up and
running.
I would think of a problem with the crush map.
> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>
> e.g.,
> OSD7 / 3 / 0 are in the
> Op 9 januari 2017 om 13:02 schreef Stuart Harland
> :
>
>
> Hi,
>
> We’ve been operating a ceph storage system storing files using librados
> (using a replicated pool on rust disks). We implemented a cache over the top
> of this with SSDs, however we now want to turn this off.
>
> The doc
27 matches
Mail list logo
