On 30/10/2020 11:28, Wido den Hollander wrote:
On 29/10/2020 18:58, Dan van der Ster wrote:
Hi Wido,
Could it be one of these?
mon osd min up ratio
mon osd min in ratio
36/120 is 0.3 so it might be one of those magic ratios at play.
I thought of those settings and looked at it. The
On 08/12/2020 20:17, Wesley Dillingham wrote:
We rebuilt all of our mons in one cluster such that they bind only to port 3300
with msgrv2. Previous to this we were binding to both 6789 and 3300. All of our
server and client components are sufficiently new (14.2.x) and we haven’t
observed an
On 11/12/2020 00:12, David Orman wrote:
Hi Janek,
We realize this, we referenced that issue in our initial email. We do want
the metrics exposed by Ceph internally, and would prefer to work towards a
fix upstream. We appreciate the suggestion for a workaround, however!
Again, we're happy to
In addition: Make sure you are using kernels with the proper fixes.
CephFS is a co-operation between the MDS, OSDs and (Kernel) clients. If
the clients are outdated they can cause all kinds of troubles.
So make sure you are able to update clients to recent versions.
Although a stock CentOS or Ub
On 01/02/2021 22:48, Tony Liu wrote:
Hi,
With 3 replicas, a pg hs 3 osds. If all those 3 osds are down,
the pg becomes unknow. Is that right?
Yes. As no OSD can report the status to the MONs.
If those 3 osds are replaced and in and on, is that pg going to
be eventually back to active? Or
On 28/01/2021 18:09, Andrei Mikhailovsky wrote:
Hello everyone,
Could some one please let me know what is the recommended modern kernel disk
scheduler that should be used for SSD and HDD osds? The information in the
manuals is pretty dated and refer to the schedulers which have been deprec
Hi,
There are many talks and presentations out there about Ceph's
performance. Ceph is great when it comes to parallel I/O, large queue
depths and many applications sending I/O towards Ceph.
One thing where Ceph isn't the fastest are 4k blocks written at Queue
Depth 1.
Some applications benefit
On 04/02/2021 18:57, Adam Boyhan wrote:
All great input and points guys.
Helps me lean towards 3 copes a bit more.
I mean honestly NVMe cost per TB isn't that much more than SATA SSD now.
Somewhat surprised the salesmen aren't pitching 3x replication as it makes them
more money.
To add to
On 03/03/2021 00:55, Lincoln Bryant wrote:
Hi list,
We recently had a cluster outage over the weekend where several OSDs were
inaccessible over night for several hours. When I found the cluster in the
morning, the monitors' root disks (which contained both the monitor's leveldb
and the Cep
On 18/03/2021 09:09, Janne Johansson wrote:
Den ons 17 mars 2021 kl 20:17 skrev Matthew H :
"A containerized environment just makes troubleshooting more difficult, getting
access and retrieving details on Ceph processes isn't as straightforward as with a non
containerized infrastructure. I
On 08/04/2021 14:09, Ha, Son Hai wrote:
Hi everyone,
We encountered an issue with KRBD mounting after mapping it to the host with
read-only option.
We try to pinpoint where the problem is, but not able to do it.
See my reply down below.
The image is mounted well if we map it without the
On 4/12/21 5:46 PM, Dan van der Ster wrote:
> Hi all,
>
> bdev_enable_discard has been in ceph for several major releases now
> but it is still off by default.
> Did anyone try it recently -- is it safe to use? And do you have perf
> numbers before and after enabling?
>
I have done so on SATA
On 13/04/2021 11:07, Dan van der Ster wrote:
On Tue, Apr 13, 2021 at 9:00 AM Wido den Hollander wrote:
On 4/12/21 5:46 PM, Dan van der Ster wrote:
Hi all,
bdev_enable_discard has been in ceph for several major releases now
but it is still off by default.
Did anyone try it recently
On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
Hi,
Is there any clusterwise metric regarding object sizes?
I'd like to collect some information about the users what is the object sizes
in their buckets.
Are you talking about RADOS objects or objects inside RGW buckets?
I think you are
Hi,
While doing some benchmarks I have two identical Ceph clusters:
3x SuperMicro 1U
AMD Epyc 7302P 16C
256GB DDR
4x Samsung PM983 1,92TB
100Gbit networking
I tested on such a setup with v16.2.4 with fio:
bs=4k
qd=1
IOps: 695
That was very low as I was expecting at least >1000 IOps.
I check
On 09/06/2021 14:33, Ilya Dryomov wrote:
On Wed, Jun 9, 2021 at 1:38 PM Wido den Hollander wrote:
Hi,
While doing some benchmarks I have two identical Ceph clusters:
3x SuperMicro 1U
AMD Epyc 7302P 16C
256GB DDR
4x Samsung PM983 1,92TB
100Gbit networking
I tested on such a setup with
On 6/9/21 8:51 PM, Vladimir Brik wrote:
> Hello
>
> My attempt to upgrade from Octopus to Pacific ran into issues, and I
> currently have one 16.2.4 mon and two 15.2.12 mons. Is this safe to run
> the cluster like this or should I shut down the 16.2.4 mon until I
> figure out what to do next wi
Op 27-07-2021 om 05:11 schreef Fyodor Ustinov:
Hi!
docs.ceph.io ? If there’s something that you’d like to see added there, you’re
welcome to submit a tracker ticket, or write to me privately. It is not
uncommon for documentation enhancements to be made based on mailing list
feedback.
Docu
Op 10-08-2021 om 22:32 schreef Xiaolong Jiang:
Thank you for your response.
We are making a choice between go and java. Internally, our company has
very good java ecosystem support with spring which I don't want to lose.
Go binding is being actively developed/supported, it's a better choice
Op 15-10-2021 om 16:40 schreef Sage Weil:
This fall I will be stepping back from a leadership role in the Ceph
project. My primary focus during the next two months will be to work with
developers and community members to ensure a smooth transition to a more
formal system of governance for the C
On 5/17/20 4:49 PM, Harald Staub wrote:
> tl;dr: this cluster is up again, thank you all (Mark, Wout, Paul
> Emmerich off-list)!
>
Awesome!
> First we tried to lower max- and min_pg_log_entries on a single running
> OSD, without and with restarting it. There was no effect. Maybe because
> of t
ample: video-2020-05
RGW can serve objects which are 50kB in size, but there is overhead
involved. Storing a lot of such small objects comes at a price of overhead.
Wido
>
> Wido den Hollander mailto:w...@42on.com>> 于2020年5月12
> 日周二 下午2:41写道：
>
>
>
> On 5/12/20 4:22 A
On 5/23/20 12:02 PM, Amudhan P wrote:
> Hi,
>
> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500
> (default) recently i tried to update MTU size to 9000.
> After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is p
On 6/1/20 6:46 AM, Nghia Viet Tran wrote:
> Hi everyone,
>
>
>
> Currently, our client application and Ceph cluster are running on the
> primary datacenter. We’re planning to deploy Ceph on the secondary
> datacenter for DR. The secondary datacenter is in the standby mode. If
> something went
On 6/2/20 5:44 AM, Brent Kennedy wrote:
> We are rebuilding servers and before luminous our process was:
>
>
>
> 1. Reweight the OSD to 0
>
> 2. Wait for rebalance to complete
>
> 3. Out the osd
>
> 4. Crush remove osd
>
> 5. Auth del osd
>
> 6. Ceph
On 6/3/20 4:49 PM, Simon Sutter wrote:
> Hello,
>
>
> I think I missunderstood the internal / public network concepts in the docs
> https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/.
>
> Now there are two questions:
>
> - Is it somehow possible to bind the MON daemon
On 6/4/20 12:24 AM, Frank Schilder wrote:
> You can use the command-line without editing the crush map. Look at the
> documentation of commands like
>
> ceph osd crush add-bucket ...
> ceph osd crush move ...
>
> Before starting this, set "ceph osd set norebalance" and unset after you are
>
rong.
Wido
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
>
> From: Wido den Hollander
> Sent: 04 June 2020 08:50:16
> To: Frank Schilder; Kyriazis, George; ceph-users
> Subject: Re: [ceph-users]
Hi,
On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing
OSDs randomly drop out of the cluster.
Usually it's 2 to 4 OSDs spread out over different nodes. Each node has
16 OSDs and not all the failing OSDs are on the same node.
The OSDs are marked as down and all they keep pri
> Op 5 jul. 2020 om 15:26 heeft Wout van Heeswijk het volgende
> geschreven:
>
> Good point, we've looked at that, but can't see any message regarding OOM
> Killer:
>
Have to add here that we looked at changing osd memory target as well, but that
did not make a difference.
tcmalloc seems
Hi,
I got involved in a case where a Nautilus cluster was experiencing MDSes
asserting showing the backtrace mentioned in this ticket:
https://tracker.ceph.com/issues/36349
ceph_assert(follows >= realm->get_newest_seq());
In the end we needed to use these tooling to get one MDS running again:
ht
On 7/21/20 6:30 PM, Jean-Sebastien Landry wrote:
> Hi everyone, we have a ceph cluster for object storage only, the rgws are
> accessible from the internet, and everything is ok.
Is there a HTTP proxy in between?
>
> Now, one of our team/client required that their data should not ever be
>
Hi,
I'm trying to have clients read the 'rbd_default_data_pool' config
option from the config store when creating a RBD image.
This doesn't seem to work and I'm wondering if somebody knows why.
I tried:
$ ceph config set client rbd_default_data_pool rbd-data
$ ceph config set global rbd_defa
On 29/07/2020 14:54, Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 6:23 AM Wido den Hollander wrote:
Hi,
I'm trying to have clients read the 'rbd_default_data_pool' config
option from the config store when creating a RBD image.
This doesn't seem to work and I
On 29/07/2020 14:52, Raffael Bachmann wrote:
Hi All,
I'm kind of crossposting this from here:
https://forum.proxmox.com/threads/i-o-wait-after-upgrade-5-x-to-6-2-and-ceph-luminous-to-nautilus.73581/
But since I'm more and more sure that it's a ceph problem I'll try my
luck here.
Since up
On 29/07/2020 16:00, Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 9:07 AM Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 9:03 AM Wido den Hollander wrote:
On 29/07/2020 14:54, Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 6:23 AM Wido den Hollander wrote:
Hi,
I'm trying to
On 29/07/2020 16:54, Wido den Hollander wrote:
On 29/07/2020 16:00, Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 9:07 AM Jason Dillaman
wrote:
On Wed, Jul 29, 2020 at 9:03 AM Wido den Hollander
wrote:
On 29/07/2020 14:54, Jason Dillaman wrote:
On Wed, Jul 29, 2020 at 6:23 AM Wido
Hi,
On a Nautilus 14.2.8 cluster I'm seeing a large amount of GC data and
the GC on the RGW does not seem to make progress.
The .rgw.gc pool contains 39GB of data spread out over 32 objects.
In the logs we do see references of the RGW GC doing work and it says it
is removing objects.
Those
On 11/08/2020 00:40, Michael Thomas wrote:
On my relatively new Octopus cluster, I have one PG that has been
perpetually stuck in the 'unknown' state. It appears to belong to the
device_health_metrics pool, which was created automatically by the mgr
daemon(?).
The OSDs that the PG maps to
On 11/08/2020 20:41, Kevin Myers wrote:
Replica count of 2 is a sure fire way to a crisis !
It is :-)
Sent from my iPad
On 11 Aug 2020, at 18:45, Martin Palma wrote:
Hello,
after an unexpected power outage our production cluster has 5 PGs
inactive and incomplete. The OSDs on which the
On 26/08/2020 15:59, Stefan Kooman wrote:
On 2020-08-26 15:20, Marcel Kuiper wrote:
Hi Vladimir,
no it is the same on all monitors. Actually I got triggered because I got
slow responses on my rados gateway with the radosgw-admin command and
narrowed it down to slow respons for rados commands
On 27/08/2020 14:23, Marc Roos wrote:
Can someone shed a light on this? Because it is the difference of
running multiple instances of one task, or running multiple different
tasks.
As far as I know this is still required because the client talk to each
other using RADOS notifies and thus
Hello,
On a Nautilus 14.2.8 cluster I am seeing large RocksDB database with
many slow DB bytes in use.
To investigate this further I marked one OSD as out and waited for the
all the backfilling to complete.
Once the backfilling was completed I exported BlueFS and investigated
the RocksDB u
ith Luminous and now upgraded to Nautilus are
suffering from this.
It kind of seems like that garbage data stays behind in RocksDB which is
never clean up.
Wido
Thanks,
Igor
On 8/31/2020 10:57 AM, Wido den Hollander wrote:
Hello,
On a Nautilus 14.2.8 cluster I am seeing large RocksDB data
On 31/08/2020 15:44, Francois Legrand wrote:
Thanks Igor for your answer,
We could try do a compaction of RocksDB manually, but it's not clear to
me if we have to compact on the mon with something like
ceph-kvstore-tool rocksdb /var/lib/ceph/mon/mon01/store.db/ compact
or on the concerned o
On 31/08/2020 11:00, Dennis Benndorf wrote:
Hi,
today I recognized bad performance in our cluster. Running "watch ceph
osd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow on
commit and that the commit timings are equal to their apply timings:
For example
Every 2.0s: ceph osd pe
On 01/09/2020 08:15, Simon Sutter wrote:
Hello again
So I have changed the network configuration.
Now my Ceph is reachable from outside, this also means all osd’s of all nodes
are reachable.
I still have the same behaviour which is a timeout.
The client can resolve all nodes with their hostn
On 02/09/2020 12:07, Stefan Kooman wrote:
On 2020-09-01 10:51, Marcel Kuiper wrote:
As a matter of fact we did. We doubled the storage nodes from 25 to 50.
Total osds now 460.
You want to share your thoughts on that?
Yes. We observed the same thing with expansions. The OSDs will be very
bu
Hi,
The cluster I'm writing about has a long history (months) of instability
mainly related to large RocksDB database and high memory consumption.
The use-case is RGW with an EC8+3 pool for data.
In the last months this cluster has been suffering from OSDs using much
more memory then osd_mem
On 9/3/20 3:38 PM, pso...@alticelabs.com wrote:
> Hello people,
>I am trying to change the cluster network in a production ceph. I'm having
> problems, after changing the ceph.conf file and restarting a osd the cluster
> is always going to HEALTH_ERROR with blocked requests. Only by return
Hi,
Last night I've spend a couple of hours debugging a issue where OSDs
would be marked as 'up', but then PGs stayed in the 'peering' state.
Looking through the admin socket I saw these OSDs were in the 'booting'
state.
Looking at the OSDMap I saw this:
osd.3 up in weight 1 up_from 26 up_th
On 9/3/20 3:55 PM, Dan van der Ster wrote:
> Hi Wido,
>
> Out of curiosity, did you ever work out how to do this?
Nope, never did this. So there are two clusters running with the same
fsid :-)
Wido
>
> Cheers, Dan
>
> On Tue, Feb 12, 2019 at 6:17 PM Wido den Holla
On 04/09/2020 13:50, Eugen Block wrote:
Hi,
Wido had an idea in a different thread [1], you could try to advise the
OSDs to compact at boot:
[osd]
osd_compact_on_start = true
This is in master only, not yet in any release.
Can you give that a shot?
Wido also reported something about l
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB per osd.
(This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
PGs are active+clean).
After readi
On 07/10/2020 16:00, Dan van der Ster wrote:
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB
Hi,
Most of it is described here: https://tracker.ceph.com/issues/22928
Buckets created under Jewel don't always have the *placement_rule* set
in their bucket metadata and this causes Nautilus RGWs to not serve
requests for them.
Snippet from the metadata:
{
"key": "bucket.instance:pbx:
Hi,
I already submitted a ticket: https://tracker.ceph.com/issues/47951
Maybe other people noticed this as well.
Situation:
- Cluster is running IPv6
- mon_host is set to a DNS entry
- DNS entry is a Round Robin with three -records
root@wido-standard-benchmark:~# ceph -s
unable to parse add
Hi,
I'm investigating an issue where 4 to 5 OSDs in a rack aren't marked as
down when the network is cut to that rack.
Situation:
- Nautilus cluster
- 3 racks
- 120 OSDs, 40 per rack
We performed a test where we turned off the network Top-of-Rack for each
rack. This worked as expected with
On 30/10/2020 06:09, Ing. Luis Felipe Domínguez Vega wrote:
Hi:
I tried get info from a RBD image but:
-
root@fond-beagle:/# rbd list --pool cinder-ceph | grep
volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda
volume-dfcca6c8
On 30/10/2020 05:20, Ing. Luis Felipe Domínguez Vega wrote:
Great and thanks, i fixed all unknowns with the command, now left the
incomplete, down, etc.
Start with a query:
$ ceph pg query
That will tell you why it's down and incomplete.
The force-create-pg has probably corrupted and de
On 29/10/2020 19:29, Zhenshi Zhou wrote:
Hi Alex,
We found that there were a huge number of keys in the "logm" and "osdmap"
table
while using ceph-monstore-tool. I think that could be the root cause.
But that is exactly how Ceph works. It might need that very old OSDMap
to get all the PGs
arked as down after 15 minutes because
it didn't send a beacon to the MON. Other OSDs kept sending reports that
it was down, but the MONs simply didn't act on it.
Wido
Cheers,
Dan
On Thu, 29 Oct 2020, 18:05 Wido den Hollander, <mailto:w...@42on.com>> wrote:
Hi,
Thursday, October 22, 2020 12:54 PM
To: Wido den Hollander mailto:w...@42on.com>>
Cc: ceph-users@ceph.io <mailto:ceph-users@ceph.io>
mailto:ceph-users@ceph.io>>
Subject: [EXTERNAL] [ceph-users] Re: 14.2.12 breaks mon_host
pointing to Round Robin DNS entry
T
On 03/11/2020 10:02, Dan van der Ster wrote:
Hi all,
We still have legacy caps on our nautilus rbd cluster. I just wanted
to check if this is totally safe (and to post here ftr because I don't
think this has ever been documented)
Here are the current caps:
[client.images]
key = xxx
caps mgr
On 8/14/19 5:46 PM, Lorenz Kiefner wrote:
> Hi,
>
> this was the first thing I was thinking about (and yes, there had been
> some issues, but they are resolved - double checked!).
>
> MTU is consistent throughout the whole net and pings in all sizes are
> handled well. And MTU problems wouldn'
> Op 24 aug. 2019 om 16:36 heeft Darren Soothill het
> volgende geschreven:
>
> So can you do it.
>
> Yes you can.
>
> Should you do it is the bigger question.
>
> So my first question would be what type of drives are you using? Enterprise
> class drives with a low failure rate?
>
Doesn’
> Op 27 aug. 2019 om 11:38 heeft Max Krasilnikov het
> volgende geschreven:
>
> Hello!
>
> Sat, Aug 24, 2019 at 10:47:55PM +0200, wido wrote:
>
>>> Op 24 aug. 2019 om 16:36 heeft Darren Soothill
>>> het volgende geschreven:
>>>
>>> So can you do it.
>>>
>>> Yes you can.
>>>
>>> Should
On 9/1/19 9:51 PM, Zoltan Arnold Nagy wrote:
> On 2019-09-01 05:57, Konstantin Shalygin wrote:
>> On 8/31/19 4:14 PM, Zoltan Arnold Nagy wrote:
>>> Could you elaborate a bit more? upmap is used to map specific PGs to
>>> specific OSDs
>>> in order to deal with CRUSH inefficiencies.
>>>
>>> Why w
On 9/2/19 11:53 AM, Zoltan Arnold Nagy wrote:
> On 2019-09-02 08:43, Wido den Hollander wrote:
>> On 9/1/19 9:51 PM, Zoltan Arnold Nagy wrote:
>>> On 2019-09-01 05:57, Konstantin Shalygin wrote:
>>>> On 8/31/19 4:14 PM, Zoltan Arnold Nagy wrote:
>>>>
Hi,
The CFP is ending today for the Ceph Day London on October 24th.
If you have a talk you would like to submit, please follow the link below!
Wido
On 7/18/19 3:43 PM, Wido den Hollander wrote:
> Hi,
>
> We will be having Ceph Day London October 24th!
>
> https://ceph.com/ce
On 9/16/19 11:36 AM, Eikermann, Robert wrote:
>
> Hi,
>
>
>
> I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d
> like to enable writeback caching tier, like described here:
> https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/.
>
>
>
Can you explain why? The
On 9/17/19 8:46 AM, Ronny Aasen wrote:
> On 17.09.2019 06:54, Ashley Merrick wrote:
>> Have just noticed their is packages available for 14.2.4..
>>
>> I know with the whole 14.2.3 release and the notes not going out to a
>> good day or so later.. but this is not long after the 14.2.3 release..?
On 9/17/19 6:54 AM, Ashley Merrick wrote:
> Have just noticed their is packages available for 14.2.4..
>
> I know with the whole 14.2.3 release and the notes not going out to a
> good day or so later.. but this is not long after the 14.2.3 release..?
>
It seems that the release is intentional
On 9/25/19 3:22 PM, nalexand...@innologica.com wrote:
> Hi everyone,
>
> We are running Nautilus 14.2.2 with 6 nodes and a total of 44 OSDs, all are
> 2TB spinning disks.
> # ceph osd count-metadata osd_objectstore
> "bluestore": 44
> # ceph osd pool get one size
> size: 3
> # ceph df
> R
On 9/25/19 6:52 PM, Nathan Fish wrote:
> You don't need more mons to scale; but going to 5 mons would make the
> cluster more robust, if it is cheap for you to do so.
> If you assume that 1 mon rebooting for updates or maintenance is
> routine, then 2/3 is vulnerable to one failure. 4/5 can survi
On 9/26/19 5:05 AM, zhanrzh...@teamsun.com.cn wrote:
> Thanks for your reply.
> We don't maintain it frequently.
> My confusion is whether the more monitor is more advantage for
> client(osd,rbdclient...) to get clustermap.
> Do All clients comunicate with one monitor of the cluster at the m
On 10/1/19 12:16 PM, Raymond Berg Hansen wrote:
> Hi. I am new to ceph but have set it up on my homelab and started using it.
> It seemed very good intil I desided to try pg autoscale.
> After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the
> number of PGs and the pools a
On 10/7/19 9:15 AM, i.schm...@langeoog.de wrote:
> Hi Folks
>
> We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To
> Monitor our systems we use Zabbix and i would like to get some Ceph Data into
> our Zabbix to get some alarms when something goes wrong.
>
> Ceph mg
On 10/9/19 5:20 PM, i.schm...@langeoog.de wrote:
> Thank you very much! This helps a lot!
>
> I'm thinking if it is a good idea at all, to tie ceph data input to a
> specific host of that cluster in zabbix. I could try and set up a new host in
> zabbix called "Ceph", representing the cluster
On 10/15/19 1:29 PM, Vitaliy Filippov wrote:
> Wow, does it really work?
>
> And why is it not supported by RBD?
>
> Can you show us the latency graphs before and after and tell the I/O
> pattern to which the latency applies? Previous common knowledge was that
> RDMA almost doesn't affect laten
Hi,
I have a Mimic 13.2.6 cluster which is throwing an error on a PG that
it's inconsistent.
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 21.e6d is active+clean+inconsistent, acting [988,508,825]
I checked 'list-inconsistent-obj' (See below) and it shows:
selected_object_info: "dat
On 10/17/19 7:48 PM, Alex L wrote:
> Final update.
>
> I switched the below from false and everything magically started working!
> cephx_require_signatures = true
> cephx_cluster_require_signatures = true
> cephx_sign_messages = true
Are you sure the time is in sync in your cluster after the p
On 10/25/19 7:14 PM, Randall Smith wrote:
> Greetings,
>
> I am running a mimic cluster. I noticed that I suddenly have over 200
> rbd images that have seemingly random 64 character names. They were
> all created within a short time period. rbd info on one of the rbds
> looks like this:
>
> rb
On 11/15/19 11:38 AM, Thomas Schneider wrote:
> Hi,
>
> when I execute this command
> rbd ls -l
> to list all RBDs I get spamming errors:
>
Those errors are weird. Can you share the Ceph cluster version and the
clients?
$ ceph versions
And then also use rpm/dpkg to check which version of Ce
On 11/15/19 11:22 AM, Thomas Schneider wrote:
> Hi,
> ceph health is reporting: pg 59.1c is creating+down, acting [426,438]
>
> root@ld3955:~# ceph health detail
> HEALTH_WARN 1 MDSs report slow metadata IOs; noscrub,nodeep-scrub
> flag(s) set; Reduced data availability: 1 pg inactive, 1 pg down
bjects, 241 TiB
> usage: 723 TiB used, 796 TiB / 1.5 PiB avail
> pgs: 0.012% pgs not active
> 8431 active+clean
> 1 creating+down
>
> io:
> client: 33 MiB/s rd, 14.20k op/s rd, 0 op/s wr
>
>
> Am 15.11.2019 um 13:2
On 11/28/19 6:41 AM, Alokkumar Mahajan wrote:
> Hello,
> I am new to Ceph and currently i am working on setting up CephFs and RBD
> environment. I have successfully setup Ceph Cluster with 4 OSD's (2
> OSD's with size 50GB and 2 OSD's with size 300GB).
>
> But while setting up CephFs the size wh
On 11/28/19 5:23 PM, Rodrigo Severo - Fábrica wrote:
> Em qui., 28 de nov. de 2019 às 00:34, Konstantin Shalygin
> escreveu:
>>
>>> My servers have 2 network boards each. I would like to use the current
>>> local one to talk to Cephs clients (both CephFS and Object Storage)
>>> and use the secon
On 12/24/19 9:19 PM, Neha Ojha wrote:
> The root cause of this issue is the overhead added by the network ping
> time monitoring feature for the mgr to process.
> We have a fix that disables sending the network ping times related
> stats to the mgr and Eric has helped verify the fix(Thanks Eric!)
On 1/10/20 9:09 PM, Frank R wrote:
> Hi all,
>
> When upgrading from Luminous to Nautilus the global configmap options
> for cluster_network and public_network were inadvertently set to an
> incorrect value (10.192.80.0/24):
>
> -
>
> [root@ceph-osd134 ceph]# ceph config dump | grep netwo
Täuber wrote:
> yesterday:
> https://ceph.io/releases/v14-2-6-nautilus-released/
>
>
> Cheers,
> Lars
>
> Thu, 9 Jan 2020 10:10:12 +0100
> Wido den Hollander ==> Neha Ojha , Sasha
> Litvak :
>> On 12/24/19 9:19 PM, Neha Ojha wrote:
>>> The ro
On 1/20/20 1:07 AM, 徐蕴 wrote:
> Hi,
>
> We upgraded our cluster from Jewel to Luminous, and it turned out that more
> than 80% object misplaced. Since our cluster has 130T data, backfilling seems
> take forever. We didn’t modify any crushmap. Any thoughts about this issue?
Did you reweight yo
On 1/19/20 12:07 PM, Stefan Kooman wrote:
> Hi,
>
> Is there any logic / filtering which PGs to backfill at any given time
> that takes into account the OSD the PG is living on?
>
> Our cluster is backfilling a complete pool now (512 PGs) and (currently)
> of the 7 active+remapped+backfilling
Hi,
Keep in mind that /var/lib/ceph/osd/ is a tmpfs which is created by
'ceph-bluestore-tool' on OSD startup.
All the data in there comes from the lvtags set on the LVs.
So I *think* you can just rename the Volume Group and rescan with
ceph-volume.
Wido
On 1/28/20 10:25 AM, Stolte, Felix wrote
On 1/28/20 11:19 AM, Marc Roos wrote:
>
> Say one is forced to move a production cluster (4 nodes) to a different
> datacenter. What options do I have, other than just turning it off at
> the old location and on on the new location?
>
> Maybe buying some extra nodes, and move one node at a t
On 1/28/20 6:58 PM, Anthony D'Atri wrote:
>
>
>> I did this ones. This cluster was running IPv6-only (still is) and thus
>> I had the flexibility of new IPs.
>
> Dumb question — how was IPv6 a factor in that flexibility? Was it just that
> you had unused addresses within an existing block?
>
On 1/28/20 7:03 PM, David DELON wrote:
> Hi,
>
> i had a problem with one application (seafile) which uses CEPH backend with
> librados.
> The corresponding pools are defined with size=3 and each object copy is on a
> different host.
> The cluster health is OK: all the monitors see all the
On 1/30/20 1:34 PM, vis...@denovogroup.org wrote:
> Iam testing failure scenarios for my cluster. I have 3 monitors. Lets say if
> mons 1 and 2 go down and so monitors can't form a quorum, how can I recover?
>
> Are the instructions at followling link valid for deleting mons 1 and 2 from
> m
On 1/30/20 1:55 PM, Gregory Farnum wrote:
> On Thu, Jan 30, 2020 at 1:38 PM Wido den Hollander wrote:
>>
>>
>>
>> On 1/30/20 1:34 PM, vis...@denovogroup.org wrote:
>>> Iam testing failure scenarios for my cluster. I have 3 monitors. Lets say
>>>
On 1/31/20 12:09 AM, Nigel Williams wrote:
> Did you end up having all new IPs for your MONs? I've wondered how
> should a large KVM deployment be handled when the instance-metadata
> has a hard-coded list of MON IPs for the cluster? how are they changed
> en-masse with running VMs? or do these
1 - 100 of 140 matches
Mail list logo
