[ceph-users] Re: CephFS mirror very slow (maybe for small files?)

[Jos Collin]

Hi Stuart,

I would highly recommend you to have this [1] fix, so that the mirroring 
works as expected and uses the prev snapshot for syncing.

Having multiple mirror daemons also improves the speed.

[1] https://github.com/ceph/ceph/pull/54405

- Jos Collin

On 13/11/23 21:31, Stuart Cornell wrote:

Hi all.
I have successfully configured an operational mirror between 2 sites for Ceph 
FS. The mirroring is running but the speed of data transfer is varying a lot 
over time (200KB/s – 120MB/s). The network infrastructure between the two Ceph 
clusters is reliable and should not be the cause of this speed variation. The 
FS in question, has a lot of small files in it and I suspect this is the cause 
of the variability – ie, the transfer of many small files will be more impacted 
by greater site-site latency.
If this suspicion is true, what options do I have to improve the overall 
throughput?

   *   Is it possible to parallelise or “chunk” the transfers with some options 
to the mirror daemon?
   *   Would the use of multiple snapshot mirror points help?
  *   Note that I am currently forced to use only a single point ( ceph fs snapshot 
mirror add   ) because the part of the FS in use is managed by 
Openstack Manila which created a subvolume. Requests to add mirrors for sub-directories are 
therefore denied.
Any suggestions for how I can improve this throughput would be most welcome.
Currrently I am running Pacific (16.2.10) on the sender and Quincy (17.2.6) on 
the target.

Stuart Cornell
Cloud Development Director
http://graphcore.ai



** We have updated our privacy policy, which contains important information about how 
we collect and process your personal data. To read the policy, please click 
here<http://www.graphcore.ai/privacy> **

This email and its attachments are intended solely for the addressed recipients 
and may contain confidential or legally privileged information.
If you are not the intended recipient you must not copy, distribute or 
disseminate this email in any way; to do so may be unlawful.

Any personal data/special category personal data herein are processed in 
accordance with UK data protection legislation.
All associated feasible security measures are in place. Further details are 
available from the Privacy Notice on the website and/or from the Company.

Graphcore Limited (registered in England and Wales with registration number 
10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN.
This message was scanned for viruses upon transmission. However Graphcore 
accepts no liability for any such transmission.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top causes 16 mgr modules have recently crashed

[Jos Collin]

Please have this fix: https://tracker.ceph.com/issues/59551. It's 
backported to quincy.


On 23/01/24 03:11, Özkan Göksu wrote:

Hello

When I run cephfs-top it causes mgr module crash. Can you please tell me
the reason?

My environment:
My ceph version 17.2.6
Operating System: Ubuntu 22.04.2 LTS
Kernel: Linux 5.15.0-84-generic

I created the cephfs-top user with the following command:
ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow
r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring

This is the crash report:

root@ud-01:~# ceph crash info
2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801
{
 "backtrace": [
 "  File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in
notify\nself.fs_perf_stats.notify_cmd(notify_id)",
 "  File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177,
in notify_cmd\nmetric_features =
int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"],
16)",
 "ValueError: invalid literal for int() with base 16: '0x'"
 ],
 "ceph_version": "17.2.6",
 "crash_id":
"2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801",
 "entity_name": "mgr.ud-01.qycnol",
 "mgr_module": "stats",
 "mgr_module_caller": "ActivePyModule::notify",
 "mgr_python_exception": "ValueError",
 "os_id": "centos",
 "os_name": "CentOS Stream",
 "os_version": "8",
 "os_version_id": "8",
 "process_name": "ceph-mgr",
 "stack_sig":
"971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c",
 "timestamp": "2024-01-22T21:25:59.313305Z",
 "utsname_hostname": "ud-01",
 "utsname_machine": "x86_64",
 "utsname_release": "5.15.0-84-generic",
 "utsname_sysname": "Linux",
 "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023"
}


Best regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top causes 16 mgr modules have recently crashed

[Jos Collin]

This fix is in the mds.
I think you need to read 
https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade.


On 23/01/24 12:19, Özkan Göksu wrote:

Hello Jos.
Thank you for the reply.

I can upgrade to 17.2.7 but I wonder can I only upgrade MON+MGR for 
this issue or do I need to upgrade all the parts?
Otherwise I need to wait few weeks. I don't want to request 
maintenance during delivery time.


root@ud-01:~# ceph orch upgrade ls
{
    "image": "quay.io/ceph/ceph <http://quay.io/ceph/ceph>",
    "registry": "quay.io <http://quay.io>",
    "bare_image": "ceph/ceph",
    "versions": [
        "18.2.1",
        "18.2.0",
        "18.1.3",
        "18.1.2",
        "18.1.1",
        "18.1.0",
        "17.2.7",
        "17.2.6",
        "17.2.5",
        "17.2.4",
        "17.2.3",
        "17.2.2",
        "17.2.1",
        "17.2.0"
    ]
}

Best regards

Jos Collin , 23 Oca 2024 Sal, 07:42 tarihinde şunu 
yazdı:


Please have this fix: https://tracker.ceph.com/issues/59551. It's
backported to quincy.

On 23/01/24 03:11, Özkan Göksu wrote:
> Hello
>
> When I run cephfs-top it causes mgr module crash. Can you please
tell me
> the reason?
>
> My environment:
> My ceph version 17.2.6
> Operating System: Ubuntu 22.04.2 LTS
> Kernel: Linux 5.15.0-84-generic
>
> I created the cephfs-top user with the following command:
> ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r'
osd 'allow
> r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring
>
> This is the crash report:
>
> root@ud-01:~# ceph crash info
> 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801
> {
>      "backtrace": [
>          "  File \"/usr/share/ceph/mgr/stats/module.py\", line
32, in
> notify\n    self.fs_perf_stats.notify_cmd(notify_id)",
>          "  File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\",
line 177,
> in notify_cmd\n    metric_features =
>

int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"],
> 16)",
>          "ValueError: invalid literal for int() with base 16: '0x'"
>      ],
>      "ceph_version": "17.2.6",
>      "crash_id":
> "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801",
>      "entity_name": "mgr.ud-01.qycnol",
>      "mgr_module": "stats",
>      "mgr_module_caller": "ActivePyModule::notify",
>      "mgr_python_exception": "ValueError",
>      "os_id": "centos",
>      "os_name": "CentOS Stream",
>      "os_version": "8",
>      "os_version_id": "8",
>      "process_name": "ceph-mgr",
>      "stack_sig":
> "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c",
>      "timestamp": "2024-01-22T21:25:59.313305Z",
>      "utsname_hostname": "ud-01",
>      "utsname_machine": "x86_64",
>      "utsname_release": "5.15.0-84-generic",
>      "utsname_sysname": "Linux",
>      "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023"
> }
>
>
> Best regards.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Linux Laptop Losing CephFS mounts on Sleep/Hibernate

[Jos Collin]

Need logs to check more on this.
Or just call a script during wakeup?

On Tue, 26 Mar 2024 at 04:16,  wrote:

> Hi All,
>
> So I've got a Ceph Reef Cluster (latest version) with a CephFS system set
> up with a number of directories on it.
>
> On a Laptop (running Rocky Linux (latest version)) I've used fstab to
> mount a number of those directories - all good, everything works, happy
> happy joy joy! :-)
>
> However, when the laptop goes into sleep or hibernate mode (ie when I
> close the lid) and then bring it back out of sleep/hibernate (ie open the
> lid) the CephFS mounts are "not present". The only way to get them back is
> to run `mount -a` as either root or as sudo. This, as I'm sure you'll
> agree, is less than ideal - especially as this is a pilot project for
> non-admin users (ie they won't have access to the root account or sudo on
> their own (corporate) laptops).
>
> So, my question to the combined wisdom of the Community is what's the best
> way to resolve this issue?
>
> I've looked at autofs, and even tried (half-heartedly - it was late, and I
> wanted to go home  :-) ) to get this running, but I'm note sure if this is
> the best way to resolve things.
>
> All help and advice on this greatly appreciated - thank in advance
>
> Cheers
>
> Dulux-Oz
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: "ceph fs perf stats" and "cephfs-top" don't work

[Jos Collin]

Contradicting outputs, it seems immediate ones. You need to wait for a 
few seconds for the 'perf stats' (and of-course the cephfs-top) to 
display the correct the metrics. I hope you have ongoing IO while 
running 'perf stats' and cephfs-top. What is your kernel version?


On 15/07/21 5:17 pm, Eugen Block wrote:

Hi,

I just setup a virtual one-node cluster (16.2.5) to check out 
cephfs-top. Regarding the number of clients I was a little surprised, 
too, in the first couple of minutes the number switched back and forth 
between 0 and 1 although I had not connected any client yet. But after 
a while the number got stable and correct. I have two clients 
connected now, but I don't see any stats despite having the stats 
module enabled:


---snip---
cephfs-top - Thu Jul 15 13:35:41 2021
Client(s): 2 - 0 FUSE, 0 kclient, 2 libcephfs

  client_id mount_root chit(%) rlat(s) wlat(s) mlat(s) dlease(%) 
ofiles oicaps oinodes mount_point@host/addr
  24835 /  N/A N/A N/A N/A N/A N/A    
N/A    N/A N/A@pacific/v1:192.168.124.35
  24846 /client    N/A N/A N/A N/A N/A N/A    
N/A    N/A N/A@host-192-168-124-168/v1:192.168.124.168

---snip---


The command 'ceph fs perf stats' also only shows this:

---snip---
pacific:~ # ceph fs perf stats
{"version": 1, "global_counters": ["cap_hit", "read_latency", 
"write_latency", "metadata_latency", "dentry_lease", "opened_files", 
"pinned_icaps", "opened_inodes"], "counters": [], "client_metadata": 
{"client.24835": {"IP": "v1:192.168.124.35", "hostname": "pacific", 
"root": "/", "mount_point": "N/A"}, "client.24846": {"IP": 
"v1:192.168.124.168", "hostname": "host-192-168-124-168", "root": 
"/client", "mount_point": "N/A"}}, "global_metrics": {"client.24835": 
[[0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0]], 
"client.24846": [[0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 
0], [0, 0]]}, "metrics": {"delayed_ranks": [], "mds.0": 
{"client.24835": [], "client.24846": []}}}

---snip---

although I have written a couple of GB into the cephfs.

Regards,
Eugen


Zitat von Erwin Bogaard :


Hi,

I just upgraded our cluster to pacific 16.2.5.
As I'm curious what cephfs-top could give for insights, I followed the
steps in the documentation.
After enabling the mgr module "stats":

# ceph mgr module ls
...
    "enabled_modules": [
    "dashboard",
    "iostat",
    "restful",
    "stats",
    "zabbix"
...

I tried the following command:
# ceph fs perf stats
{"version": 1, "global_counters": ["cap_hit", "read_latency",
"write_latency", "metadata_latency", "dentry_lease", "opened_files",
"pinned_icaps", "opened_inodes"], "counters": [], "client_metadata": {},
"global_metrics": {}, "metrics": {"delayed_ranks": []}}

As you can see, this returns no info whatsoever. The same with:

# cephfs-top
cluster ceph does not exist

The actual cluster name is "ceph".

So I don't understand why "ceph fs perf stats" isn't showing any
information.
Maybe another indicator something isn't ritght:

# ceph fs status
cephfs - 0 clients
==
RANK  STATE  MDS    ACTIVITY DNS    INOS   DIRS CAPS
...

I see "0 clients". When I take a look in the mgr dashboard, I can 
actually

see all clients. Which are RHEL 7 & 8 cephfs kernel clients.
There is only 1 mds active, and 1 in standby-replay.
I have multiple pools active, but only 1 fs.

Does anyone have a suggestion where I can take a look enable 
gathering the

stats?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: "ceph fs perf stats" and "cephfs-top" don't work

[Jos Collin]

On 14/07/21 1:20 pm, Erwin Bogaard wrote:

Hi,

I just upgraded our cluster to pacific 16.2.5.
As I'm curious what cephfs-top could give for insights, I followed the
steps in the documentation.
After enabling the mgr module "stats":

# ceph mgr module ls
...
 "enabled_modules": [
 "dashboard",
 "iostat",
 "restful",
 "stats",
 "zabbix"
...

I tried the following command:
# ceph fs perf stats
{"version": 1, "global_counters": ["cap_hit", "read_latency",
"write_latency", "metadata_latency", "dentry_lease", "opened_files",
"pinned_icaps", "opened_inodes"], "counters": [], "client_metadata": {},
"global_metrics": {}, "metrics": {"delayed_ranks": []}}

As you can see, this returns no info whatsoever.
Probably there is no IO. Even if you have, the queries would expire - we 
have QUERY_EXPIRE_INTERVAL in perf_stats.

> QUERY_EXPIRE_INTERVAL = timedelta(minutes=1)

  The same with:

# cephfs-top
cluster ceph does not exist
Could you please use the options: --cluster, --id and --conffile and 
check if that connects?

Please send me the mgr logs to check.



The actual cluster name is "ceph".

So I don't understand why "ceph fs perf stats" isn't showing any
information.
Maybe another indicator something isn't ritght:

# ceph fs status
cephfs - 0 clients
==
RANK  STATE  MDSACTIVITY DNSINOS   DIRS   CAPS
...

I see "0 clients". When I take a look in the mgr dashboard, I can actually
see all clients. Which are RHEL 7 & 8 cephfs kernel clients.
There is only 1 mds active, and 1 in standby-replay.
I have multiple pools active, but only 1 fs.

Does anyone have a suggestion where I can take a look enable gathering the
stats?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: docs.ceph.com certificate expired?

[Jos Collin]

Fixed

On 22/04/20 6:57 pm, Bobby wrote:


Thanks ! When will it be back?

On Wed, Apr 22, 2020 at 3:03 PM > wrote:


Hello,

trying to access the documentation on docs.ceph.com
 now results in an error:  The certificate
expired on April 22, 2020, 8:46 AM.

Bye,
Ulrich
___
Dev mailing list -- d...@ceph.io 
To unsubscribe send an email to dev-le...@ceph.io



___
Dev mailing list -- d...@ceph.io
To unsubscribe send an email to dev-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top doesn't work

[Jos Collin]

Do you have mounted clients? How many clients do you have?

Please see: https://tracker.ceph.com/issues/55197

On 19/04/22 01:13, Vladimir Brik wrote:
Does anybody know why cephfs-top may only display header lines (date, 
client types, metric names) but no actual data?


When I run it, cephfs-top consumes quite a bit of the CPU and 
generates quite a bit of network traffic, but it doesn't actually 
display the data.


I poked around in the source code and it seems like it might be curses 
issue, but I am not sure.



Vlad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top doesn't work

[Jos Collin]

This doesn't break anything, but the current version of cephfs-top 
cannot accommodate great number of clients. The workaround is to limit 
the number of clients (if that's possible) or reduce the terminal 
zoom/font size to accommodate 100 clients.

We have a tracker [1] to implement the limit also.

[1] https://tracker.ceph.com/issues/55121

On 19/04/22 20:30, Vladimir Brik wrote:

Yes, `ceph fs perf stats` works.

Reverting to older versions I get "exception: addwstr() returned ERR"

If I manually set self.height, self.width to something large in 
refresh_window_size I can see some data, but there is no way to 
scroll, so I'll probably need to write something myself


Vlad

On 4/18/22 21:20, Xiubo Li wrote:


On 4/19/22 3:43 AM, Vladimir Brik wrote:
Does anybody know why cephfs-top may only display header lines 
(date, client types, metric names) but no actual data?


When I run it, cephfs-top consumes quite a bit of the CPU and 
generates quite a bit of network traffic, but it doesn't actually 
display the data.


I poked around in the source code and it seems like it might be 
curses issue, but I am not sure.



Does there any data from `ceph fs perf stats` ?

Before I hit the same issue it was caused by the curses and windows 
issue, we have fixed that long time ago, you can try to enlarge your 
terminal size and try again.


If that still doesn't work please try to revert some recent commits 
of the cephfs-top to see whether will it work for you. Recently there 
have some new features supported.


-- Xiubo



Vlad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Reasonable MDS rejoin time?

[Jos Collin]

It's hard to suggest without the logs. Do verbose logging debug_mds=20. 
What's the ceph version? Do you have the logs why the MDS crashed?


On 16/05/22 11:20, Felix Lee wrote:

Dear all,
We currently have 7 multi-active MDS, with another 7 standby-replay.
We thought this should cover most of disasters, and it actually did. 
But things just got happened, here is the story:
One of MDS crashed and standby-replay took over, but got stuck at 
resolve state.
Then, the other two MDS(rank 0 and 5) received tones of slow requests, 
and my colleague restarted them, thinking the standby-replay would 
take over immediately (this seemed to be wrong or at least unnecessary 
action, I guess...). Then, it resulted three of them in resolve state...
In the meanwhile, I realized that the first failed rank(rank 2) had 
abnormal memory usage and kept getting crashed, after couple 
restarting, the memory usage was back to normal, and then, those tree 
MDS entered into rejoin state.
Now, this rejoin state is there for three days and keeps going as 
we're speaking. Here, no significant error message shows up even with 
"debug_mds 10", so, we have no idea when it's gonna end and if it's 
really running on the track.
So, I am wondering how do we check MDS rejoin progress/status to make 
sure if it's running normally? Or, how do we estimate the rejoin time 
and maybe improve it? because we always need to tell user the time 
estimation of its recovery.



Thanks
&
Best regards,
Felix Lee ~



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Reasonable MDS rejoin time?

[Jos Collin]

I suggest you to upgrade the cluster to the latest release [1], as 
nautilus reached EOL.


[1] https://docs.ceph.com/en/latest/releases/

On 16/05/22 13:29, Felix Lee wrote:

Hi, Jos,
Many thanks for your reply.
And sorry, I missed to mention the version, which is 14.2.22.

Here is the log:
https://drive.google.com/drive/folders/1qzPf64qw16VJDKSzcDoixZ690KL8XSoc?usp=sharing 



Here, the ceph01(active) and ceph11(standby-replay) were the ones what 
suffered crash. The log didn't tell us much but several slow request 
were occurring. And, the ceph11 had "cache is too large" warning by 
the time it went crashed, suppose it could happen when doing recovery. 
(each MDS has 64GB memory, BTW )
The ceph16 is current rejoin one, I've turned debug_mds to 20 for a 
while as ceph-mds.ceph16.log-20220516.gz



Thanks
&
Best regards,
Felix Lee ~



On 5/16/22 14:45, Jos Collin wrote:
It's hard to suggest without the logs. Do verbose logging 
debug_mds=20. What's the ceph version? Do you have the logs why the 
MDS crashed?


On 16/05/22 11:20, Felix Lee wrote:

Dear all,
We currently have 7 multi-active MDS, with another 7 standby-replay.
We thought this should cover most of disasters, and it actually did. 
But things just got happened, here is the story:
One of MDS crashed and standby-replay took over, but got stuck at 
resolve state.
Then, the other two MDS(rank 0 and 5) received tones of slow 
requests, and my colleague restarted them, thinking the 
standby-replay would take over immediately (this seemed to be wrong 
or at least unnecessary action, I guess...). Then, it resulted three 
of them in resolve state...
In the meanwhile, I realized that the first failed rank(rank 2) had 
abnormal memory usage and kept getting crashed, after couple 
restarting, the memory usage was back to normal, and then, those 
tree MDS entered into rejoin state.
Now, this rejoin state is there for three days and keeps going as 
we're speaking. Here, no significant error message shows up even 
with "debug_mds 10", so, we have no idea when it's gonna end and if 
it's really running on the track.
So, I am wondering how do we check MDS rejoin progress/status to 
make sure if it's running normally? Or, how do we estimate the 
rejoin time and maybe improve it? because we always need to tell 
user the time estimation of its recovery.



Thanks
&
Best regards,
Felix Lee ~



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io





___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top doesn't work

[Jos Collin]

This issue is fixed in https://github.com/ceph/ceph/pull/48090. Could you
please check it out and let me know?

Thanks.

On Tue, 19 Apr 2022 at 01:14, Vladimir Brik 
wrote:

> Does anybody know why cephfs-top may only display header
> lines (date, client types, metric names) but no actual data?
>
> When I run it, cephfs-top consumes quite a bit of the CPU
> and generates quite a bit of network traffic, but it doesn't
> actually display the data.
>
> I poked around in the source code and it seems like it might
> be curses issue, but I am not sure.
>
>
> Vlad
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs-top doesn't work

[Jos Collin]

Yes, you need perf stats version 2 for the latest cephfs-top UI to work.

On Wed, 5 Oct 2022 at 20:03, Vladimir Brik 
wrote:

> It looks like my cluster is too old. I am getting "perf
> stats version mismatch!"
>
> Vlad
>
> On 10/5/22 08:37, Jos Collin wrote:
> > This issue is fixed in
> > https://github.com/ceph/ceph/pull/48090
> > <https://github.com/ceph/ceph/pull/48090>. Could you please
> > check it out and let me know?
> >
> > Thanks.
> >
> > On Tue, 19 Apr 2022 at 01:14, Vladimir Brik
> >  > <mailto:vladimir.b...@icecube.wisc.edu>> wrote:
> >
> > Does anybody know why cephfs-top may only display header
> > lines (date, client types, metric names) but no actual data?
> >
> > When I run it, cephfs-top consumes quite a bit of the CPU
> > and generates quite a bit of network traffic, but it
> > doesn't
> > actually display the data.
> >
> > I poked around in the source code and it seems like it
> > might
> > be curses issue, but I am not sure.
> >
> >
> > Vlad
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > <mailto:ceph-users-le...@ceph.io>
> >
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Getting started with cephfs-top, how to install

[Jos Collin]

How many clients do you have?  If you have several clients and issues 
viewing them, please checkout [1] patch.


[1] https://github.com/ceph/ceph/pull/48090

On 18/10/22 01:50, Zach Heise (SSCC) wrote:


I'd like to see what CephFS clients are doing the most IO. According 
to this page: https://docs.ceph.com/en/quincy/cephfs/cephfs-top/ - 
cephfs-top is the simplest way to do this? I enabled 'ceph mgr module 
enable stats' today, but I'm a bit confused about what the best way is 
to get the cephfs-top package to use this perf stats.


The ceph doc page linked above just mentions "cephfs-top is available 
as part of cephfs-top package" but it does not list what repo is 
required to access this. Anyone using cephfs-top themselves, and know 
the missing parts of this document that should be added?


--
Zach Heise

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How to call cephfs-top

[Jos Collin]

On 28/04/23 13:51, E Taka wrote:

I'm using a dockerized Ceph 17.2.6 under Ubuntu 22.04.

Presumably I'm missing a very basic thing, since this seems a very simple
question: how can I call cephfs-top in my environment? It is not inckuded
in the Docker Image which is accessed by "cephadm shell".

And calling the version found in the source code always fails with "[errno
13] RADOS permission denied", even when using "--cluster" with the correct
ID, "--conffile" and "--id".


To run from the source code, you need to set PYTHONPATH to 
ceph/build/lib/cython_modules/lib.3/




The auth user client.fstop exists, and "ceph fs perf stats" runs.
What am I missing?

Thanks!
___
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph fs perf stats output is empty

[Jos Collin]

Additionally, the first `ceph fs perf stats` output would be empty or
outdated in most cases. You need to query a few times to get the latest
values. So try `watch ceph fs perf stats`.

On Mon, 12 Jun 2023 at 06:30, Xiubo Li  wrote:

>
> On 6/10/23 05:35, Denis Polom wrote:
> > Hi
> >
> > I'm running latest Ceph Pacific 16.2.13 with Cephfs. I need to collect
> > performance stats per client, but getting empty list without any numbers
> >
> > I even run dd on client against mounted ceph fs, but output is only
> > like this:
> >
> > #> ceph fs perf stats 0 4638 192.168.121.1
> >
> > {"version": 2, "global_counters": ["cap_hit", "read_latency",
> > "write_latency", "metadata_latency", "dentry_lease", "opened_files",
> > "pinned_icaps", "
> > opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency",
> > "stdev_read_latency", "avg_write_latency", "stdev_write_latency",
> > "avg_metada
> > ta_latency", "stdev_metadata_latency"], "counters": [],
> > "client_metadata": {}, "global_metrics": {}, "metrics":
> > {"delayed_ranks": []}}
> >
> > Do I need to set some extra options?
> >
> Were you using the ceph-fuse/libcephfs user space clients ? if so you
> need to manually enable the 'client_collect_and_send_global_metrics'
> option, which is disabled in Ceph Pacific 16.2.13.
>
> While if you were using the kclient you need to make sure the
> 'disable_send_metrics' module parameter is 'false' for ceph.ko module.
>
> Thanks
>
> - Xiubo
>
>
> > Does it work for some of you guys?
> >
> > Thank you
> >
> > dp
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph fs perf stats output is empty

[Jos Collin]

Each array has different types. You need to look at the below cephfs-top 
code and see how they are interpreted.


[1] 
https://github.com/ceph/ceph/blob/main/src/tools/cephfs/top/cephfs-top#L66-L83
[2] 
https://github.com/ceph/ceph/blob/main/src/tools/cephfs/top/cephfs-top#L641-L714
[3] 
https://github.com/ceph/ceph/blob/main/src/tools/cephfs/top/cephfs-top#L110-L143
[4] 
https://github.com/ceph/ceph/blob/main/src/tools/cephfs/top/cephfs-top#L775-L823


On 12/06/23 14:23, Denis Polom wrote:


Hi,

yes, I've found that trick was I had to wait for about 15 sec to see 
the metrics.


Now I can see some numbers. Are units there in miliseconds? And also I 
see 2 numbers reported - is the first value actual and second is delta?


  "client.4636": [
    [
  924,
  4
    ],
    [
  0,
  0
    ],
    [
  10,
  981017512
    ],
    [
  0,
  33484910
    ],
    [
  136,
  0
    ],
    [
  1,
  2
    ],
    [
  2,
  2
    ],
    [
  1,
  2
    ],
    [
  0,
  0
    ],
    [
  302,
  1266679808
    ],
    [
  0,
  0
    ],
    [
  0,
  0
    ],
    [
  0,
  36361015
    ],
    [
  4205537661271535,
  302
    ],
    [
  0,
  11161636
    ],
    [
  190208004421472,
  3
    ]
  ]
    }
  },

Thx!



On 6/12/23 06:36, Jos Collin wrote:
Additionally, the first `ceph fs perf stats` output would be empty or 
outdated in most cases. You need to query a few times to get the 
latest values. So try `watch ceph fs perf stats`.


On Mon, 12 Jun 2023 at 06:30, Xiubo Li  wrote:


On 6/10/23 05:35, Denis Polom wrote:
> Hi
>
> I'm running latest Ceph Pacific 16.2.13 with Cephfs. I need to
collect
> performance stats per client, but getting empty list without
any numbers
>
> I even run dd on client against mounted ceph fs, but output is
only
> like this:
>
> #> ceph fs perf stats 0 4638 192.168.121.1
>
> {"version": 2, "global_counters": ["cap_hit", "read_latency",
> "write_latency", "metadata_latency", "dentry_lease",
"opened_files",
> "pinned_icaps", "
> opened_inodes", "read_io_sizes", "write_io_sizes",
"avg_read_latency",
> "stdev_read_latency", "avg_write_latency", "stdev_write_latency",
> "avg_metada
> ta_latency", "stdev_metadata_latency"], "counters": [],
> "client_metadata": {}, "global_metrics": {}, "metrics":
> {"delayed_ranks": []}}
>
> Do I need to set some extra options?
>
Were you using the ceph-fuse/libcephfs user space clients ? if so
you
need to manually enable the 'client_collect_and_send_global_metrics'
option, which is disabled in Ceph Pacific 16.2.13.

While if you were using the kclient you need to make sure the
'disable_send_metrics' module parameter is 'false' for ceph.ko
module.

Thanks

- Xiubo


> Does it work for some of you guys?
>
> Thank you
>
> dp
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io