[ceph-users] blustore osd nearfull but no pgs on it
Hi, after a massive rebalance(tunables) my small SSD-OSDs are getting full, i changed my crush rules so there are actual no pgs/pools on it, but the disks stay full: ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 158 ssd 0.21999 1.0 224 GiB 194 GiB 193 GiB 22 MiB 1002 MiB 30 GiB 86.68 1.49 0 up osd.158 inferring bluefs devices from bluestore path 1 : device size 0x37e440 : own 0x[1ad3f0~23c60] = 0x23c60 : using 0x3963(918 MiB) : bluestore has 0x46e2d(18 GiB) available when i recreate the osd the osd gets full again any suggestion? thx & best regards ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: blustore osd nearfull but no pgs on it
thx for your reply, it shows nothing,... there are no pgs on the osd,...
best regards
On 17.11.23 23:09, Eugen Block wrote:
After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
show you which PGs are created there and then you’ll know which pool
they belong to, then check again the crush rule for that pool. You can
paste the outputs here.
Zitat von Debian :
Hi,
after a massive rebalance(tunables) my small SSD-OSDs are getting
full, i changed my crush rules so there are actual no pgs/pools on
it, but the disks stay full:
ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
nautilus (stable)
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
158 ssd 0.21999 1.0 224 GiB 194 GiB 193 GiB 22 MiB 1002
MiB 30 GiB 86.68 1.49 0 up osd.158
inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] =
0x23c60 : using 0x3963(918 MiB) : bluestore has
0x46e2d(18 GiB) available
when i recreate the osd the osd gets full again
any suggestion?
thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: blustore osd nearfull but no pgs on it
Hi,
the block.db size ist default and not custom configured:
current:
bluefs.db_used_bytes: 9602859008
bluefs.db_used_bytes: 469434368
ceph daemon osd.149 config show
"bluestore_bitmapallocator_span_size": "1024",
"bluestore_block_db_size": "0",
"bluestore_block_size": "107374182400",
"bluestore_block_wal_size": "100663296",
"bluestore_cache_size": "0",
"bluestore_cache_size_hdd": "1073741824",
"bluestore_cache_size_ssd": "3221225472",
"bluestore_compression_max_blob_size": "0",
"bluestore_compression_max_blob_size_hdd": "524288",
"bluestore_compression_max_blob_size_ssd": "65536",
"bluestore_compression_min_blob_size": "0",
"bluestore_compression_min_blob_size_hdd": "131072",
"bluestore_compression_min_blob_size_ssd": "8192",
"bluestore_extent_map_inline_shard_prealloc_size": "256",
"bluestore_extent_map_shard_max_size": "1200",
"bluestore_extent_map_shard_min_size": "150",
"bluestore_extent_map_shard_target_size": "500",
"bluestore_extent_map_shard_target_size_slop": "0.20",
"bluestore_max_alloc_size": "0",
"bluestore_max_blob_size": "0",
"bluestore_max_blob_size_hdd": "524288",
"bluestore_max_blob_size_ssd": "65536",
"bluestore_min_alloc_size": "0",
"bluestore_min_alloc_size_hdd": "65536",
"bluestore_min_alloc_size_ssd": "4096",
"bluestore_prefer_deferred_size": "0",
"bluestore_prefer_deferred_size_hdd": "32768",
"bluestore_prefer_deferred_size_ssd": "0",
"bluestore_rocksdb_options":
"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2",
"bluefs_alloc_size": "1048576",
"bluefs_allocator": "hybrid",
"bluefs_buffered_io": "false",
"bluefs_check_for_zeros": "false",
"bluefs_compact_log_sync": "false",
"bluefs_log_compact_min_ratio": "5.00",
"bluefs_log_compact_min_size": "16777216",
"bluefs_max_log_runway": "4194304",
"bluefs_max_prefetch": "1048576",
"bluefs_min_flush_size": "524288",
"bluefs_min_log_runway": "1048576",
"bluefs_preextend_wal_files": "false",
"bluefs_replay_recovery": "false",
"bluefs_replay_recovery_disable_compact": "false",
"bluefs_shared_alloc_size": "65536",
"bluefs_sync_write": "false",
which the osd performance counter i cannot determine who is using the
memory,...
thx & best regards
On 18.11.23 09:05, Eugen Block wrote:
Do you have a large block.db size defined in the ceph.conf (or config
store)?
Zitat von Debian :
thx for your reply, it shows nothing,... there are no pgs on the osd,...
best regards
On 17.11.23 23:09, Eugen Block wrote:
After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
show you which PGs are created there and then you’ll know which pool
they belong to, then check again the crush rule for that pool. You
can paste the outputs here.
Zitat von Debian :
Hi,
after a massive rebalance(tunables) my small SSD-OSDs are getting
full, i changed my crush rules so there are actual no pgs/pools on
it, but the disks stay full:
ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
nautilus (stable)
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
158 ssd 0.21999 1.0 224 GiB 194 GiB 193 GiB 22 MiB 1002
MiB 30 GiB 86.68 1.49 0 up osd.158
inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] =
0x23c60 : using 0x3963(918 MiB) : bluestore has
0x46e2d(18 GiB) available
when i recreate the osd the osd gets full again
any suggestion?
thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: blustore osd nearfull but no pgs on it
Hi,
ohh that is exactly my problem, my Cluster is healthy and no rebalance
active.
I have only to wait that the old osdmaps get cleaned up,...
thx!
On 20.11.23 10:42, Michal Strnad wrote:
Hi.
Try to look on thread "Disks are filling up even if there is not a
single placement group on them" in this mailing list. Maybe you
encounter the same problem as me.
Michal
On 11/20/23 08:56, Debian wrote:
Hi,
the block.db size ist default and not custom configured:
current:
bluefs.db_used_bytes: 9602859008
bluefs.db_used_bytes: 469434368
ceph daemon osd.149 config show
"bluestore_bitmapallocator_span_size": "1024",
"bluestore_block_db_size": "0",
"bluestore_block_size": "107374182400",
"bluestore_block_wal_size": "100663296",
"bluestore_cache_size": "0",
"bluestore_cache_size_hdd": "1073741824",
"bluestore_cache_size_ssd": "3221225472",
"bluestore_compression_max_blob_size": "0",
"bluestore_compression_max_blob_size_hdd": "524288",
"bluestore_compression_max_blob_size_ssd": "65536",
"bluestore_compression_min_blob_size": "0",
"bluestore_compression_min_blob_size_hdd": "131072",
"bluestore_compression_min_blob_size_ssd": "8192",
"bluestore_extent_map_inline_shard_prealloc_size": "256",
"bluestore_extent_map_shard_max_size": "1200",
"bluestore_extent_map_shard_min_size": "150",
"bluestore_extent_map_shard_target_size": "500",
"bluestore_extent_map_shard_target_size_slop": "0.20",
"bluestore_max_alloc_size": "0",
"bluestore_max_blob_size": "0",
"bluestore_max_blob_size_hdd": "524288",
"bluestore_max_blob_size_ssd": "65536",
"bluestore_min_alloc_size": "0",
"bluestore_min_alloc_size_hdd": "65536",
"bluestore_min_alloc_size_ssd": "4096",
"bluestore_prefer_deferred_size": "0",
"bluestore_prefer_deferred_size_hdd": "32768",
"bluestore_prefer_deferred_size_ssd": "0",
"bluestore_rocksdb_options":
"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2",
"bluefs_alloc_size": "1048576",
"bluefs_allocator": "hybrid",
"bluefs_buffered_io": "false",
"bluefs_check_for_zeros": "false",
"bluefs_compact_log_sync": "false",
"bluefs_log_compact_min_ratio": "5.00",
"bluefs_log_compact_min_size": "16777216",
"bluefs_max_log_runway": "4194304",
"bluefs_max_prefetch": "1048576",
"bluefs_min_flush_size": "524288",
"bluefs_min_log_runway": "1048576",
"bluefs_preextend_wal_files": "false",
"bluefs_replay_recovery": "false",
"bluefs_replay_recovery_disable_compact": "false",
"bluefs_shared_alloc_size": "65536",
"bluefs_sync_write": "false",
which the osd performance counter i cannot determine who is using the
memory,...
thx & best regards
On 18.11.23 09:05, Eugen Block wrote:
Do you have a large block.db size defined in the ceph.conf (or
config store)?
Zitat von Debian :
thx for your reply, it shows nothing,... there are no pgs on the
osd,...
best regards
On 17.11.23 23:09, Eugen Block wrote:
After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
show you which PGs are created there and then you’ll know which
pool they belong to, then check again the crush rule for that
pool. You can paste the outputs here.
Zitat von Debian :
Hi,
after a massive rebalance(tunables) my small SSD-OSDs are getting
full, i changed my crush rules so there are actual no pgs/pools
on it, but the disks stay full:
ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
nautilus (stable)
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
158 ssd 0.21999 1.0 224 GiB 194 GiB 193 GiB 22 MiB 1002
MiB 30 GiB 86.68 1.49 0 up osd.158
inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23
[ceph-users] Re: blustore osd nearfull but no pgs on it
Hi,
yes all of my small osds are affected
i found the issue, my cluster is healthy and my rebalance finished - i
have only to wait that my old osdmaps get cleaned up.
like in the thread "Disks are filling up even if there is not a single
placement group on them"
thx!
On 20.11.23 11:36, Eugen Block wrote:
You provide only a few details at a time, it would help to get a full
picture if you provided the output Wesley asked for (ceph df detail,
ceph tell osd.158 status, ceph osd df tree). Is osd.149 now the
problematic one or did you just add output from a different osd?
It's not really clear what you're doing without the necessary context.
You can just add the 'ceph daemon osd.{OSD} perf dump' output here or
in some pastebin.
Zitat von Debian :
Hi,
the block.db size ist default and not custom configured:
current:
bluefs.db_used_bytes: 9602859008
bluefs.db_used_bytes: 469434368
ceph daemon osd.149 config show
"bluestore_bitmapallocator_span_size": "1024",
"bluestore_block_db_size": "0",
"bluestore_block_size": "107374182400",
"bluestore_block_wal_size": "100663296",
"bluestore_cache_size": "0",
"bluestore_cache_size_hdd": "1073741824",
"bluestore_cache_size_ssd": "3221225472",
"bluestore_compression_max_blob_size": "0",
"bluestore_compression_max_blob_size_hdd": "524288",
"bluestore_compression_max_blob_size_ssd": "65536",
"bluestore_compression_min_blob_size": "0",
"bluestore_compression_min_blob_size_hdd": "131072",
"bluestore_compression_min_blob_size_ssd": "8192",
"bluestore_extent_map_inline_shard_prealloc_size": "256",
"bluestore_extent_map_shard_max_size": "1200",
"bluestore_extent_map_shard_min_size": "150",
"bluestore_extent_map_shard_target_size": "500",
"bluestore_extent_map_shard_target_size_slop": "0.20",
"bluestore_max_alloc_size": "0",
"bluestore_max_blob_size": "0",
"bluestore_max_blob_size_hdd": "524288",
"bluestore_max_blob_size_ssd": "65536",
"bluestore_min_alloc_size": "0",
"bluestore_min_alloc_size_hdd": "65536",
"bluestore_min_alloc_size_ssd": "4096",
"bluestore_prefer_deferred_size": "0",
"bluestore_prefer_deferred_size_hdd": "32768",
"bluestore_prefer_deferred_size_ssd": "0",
"bluestore_rocksdb_options":
"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2",
"bluefs_alloc_size": "1048576",
"bluefs_allocator": "hybrid",
"bluefs_buffered_io": "false",
"bluefs_check_for_zeros": "false",
"bluefs_compact_log_sync": "false",
"bluefs_log_compact_min_ratio": "5.00",
"bluefs_log_compact_min_size": "16777216",
"bluefs_max_log_runway": "4194304",
"bluefs_max_prefetch": "1048576",
"bluefs_min_flush_size": "524288",
"bluefs_min_log_runway": "1048576",
"bluefs_preextend_wal_files": "false",
"bluefs_replay_recovery": "false",
"bluefs_replay_recovery_disable_compact": "false",
"bluefs_shared_alloc_size": "65536",
"bluefs_sync_write": "false",
which the osd performance counter i cannot determine who is using the
memory,...
thx & best regards
On 18.11.23 09:05, Eugen Block wrote:
Do you have a large block.db size defined in the ceph.conf (or
config store)?
Zitat von Debian :
thx for your reply, it shows nothing,... there are no pgs on the
osd,...
best regards
On 17.11.23 23:09, Eugen Block wrote:
After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
show you which PGs are created there and then you’ll know which
pool they belong to, then check again the crush rule for that
pool. You can paste the outputs here.
Zitat von Debian :
Hi,
after a massive rebalance(tunables) my small SSD-OSDs are getting
full, i changed my crush rules so there are actual no pgs/pools
on it, but the disks stay full:
ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
naut
[ceph-users] blustore osd nearfull but no pgs on it
Hi, after a massive rebalance(tunables) my small SSD-OSDs are getting full, i changed my crush rules so there are actual no pgs/pools on it, but the disks stay full: ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable) ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 158 ssd 0.21999 1.0 224 GiB 194 GiB 193 GiB 22 MiB 1002 MiB 30 GiB 86.68 1.49 0 up osd.158 inferring bluefs devices from bluestore path 1 : device size 0x37e440 : own 0x[1ad3f0~23c60] = 0x23c60 : using 0x3963(918 MiB) : bluestore has 0x46e2d(18 GiB) available when i recreate the osd the osd gets full again any suggestion? thx & best regards ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph and raid 1 replication
Hi every one, I'm new to ceph and I'm still studying it. In my company we decided to test ceph for possible further implementations. Although I undestood its capabilities I'm still doubtful about how to setup replication. Once implemented in production I can accept a little lacking of performance in favor of stability and night sleep, hence, if in testing area I can introduce ceph as network storage, I'd like to replicate some osds' drives as I'd do with raid 1 once in production. The goal would be hosting data for kubernetes storage classes. so questions are: 1) what do you think about this kind of solution 2) How can I setup full replication between osds? thanks in advance Rob ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph and raid 1 replication
Thanks for considerations. On 4/3/24 13:08, Janne Johansson wrote: Hi every one, I'm new to ceph and I'm still studying it. In my company we decided to test ceph for possible further implementations. Although I undestood its capabilities I'm still doubtful about how to setup replication. Default settings in ceph will give you replication = 3, which is like RAID-1 but with three drives having the same data. It is just not made on per-disk basis, but all stored data will have two extra copies on other drives (on separate hosts) Once implemented in production I can accept a little lacking of performance in favor of stability and night sleep, hence, if in testing area I can introduce ceph as network storage, I'd like to replicate some osds' drives as I'd do with raid 1 once in production. As with zfs (and btrfs and other storage solutions) you are most often best served by handing over all drives raw as they are, and let the storage system handle the redundancy on a higher level and not build raid-1s and hand those over to the storage. The goal would be hosting data for kubernetes storage classes. so questions are: 1) what do you think about this kind of solution Bad idea 2) How can I setup full replication between osds? Not really needed. Go with the defaults, allow ceph to place 3 copies of each piece of data spread out on three or more separate OSD hosts and move on to the interesting parts of actually using the storage instead of trying to "fix" something which isn't broken by default. Ceph will not make full copies of whole OSDs, rather pools will be made up of many PGs and each PG will be replicated as needed to give you three copies of each, just not to the same OSDs. It will also auto-repair to other drives and hosts and auto-balance data, which a raid-1 set would not do unless you have unused hot spares waiting for disasters. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph recipe for nfs exports
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the
official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found here
below.
###
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81
--cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \
--initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate
--allow-fqdn-hostname --ssl-dashboard-port 443 \
--dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key
/etc/ssl/wildcard.it/wildcard.it.key \
--allow-overwrite --cleanup-on-failure
cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k
/etc/ceph/ceph.client.admin.keyring
cephadm add-repo --release reef && cephadm install ceph-common
--> adding hosts and set labels
for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f
-i /etc/ceph/ceph.pub root@$IP ; done
ceph orch host add cephstage01 10.20.20.81 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstage02 10.20.20.82 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstage03 10.20.20.83 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstagedatanode01 10.20.20.84 --labels
osd,nfs,prometheus
ceph orch host add cephstagedatanode02 10.20.20.85 --labels
osd,nfs,prometheus
ceph orch host add cephstagedatanode03 10.20.20.86 --labels
osd,nfs,prometheus
--> network setup and daemons deploy
ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24
ceph orch apply mon
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply mgr
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply prometheus
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
ceph orch apply grafana
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
ceph orch apply node-exporter
ceph orch apply alertmanager
ceph config set mgr mgr/cephadm/secure_monitoring_stack true
--> disks and osd setup
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do
ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f
/dev/sdc"; done
ceph config set mgr mgr/cephadm/device_enhanced_scan true
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ;
doceph orch device ls --hostname=$IP --wide --refresh ; done
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ;
doceph orch device zap $IP /dev/sdb; done
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ;
doceph orch device zap $IP /dev/sdc ; done
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ;
doceph orch daemon add osd $IP:/dev/sdb ; done
for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ;
doceph orch daemon add osd $IP:/dev/sdc ; done
--> ganesha nfs cluster
ceph mgr module enable nfs
ceph fs volume create vol1
ceph nfs cluster create nfs-cephfs
"cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress
--virtual-ip 192.168.7.80 --ingress-mode default
ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt
--fsname vol1
--> nfs mount
mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the
latters I installed one 3tb ssd, for the data, and one 300gb ssd for the
journaling but
my problems are :
- Although I can mount the export I can't write on it
- I can't understand how to use the sdc disks for journaling
- I can't understand the concept of "pseudo path"
here below you can find the json output of the exports
--> check
ceph nfs export ls nfs-cephfs
ceph nfs export info nfs-cephfs /mnt
json file
-
{
"export_id": 1,
"path": "/",
"cluster_id": "nfs-cephfs",
"pseudo": "/mnt",
"access_type": "RW",
"squash": "none",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "nfs.nfs-cephfs.1",
"fs_name": "vol1"
},
"clients": []
}
Thanks in advance
Rob
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph recipe for nfs exports
first of all thanks to all!
Al supposed by Robert Sander, I get "permission denied" but even writing
with root privileges I get the same error.
As soon as I can I'll test your suggestions and update the thread.
Thanks again
On 4/24/24 16:05, Adam King wrote:
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set
things up doesn't look to different than one of our integration tests
for ingress over nfs
(https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress.yaml)
and that test tests a simple read/write to the export after
creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD
spec. For example
*service_type: osd
service_id: foo
placement:
hosts:
- vm-00
spec:
data_devices:
paths:
- /dev/vdb
journal_devices:
paths:
- /dev/vdc*
that will make non-colocated OSDs where the devices from the
journal_devices section are used as journal devices for the OSDs on
the devices in the data_devices section. Although I'd recommend
looking through
https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications
<https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and
see if there are any other filtering options than the path that can be
used first. It's possible the path the device gets can change on
reboot and you could end up with cepadm using a device you don't want
it to for this as that other device gets the path another device held
previously.
- I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path
nfs-ganesha will present to the user. There is another argument to
`ceph nfs export create` which is just "path" rather than pseudo-path
that marks what actual path within the cephfs the export is mounted
on. It's optional and defaults to "/" (so the export you made is
mounted at the root of the fs). I think that's the one that really
matters. The pseudo-path seems to just act like a user facing name for
the path.
On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian
wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the
official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found
here
below.
###
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81
--cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid
$FSID --initial-dashboard-user adm \
--initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate
--allow-fqdn-hostname --ssl-dashboard-port 443 \
--dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt
<http://wildcard.it/wildcard.it.crt> --dashboard-key
/etc/ssl/wildcard.it/wildcard.it.key
<http://wildcard.it/wildcard.it.key> \
--allow-overwrite --cleanup-on-failure
cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k
/etc/ceph/ceph.client.admin.keyring
cephadm add-repo --release reef && cephadm install ceph-common
--> adding hosts and set labels
for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do
ssh-copy-id -f
-i /etc/ceph/ceph.pub root@$IP ; done
ceph orch host add cephstage01 10.20.20.81 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstage02 10.20.20.82 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstage03 10.20.20.83 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstagedatanode01 10.20.20.84 --labels
osd,nfs,prometheus
ceph orch host add cephstagedatanode02 10.20.20.85 --labels
osd,nfs,prometheus
ceph orch host add cephstagedatanode03 10.20.20.86 --labels
osd,nfs,prometheus
--> network setup and daemons deploy
ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24
<http://10.20.20.0/24,192.168.7.0/24>
ceph orch apply mon
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply mgr
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply prometheus
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
ceph orch apply grafana
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
[ceph-users] Re: ceph recipe for nfs exports
Hi you all,
I did the changes suggested but the situation is still the same, I set
the squash to "all", since I want only "nobody:nogroup" ids but I can't
understand where the path should point.
If I understood it well, I pass a far disks ( unpartitioined and so
unformatted ) to the osd daemon, and then I create nfs daemons, ceph
will autonomously link the nfs shares on the fs managed by the osds, is
it correct?
Don't I have to create any filesystem on the osd?
By the way this is the dump
root@cephstage01:~# ceph nfs export info nfs-cephfs /mnt
{
"access_type": "RW",
"clients": [],
"cluster_id": "nfs-cephfs",
"export_id": 1,
"fsal": {
"fs_name": "vol1",
"name": "CEPH",
"user_id": "nfs.nfs-cephfs.1"
},
"path": "/",
"protocols": [
4
],
"pseudo": "/mnt",
"security_label": true,
"squash": "all",
"transports": [
"TCP"
]
}
I can mount it correctly, but when I try to write or touch any file in
it, it returns me "Permission denied"
❯ sudo mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
❯ touch /mnt/ceph/pino
touch: cannot touch '/mnt/ceph/pino': Permission denied
any suggestion will be appreciated
Rob
On 4/24/24 16:05, Adam King wrote:
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set
things up doesn't look to different than one of our integration tests
for ingress over nfs
(https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress.yaml)
and that test tests a simple read/write to the export after
creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD
spec. For example
*service_type: osd
service_id: foo
placement:
hosts:
- vm-00
spec:
data_devices:
paths:
- /dev/vdb
journal_devices:
paths:
- /dev/vdc*
that will make non-colocated OSDs where the devices from the
journal_devices section are used as journal devices for the OSDs on
the devices in the data_devices section. Although I'd recommend
looking through
https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications
<https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and
see if there are any other filtering options than the path that can be
used first. It's possible the path the device gets can change on
reboot and you could end up with cepadm using a device you don't want
it to for this as that other device gets the path another device held
previously.
- I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path
nfs-ganesha will present to the user. There is another argument to
`ceph nfs export create` which is just "path" rather than pseudo-path
that marks what actual path within the cephfs the export is mounted
on. It's optional and defaults to "/" (so the export you made is
mounted at the root of the fs). I think that's the one that really
matters. The pseudo-path seems to just act like a user facing name for
the path.
On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian
wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the
official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found
here
below.
###
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81
--cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid
$FSID --initial-dashboard-user adm \
--initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate
--allow-fqdn-hostname --ssl-dashboard-port 443 \
--dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt
<http://wildcard.it/wildcard.it.crt> --dashboard-key
/etc/ssl/wildcard.it/wildcard.it.key
<http://wildcard.it/wildcard.it.key> \
--allow-overwrite --cleanup-on-failure
cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k
/etc/ceph/ceph.client.admin.keyring
cephadm add-repo --release reef && cephadm install ceph-common
--> adding hosts and set labels
for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do
ssh-copy-id -f
-i /etc/ceph/ceph.pub root@$IP ; done
ceph orch host add cephstage01 10.20.20.81 --labels
_admin,mon,mgr,prometheus,grafana
ceph orch host add cephstage02 10.20.20.82 --labels
_admin,mon,mgr,prometheus
[ceph-users] service:mgr [ERROR] "Failed to apply:
Hi you all, it is a couple of days I'm facing this problem. Although I already destroyed the cluster a couple of times I continuously get these error I instruct ceph to place 3 daemons ceph orch apply mgr 3 --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" but root@cephstage01:/# ceph -s cluster: id: 0827effc-302a-45a6-bb97-758d2a9e6835 health: HEALTH_ERR Failed to apply 6 service(s): alertmanager,grafana,mgr,mon,node-exporter,prometheus failed to probe daemons or devices 2 mgr modules have failed 1 mgr modules have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 2 daemons, quorum cephstage01,cephstagedatanode01 (age 21m) mgr: cephstage01.ukwnlo(active, since 5h), standbys: cephstagedatanode01.vqbwbz osd: 0 root@cephstage01:/# ceph orch ls --service_name mgr --format yaml service_type: mgr service_name: mgr placement: hosts: - cephstage01:10.20.20.81 - cephstage02:10.20.20.82 - cephstage03:10.20.20.83 status: created: '2024-05-02T13:52:41.346547Z' last_refresh: '2024-05-02T14:06:31.823024Z' running: 2 size: 1 events: - 2024-05-02T13:51:49.993145Z service:mgr [INFO] "service was created" - '2024-05-02T13:53:45.566950Z service:mgr [ERROR] "Failed to apply: Cannot place on cephstage02, cephstage03: Unknown hosts"' and when I check those hosts root@cephstage01:/# ceph cephadm check-host cephstage03 cephstage03 (None) ok docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Hostname "cephstage03" matches what is expected. Host looks OK they look fine. every time I recreate the cluster it gives me the same errors: does anybody have any idea? thanks in advance ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Prefered distro for Ceph
Hi, I never tried anything else than debian. On 9/5/24 12:33 PM, Boris wrote: Didn't you already got the answer from the reddit thread? https://www.reddit.com/r/ceph/comments/1f88u6m/prefered_distro_for_ceph/ I always point here: https://docs.ceph.com/en/latest/start/os-recommendations/ and we are running very well with Ubuntu with, and without, the orchestrator. Am Do., 5. Sept. 2024 um 12:26 Uhr schrieb Denis Polom : Hi guys, what distro would you prefer and why for the production Ceph? We use Ubuntu on most of our Ceph clusters and some are Debian. Now we are thinking about unifying it by using only Debian or Ubuntu. I personally prefer Debian mainly for its stability and easy upgrade-in-place. What are yours preferences? Thank you ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
