[ceph-users] Re: Performance issues RGW (S3)
> > I am not sure adding more RGW's will increase the performance. That was a tangent. > To be clear, that means whatever.rgw.buckets.index ? >>> No, sorry my bad. .index is 32 and .data is 256. >> Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas >> on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's >> pg_num to a power of 2 ideally. > > No, my RBD pool is larger. My average PG per OSD is round 60-70. Ah. Aim for 100-200 with spinners. > >> Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index >> and 8192 for .data, assuming you have only RGW pools. And would be included >> to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump >> .log and .non-ec to 128 or 256 as well. >> If you have RBD or other pools colocated, those numbers would change. >> ^ above assume disabling the autoscaler > > I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Your index pool still only benefits from half of your OSDs with a value of 128. > Also doubled the .non-e and .log pools. Performance wise I don't see any > improvement. If I would see 10-20% improvement, I definitely would increase > it to 512 / 8192. > With 0.5MB object size I am still limited at about 150 up to 250 objects/s. > > The disks aren't saturated. The wr await is mostly around 1ms and does not > get higher when benchmarking with S3. Trust iostat about as far as you can throw it. > > Other suggestions, or does anyone else has suggestions? > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Performance issues RGW (S3)
On 2024-06-11 01:01, Anthony D'Atri wrote: To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each? Might be an option, just don't like the idea to host multiple components on nodes. But I'll consider it. I really don't like mixing mon/mgr with other components because of coupled failure domains, and past experience with mon misbehavior, but many people do that. ymmv. With a bunch of RGWs none of them need grow to consume significant resources, and it can be difficult to get an RGW daemon to itself really use all of a dedicated node. I am not sure adding more RGW's will increase the performance. Just tested with 1 and with 2 RGW's: Client 1 -> RGW Node A = 150-250 objects/s Client 1 -> RGW Node A = 60-120 objects/s and simultaneously Client 2 -> RGW Node B = 60-120 objects/s. Together makes 150-250 objects/s. So, it does not matter performance wise if I am using 1 or 2 RGW nodes. Client 1 -> HAProxy -> 3 RGW's = 150-250 objects/s. There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs? Index as the data pool has 256 PG's. To be clear, that means whatever.rgw.buckets.index ? No, sorry my bad. .index is 32 and .data is 256. Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. No, my RBD pool is larger. My average PG per OSD is round 60-70. Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. If you have RBD or other pools colocated, those numbers would change. ^ above assume disabling the autoscaler I bumped my .data pool from 256 to 1024 and .index from 32 to 128. Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. With 0.5MB object size I am still limited at about 150 up to 250 objects/s. The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. Other suggestions, or does anyone else has suggestions? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Attention: Documentation - mon states and names
Custom names were never really 100% implemented, and I would not be surprised if they don't work in Reef. > On Jun 11, 2024, at 14:02, Joel Davidow wrote: > > Zac, > > Thanks for your super-fast response and action on this. Those four items > are great and the corresponding email as reformatted looks good. > > Jana's point about cluster names is a good one. The deprecation of custom > cluster names, which appears to have started in octopus per > https://docs.ceph.com/en/octopus/rados/configuration/common/, alleviates > that confusion going forward but does not help with clusters already > deployed with custom names. > > Thanks again, > Joel > > On Tue, Jun 11, 2024 at 2:26 AM Janne Johansson wrote: > >>> Note the difference of convention in ceph command presentation. In >>> >> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#understanding-mon-status >> , >>> mon.X uses X to represent the portion of the command to be replaced by >> the >>> operator with a specific value. However, that may not be clear to all >>> readers, some of whom may read that as a literal X. I recommend switching >>> convention to something that makes visually explicit any portion of a >>> command that an operator has to replace with a specific value. One such >>> convention is to use <> as delimiters marking the portion of a command >> that >>> an operator has to replace with a specific value, minus the delimiters >>> themselves. I'm sure there are other conventions that would accomplish >> the >>> same goal and provide the <> convention as an example only. >> >> Yes, this is one of my main gripes. Many of the doc parts should more >> visibly point out which words or parts of names are the ones that you >> chose (by selecting a hostname for instance), it gets weird when you >> see "mon-1" or "client.rgw.rgw1" and you don't know which of those are >> to be changed to suit your environment and which are not. Sometimes >> the "ceph" word sneaks into paths because it is the name of the >> software (duh) but sometimes because it is the clustername. Now I >> don't hope many people change their clustername, but if you did, docs >> would be hard to follow in order to figure out where to replace "ceph" >> with your cluster name. >> >>> Also, the actual name of a mon is not clear due to the variety of mon >> name >>> formats. The value of the NAME column returned by ceph orch ps >>> --daemon-type mon and the return from ceph mon dump follow the format of >>> mon. whereas the value of name returned by ceph tell >>> mon_status, the mon line returned by ceph -s, and the return from ceph >> mon >>> stat follow the format of . Unifying the return for the mon name >>> value of all those commands could be helpful in establishing the format >> of >>> a mon name, though that is probably easier said than done. >>> >>> In addition, in >>> >> https://docs.ceph.com/en/latest/rados/configuration/mon-config-ref/#configuring-monitors >> , >>> mon names are stated to use alpha notation by convention, but that >>> convention is not followed by cephadm in the clusters that I've deployed. >>> Cephadm also uses a minimal ceph.conf file with configs in the mon >>> database. I recommend this section be updated to mention those changes. >> If >>> there is a way to explain what a mon name is or how it is formatted, >>> perhaps adding that to that same section would be good. >> >> >> >> -- >> May the most significant bit of your life be positive. >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Attention: Documentation - mon states and names
Zac, Thanks for your super-fast response and action on this. Those four items are great and the corresponding email as reformatted looks good. Jana's point about cluster names is a good one. The deprecation of custom cluster names, which appears to have started in octopus per https://docs.ceph.com/en/octopus/rados/configuration/common/, alleviates that confusion going forward but does not help with clusters already deployed with custom names. Thanks again, Joel On Tue, Jun 11, 2024 at 2:26 AM Janne Johansson wrote: > > Note the difference of convention in ceph command presentation. In > > > https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#understanding-mon-status > , > > mon.X uses X to represent the portion of the command to be replaced by > the > > operator with a specific value. However, that may not be clear to all > > readers, some of whom may read that as a literal X. I recommend switching > > convention to something that makes visually explicit any portion of a > > command that an operator has to replace with a specific value. One such > > convention is to use <> as delimiters marking the portion of a command > that > > an operator has to replace with a specific value, minus the delimiters > > themselves. I'm sure there are other conventions that would accomplish > the > > same goal and provide the <> convention as an example only. > > Yes, this is one of my main gripes. Many of the doc parts should more > visibly point out which words or parts of names are the ones that you > chose (by selecting a hostname for instance), it gets weird when you > see "mon-1" or "client.rgw.rgw1" and you don't know which of those are > to be changed to suit your environment and which are not. Sometimes > the "ceph" word sneaks into paths because it is the name of the > software (duh) but sometimes because it is the clustername. Now I > don't hope many people change their clustername, but if you did, docs > would be hard to follow in order to figure out where to replace "ceph" > with your cluster name. > > > Also, the actual name of a mon is not clear due to the variety of mon > name > > formats. The value of the NAME column returned by ceph orch ps > > --daemon-type mon and the return from ceph mon dump follow the format of > > mon. whereas the value of name returned by ceph tell > > mon_status, the mon line returned by ceph -s, and the return from ceph > mon > > stat follow the format of . Unifying the return for the mon name > > value of all those commands could be helpful in establishing the format > of > > a mon name, though that is probably easier said than done. > > > > In addition, in > > > https://docs.ceph.com/en/latest/rados/configuration/mon-config-ref/#configuring-monitors > , > > mon names are stated to use alpha notation by convention, but that > > convention is not followed by cephadm in the clusters that I've deployed. > > Cephadm also uses a minimal ceph.conf file with configs in the mon > > database. I recommend this section be updated to mention those changes. > If > > there is a way to explain what a mon name is or how it is formatted, > > perhaps adding that to that same section would be good. > > > > -- > May the most significant bit of your life be positive. > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Uttam Nagar Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Uttam Nagar Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Janakpuri Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Janakpuri Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] YounG_ Genuine Call Girls In Dwarka Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Dwarka Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Dwarka Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Dwarka Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Karol Bagh Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Karol Bagh Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Majnu Ka tilla Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Majnu Ka tilla Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Majnu Ka tilla Delhi 8448079011 Escort In Delhi
Call Vicky 8448079011 Book Call Girls In Majnu Ka tilla Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Genuine Call Girls In Mahipalpur Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Mahipalpur Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Mahipalpur Delhi 8448079011 Escort Service In Delhi
Call Vicky 8448079011 Book Call Girls In Mahipalpur Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Announcing go-ceph v0.28.0
Hi, We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.28.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] [community] Announcing Ceph Days London 2024 - July 17th
Hi All, Ceph Days is coming to London on July 17th 2024, co-hosted by IBM and Canonical. We're planning a full day of talks, lighting talks, and panel discussions all relating to various parts of the Ceph project. The CFP is now open and will close on the 30th June, so get your talks in ASAP! Event registration is also open now and space is limited, so book now to reserve your seat! Links to both registration and CFP are here: https://ceph.io/en/community/events/2024/ceph-days-london/ Looking forward to seeing you there! Thanks, Danny ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] 100 % Genuine Call Girls In Paharganj Delhi 8448079011 Escort In Delhi
Call Vicky 8448079011 Book Call Girls In Paharganj Delhi Independent /Mahipalpur, Available inbound and outbound escorts in India for home or hotel service at reasonable charges. The need for escorts or a professional companion in India is growing with the increasing number of city people. The person who has to be a companion should have a good sense of humor and should be able to understand the needs of the person they are accompanying. The person who has to be a companion should also have the ability to communicate well with this person. Service providers with us are much expected to follow all best Indian level qualities. High standards level quality is available to the clients. The person who is accompanying someone should be able to offer the companionship that the other needs and provide a sense of security. This will require them to be patient, friendly and respectful. The male fantasy is what a man would like to see in a woman. It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal It is the opposite of the female fantasy. The male fantasy is most often associated with physical attractiveness and sex appeal. Male fantasies are not just limited to visual aspects, but also include personality traits such as intelligence, humor and kindness. Fantasy of a partner who is physically attractive and smart (attractive) In the same way men are more likely than women to fantasize about being with an intelligent partner, men also tend to fantasize about having an attractive partner. Men may fantasize about their female partners being physically attractive and as such demonstrating high levels of tight bulging figure with ultimate sex appeal but also include personality traits such as intelligence, humor and kindness. Fanta About Us Green Park Escort Delhi Okhla Escort Delhi Lajpat Nagar Escort Delhi Delhi/New Delhi Escort Dwarka Escort Delhi Call Girls In Karol Bagh Delhi Paharganj Call Girls Delhi Majnu Ka tilla Escort Delhi Pitampura Escort Delhi GTB Nagar Escort Delhi Badarpur Escort Delhi Sarita Vihar Escort Delhi/ Nehru Place Call Girls Delhi Hauz Khas Escort Delhi Katwaria Sarai Escort Delhi Lado sarai Escort delhi Kalkaji Escort Delhi Safadarjung Escort Service Delhi Saket Escort Service Delhi Vasant Kunj Escort Delhi/ RK Puram Escort Delhi/ Bhikaji Cama Palace Escort Delhi Aerocity Escort Service Delhi Mahipalpur Escort Delhi Dhaula Kuan Escort Service Delhi Tilak Nagar Escort Service Delhi Chanakyapuri Escort Service Delhi Janakpuri Escort Service In Delhi Kirti Nagar Escort Delhi Patel Nagar Escort Service Delhi Patel Chowk Escort Service Delhi Connaught Place Escort Service Delhi Rajouri Garden Escort Delhi Anand Vihar Escort Service Delhi Kashmeri Gate Escort Service Delhi Ajmiri Gate Escort Service Delhi Jor Bagh Escort Service Delhi ALL Delhi Ncr ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Documentation for meaning of "tag cephfs" in OSD caps
There is a tiny bit more to it. The idea is that, when adding a data pool, any
cephfs client can access the new pool without changing and updating the caps.
To this end, the fs-caps must include 2 pieces of information, the application
name "cephfs" and the file system name (ceph can have multiple file systems).
Any cephfs enabled pool with the correct file system name will be accessible to
a properly authorized client of that file system without having to add that
pool to the client caps explicitly, as was necessary in older versions.
The 2 pieces of information are provided like:
application name cephfs: "tag cephfs"
file system name: "data=con-fs2"
One can check what is encoded for each pool using
ceph osd pool ls detail --format=json | jq '.[] | .pool_name,
.application_metadata'
For a ceph-fs pool, it will look something like
"con-fs2-data2"
{
"cephfs": {
"data": "con-fs2"
}
}
As of today, it seems indeed undocumented black magic and you need to search
very carefully to find ceph-user cases that discuss (issues with) these tags,
thereby explaining it as a side effect.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Eugen Block
Sent: Tuesday, June 11, 2024 2:14 PM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Documentation for meaning of "tag cephfs" in OSD caps
I assume it means that pools with an enabled application "cephfs" can
be targeted by specifying this tag instead of listing each pool
separately. Browsing through the code [1] seems to confirm that
(somehow, I'm not a dev):
> if (g.match.pool_tag.application == ng.match.pool_tag.application
But I agree, it's worth adding that to the docs.
[1]
https://github.com/ceph/ceph/blob/09e81319648dd504cfd94edfdd321c7163cefa98/src/osd/OSDCap.cc#L549
Zitat von Petr Bena :
> Hello
>
> In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that
>
> ceph fs authorize cephfs_a client.foo / r /bar rw Results in
>
> client.foo
> key: *key*
> caps: [mds] allow r, allow rw path=/bar
> caps: [mon] allow r
> caps: [osd] allow rw tag cephfs data=cephfs_a
>
>
> What is this "tag cephfs" thing? It seems like some undocumented
> black magic to me, since I can't find anything that documents it.
> Can someone explain how it works under the hood? What does it expand
> to? What does it limit and how?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS metadata pool size
Only in warning mode. And there were no PG splits or merges in the last 2 month. [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koep...@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Tue, Jun 11, 2024 at 3:32 PM Eugen Block wrote: > I don't think scrubs can cause this. Do you have autoscaler enabled? > > Zitat von Lars Köppel : > > > Hi, > > > > thank you for your response. > > > > I don't think this thread covers my problem, because the OSDs for the > > metadata pool fill up at different rates. So I would think this is no > > direct problem with the journal. > > Because we had earlier problems with the journal I changed some > > settings(see below). I already restarted all MDS multiple times but no > > change here. > > > > The health warnings regarding cache pressure resolve normally after a > > short period of time, when the heavy load on the client ends. Sometimes > it > > stays a bit longer because an rsync is running and copying data on the > > cluster(rsync is not good at releasing the caps). > > > > Could it be a problem if scrubs run most of the time in the background? > Can > > this block any other tasks or generate new data itself? > > > > Best regards, > > Lars > > > > > > global basic mds_cache_memory_limit > > 17179869184 > > global advanced mds_max_caps_per_client > >16384 > > global advanced > mds_recall_global_max_decay_threshold > >262144 > > global advanced mds_recall_max_decay_rate > >1.00 > > global advanced mds_recall_max_decay_threshold > > 262144 > > mds advanced mds_cache_trim_threshold > > 131072 > > mds advanced mds_heartbeat_grace > >120.00 > > mds advanced mds_heartbeat_reset_grace > >7400 > > mds advanced mds_tick_interval > >3.00 > > > > > > [image: ariadne.ai Logo] Lars Köppel > > Developer > > Email: lars.koep...@ariadne.ai > > Phone: +49 6221 5993580 <+4962215993580> > > ariadne.ai (Germany) GmbH > > Häusserstraße 3, 69115 Heidelberg > > Amtsgericht Mannheim, HRB 744040 > > Geschäftsführer: Dr. Fabian Svara > > https://ariadne.ai > > > > > > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block wrote: > > > >> Hi, > >> > >> can you check if this thread [1] applies to your situation? You don't > >> have multi-active MDS enabled, but maybe it's still some journal > >> trimming, or maybe misbehaving clients? In your first post there were > >> health warnings regarding cache pressure and cache size. Are those > >> resolved? > >> > >> [1] > >> > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE > >> > >> Zitat von Lars Köppel : > >> > >> > Hello everyone, > >> > > >> > short update to this problem. > >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size > >> ~50%). > >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up > and > >> > down a lot but the higher one has now also reached 'nearfull' status. > How > >> > is this possible? What is going on? > >> > > >> > Does anyone have a solution how to fix this without zapping the OSD? > >> > > >> > Best regards, > >> > Lars > >> > > >> > > >> > [image: ariadne.ai Logo] Lars Köppel > >> > Developer > >> > Email: lars.koep...@ariadne.ai > >> > Phone: +49 6221 5993580 <+4962215993580> > >> > ariadne.ai (Germany) GmbH > >> > Häusserstraße 3, 69115 Heidelberg > >> > Amtsgericht Mannheim, HRB 744040 > >> > Geschäftsführer: Dr. Fabian Svara > >> > https://ariadne.ai > >> > ___ > >> > ceph-users mailing list -- ceph-users@ceph.io > >> > To unsubscribe send an email to ceph-users-le...@ceph.io > >> > >> > >> ___ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS metadata pool size
I don't think scrubs can cause this. Do you have autoscaler enabled? Zitat von Lars Köppel : Hi, thank you for your response. I don't think this thread covers my problem, because the OSDs for the metadata pool fill up at different rates. So I would think this is no direct problem with the journal. Because we had earlier problems with the journal I changed some settings(see below). I already restarted all MDS multiple times but no change here. The health warnings regarding cache pressure resolve normally after a short period of time, when the heavy load on the client ends. Sometimes it stays a bit longer because an rsync is running and copying data on the cluster(rsync is not good at releasing the caps). Could it be a problem if scrubs run most of the time in the background? Can this block any other tasks or generate new data itself? Best regards, Lars global basic mds_cache_memory_limit 17179869184 global advanced mds_max_caps_per_client 16384 global advanced mds_recall_global_max_decay_threshold 262144 global advanced mds_recall_max_decay_rate 1.00 global advanced mds_recall_max_decay_threshold 262144 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_heartbeat_grace 120.00 mds advanced mds_heartbeat_reset_grace 7400 mds advanced mds_tick_interval 3.00 [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koep...@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Tue, Jun 11, 2024 at 2:05 PM Eugen Block wrote: Hi, can you check if this thread [1] applies to your situation? You don't have multi-active MDS enabled, but maybe it's still some journal trimming, or maybe misbehaving clients? In your first post there were health warnings regarding cache pressure and cache size. Are those resolved? [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE Zitat von Lars Köppel : > Hello everyone, > > short update to this problem. > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and > down a lot but the higher one has now also reached 'nearfull' status. How > is this possible? What is going on? > > Does anyone have a solution how to fix this without zapping the OSD? > > Best regards, > Lars > > > [image: ariadne.ai Logo] Lars Köppel > Developer > Email: lars.koep...@ariadne.ai > Phone: +49 6221 5993580 <+4962215993580> > ariadne.ai (Germany) GmbH > Häusserstraße 3, 69115 Heidelberg > Amtsgericht Mannheim, HRB 744040 > Geschäftsführer: Dr. Fabian Svara > https://ariadne.ai > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS metadata pool size
Hi, thank you for your response. I don't think this thread covers my problem, because the OSDs for the metadata pool fill up at different rates. So I would think this is no direct problem with the journal. Because we had earlier problems with the journal I changed some settings(see below). I already restarted all MDS multiple times but no change here. The health warnings regarding cache pressure resolve normally after a short period of time, when the heavy load on the client ends. Sometimes it stays a bit longer because an rsync is running and copying data on the cluster(rsync is not good at releasing the caps). Could it be a problem if scrubs run most of the time in the background? Can this block any other tasks or generate new data itself? Best regards, Lars global basic mds_cache_memory_limit 17179869184 global advanced mds_max_caps_per_client 16384 global advanced mds_recall_global_max_decay_threshold 262144 global advanced mds_recall_max_decay_rate 1.00 global advanced mds_recall_max_decay_threshold 262144 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_heartbeat_grace 120.00 mds advanced mds_heartbeat_reset_grace 7400 mds advanced mds_tick_interval 3.00 [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koep...@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Tue, Jun 11, 2024 at 2:05 PM Eugen Block wrote: > Hi, > > can you check if this thread [1] applies to your situation? You don't > have multi-active MDS enabled, but maybe it's still some journal > trimming, or maybe misbehaving clients? In your first post there were > health warnings regarding cache pressure and cache size. Are those > resolved? > > [1] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE > > Zitat von Lars Köppel : > > > Hello everyone, > > > > short update to this problem. > > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size > ~50%). > > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and > > down a lot but the higher one has now also reached 'nearfull' status. How > > is this possible? What is going on? > > > > Does anyone have a solution how to fix this without zapping the OSD? > > > > Best regards, > > Lars > > > > > > [image: ariadne.ai Logo] Lars Köppel > > Developer > > Email: lars.koep...@ariadne.ai > > Phone: +49 6221 5993580 <+4962215993580> > > ariadne.ai (Germany) GmbH > > Häusserstraße 3, 69115 Heidelberg > > Amtsgericht Mannheim, HRB 744040 > > Geschäftsführer: Dr. Fabian Svara > > https://ariadne.ai > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: degraded objects when setting different CRUSH rule on a pool, why?
Hi Stefan,
I assume the number of dropped replicas is related to the pool's
min_size. If you increase min_size to 3 you should see only one
replica dropped from the acting set. I didn't run too detailed tests,
but a first quick one seems to confirm that:
# Test with min_size 2, size 4
48.7 [6,2,11,10]p6 [6,2]p6
Changed the rule back to the previous rule, then changed it again with
new min_size:
# Test with min_size 3, size 4
48.7 [6,2,11,10]p6 [6,2,10]p6
I don't really have an explanation why they are degraded, though. I
think Frank already had some open bug reports for that, this topic
comes up every now and then without any reasonable explanation.
Regards,
Eugen
Zitat von Stefan Kooman :
Hi,
TL;DR:
Selecting a different CRUSH rule (stretch_rule, no device class) for
pool SSD results in degraded objects (unexpected) and misplaced
objects (expected). Why would Ceph drop up to two healthy copies?
Consider this two data center cluster:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.78400 root default
-10 0.39200 datacenter DC1
-3 0.39200 host pve1
0hdd 0.09799 osd.0 up 1.0 1.0
1hdd 0.09799 osd.1 up 1.0 1.0
4ssd 0.09799 osd.4 up 1.0 1.0
5ssd 0.09799 osd.5 up 1.0 1.0
-11 0.39200 datacenter DC2
-5 0.39200 host pve2
2hdd 0.09799 osd.2 up 1.0 1.0
3hdd 0.09799 osd.3 up 1.0 1.0
6ssd 0.09799 osd.6 up 1.0 1.0
7ssd 0.09799 osd.7 up 1.0 1.0
Pools available:
device_health_metrics
HDD
SSD
Let's focus on SSD for now. Crush rule in use for SSD pool:
rule SSD {
id 2
type replicated
min_size 1
max_size 10
step take default class ssd
step choose firstn 0 type host
step chooseleaf firstn 0 type osd
step emit
}
SSD pool replication settings: min_size=2, size=4
The new stretch rule to use:
rule stretch_rule {
id 3
type replicated
min_size 1
max_size 10
step take default
step take DC1
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
step take default
step take DC2
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
step emit
}
ceph pg ls for pool 3 (SSD):
3.0 184 0 00 7380705280
0 7458 active+clean 9m 926'7458 951:18695
[7,6,5,4]p7 [7,6,5,4]p7 2024-06-05T08:52:51.951936+0200
2024-06-05T08:52:51.951936+0200
3.1 169 0 00 6671566550
0 3242 active+clean18m 926'3242 951:14151
[4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:42.948682+0200
2024-06-05T08:54:42.948682+0200
3.2 221 0 00 8856419680
0 6989 active+clean 9m 926'6989 951:17645
[4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:53:27.981787+0200
2024-06-05T08:53:27.981787+0200
3.3 180 0 00 7165091840
0 3194 active+clean 9m 926'3194 951:14191
[4,5,6,7]p4 [4,5,6,7]p4 2024-06-05T08:54:29.584216+0200
2024-06-05T08:54:29.584216+0200
3.4 189 0 00 754417698 137
8 3616 active+clean 9m 926'3616 951:19245
[6,7,4,5]p6 [6,7,4,5]p6 2024-06-05T08:54:02.307323+0200
2024-06-05T08:54:02.307323+0200
3.5 188 0 00 7425433770
0 5992 active+clean 9m 926'5992 951:18862
[6,7,5,4]p6 [6,7,5,4]p6 2024-06-05T08:53:09.483136+0200
2024-06-05T08:53:09.483136+0200
3.6 191 0 00 769482752 150
16 6810 active+clean 9m 926'6810 951:30043
[7,6,4,5]p7 [7,6,4,5]p7 2024-06-05T08:53:46.646517+0200
2024-06-05T08:53:46.646517+0200
3.7 170 0 00 6815873790
0 10081 active+clean 9m 926'21581 951:28473
[4,5,7,6]p4 [4,5,7,6]p4 2024-06-05T08:54:16.047967+0200
2024-06-05T08:54:16.047967+0200
ceph pg ls when the new crush rule is selected for pool SSD:
3.0 184 3721860 7380705280
0 7458
active+recovery_wait+undersized+degraded+remapped14s 926'7458
955:18688 [1,0,3,7]p1 [4,7]p4
2024-06-05T08:52:51.951936+0200 2024-06-05T08:52:51.951936+0200
3.1 169 0 00 6671566550
0 3242
active+clean19m 926'3242 954:14154 [4,5,6,7]p4 [4,5,6,7]p4
2024-06-05T08:54:42.948682+0200 2024-06-05T08:54:42.948682+0200
3.2 221 444 0
[ceph-users] Re: [SPAM] Re: Ceph RBD, MySQL write IOPs - what is possible?
Hi, don’t expect solution on group, just direction. Here is link to the blog post https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ on youtube is presentation from nyc ceph days View performance from the client's perspective, run the measurement tools from inside the virtual machine. This approach will provide you the performance as experienced by the client. The most commonly used tool for performance measurement is fio. I strongly recommend using fio for your evaluation. also use ioping to measure latency. While fio will provide IOPS/ and latency metrics during load, ioping offers view of latency behavior when the machine is not under heavy load. Based on my previous experiences (not only mine, but also my team), many performance issues were related to network configurations or problems around the network infrastructure. As example we encountered a situation where a change made by the network team to the spine switches caused disk latency to increase from 3ms to 80-120ms. Other example which almost burn me was issue with one spine cards, which was not fully broken, monitoring not discovered it, tests shows everything is ok but on ceph we had many, many issues like flapping osd’s, like half of osds form 500 goes down, latency spikes time to time. Card had problems time to time but not during tests :) and of course AMD nodes before I discovered iommu=pt for kernel params. Belive me this c-states and power management on nodes are important. You already received very good advices from others, not much to add, look on your network drivers, rx queue, tx queue. for your information this cluster was not fine tuned, also e2e enc. is enabled 6 node cluster, all nvme 8x nvme per node, 512gb ram, 4x25GB lacp for public and another 4x25GB for cluster net. (malleanox cards) # rados bench -p test 10 write -t 8 -b 16K Rados bench results: Total time run: 10.0003 Total writes made: 113195 Write size: 16384 Object size:16384 Bandwidth (MB/sec): 176.862 Stddev Bandwidth: 27.047 Max bandwidth (MB/sec): 195.828 Min bandwidth (MB/sec): 107.906 Average IOPS: 11319 Stddev IOPS:1731.01 Max IOPS: 12533 Min IOPS: 6906 Average Latency(s): 0.000705734 Stddev Latency(s): 0.00224331 Max latency(s): 0.325178 Min latency(s): 0.000413413 This is test from fio with librbd it shows more or less vm performance. [test] ioengine=rbd clientname=admin pool=test rbdname=bench rw=randwrite bs=4k iodepth=256 direct=1 numjobs=1 fsync=0 size=10G runtime=300 time_based invalidate=0 test: (groupid=0, jobs=1): err= 0: pid=3495143: Tue Jun 11 11:56:04 2024 write: IOPS=83.6k, BW=326MiB/s (342MB/s)(95.6GiB/32msec); 0 zone resets slat (nsec): min=975, max=2665.0k, avg=3943.68, stdev=2820.21 clat (usec): min=399, max=225434, avg=3058.67, stdev=1801.25 and for iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=3503647: Tue Jun 11 11:57:48 2024 write: IOPS=1845, BW=7382KiB/s (7559kB/s)(159MiB/22033msec); 0 zone resets slat (nsec): min=2966, max=41133, avg=4381.81, stdev=1062.40 clat (usec): min=367, max=202364, avg=537.05, stdev=1009.49 and iodepth=256 and bs=16k test: (groupid=0, jobs=1): err= 0: pid=3505339: Tue Jun 11 12:03:27 2024 write: IOPS=79.6k, BW=1244MiB/s (1305MB/s)(365GiB/32msec); 0 zone resets slat (nsec): min=1815, max=4497.4k, avg=5671.20, stdev=3540.33 clat (usec): min=446, max=267567, avg=3208.34, stdev=2038.58 lat (usec): min=451, max=267571, avg=3214.01, stdev=2038.60 BR, Sebastian > On 11 Jun 2024, at 02:23, Mark Lehrer wrote: > > If they can do 1 TB/s with a single 16K write thread, that will be > quite impressive :DOtherwise not really applicable. Ceph scaling > has always been good. > > More seriously, would you mind sending a link to this? > > > Thanks! > > Mark > > On Mon, Jun 10, 2024 at 12:01 PM Anthony D'Atri > wrote: >> >> Eh? cf. Mark and Dan's 1TB/s presentation. >> >> On Jun 10, 2024, at 13:58, Mark Lehrer wrote: >> >> It >> seems like Ceph still hasn't adjusted to SSD performance. >> >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Documentation for meaning of "tag cephfs" in OSD caps
I assume it means that pools with an enabled application "cephfs" can be targeted by specifying this tag instead of listing each pool separately. Browsing through the code [1] seems to confirm that (somehow, I'm not a dev): if (g.match.pool_tag.application == ng.match.pool_tag.application But I agree, it's worth adding that to the docs. [1] https://github.com/ceph/ceph/blob/09e81319648dd504cfd94edfdd321c7163cefa98/src/osd/OSDCap.cc#L549 Zitat von Petr Bena : Hello In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that ceph fs authorize cephfs_a client.foo / r /bar rw Results in client.foo key: *key* caps: [mds] allow r, allow rw path=/bar caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_a What is this "tag cephfs" thing? It seems like some undocumented black magic to me, since I can't find anything that documents it. Can someone explain how it works under the hood? What does it expand to? What does it limit and how? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS metadata pool size
Hi, can you check if this thread [1] applies to your situation? You don't have multi-active MDS enabled, but maybe it's still some journal trimming, or maybe misbehaving clients? In your first post there were health warnings regarding cache pressure and cache size. Are those resolved? [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE Zitat von Lars Köppel : Hello everyone, short update to this problem. The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and down a lot but the higher one has now also reached 'nearfull' status. How is this possible? What is going on? Does anyone have a solution how to fix this without zapping the OSD? Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koep...@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Documentation for meaning of "tag cephfs" in OSD caps
Hello In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that ceph fs authorize cephfs_a client.foo / r /bar rw Results in client.foo key: *key* caps: [mds] allow r, allow rw path=/bar caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_a What is this "tag cephfs" thing? It seems like some undocumented black magic to me, since I can't find anything that documents it. Can someone explain how it works under the hood? What does it expand to? What does it limit and how? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS metadata pool size
Hello everyone, short update to this problem. The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and down a lot but the higher one has now also reached 'nearfull' status. How is this possible? What is going on? Does anyone have a solution how to fix this without zapping the OSD? Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koep...@ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Attention: Documentation - mon states and names
> Note the difference of convention in ceph command presentation. In > https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#understanding-mon-status, > mon.X uses X to represent the portion of the command to be replaced by the > operator with a specific value. However, that may not be clear to all > readers, some of whom may read that as a literal X. I recommend switching > convention to something that makes visually explicit any portion of a > command that an operator has to replace with a specific value. One such > convention is to use <> as delimiters marking the portion of a command that > an operator has to replace with a specific value, minus the delimiters > themselves. I'm sure there are other conventions that would accomplish the > same goal and provide the <> convention as an example only. Yes, this is one of my main gripes. Many of the doc parts should more visibly point out which words or parts of names are the ones that you chose (by selecting a hostname for instance), it gets weird when you see "mon-1" or "client.rgw.rgw1" and you don't know which of those are to be changed to suit your environment and which are not. Sometimes the "ceph" word sneaks into paths because it is the name of the software (duh) but sometimes because it is the clustername. Now I don't hope many people change their clustername, but if you did, docs would be hard to follow in order to figure out where to replace "ceph" with your cluster name. > Also, the actual name of a mon is not clear due to the variety of mon name > formats. The value of the NAME column returned by ceph orch ps > --daemon-type mon and the return from ceph mon dump follow the format of > mon. whereas the value of name returned by ceph tell > mon_status, the mon line returned by ceph -s, and the return from ceph mon > stat follow the format of . Unifying the return for the mon name > value of all those commands could be helpful in establishing the format of > a mon name, though that is probably easier said than done. > > In addition, in > https://docs.ceph.com/en/latest/rados/configuration/mon-config-ref/#configuring-monitors, > mon names are stated to use alpha notation by convention, but that > convention is not followed by cephadm in the clusters that I've deployed. > Cephadm also uses a minimal ceph.conf file with configs in the mon > database. I recommend this section be updated to mention those changes. If > there is a way to explain what a mon name is or how it is formatted, > perhaps adding that to that same section would be good. -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: About disk disk iops and ultil peak
Hi Anthony, I have 15 nodes 18HDD disk and 6 ssd disk per node Vào Th 3, 11 thg 6, 2024 vào lúc 10:29 Anthony D'Atri < anthony.da...@gmail.com> đã viết: > What specifically are your OSD devices? > > On Jun 10, 2024, at 22:23, Phong Tran Thanh > wrote: > > Hi ceph user! > > I am encountering a problem with IOPS and disk utilization of OSD. > Sometimes, my disk peaks in IOPS and utilization become too high, which > affects my cluster and causes slow operations to appear in the logs. > > 6/6/24 9:51:46 AM[WRN]Health check update: 0 slow ops, oldest one blocked > for 36 sec, osd.268 has slow ops (SLOW_OPS) > > 6/6/24 9:51:37 AM[WRN]Health check update: 0 slow ops, oldest one blocked > for 31 sec, osd.268 has slow ops (SLOW_OPS) > > > This is config tu reduce it, but its not resolve my problem > global advanced osd_mclock_profile > custom > > global advanced > osd_mclock_scheduler_background_best_effort_lim 0.10 > > global advanced > osd_mclock_scheduler_background_best_effort_res 0.10 > > global advanced > osd_mclock_scheduler_background_best_effort_wgt 1 > > global advanced osd_mclock_scheduler_background_recovery_lim > 0.10 > > global advanced osd_mclock_scheduler_background_recovery_res > 0.10 > > global advanced osd_mclock_scheduler_background_recovery_wgt > 1 > > global advanced osd_mclock_scheduler_client_lim > 0.40 > > global advanced osd_mclock_scheduler_client_res > 0.40 > > global advanced osd_mclock_scheduler_client_wgt 4 > > Hope someone can help me > > Thanks so much! > -- > > Email: tranphong...@gmail.com > Skype: tranphong079 > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > > -- Trân trọng, *Tran Thanh Phong* Email: tranphong...@gmail.com Skype: tranphong079 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: About disk disk iops and ultil peak
Hi Anthony! My osd is HDD 12TB 7200 and SSD 960GB for wal/db Thanks Anthony! Vào Th 3, 11 thg 6, 2024 vào lúc 10:29 Anthony D'Atri < anthony.da...@gmail.com> đã viết: > What specifically are your OSD devices? > > On Jun 10, 2024, at 22:23, Phong Tran Thanh > wrote: > > Hi ceph user! > > I am encountering a problem with IOPS and disk utilization of OSD. > Sometimes, my disk peaks in IOPS and utilization become too high, which > affects my cluster and causes slow operations to appear in the logs. > > 6/6/24 9:51:46 AM[WRN]Health check update: 0 slow ops, oldest one blocked > for 36 sec, osd.268 has slow ops (SLOW_OPS) > > 6/6/24 9:51:37 AM[WRN]Health check update: 0 slow ops, oldest one blocked > for 31 sec, osd.268 has slow ops (SLOW_OPS) > > > This is config tu reduce it, but its not resolve my problem > global advanced osd_mclock_profile > custom > > global advanced > osd_mclock_scheduler_background_best_effort_lim 0.10 > > global advanced > osd_mclock_scheduler_background_best_effort_res 0.10 > > global advanced > osd_mclock_scheduler_background_best_effort_wgt 1 > > global advanced osd_mclock_scheduler_background_recovery_lim > 0.10 > > global advanced osd_mclock_scheduler_background_recovery_res > 0.10 > > global advanced osd_mclock_scheduler_background_recovery_wgt > 1 > > global advanced osd_mclock_scheduler_client_lim > 0.40 > > global advanced osd_mclock_scheduler_client_res > 0.40 > > global advanced osd_mclock_scheduler_client_wgt 4 > > Hope someone can help me > > Thanks so much! > -- > > Email: tranphong...@gmail.com > Skype: tranphong079 > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > > -- Trân trọng, *Tran Thanh Phong* Email: tranphong...@gmail.com Skype: tranphong079 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
