[389-devel] Re: 300s delay when query cn=monitor
HI Erwin, Just seeing your mail, I got a wild idea: If I remember correctly a cn=monitor query collects some disk statistics and to do that it performs a stat on all mount points. A problem with some nfs mounted file system could then explain the delay. It would be interesting to see if "df" command is responsive ... (could be interesting to check the df output in sos report if you have it ) Regards, Pierre On Mon, Jul 12, 2021 at 8:25 AM Erwin Weitlaner wrote: > Thank you Thierry for your support. > > So I will try to get the thread dump .. If the problem comes with a > connection timeout from a (not listening) client, shoudn´t I see an error > log entry somewhere? I searched for that for hours but no hints .. Maybe > not the right idea but the problem came after our last linux patch, so if > others have similar problems maybe a code change issue? > > SG Erwin > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: PR 4783 - Issue 4764 - replicated operation sometime checks ACI
https://github.com/389ds/389-ds-base/pull/4783 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: Issue 4765 - database suffix unexpectdly changed from .db to .db4 #4766
A simple fix to solve a nasty side effect of moving include db.h to the db-bdb plugin: https://github.com/389ds/389-ds-base/pull/4766 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] PR test are failing because lmdb pacakages are not installed.
Hi all, When pushing commits on backend redesign phase 4 I got compile error on all builds because lmdb package are not installed (==> lmdb.h is missing) COPR build is successful (because I updated the rpm/389-ds-base.spec.in spec file and COPR use it to install the prerequisites but obviously PR test builds are different. ( I suspect that it is using some pre-built image ) Does anyone know how to fix that ? Regards Pierre -- -- 389 Directory Server Development Team --- Begin Message --- [389ds/389-ds-base] Compile workflow run Repository: 389ds/389-ds-base Workflow: Compile Duration: 4 minutes and 22.0 seconds Finished: 2021-04-30 09:19:17 UTC View results: https://github.com/389ds/389-ds-base/actions/runs/798944212 Jobs: * compile (Fedora 33 GCC) failed (3 annotations) * compile (Fedora 33 GCC Strict) failed (12 annotations) * compile (Fedora 33 GCC Static Analyzer) failed (11 annotations) * compile (Fedora 33 Clang) failed (6 annotations) * compile (Fedora 33 Clang -Weverything) failed (12 annotations) -- You are receiving this because this workflow ran on your branch. Manage your GitHub Actions notifications: https://github.com/settings/notifications --- End Message --- ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] please review: Issue 4653 - refactor ldbm backend to allow replacement of BDB - phase 3e - dbscan #4709
FYI: William already reviewed it but would like a second opinion. https://github.com/389ds/389-ds-base/pull/4709 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: Issue 4680 - 389ds coredump (@389ds/389-ds-base-nightly) in replica install with CA - PR 4715
https://github.com/389ds/389-ds-base/pull/4715 FYI a 1 liner fix. -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: PR 4665: issue 4585 - backend redesign phase 3c - dbregion test removal #4665
FYI: A second review is needed on that one (as most of the changes reviewed by Mark the first time have been removed) https://github.com/389ds/389-ds-base/pull/4665 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: PR 4622 - Backend redesign phase 3b - use dbimpl in replicatin plugin #4622
This one is much more simpler and easy to understand than phase 3a ! -;) https://github.com/389ds/389-ds-base/pull/4622 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[389-devel] Please review: issue 4469 Backend redesign phase 3
https://github.com/389ds/389-ds-base/pull/4546 FYI: FirstYear has already reviewed the code (Thank you William !) but due to the size and the complexity of the change, it would be better if two other reviewers could also have a look at it. -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Deref plugin entries == NULL #4525
As I understand things: yes. (I do not know if there is an explicit way to tells ldbm that a memory block is no more needed, in which case we could do something about it ... ) On Fri, Jan 15, 2021 at 12:11 PM thierry bordaz wrote: > > > On 1/15/21 11:53 AM, Pierre Rogier wrote: > > Hi Thierry, > > I was rather thinking about the key and value duplication when querying > the DB: > > When using bdb functions that is done implicitly. > bdb either copies the values in the DBT buffer or it alloc/realloc it > When mimicking bdb behavior with LDBM we will have to do that explicitly > in the LDBM plugin: > LDMB returns a memory mapped address that may be ummapped >once the txn is ended. So we must copy the result before closing the > txn. > If we have a read txn that protects the full operation lifespan, then we > could directly use the mapped address without needing to duplicate them. > > ah okay ! nice. > Just a question, if we have a txn covering a search with large candidate > list (unindexed), does that mean that by default each db key/value will > remain mapped until txn commit ? > > thanks > thierry > > > Pierre > > On Fri, Jan 15, 2021 at 10:53 AM thierry bordaz > wrote: > >> >> >> On 1/14/21 12:32 PM, Pierre Rogier wrote: >> >> Hi William, >> >> > It's a scenario we will need to fix via your BE work because of the >> MVCC transaction model that >> > LMDB will force us to adopt :) >> >> As I see things in the early phases the lmdb read txn will probably only >> be managed at the db plugin level rather than at backend level. That >> means that we will have the same inconsistency risk than today (i.e as if >> using bdb and the implicit txn). >> The txn model redesign you are speaking about should only occur in one of >> the last phases (once bdb does no more coexists with lmdb). >> It must be done because it could provide a serious performance boost for >> read operations (IMHO, In most cases we could avoid to duplicate the db >> data) >> >> Pierre, what duplicate are you thinking of ? str2entry ? >> >> But we should not do it while bdb is still around because of the risk of >> lock issue and excessive retries. >> >> Note I put a phasing section in >> >> https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing >> explaining that. But I guess I should move it within Ludwig's document >> that englobs it. >> >> >> Pierre >> >> On Thu, Jan 14, 2021 at 12:01 AM William Brown wrote: >> >>> >>> >>> > On 13 Jan 2021, at 21:24, Pierre Rogier wrote: >>> > >>> > Thank you Willian, >>> > So far your scenario (entry found when reading base entry but no more >>> existing when computing the candidates) is the only one that matches the >>> symptoms. >>> >>> It's a scenario we will need to fix via your BE work because of the MVCC >>> transaction model that LMDB will force us to adopt :) >>> >>> > And that triggered a thought: >>> > We cannot do anything for SUBTREE and ONE_LEVEL searches >>> > because the fact that the base entry id is not in the candidate may >>> be normal >>> > but IMHO we should improve the BASE search case. >>> > In this case the candidate list is directly set to the base entry id >>> > ==> if the candidate entry (in ldbm_back_next_search_entry) is not >>> found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY >>> error .. >>> >>> I suspect that Mark has seen this email and submitted a PR to resolve >>> this exact case :) >>> >>> >>> > >>> >Pierre >>> > >>> > >>> > On Wed, Jan 13, 2021 at 1:45 AM William Brown wrote: >>> > Hey there, >>> > >>> > https://github.com/389ds/389-ds-base/pull/4525/files >>> > >>> > I had a look and I can see a few possible contributing factors, but >>> without a core and the exact state I can't be sure if this is correct. It's >>> all just hypothetical from reading the code. >>> > >>> > >>> > The crash is in deref_do_deref_attr() which is called as part of >>> deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by >>> "./ldap/servers/slapd/result.c:1488:rc = plugin_call_plugins(pb, >>> SLAPI_PLUGIN_PRE_ENTRY_FN);" >>> > >>> > >>> > I think what's impo
[389-devel] Re: Deref plugin entries == NULL #4525
Hi Thierry, I was rather thinking about the key and value duplication when querying the DB: When using bdb functions that is done implicitly. bdb either copies the values in the DBT buffer or it alloc/realloc it When mimicking bdb behavior with LDBM we will have to do that explicitly in the LDBM plugin: LDMB returns a memory mapped address that may be ummapped once the txn is ended. So we must copy the result before closing the txn. If we have a read txn that protects the full operation lifespan, then we could directly use the mapped address without needing to duplicate them. Pierre On Fri, Jan 15, 2021 at 10:53 AM thierry bordaz wrote: > > > On 1/14/21 12:32 PM, Pierre Rogier wrote: > > Hi William, > > > It's a scenario we will need to fix via your BE work because of the MVCC > transaction model that > > LMDB will force us to adopt :) > > As I see things in the early phases the lmdb read txn will probably only > be managed at the db plugin level rather than at backend level. That > means that we will have the same inconsistency risk than today (i.e as if > using bdb and the implicit txn). > The txn model redesign you are speaking about should only occur in one of > the last phases (once bdb does no more coexists with lmdb). > It must be done because it could provide a serious performance boost for > read operations (IMHO, In most cases we could avoid to duplicate the db > data) > > Pierre, what duplicate are you thinking of ? str2entry ? > > But we should not do it while bdb is still around because of the risk of > lock issue and excessive retries. > > Note I put a phasing section in > > https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing > explaining that. But I guess I should move it within Ludwig's document > that englobs it. > > > Pierre > > On Thu, Jan 14, 2021 at 12:01 AM William Brown wrote: > >> >> >> > On 13 Jan 2021, at 21:24, Pierre Rogier wrote: >> > >> > Thank you Willian, >> > So far your scenario (entry found when reading base entry but no more >> existing when computing the candidates) is the only one that matches the >> symptoms. >> >> It's a scenario we will need to fix via your BE work because of the MVCC >> transaction model that LMDB will force us to adopt :) >> >> > And that triggered a thought: >> > We cannot do anything for SUBTREE and ONE_LEVEL searches >> > because the fact that the base entry id is not in the candidate may >> be normal >> > but IMHO we should improve the BASE search case. >> > In this case the candidate list is directly set to the base entry id >> > ==> if the candidate entry (in ldbm_back_next_search_entry) is not >> found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY >> error .. >> >> I suspect that Mark has seen this email and submitted a PR to resolve >> this exact case :) >> >> >> > >> >Pierre >> > >> > >> > On Wed, Jan 13, 2021 at 1:45 AM William Brown wrote: >> > Hey there, >> > >> > https://github.com/389ds/389-ds-base/pull/4525/files >> > >> > I had a look and I can see a few possible contributing factors, but >> without a core and the exact state I can't be sure if this is correct. It's >> all just hypothetical from reading the code. >> > >> > >> > The crash is in deref_do_deref_attr() which is called as part of >> deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by >> "./ldap/servers/slapd/result.c:1488:rc = plugin_call_plugins(pb, >> SLAPI_PLUGIN_PRE_ENTRY_FN);" >> > >> > >> > I think what's important here is that the search is conducted in >> ./ldap/servers/slapd/opshared.c:818 rc = (*be->be_search)(pb); Is *not* >> in a transaction. That means that while the single search in be_search() is >> consistent due to an implied transaction, the subsequent search in >> deref_pre_entry() is likely conducted in a seperate transaction. This >> allows for other operations to potentially interleave and cause changes - >> modrdn or delete would certainly be candidates to cause a DN to be remove >> between these two points. It would be extremely hard to reproduce as a race >> condition of course. >> > >> > >> > A question you asked is why don't we get a "no such entry" error or >> similar? I think that this is because build_candidate_list in ldbm_search.c >> doesn't actually create an error if the base_candidates list is empty, >> because an IDL is a
[389-devel] Re: Deref plugin entries == NULL #4525
Hi William, > It's a scenario we will need to fix via your BE work because of the MVCC transaction model that > LMDB will force us to adopt :) As I see things in the early phases the lmdb read txn will probably only be managed at the db plugin level rather than at backend level. That means that we will have the same inconsistency risk than today (i.e as if using bdb and the implicit txn). The txn model redesign you are speaking about should only occur in one of the last phases (once bdb does no more coexists with lmdb). It must be done because it could provide a serious performance boost for read operations (IMHO, In most cases we could avoid to duplicate the db data) But we should not do it while bdb is still around because of the risk of lock issue and excessive retries. Note I put a phasing section in https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing explaining that. But I guess I should move it within Ludwig's document that englobs it. Pierre On Thu, Jan 14, 2021 at 12:01 AM William Brown wrote: > > > > On 13 Jan 2021, at 21:24, Pierre Rogier wrote: > > > > Thank you Willian, > > So far your scenario (entry found when reading base entry but no more > existing when computing the candidates) is the only one that matches the > symptoms. > > It's a scenario we will need to fix via your BE work because of the MVCC > transaction model that LMDB will force us to adopt :) > > > And that triggered a thought: > > We cannot do anything for SUBTREE and ONE_LEVEL searches > > because the fact that the base entry id is not in the candidate may be > normal > > but IMHO we should improve the BASE search case. > > In this case the candidate list is directly set to the base entry id > > ==> if the candidate entry (in ldbm_back_next_search_entry) is not > found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY > error .. > > I suspect that Mark has seen this email and submitted a PR to resolve this > exact case :) > > > > > >Pierre > > > > > > On Wed, Jan 13, 2021 at 1:45 AM William Brown wrote: > > Hey there, > > > > https://github.com/389ds/389-ds-base/pull/4525/files > > > > I had a look and I can see a few possible contributing factors, but > without a core and the exact state I can't be sure if this is correct. It's > all just hypothetical from reading the code. > > > > > > The crash is in deref_do_deref_attr() which is called as part of > deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by > "./ldap/servers/slapd/result.c:1488:rc = plugin_call_plugins(pb, > SLAPI_PLUGIN_PRE_ENTRY_FN);" > > > > > > I think what's important here is that the search is conducted in > ./ldap/servers/slapd/opshared.c:818 rc = (*be->be_search)(pb); Is *not* > in a transaction. That means that while the single search in be_search() is > consistent due to an implied transaction, the subsequent search in > deref_pre_entry() is likely conducted in a seperate transaction. This > allows for other operations to potentially interleave and cause changes - > modrdn or delete would certainly be candidates to cause a DN to be remove > between these two points. It would be extremely hard to reproduce as a race > condition of course. > > > > > > A question you asked is why don't we get a "no such entry" error or > similar? I think that this is because build_candidate_list in ldbm_search.c > doesn't actually create an error if the base_candidates list is empty, > because an IDL is allocated with a value of 0 (no matching entries). this > allows the search to proceed, and there are no errors, and the result set > is set to NULL with size 0. I can't see where LDAP_NO_SUCH_OBJECT is set in > this process, but without looking further into it, my suspicion is that > entries of size 0 WONT return an error condition to internal_search_pb, so > it's valid for this to be empty. > > > > Anyway, again, this is just reading the code for 20 minutes, and is not > a complete in depth investigation, but maybe it's some ideas about what > happened? > > > > Hope it helps :) > > > > > > > > — > > Sincerely, > > > > William Brown > > > > Senior Software Engineer, 389 Directory Server > > SUSE Labs, Australia > > ___ > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fed
[389-devel] Re: Deref plugin entries == NULL #4525
Thank you Willian, So far your scenario (entry found when reading base entry but no more existing when computing the candidates) is the only one that matches the symptoms. And that triggered a thought: We cannot do anything for SUBTREE and ONE_LEVEL searches because the fact that the base entry id is not in the candidate may be normal but IMHO we should improve the BASE search case. In this case the candidate list is directly set to the base entry id ==> if the candidate entry (in ldbm_back_next_search_entry) is not found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY error .. Pierre On Wed, Jan 13, 2021 at 1:45 AM William Brown wrote: > Hey there, > > https://github.com/389ds/389-ds-base/pull/4525/files > > I had a look and I can see a few possible contributing factors, but > without a core and the exact state I can't be sure if this is correct. It's > all just hypothetical from reading the code. > > > The crash is in deref_do_deref_attr() which is called as part of > deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by > "./ldap/servers/slapd/result.c:1488:rc = plugin_call_plugins(pb, > SLAPI_PLUGIN_PRE_ENTRY_FN);" > > > I think what's important here is that the search is conducted in > ./ldap/servers/slapd/opshared.c:818 rc = (*be->be_search)(pb); Is *not* > in a transaction. That means that while the single search in be_search() is > consistent due to an implied transaction, the subsequent search in > deref_pre_entry() is likely conducted in a seperate transaction. This > allows for other operations to potentially interleave and cause changes - > modrdn or delete would certainly be candidates to cause a DN to be remove > between these two points. It would be extremely hard to reproduce as a race > condition of course. > > > A question you asked is why don't we get a "no such entry" error or > similar? I think that this is because build_candidate_list in ldbm_search.c > doesn't actually create an error if the base_candidates list is empty, > because an IDL is allocated with a value of 0 (no matching entries). this > allows the search to proceed, and there are no errors, and the result set > is set to NULL with size 0. I can't see where LDAP_NO_SUCH_OBJECT is set in > this process, but without looking further into it, my suspicion is that > entries of size 0 WONT return an error condition to internal_search_pb, so > it's valid for this to be empty. > > Anyway, again, this is just reading the code for 20 minutes, and is not a > complete in depth investigation, but maybe it's some ideas about what > happened? > > Hope it helps :) > > > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs, Australia > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: Issue 4534 - libasan read buffer overflow in filtercmp
https://github.com/389ds/389-ds-base/pull/4541 Regards, -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review - Issue 4504 - Fix pytest test_dsconf_replication_monitor
https://github.com/389ds/389-ds-base/pull/4505 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: PR 4451 - dsconf replication monitor fails to retrieve consumer RUV
https://github.com/389ds/389-ds-base/pull/4451 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: PR 4440 - Using ldifgen with --start-idx option fails with unsupported operand
https://github.com/389ds/389-ds-base/pull/ -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: PR 2054 - Do not add referrals for masters with different data generation
https://github.com/389ds/389-ds-base/pull/4427 -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Mapping tree rework
Hi, As we are speaking about the test, we should not forget to test the different mapping tree usages: the common ones (like standard, chaining or referral on update backends) and the uncommon ones (like those involving the various distribution plugins) I am a bit uneasy about thoses, as I remember they sometimes involved weird configuration and custom plugins), that said it is old stories and things have probably changed. Let's hope that we now have a nice list of supported distribution scenarios that could be tested ... Regards, Pierre On Mon, Oct 19, 2020 at 10:10 PM Mark Reynolds wrote: > Hi, > > So some of the arguments here is that we are introducing risk for > something that is not really a big problem. Or, simply not worth investing > in. From a Red Hat perspective "we" would *never* fix this, it's just not > a problem that comes up enough to justify the work and time. But... The > initial work has been done by the upstream community (William). So from a > RH perspective we are getting this work for free. Personally I don't see > this code change as "very" risky, but this is a very sensitive area of the > code. That being said, I am not opposed to adding it, but... I think we > need much more testing around it to build confidence in the patch. I would > want tests that deal with suffixes of varying size, names, nested > levels/complexity: > > o=my_server.com > > dc=example,dc=com > > dc=abcdef,dc=abc (same length as suffix above - since the patch uses > sizing as a way of sorting) > > dc=test,dc=this,dc=patch > > > I want tests that are adding and removing subsuffixes, and > sub-subsuffixes, and making sure ldap ops work, and replication, etc. I > want tests that use many different suffixes at the same time and many > subsuffixes - some customers have 50 subsuffixes. Our current CI test > suite does not have these kinds of tests, and we need them. > > As of today I'm not comfortable with the current CI tests to ack this > patch, but if we can ramp it up and cover more scenarios it would be a step > in the right direction. This is all just my humble opinion, we are all > still just talking :-) > > Mark > > > > > On 10/19/20 6:47 AM, Pierre Rogier wrote: > > Hi William, > > Things are not black and white: >there is a huge difference between a fix with limited impact (like > adding some check in configuration tools or in the server) and redesigning > something that is used in many different contexts for every request handled > by the server ... > > In the first case we could easily mitigate the risk by testing and be > fairly confident, in the second case the tests are too complex to achieve > the same confidence and we should take this kind of risk only if there were > a serious benefit to balance it, but in this case, there are other > solutions with less risks. > > I can understand it could seem too conservervative and frustrating but > that is the price when working on mature projects. If you do not do that, > the product becomes unstable, and users quickly abandon it. > > Regards, >Pierre > > > On Mon, Oct 19, 2020 at 1:27 AM William Brown wrote: > >> >> >> > On 16 Oct 2020, at 17:48, Pierre Rogier wrote: >> > >> > Hi William, >> > I agree with your architecture points and that is why I said my >> proposal is a less appealing trade off. >> > >> > My real concern is your last point: >> > we just do not know and IMHO we are unable to predict what (or if) >> config will cause problems, and I am afraid we will only discover it when >> people start to complain. >> > So I still think that the benefit/risk ratio is bad) >> > >> >> I think this wasn't my point. The thing is *any* change will have that >> "unknown" risk. Our job is to qualify and identify as many of those risks >> as we can, to remove them as unknowns. Think about the work recently to >> merge the changelog to the main db, or BDB to LMDB work, even changing from >> perl to python for installation. These are all significantly larger >> changes, which would be "much riskier" but all of them have been managed >> effectively by the team communicating, coordinating, analysing, designing >> and testing changes. >> >> So I really don't accept this "unknown" risk argument. I have laid out a >> design that explores the configuration, how it works today and how the >> values are currently trusted, and a process to manage and understand this >> change in a way to minimise the risk. There are associated tests, and it >> passes with address sanit
[389-devel] Re: Mapping tree rework
Hi William, Things are not black and white: there is a huge difference between a fix with limited impact (like adding some check in configuration tools or in the server) and redesigning something that is used in many different contexts for every request handled by the server ... In the first case we could easily mitigate the risk by testing and be fairly confident, in the second case the tests are too complex to achieve the same confidence and we should take this kind of risk only if there were a serious benefit to balance it, but in this case, there are other solutions with less risks. I can understand it could seem too conservervative and frustrating but that is the price when working on mature projects. If you do not do that, the product becomes unstable, and users quickly abandon it. Regards, Pierre On Mon, Oct 19, 2020 at 1:27 AM William Brown wrote: > > > > On 16 Oct 2020, at 17:48, Pierre Rogier wrote: > > > > Hi William, > > I agree with your architecture points and that is why I said my proposal > is a less appealing trade off. > > > > My real concern is your last point: > > we just do not know and IMHO we are unable to predict what (or if) > config will cause problems, and I am afraid we will only discover it when > people start to complain. > > So I still think that the benefit/risk ratio is bad) > > > > I think this wasn't my point. The thing is *any* change will have that > "unknown" risk. Our job is to qualify and identify as many of those risks > as we can, to remove them as unknowns. Think about the work recently to > merge the changelog to the main db, or BDB to LMDB work, even changing from > perl to python for installation. These are all significantly larger > changes, which would be "much riskier" but all of them have been managed > effectively by the team communicating, coordinating, analysing, designing > and testing changes. > > So I really don't accept this "unknown" risk argument. I have laid out a > design that explores the configuration, how it works today and how the > values are currently trusted, and a process to manage and understand this > change in a way to minimise the risk. There are associated tests, and it > passes with address sanitiser, and other test cases for mapping trees, > replication and others. > > If we just say "unknown risk" at every change we make we'd never progress. > We may as well packup and go home, the project is completed. > > So I still stand by my design and the PR I have submitted in this case, > and if there are concerns about esoteric configurations, then we should > identify and understand them too beyond the testing I have already provided. > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs, Australia > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Mapping tree rework
Hi William, I agree with your architecture points and that is why I said my proposal is a less appealing trade off. My real concern is your last point: we just do not know and IMHO we are unable to predict what (or if) config will cause problems, and I am afraid we will only discover it when people start to complain. So I still think that the benefit/risk ratio is bad) Regards Pierre On Fri, Oct 16, 2020 at 1:35 AM William Brown wrote: > > > > > Hi, > > > > For my part, I have seen mapping tree misconfiguration popping from time > to time but it is quite rare (maybe 7 or 8 times in 15 years) > > So although in pure term or architecture your proposal is IMHO the > cleanest, in term of risks it is not a good solution: > > there will likely be regressions (it always happens when a component > get redesigned) > > and that means that the number of people that will be annoyed by the > change will be far greater than the few people that will be helped by the > change. > > Today, we already rely upon the attribute of cn to determine what suffix > the mapping tree element provides. We must trust this value, and already > do. This also implies that all current deployments, also trust this value > to be correct. > > It is also provable by tests that *invalid* nsslapd-parent-suffix configs > cause backends to "not appear" in the mapping tree. So invalid parent > suffixes already fail "noisly". This means, yes, that most deployments have > both valid cn's for suffixes and valid nsslapd-parent-suffix values. > > While it may be rare, we have had a few cases here at suse because the > lib389 tools make a mistake in configuration that can easily and trivially > cause this failure. > > So changing this to trust a value of cn to provide ordering, this is > already a value we rely on to be correct *and* we remove an obvious > configuration consistency failure that has occurred. It is because of these > factors that I am more confident that we can make this change with low risk. > > > > > So my feeling is that we should rather go for a trade off. > > Since the goal is to prevent that user misconfigures the mapping tree > without warning, > > we should do that without changing the way the mapping tree is handled > internally. > > simply by adding a consistency check when starting the server or > changing the mapping tree configuration. > > > > If we want to limit the risk we could phase things: > > in first phase we only log warning if inconsistency are found > > and latter on when we get more confident about the code we could > reject the configuration change in case of inconsistency > > > > I know that my proposal is less appealing in term of architecture but > such a solution is safer because it does not change the way mapping tree > are handled and that drastically limits the regression risks > > Generally it's my view that we should always prioritise constraint and > "inability to make mistakes" over "warning about mistakes". There is > certainly value in providing warnings about mistakes when they occur, but > preventing the mistake from ever occuring is a far more reliable option. > Our server should ensure that there is "no way to hold it incorrectly". > > In the situation where we "warn", that would then actually mean we have to > redesign some parts of lib389 to parse and generate a mapping tree itself > so it can then correctly determine the parent-suffixs to emit into configs > when it attaches a backend into the tree. This would also itself be a > significant chunk of work, and a risk of breaking our cli tools too, while > also "not preventing" the issue for any other administration methods that > exist. > > I think that your suggestion is moving the burden of correctness from our > work in the server as engineers, onto administrators and our tooling to > "understand and use it correctly", and that doesn't really sit right with > me. So I'd rather continue with the suggestion I have made as we eliminate > an entire class of potential problems. > > In order to really see understand the percieved risks of this change, I'd > really like to see configurations that would cause this proposal to fail, > and then those can become tests that we can understand and resolve issues > with. > > Thanks, > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs, Australia > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to
[389-devel] Re: Mapping tree rework
Hi, For my part, I have seen mapping tree misconfiguration popping from time to time but it is quite rare (maybe 7 or 8 times in 15 years) So although in pure term or architecture your proposal is IMHO the cleanest, in term of risks it is not a good solution: there will likely be regressions (it always happens when a component get redesigned) and that means that the number of people that will be annoyed by the change will be far greater than the few people that will be helped by the change. So my feeling is that we should rather go for a trade off. Since the goal is to prevent that user misconfigures the mapping tree without warning, we should do that without changing the way the mapping tree is handled internally. simply by adding a consistency check when starting the server or changing the mapping tree configuration. If we want to limit the risk we could phase things: in first phase we only log warning if inconsistency are found and latter on when we get more confident about the code we could reject the configuration change in case of inconsistency I know that my proposal is less appealing in term of architecture but such a solution is safer because it does not change the way mapping tree are handled and that drastically limits the regression risks Regards, Pierre On Wed, Oct 14, 2020 at 11:23 PM William Brown wrote: > This has come up because there is a set of customer cases where they have > configured it incorrectly, due to bugs in lib389. The issues in lib389 > arise from a lack of validation/constraint in the checking of the > nsslapd-parent-suffix value in the server, allowing the client to create > invalid configurations. > > So today, our own tools can easily, and trivially cause this situation. > > One thought is to either document this issue or to fix lib389 - but > neither of the actions really fix the underlying problem, which is that our > server accepts an invalid configuration silently. > > So the best thing for us to do is to make it impossible for the server to > get it wrong, which means we fix lib389 *and* any other admin > tooling/scripts in a single pass. > > Which is what led to the interest in changing something that has been > "working" for a long time :) > > > > > On 14 Oct 2020, at 19:47, Ludwig Krispenz wrote: > > > > Hi, > > > > you are right that it is possible to configure suffix hierarchies which > are broken, but in my experience this wasn't an issue. people using sub > suffixes did get it right. > > > > So is there really a need to change something that is working for a long > time ? > > > > > > Regards, > > > > Ludwig > > > > > > On 14.10.20 08:12, William Brown wrote: > >> https://github.com/mreynolds389/389wiki/pull/48 > >> > >> This is a draft design, and probably of interest to thierry whom I > discussed this with last night :) > >> > >> Thanks! > >> > >> — > >> Sincerely, > >> > >> William Brown > >> > >> Senior Software Engineer, 389 Directory Server > >> SUSE Labs, Australia > >> ___ > >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org > >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > >> Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > ___ > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs, Australia > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > -- -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
