Hi Thierry,
I was rather thinking about the key and value duplication when querying the
DB:
When using bdb functions that is done implicitly.
bdb either copies the values in the DBT buffer or it alloc/realloc it
When mimicking bdb behavior with LDBM we will have to do that explicitly in
the LDBM plugin:
LDMB returns a memory mapped address that may be ummapped
once the txn is ended. So we must copy the result before closing the txn.
If we have a read txn that protects the full operation lifespan, then we
could directly use the mapped address without needing to duplicate them.
Pierre
On Fri, Jan 15, 2021 at 10:53 AM thierry bordaz <tbor...@redhat.com> wrote:
>
>
> On 1/14/21 12:32 PM, Pierre Rogier wrote:
>
> Hi William,
>
> > It's a scenario we will need to fix via your BE work because of the MVCC
> transaction model that
> > LMDB will force us to adopt :)
>
> As I see things in the early phases the lmdb read txn will probably only
> be managed at the db plugin level rather than at backend level. That
> means that we will have the same inconsistency risk than today (i.e as if
> using bdb and the implicit txn).
> The txn model redesign you are speaking about should only occur in one of
> the last phases (once bdb does no more coexists with lmdb).
> It must be done because it could provide a serious performance boost for
> read operations (IMHO, In most cases we could avoid to duplicate the db
> data)
>
> Pierre, what duplicate are you thinking of ? str2entry ?
>
> But we should not do it while bdb is still around because of the risk of
> lock issue and excessive retries.
>
> Note I put a phasing section in
>
> https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing
> explaining that. But I guess I should move it within Ludwig's document
> that englobs it.
>
>
> Pierre
>
> On Thu, Jan 14, 2021 at 12:01 AM William Brown <wbr...@suse.de> wrote:
>
>>
>>
>> > On 13 Jan 2021, at 21:24, Pierre Rogier <prog...@redhat.com> wrote:
>> >
>> > Thank you Willian,
>> > So far your scenario (entry found when reading base entry but no more
>> existing when computing the candidates) is the only one that matches the
>> symptoms.
>>
>> It's a scenario we will need to fix via your BE work because of the MVCC
>> transaction model that LMDB will force us to adopt :)
>>
>> > And that triggered a thought:
>> > We cannot do anything for SUBTREE and ONE_LEVEL searches
>> > because the fact that the base entry id is not in the candidate may
>> be normal
>> > but IMHO we should improve the BASE search case.
>> > In this case the candidate list is directly set to the base entry id
>> > ==> if the candidate entry (in ldbm_back_next_search_entry) is not
>> found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY
>> error ..
>>
>> I suspect that Mark has seen this email and submitted a PR to resolve
>> this exact case :)
>>
>>
>> >
>> > Pierre
>> >
>> >
>> > On Wed, Jan 13, 2021 at 1:45 AM William Brown <wbr...@suse.de> wrote:
>> > Hey there,
>> >
>> > https://github.com/389ds/389-ds-base/pull/4525/files
>> >
>> > I had a look and I can see a few possible contributing factors, but
>> without a core and the exact state I can't be sure if this is correct. It's
>> all just hypothetical from reading the code.
>> >
>> >
>> > The crash is in deref_do_deref_attr() which is called as part of
>> deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by
>> "./ldap/servers/slapd/result.c:1488: rc = plugin_call_plugins(pb,
>> SLAPI_PLUGIN_PRE_ENTRY_FN);"
>> >
>> >
>> > I think what's important here is that the search is conducted in
>> ./ldap/servers/slapd/opshared.c:818 rc = (*be->be_search)(pb); Is *not*
>> in a transaction. That means that while the single search in be_search() is
>> consistent due to an implied transaction, the subsequent search in
>> deref_pre_entry() is likely conducted in a seperate transaction. This
>> allows for other operations to potentially interleave and cause changes -
>> modrdn or delete would certainly be candidates to cause a DN to be remove
>> between these two points. It would be extremely hard to reproduce as a race
>> condition of course.
>> >
>> >
>> > A question you asked is why don't we get a "no such entry" error or
>> similar? I think that this is because build_candidate_list in ldbm_search.c
>> doesn't actually create an error if the base_candidates list is empty,
>> because an IDL is allocated with a value of 0 (no matching entries). this
>> allows the search to proceed, and there are no errors, and the result set
>> is set to NULL with size 0. I can't see where LDAP_NO_SUCH_OBJECT is set in
>> this process, but without looking further into it, my suspicion is that
>> entries of size 0 WONT return an error condition to internal_search_pb, so
>> it's valid for this to be empty.
>> >
>> > Anyway, again, this is just reading the code for 20 minutes, and is not
>> a complete in depth investigation, but maybe it's some ideas about what
>> happened?
>> >
>> > Hope it helps :)
>> >
>> >
>> >
>> > —
>> > Sincerely,
>> >
>> > William Brown
>> >
>> > Senior Software Engineer, 389 Directory Server
>> > SUSE Labs, Australia
>> > _______________________________________________
>> > 389-devel mailing list -- 389-de...@lists.fedoraproject.org
>> > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
>> > Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
>> >
>> >
>> > --
>> > --
>> >
>> > 389 Directory Server Development Team
>> > _______________________________________________
>> > 389-devel mailing list -- 389-de...@lists.fedoraproject.org
>> > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
>> > Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
>>
>> —
>> Sincerely,
>>
>> William Brown
>>
>> Senior Software Engineer, 389 Directory Server
>> SUSE Labs, Australia
>> _______________________________________________
>> 389-devel mailing list -- 389-de...@lists.fedoraproject.org
>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
>>
>
>
> --
> --
>
> 389 Directory Server Development Team
>
> _______________________________________________
> 389-devel mailing list -- 389-de...@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
>
>
>
--
--
389 Directory Server Development Team
_______________________________________________
389-devel mailing list -- 389-de...@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
