Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

I found the commit message's description of reference resolution hard to
understand.  I think I understand it well enough now to ACK the design.
Would be nice to work some of your explanations into the commit message,
but we don't have time for that.  So I'm merely fixing your QGA vs. QSD
typo.

We might want to put an improved description into the code.  Not today.

Thanks!

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow  writes:

> On Thu, Mar 13, 2025 at 2:30 PM Markus Armbruster  wrote:
>
>> John Snow  writes:
>>
>> > On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster  wrote:
>> >
>> >> John Snow  writes:
>> >>
>> >> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster > >> > wrote:

[...]

>> >> >> Regarding "2. If contextual information ...":
>> >> >>
>> >> >> I guess "contextual information" is the context established by
>> >> >> qapi:namespace and qapi:module directives, i.e. the current
>> >> >> namespace and module, if any.
>> >> >>
>> >> >
>> >> > Yep!
>> >> >
>> >> >
>> >> >>
>> >> >> If the cross reference lacks a namespace, we substitute the current
>> >> >> namespace.  Same for module.
>> >> >>
>> >> >> We then use that "to find a direct match to the implied target
>> >> >> name".  Sounds greek to me.  Example(s) might help.
>> >> >>
>> >> >
>> >> > If namespace or module is missing from the link target, we try to fill 
>> >> > in
>> >> > the blanks with the contextual information if present.
>> >> >
>> >> > Example, we are in the block-core section of the QEMU QMP reference 
>> >> > manual
>> >> > document and we reference `block-dirty-bitmap-add`. With context, we are
>> >> > able to assemble a fully qualified name:
>> >> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
>> >> > registry directly, so it matches and no further search is performed.
>> >>
>> >> We try this lookup only when the reference lacks a namespace and we are
>> >> "in" a namespace, or when it lacks a module and we are "in" a module.
>> >> Correct?
>> >>
>> >
>> > or both: if we provided only a name but the context has both a namespace
>> > and module.
>>
>> So my or is inclusive :)
>>
>> > essentially the algorithm splits the explicit target into (ns, mod, name)
>> > and for any that are blank, we try to fill in those blanks with context
>> > info where available. Sometimes you have neither explicit nor contextual
>> > info for a component.
>> >
>> > Then we do a lookup for an exact match, in order;
>> >
>> > 1. explicit target name, whatever it was
>>
>> Fully qualified name.
>>
>
> Yes, for lookup to succeed it should be fully qualified, though if the
> target text is "ns:module", that's actually going to succeed here, too.
>
>
>>
>> If lookup succeeds, we're done.
>>
>> If lookup fails, we're also done.
>>
>
> If lookup fails, we actually continue on to #2, but whether or not this
> does anything useful depends on whether or not the original target text was
> fully qualified or not. If it was, #2 searches with the exact same text and
> will fail again and proceed to #3, where because we had a fully qualified
> name, none of the search conditions apply and we then just exit.
>
> (It just lacks an early return, but abstractly, if lookup on #1 fails with
> a fully qualified name, we are indeed done.)
>
> If lookup fails because it wasn't actually fully qualified, then #2 has
> some gaps to try to fill.

So, instead of

if ref is fully qualified:
try to look up ref
else
proceed to 2.

you do

look up ref
if found:
ref must be fully qualified
else
proceed to 2.

Since "proceed to 2. won't do anything useful when ref is fully
qualified", the two should produce the same result.

>> *Except* for the ambiguous form NAMESPACE:MYSTERY.  If lookup fails for
>> that, the name is not fully qualified after all.  Probably.  Maybe.  We
>> assume it's missing a module, and proceed to 2.
>>
>> I'm mostly ignoring this exception from now on to keep things simple.
>>
>> > 2. FQN using contextual info
>>
>> Partially qualified name, but context can fill the gaps.
>>
>> If lookup succeeds, we're done.
>>
>> Else, we proceed to 3.
>>
>
> That's right.
>
>
>>
>> > and we stop after the first hit - no chance for multiple results here, just
>> > zero-or-one each step.
>> >
>> > i.e. any explicitly provided information is never "overwritten" with
>> > context, context only fills in the blanks where that info was not provided.
>> >
>> > If neither of these work, we move on to fuzzy searching.
>> >
>> >
>> >> We then subsitute current namespace / module for the lacking one(s), and
>> >> try the same lookup as in 1.  Correct?
>> >>
>> >
>> > Yes!
>> >
>> >
>> >> If we have a reference of the form MYSTERY, it could either be a
>> >> reference to module MYSTERY in the current namespace, or to definition
>> >> MYSTERY in the current namespace and module.  How do we decide?
>> >>
>> >
>> > fqn a: NS:MYSTERY
>> > fqn b: NS:MOD:MYSTERY
>> >
>> > Given we have a current ns/mod context, it's going to pick the second one.
>> >
>> > Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
>> > this. (question is: how soon do we need it?)
>>
>> While we should try to put this on a more solid foundation, it is not a
>> blocker.
>>
>> >> >> Regarding "3. If both prior lookups fail ...":
>> >> >>
>> >> >> I guess we get here when namespace or module are

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

On Thu, Mar 13, 2025 at 2:30 PM Markus Armbruster  wrote:

> John Snow  writes:
>
> > On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster 
> wrote:
> >
> >> John Snow  writes:
> >>
> >> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster  >
> >> > wrote:
> >> >
> >> >> John Snow  writes:
> >> >>
> >> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <
> arm...@redhat.com> wrote:
> >> >> >
> >> >> >> John Snow  writes:
> >> >> >>
> >> >> >> > This patch does three things:
> >> >> >> >
> >> >> >> > 1. Record the current namespace context in pending_xrefs so it
> can be
> >> >> >> >used for link resolution later,
> >> >> >> > 2. Pass that recorded namespace context to find_obj() when
> resolving a
> >> >> >> >reference, and
> >> >> >> > 3. Wildly and completely rewrite find_obj().
> >> >> >> >
> >> >> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> >> >> > of either namespace or module, and to cope with the presence or
> absence
> >> >> >> > of contextual information for either.
> >> >> >> >
> >> >> >> > References now work like this:
> >> >> >> >
> >> >> >> > 1. If the explicit reference target is recorded in the domain's
> object
> >> >> >> >registry, we link to that target and stop looking. We do
> this lookup
> >> >> >> >regardless of how fully qualified the target is, which
> allows direct
> >> >> >> >references to modules (which don't have a module component
> to their
> >> >> >> >names) or direct references to definitions that may or may
> not belong
> >> >> >> >to a namespace or module.
> >> >> >> >
> >> >> >> > 2. If contextual information is available from qapi:namespace or
> >> >> >> >qapi:module directives, try using those components to find a
> direct
> >> >> >> >match to the implied target name.
> >> >> >> >
> >> >> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >> >> >looking for wildcard matches in order from most to least
> >> >> >> >specific. Any explicitly provided components (namespace,
> module)
> >> >> >> >*must* match exactly, but both contextual and entirely
> omitted
> >> >> >> >components are allowed to differ from the search result.
> Note that if
> >> >> >> >more than one result is found, Sphinx will emit a warning (a
> build
> >> >> >> >error for QEMU) and list all of the candidate references.
> >> >> >> >
> >> >> >> > The practical upshot is that in the large majority of cases,
> namespace
> >> >> >> > and module information is not required when creating simple
> `references`
> >> >> >> > to definitions from within the same context -- even when
> identical
> >> >> >> > definitions exist in other contexts.
> >> >> >>
> >> >> >> Can you illustrate this this examples?
> >> >> >>
> >> >> >
> >> >> > do wha?
> >> >>
> >> >> Sorry, I went into the curve too fast.
> >> >>
> >> >> The stuff under "References now work like this" confuses me.  I
> guess it
> >> >> describes a series of lookups to try one after the other.
> >> >>
> >> >> I understand a cross-reference consists of namespace (optional),
> module
> >> >> (optional), name, and role.
> >> >>
> >> >> Let's assume role is "any" for simplicity's sake.
> >> >>
> >> >> Regarding "1. If the explicit ...":
> >> >>
> >> >> What is a reference's "explicit reference target"?  Examples
> might
> >> >> help me understand.
> >> >>
> >> >
> >> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
> >> >
> >> > If that explicit target matches an object in the object database
> >> > *directly*, we match immediately and don't consider other potential
> >> > targets. This also applies to things like modules, e.g.
> `QMP:block-core`
> >> > even though the "module" is absent (it IS the module)
> >> >
> >> > We always search for the explicit target no matter how un/fully
> qualified
> >> > it is.
> >> >
> >> >
> >> >>
> >> >> What is "recorded in the domain's object registry"?
> >> >>
> >> >
> >> > domain.objects{} - essentially a record of every ObjectDefinition's
> >> > "fullname" - the return value from QAPIDefinition._get_fqn().
> >> >
> >> >
> >> >>
> >> >> Can you show me a reference where this lookup succeeds?
> >> >>
> >> >
> >> > `QMP:block-core`
> >> > `QMP:block-core.block-dirty-bitmap-add`
> >>
> >> So, for this lookup to work, the reference must either be of the form
> >> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
> >> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
> >> in that namespace.  Correct?
> >>
> >
> > Yes.
> >
> >
> >> These a "fully qualified names (FQN)" in your parlance, right?
> >>
> >
> > More or less, though as you found below...
> >
> >
> >> Note that the first form is syntactically indistinguishable from
> >> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
> >> namespace, but not the module.
> >>
> >> If the NAMESPACE:MODULE interpretation resolves, we never try the
> >> NAMESPACE:DEFN inte

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow  writes:

> On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster  wrote:
>
>> John Snow  writes:
>>
>> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster 
>> > wrote:
>> >
>> >> John Snow  writes:
>> >>
>> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster  
>> >> > wrote:
>> >> >
>> >> >> John Snow  writes:
>> >> >>
>> >> >> > This patch does three things:
>> >> >> >
>> >> >> > 1. Record the current namespace context in pending_xrefs so it can be
>> >> >> >used for link resolution later,
>> >> >> > 2. Pass that recorded namespace context to find_obj() when resolving 
>> >> >> > a
>> >> >> >reference, and
>> >> >> > 3. Wildly and completely rewrite find_obj().
>> >> >> >
>> >> >> > cross-reference support is expanded to tolerate the presence or 
>> >> >> > absence
>> >> >> > of either namespace or module, and to cope with the presence or 
>> >> >> > absence
>> >> >> > of contextual information for either.
>> >> >> >
>> >> >> > References now work like this:
>> >> >> >
>> >> >> > 1. If the explicit reference target is recorded in the domain's 
>> >> >> > object
>> >> >> >registry, we link to that target and stop looking. We do this 
>> >> >> > lookup
>> >> >> >regardless of how fully qualified the target is, which allows 
>> >> >> > direct
>> >> >> >references to modules (which don't have a module component to 
>> >> >> > their
>> >> >> >names) or direct references to definitions that may or may not 
>> >> >> > belong
>> >> >> >to a namespace or module.
>> >> >> >
>> >> >> > 2. If contextual information is available from qapi:namespace or
>> >> >> >qapi:module directives, try using those components to find a 
>> >> >> > direct
>> >> >> >match to the implied target name.
>> >> >> >
>> >> >> > 3. If both prior lookups fail, generate a series of regular 
>> >> >> > expressions
>> >> >> >looking for wildcard matches in order from most to least
>> >> >> >specific. Any explicitly provided components (namespace, module)
>> >> >> >*must* match exactly, but both contextual and entirely omitted
>> >> >> >components are allowed to differ from the search result. Note 
>> >> >> > that if
>> >> >> >more than one result is found, Sphinx will emit a warning (a build
>> >> >> >error for QEMU) and list all of the candidate references.
>> >> >> >
>> >> >> > The practical upshot is that in the large majority of cases, 
>> >> >> > namespace
>> >> >> > and module information is not required when creating simple 
>> >> >> > `references`
>> >> >> > to definitions from within the same context -- even when identical
>> >> >> > definitions exist in other contexts.
>> >> >>
>> >> >> Can you illustrate this this examples?
>> >> >>
>> >> >
>> >> > do wha?
>> >>
>> >> Sorry, I went into the curve too fast.
>> >>
>> >> The stuff under "References now work like this" confuses me.  I guess it
>> >> describes a series of lookups to try one after the other.
>> >>
>> >> I understand a cross-reference consists of namespace (optional), module
>> >> (optional), name, and role.
>> >>
>> >> Let's assume role is "any" for simplicity's sake.
>> >>
>> >> Regarding "1. If the explicit ...":
>> >>
>> >> What is a reference's "explicit reference target"?  Examples might
>> >> help me understand.
>> >>
>> >
>> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
>> >
>> > If that explicit target matches an object in the object database
>> > *directly*, we match immediately and don't consider other potential
>> > targets. This also applies to things like modules, e.g. `QMP:block-core`
>> > even though the "module" is absent (it IS the module)
>> >
>> > We always search for the explicit target no matter how un/fully qualified
>> > it is.
>> >
>> >
>> >>
>> >> What is "recorded in the domain's object registry"?
>> >>
>> >
>> > domain.objects{} - essentially a record of every ObjectDefinition's
>> > "fullname" - the return value from QAPIDefinition._get_fqn().
>> >
>> >
>> >>
>> >> Can you show me a reference where this lookup succeeds?
>> >>
>> >
>> > `QMP:block-core`
>> > `QMP:block-core.block-dirty-bitmap-add`
>>
>> So, for this lookup to work, the reference must either be of the form
>> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
>> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
>> in that namespace.  Correct?
>>
>
> Yes.
>
>
>> These a "fully qualified names (FQN)" in your parlance, right?
>>
>
> More or less, though as you found below...
>
>
>> Note that the first form is syntactically indistinguishable from
>> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
>> namespace, but not the module.
>>
>> If the NAMESPACE:MODULE interpretation resolves, we never try the
>> NAMESPACE:DEFN interpretation, because that happens in later steps.
>> Correct?
>>
>> The first form is fully qualified only if it resolves as FQN.  So,
>> whether such a reference is fully qualified is not syntactically
>> de

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster  wrote:

> John Snow  writes:
>
> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster 
> > wrote:
> >
> >> John Snow  writes:
> >>
> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster 
> wrote:
> >> >
> >> >> John Snow  writes:
> >> >>
> >> >> > This patch does three things:
> >> >> >
> >> >> > 1. Record the current namespace context in pending_xrefs so it can
> be
> >> >> >used for link resolution later,
> >> >> > 2. Pass that recorded namespace context to find_obj() when
> resolving a
> >> >> >reference, and
> >> >> > 3. Wildly and completely rewrite find_obj().
> >> >> >
> >> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> >> > of either namespace or module, and to cope with the presence or
> absence
> >> >> > of contextual information for either.
> >> >> >
> >> >> > References now work like this:
> >> >> >
> >> >> > 1. If the explicit reference target is recorded in the domain's
> object
> >> >> >registry, we link to that target and stop looking. We do this
> lookup
> >> >> >regardless of how fully qualified the target is, which allows
> direct
> >> >> >references to modules (which don't have a module component to
> their
> >> >> >names) or direct references to definitions that may or may not
> belong
> >> >> >to a namespace or module.
> >> >> >
> >> >> > 2. If contextual information is available from qapi:namespace or
> >> >> >qapi:module directives, try using those components to find a
> direct
> >> >> >match to the implied target name.
> >> >> >
> >> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >> >looking for wildcard matches in order from most to least
> >> >> >specific. Any explicitly provided components (namespace, module)
> >> >> >*must* match exactly, but both contextual and entirely omitted
> >> >> >components are allowed to differ from the search result. Note
> that if
> >> >> >more than one result is found, Sphinx will emit a warning (a
> build
> >> >> >error for QEMU) and list all of the candidate references.
> >> >> >
> >> >> > The practical upshot is that in the large majority of cases,
> namespace
> >> >> > and module information is not required when creating simple
> `references`
> >> >> > to definitions from within the same context -- even when identical
> >> >> > definitions exist in other contexts.
> >> >>
> >> >> Can you illustrate this this examples?
> >> >>
> >> >
> >> > do wha?
> >>
> >> Sorry, I went into the curve too fast.
> >>
> >> The stuff under "References now work like this" confuses me.  I guess it
> >> describes a series of lookups to try one after the other.
> >>
> >> I understand a cross-reference consists of namespace (optional), module
> >> (optional), name, and role.
> >>
> >> Let's assume role is "any" for simplicity's sake.
> >>
> >> Regarding "1. If the explicit ...":
> >>
> >> What is a reference's "explicit reference target"?  Examples might
> >> help me understand.
> >>
> >
> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
> >
> > If that explicit target matches an object in the object database
> > *directly*, we match immediately and don't consider other potential
> > targets. This also applies to things like modules, e.g. `QMP:block-core`
> > even though the "module" is absent (it IS the module)
> >
> > We always search for the explicit target no matter how un/fully qualified
> > it is.
> >
> >
> >>
> >> What is "recorded in the domain's object registry"?
> >>
> >
> > domain.objects{} - essentially a record of every ObjectDefinition's
> > "fullname" - the return value from QAPIDefinition._get_fqn().
> >
> >
> >>
> >> Can you show me a reference where this lookup succeeds?
> >>
> >
> > `QMP:block-core`
> > `QMP:block-core.block-dirty-bitmap-add`
>
> So, for this lookup to work, the reference must either be of the form
> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
> in that namespace.  Correct?
>

Yes.


> These a "fully qualified names (FQN)" in your parlance, right?
>

More or less, though as you found below...


> Note that the first form is syntactically indistinguishable from
> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
> namespace, but not the module.
>
> If the NAMESPACE:MODULE interpretation resolves, we never try the
> NAMESPACE:DEFN interpretation, because that happens in later steps.
> Correct?
>
> The first form is fully qualified only if it resolves as FQN.  So,
> whether such a reference is fully qualified is not syntactically
> decidable.  Hmm.
>

You're right. There is a weirdness here. I might need to do some more
thinking to make sure it isn't theoretically a problem, but in practice,
right now, it isn't.

Stay tuned, I guess.


> >> Regarding "2. If contextual information ...":
> >>
> >> I guess "

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster  wrote:

> John Snow  writes:
>
> > This patch does three things:
> >
> > 1. Record the current namespace context in pending_xrefs so it can be
> >used for link resolution later,
> > 2. Pass that recorded namespace context to find_obj() when resolving a
> >reference, and
> > 3. Wildly and completely rewrite find_obj().
> >
> > cross-reference support is expanded to tolerate the presence or absence
> > of either namespace or module, and to cope with the presence or absence
> > of contextual information for either.
> >
> > References now work like this:
> >
> > 1. If the explicit reference target is recorded in the domain's object
> >registry, we link to that target and stop looking. We do this lookup
> >regardless of how fully qualified the target is, which allows direct
> >references to modules (which don't have a module component to their
> >names) or direct references to definitions that may or may not belong
> >to a namespace or module.
> >
> > 2. If contextual information is available from qapi:namespace or
> >qapi:module directives, try using those components to find a direct
> >match to the implied target name.
> >
> > 3. If both prior lookups fail, generate a series of regular expressions
> >looking for wildcard matches in order from most to least
> >specific. Any explicitly provided components (namespace, module)
> >*must* match exactly, but both contextual and entirely omitted
> >components are allowed to differ from the search result. Note that if
> >more than one result is found, Sphinx will emit a warning (a build
> >error for QEMU) and list all of the candidate references.
> >
> > The practical upshot is that in the large majority of cases, namespace
> > and module information is not required when creating simple `references`
> > to definitions from within the same context -- even when identical
> > definitions exist in other contexts.
>
> Can you illustrate this this examples?
>

do wha?


>
> > Even when using simple `references` from elsewhere in the QEMU
> > documentation manual, explicit namespace info is not required if there
> > is only one definition by that name.
>
> Fails safely: if we add a second definition, doc generation errors out.
> Correct?
>

Yes, see the disambiguation patch for qapi-domain.rst for proof. Roll it
back and see what happens!


>
> > Disambiguation *will* be required from outside of the QAPI documentation
> > when referencing e.g. block-core definitions, which are shared between
> > QEMU QMP and the QEMU Storage Daemon. In that case, there are two
> > options:
> >
> > A: References can be made partially or fully explicit,
> >e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
> >the definition, while `QGA:block-dirty-bitmap-add` would link to the
> >QGA version.
>
> Do you mean "QSD:"?
>

Haha! Yes, I suppose I did.


>
> > B: If all of the references in a document are intended to go to the same
> >place, you can insert a "qapi:namespace:: QMP" directive to influence
> >the fuzzy-searching for later references.
> >
> > Signed-off-by: John Snow 
>
>

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow  writes:

> On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster 
> wrote:
>
>> John Snow  writes:
>>
>> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster  
>> > wrote:
>> >
>> >> John Snow  writes:
>> >>
>> >> > This patch does three things:
>> >> >
>> >> > 1. Record the current namespace context in pending_xrefs so it can be
>> >> >used for link resolution later,
>> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >> >reference, and
>> >> > 3. Wildly and completely rewrite find_obj().
>> >> >
>> >> > cross-reference support is expanded to tolerate the presence or absence
>> >> > of either namespace or module, and to cope with the presence or absence
>> >> > of contextual information for either.
>> >> >
>> >> > References now work like this:
>> >> >
>> >> > 1. If the explicit reference target is recorded in the domain's object
>> >> >registry, we link to that target and stop looking. We do this lookup
>> >> >regardless of how fully qualified the target is, which allows direct
>> >> >references to modules (which don't have a module component to their
>> >> >names) or direct references to definitions that may or may not belong
>> >> >to a namespace or module.
>> >> >
>> >> > 2. If contextual information is available from qapi:namespace or
>> >> >qapi:module directives, try using those components to find a direct
>> >> >match to the implied target name.
>> >> >
>> >> > 3. If both prior lookups fail, generate a series of regular expressions
>> >> >looking for wildcard matches in order from most to least
>> >> >specific. Any explicitly provided components (namespace, module)
>> >> >*must* match exactly, but both contextual and entirely omitted
>> >> >components are allowed to differ from the search result. Note that if
>> >> >more than one result is found, Sphinx will emit a warning (a build
>> >> >error for QEMU) and list all of the candidate references.
>> >> >
>> >> > The practical upshot is that in the large majority of cases, namespace
>> >> > and module information is not required when creating simple `references`
>> >> > to definitions from within the same context -- even when identical
>> >> > definitions exist in other contexts.
>> >>
>> >> Can you illustrate this this examples?
>> >>
>> >
>> > do wha?
>>
>> Sorry, I went into the curve too fast.
>>
>> The stuff under "References now work like this" confuses me.  I guess it
>> describes a series of lookups to try one after the other.
>>
>> I understand a cross-reference consists of namespace (optional), module
>> (optional), name, and role.
>>
>> Let's assume role is "any" for simplicity's sake.
>>
>> Regarding "1. If the explicit ...":
>>
>> What is a reference's "explicit reference target"?  Examples might
>> help me understand.
>>
>
> explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
>
> If that explicit target matches an object in the object database
> *directly*, we match immediately and don't consider other potential
> targets. This also applies to things like modules, e.g. `QMP:block-core`
> even though the "module" is absent (it IS the module)
>
> We always search for the explicit target no matter how un/fully qualified
> it is.
>
>
>>
>> What is "recorded in the domain's object registry"?
>>
>
> domain.objects{} - essentially a record of every ObjectDefinition's
> "fullname" - the return value from QAPIDefinition._get_fqn().
>
>
>>
>> Can you show me a reference where this lookup succeeds?
>>
>
> `QMP:block-core`
> `QMP:block-core.block-dirty-bitmap-add`

So, for this lookup to work, the reference must either be of the form
NAMESPACE:MODULE and resolve to that module in that namespace, or of the
form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
in that namespace.  Correct?

These a "fully qualified names (FQN)" in your parlance, right?

Note that the first form is syntactically indistinguishable from
NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
namespace, but not the module.

If the NAMESPACE:MODULE interpretation resolves, we never try the
NAMESPACE:DEFN interpretation, because that happens in later steps.
Correct?

The first form is fully qualified only if it resolves as FQN.  So,
whether such a reference is fully qualified is not syntactically
decidable.  Hmm.

>> Regarding "2. If contextual information ...":
>>
>> I guess "contextual information" is the context established by
>> qapi:namespace and qapi:module directives, i.e. the current
>> namespace and module, if any.
>>
>
> Yep!
>
>
>>
>> If the cross reference lacks a namespace, we substitute the current
>> namespace.  Same for module.
>>
>> We then use that "to find a direct match to the implied target
>> name".  Sounds greek to me.  Example(s) might help.
>>
>
> If namespace or module is missing from the link target, we try to fill in
> the blanks with the contextual information if present

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster 
wrote:

> John Snow  writes:
>
> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster 
> wrote:
> >
> >> John Snow  writes:
> >>
> >> > This patch does three things:
> >> >
> >> > 1. Record the current namespace context in pending_xrefs so it can be
> >> >used for link resolution later,
> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
> >> >reference, and
> >> > 3. Wildly and completely rewrite find_obj().
> >> >
> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> > of either namespace or module, and to cope with the presence or
> absence
> >> > of contextual information for either.
> >> >
> >> > References now work like this:
> >> >
> >> > 1. If the explicit reference target is recorded in the domain's object
> >> >registry, we link to that target and stop looking. We do this
> lookup
> >> >regardless of how fully qualified the target is, which allows
> direct
> >> >references to modules (which don't have a module component to their
> >> >names) or direct references to definitions that may or may not
> belong
> >> >to a namespace or module.
> >> >
> >> > 2. If contextual information is available from qapi:namespace or
> >> >qapi:module directives, try using those components to find a direct
> >> >match to the implied target name.
> >> >
> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >looking for wildcard matches in order from most to least
> >> >specific. Any explicitly provided components (namespace, module)
> >> >*must* match exactly, but both contextual and entirely omitted
> >> >components are allowed to differ from the search result. Note that
> if
> >> >more than one result is found, Sphinx will emit a warning (a build
> >> >error for QEMU) and list all of the candidate references.
> >> >
> >> > The practical upshot is that in the large majority of cases, namespace
> >> > and module information is not required when creating simple
> `references`
> >> > to definitions from within the same context -- even when identical
> >> > definitions exist in other contexts.
> >>
> >> Can you illustrate this this examples?
> >>
> >
> > do wha?
>
> Sorry, I went into the curve too fast.
>
> The stuff under "References now work like this" confuses me.  I guess it
> describes a series of lookups to try one after the other.
>
> I understand a cross-reference consists of namespace (optional), module
> (optional), name, and role.
>
> Let's assume role is "any" for simplicity's sake.
>
> Regarding "1. If the explicit ...":
>
> What is a reference's "explicit reference target"?  Examples might
> help me understand.
>

explicit lookup: `QMP:block-core:block-dirty-bitmap-add`

If that explicit target matches an object in the object database
*directly*, we match immediately and don't consider other potential
targets. This also applies to things like modules, e.g. `QMP:block-core`
even though the "module" is absent (it IS the module)

We always search for the explicit target no matter how un/fully qualified
it is.


>
> What is "recorded in the domain's object registry"?
>

domain.objects{} - essentially a record of every ObjectDefinition's
"fullname" - the return value from QAPIDefinition._get_fqn().


>
> Can you show me a reference where this lookup succeeds?
>

`QMP:block-core`
`QMP:block-core.block-dirty-bitmap-add`


>
> Regarding "2. If contextual information ...":
>
> I guess "contextual information" is the context established by
> qapi:namespace and qapi:module directives, i.e. the current
> namespace and module, if any.
>

Yep!


>
> If the cross reference lacks a namespace, we substitute the current
> namespace.  Same for module.
>
> We then use that "to find a direct match to the implied target
> name".  Sounds greek to me.  Example(s) might help.
>

If namespace or module is missing from the link target, we try to fill in
the blanks with the contextual information if present.

Example, we are in the block-core section of the QEMU QMP reference manual
document and we reference `block-dirty-bitmap-add`. With context, we are
able to assemble a fully qualified name:
"QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
registry directly, so it matches and no further search is performed.


>
> Regarding "3. If both prior lookups fail ...":
>
> I guess we get here when namespace or module are absent, and
> substituting the current namespace or module doesn't resolve.  We
> then substitute a wildcard, so to speak, i.e. look in all namespaces
> / modules, and succeed if we find exactly one resolution.  Fair?
>

More or less, though the mechanics are quite a bit more complex than your
overview (and what I wrote in qapi-domain.rst.) We can get here for a few
reasons:

(1) We didn't provide a fully qualified target, and we don't have full
c

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow  writes:

> On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster  wrote:
>
>> John Snow  writes:
>>
>> > This patch does three things:
>> >
>> > 1. Record the current namespace context in pending_xrefs so it can be
>> >used for link resolution later,
>> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >reference, and
>> > 3. Wildly and completely rewrite find_obj().
>> >
>> > cross-reference support is expanded to tolerate the presence or absence
>> > of either namespace or module, and to cope with the presence or absence
>> > of contextual information for either.
>> >
>> > References now work like this:
>> >
>> > 1. If the explicit reference target is recorded in the domain's object
>> >registry, we link to that target and stop looking. We do this lookup
>> >regardless of how fully qualified the target is, which allows direct
>> >references to modules (which don't have a module component to their
>> >names) or direct references to definitions that may or may not belong
>> >to a namespace or module.
>> >
>> > 2. If contextual information is available from qapi:namespace or
>> >qapi:module directives, try using those components to find a direct
>> >match to the implied target name.
>> >
>> > 3. If both prior lookups fail, generate a series of regular expressions
>> >looking for wildcard matches in order from most to least
>> >specific. Any explicitly provided components (namespace, module)
>> >*must* match exactly, but both contextual and entirely omitted
>> >components are allowed to differ from the search result. Note that if
>> >more than one result is found, Sphinx will emit a warning (a build
>> >error for QEMU) and list all of the candidate references.
>> >
>> > The practical upshot is that in the large majority of cases, namespace
>> > and module information is not required when creating simple `references`
>> > to definitions from within the same context -- even when identical
>> > definitions exist in other contexts.
>>
>> Can you illustrate this this examples?
>>
>
> do wha?

Sorry, I went into the curve too fast.

The stuff under "References now work like this" confuses me.  I guess it
describes a series of lookups to try one after the other.

I understand a cross-reference consists of namespace (optional), module
(optional), name, and role.

Let's assume role is "any" for simplicity's sake.

Regarding "1. If the explicit ...":

What is a reference's "explicit reference target"?  Examples might
help me understand.

What is "recorded in the domain's object registry"?

Can you show me a reference where this lookup succeeds?

Regarding "2. If contextual information ...":

I guess "contextual information" is the context established by
qapi:namespace and qapi:module directives, i.e. the current
namespace and module, if any.

If the cross reference lacks a namespace, we substitute the current
namespace.  Same for module.

We then use that "to find a direct match to the implied target
name".  Sounds greek to me.  Example(s) might help.

Regarding "3. If both prior lookups fail ...":

I guess we get here when namespace or module are absent, and
substituting the current namespace or module doesn't resolve.  We
then substitute a wildcard, so to speak, i.e. look in all namespaces
/ modules, and succeed if we find exactly one resolution.  Fair?

[...]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow  writes:

> This patch does three things:
>
> 1. Record the current namespace context in pending_xrefs so it can be
>used for link resolution later,
> 2. Pass that recorded namespace context to find_obj() when resolving a
>reference, and
> 3. Wildly and completely rewrite find_obj().
>
> cross-reference support is expanded to tolerate the presence or absence
> of either namespace or module, and to cope with the presence or absence
> of contextual information for either.
>
> References now work like this:
>
> 1. If the explicit reference target is recorded in the domain's object
>registry, we link to that target and stop looking. We do this lookup
>regardless of how fully qualified the target is, which allows direct
>references to modules (which don't have a module component to their
>names) or direct references to definitions that may or may not belong
>to a namespace or module.
>
> 2. If contextual information is available from qapi:namespace or
>qapi:module directives, try using those components to find a direct
>match to the implied target name.
>
> 3. If both prior lookups fail, generate a series of regular expressions
>looking for wildcard matches in order from most to least
>specific. Any explicitly provided components (namespace, module)
>*must* match exactly, but both contextual and entirely omitted
>components are allowed to differ from the search result. Note that if
>more than one result is found, Sphinx will emit a warning (a build
>error for QEMU) and list all of the candidate references.
>
> The practical upshot is that in the large majority of cases, namespace
> and module information is not required when creating simple `references`
> to definitions from within the same context -- even when identical
> definitions exist in other contexts.

Can you illustrate this this examples?

> Even when using simple `references` from elsewhere in the QEMU
> documentation manual, explicit namespace info is not required if there
> is only one definition by that name.

Fails safely: if we add a second definition, doc generation errors out.
Correct?

> Disambiguation *will* be required from outside of the QAPI documentation
> when referencing e.g. block-core definitions, which are shared between
> QEMU QMP and the QEMU Storage Daemon. In that case, there are two
> options:
>
> A: References can be made partially or fully explicit,
>e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
>the definition, while `QGA:block-dirty-bitmap-add` would link to the
>QGA version.

Do you mean "QSD:"?

> B: If all of the references in a document are intended to go to the same
>place, you can insert a "qapi:namespace:: QMP" directive to influence
>the fuzzy-searching for later references.
>
> Signed-off-by: John Snow 

[PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

This patch does three things:

1. Record the current namespace context in pending_xrefs so it can be
   used for link resolution later,
2. Pass that recorded namespace context to find_obj() when resolving a
   reference, and
3. Wildly and completely rewrite find_obj().

cross-reference support is expanded to tolerate the presence or absence
of either namespace or module, and to cope with the presence or absence
of contextual information for either.

References now work like this:

1. If the explicit reference target is recorded in the domain's object
   registry, we link to that target and stop looking. We do this lookup
   regardless of how fully qualified the target is, which allows direct
   references to modules (which don't have a module component to their
   names) or direct references to definitions that may or may not belong
   to a namespace or module.

2. If contextual information is available from qapi:namespace or
   qapi:module directives, try using those components to find a direct
   match to the implied target name.

3. If both prior lookups fail, generate a series of regular expressions
   looking for wildcard matches in order from most to least
   specific. Any explicitly provided components (namespace, module)
   *must* match exactly, but both contextual and entirely omitted
   components are allowed to differ from the search result. Note that if
   more than one result is found, Sphinx will emit a warning (a build
   error for QEMU) and list all of the candidate references.

The practical upshot is that in the large majority of cases, namespace
and module information is not required when creating simple `references`
to definitions from within the same context -- even when identical
definitions exist in other contexts.

Even when using simple `references` from elsewhere in the QEMU
documentation manual, explicit namespace info is not required if there
is only one definition by that name.

Disambiguation *will* be required from outside of the QAPI documentation
when referencing e.g. block-core definitions, which are shared between
QEMU QMP and the QEMU Storage Daemon. In that case, there are two
options:

A: References can be made partially or fully explicit,
   e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
   the definition, while `QGA:block-dirty-bitmap-add` would link to the
   QGA version.

B: If all of the references in a document are intended to go to the same
   place, you can insert a "qapi:namespace:: QMP" directive to influence
   the fuzzy-searching for later references.

Signed-off-by: John Snow 
---
 docs/devel/qapi-domain.rst |  34 --
 docs/sphinx/qapi_domain.py | 127 -
 2 files changed, 114 insertions(+), 47 deletions(-)

diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
index 73e13ab45ca..4705ba255e0 100644
--- a/docs/devel/qapi-domain.rst
+++ b/docs/devel/qapi-domain.rst
@@ -400,11 +400,10 @@ Namespaces
 Mimicking the `Python domain target specification syntax
 
`_,
 QAPI allows you to specify the fully qualified path for a data
-type. QAPI enforces globally unique names, so it's unlikely you'll need
-this specific feature, but it may be extended in the near future to
-allow referencing identically named commands and data types from
-different utilities; i.e. QEMU Storage Daemon vs QMP.
+type.
 
+* A namespace can be explicitly provided;
+  e.g. ``:qapi:type:`QMP:BitmapSyncMode``
 * A module can be explicitly provided;
   ``:qapi:type:`block-core.BitmapSyncMode``` will render to:
   :qapi:type:`block-core.BitmapSyncMode`
@@ -413,6 +412,28 @@ different utilities; i.e. QEMU Storage Daemon vs QMP.
   will render to: :qapi:type:`~block-core.BitmapSyncMode`
 
 
+Target resolution
+-
+
+Any cross-reference to a QAPI type, whether using the ```any``` style of
+reference or the more explicit ```:qapi:any:`target``` syntax, allows
+for the presence or absence of either the namespace or module
+information.
+
+When absent, their value will be inferred from context by the presence
+of any ``qapi:namespace`` or ``qapi:module`` directives preceding the
+cross-reference.
+
+If no results are found when using the inferred values, other
+namespaces/modules will be searched as a last resort; but any explicitly
+provided values must always match in order to succeed.
+
+This allows for efficient cross-referencing with a minimum of syntax in
+the large majority of cases, but additional context or namespace markup
+may be required outside of the QAPI reference documents when linking to
+items that share a name across multiple documented QAPI schema.
+
+
 Custom link text
 
 
@@ -492,6 +513,11 @@ directives are associated with the most recent namespace. 
This affects
 the definition's "fully qualified name", allowing two different
 namespaces to create an otherwise identically named definition.