Re: [Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Eric Blake]

On 11/12/2015 06:16 AM, Markus Armbruster wrote:

> We have one in visit_needed(), and we use its is_implicit() to skip
> implicit object types.  We could use entity.is_builtin() to skip (some)
> builtins, and handle them elsewhere, but that doesn't feel like an
> improvement over your code.
> 
> Let's take a step back and reconsider how we do builtins.
> 
>>> +self._btin += gen_enum(name, values, prefix)
>>> +if do_builtins:
>>> +self.defn += gen_enum_lookup(name, values, prefix)
>>> +else:
>>> +self._fwdecl += gen_enum(name, values, prefix)
>>> +self._fwdefn += gen_enum_lookup(name, values, prefix)
>>>
>>>  def visit_array_type(self, name, info, element_type):
>>>  if isinstance(element_type, QAPISchemaBuiltinType):
> 
> Linking generated code from multiple schemata that share names may fail,
> because multiple definitions of the same external symbol exist.
> 
> Example: two schemata both define enum BadIdea.  Both generate const
> char *BadIdea_lookup[] = { ... }, and we end up with two global symbols
> BadIdea_lookup.
> 
> Solution: don't do that then.  Easy enough, except *all* schemata share
> the builtin symbols!  Solution:
[...]

Yes, the existing solution is a bit ugly.  We don't even need the
secondary test-qapi-types.h to declare builtins, since it is relying on
the top-level qapi-types.h to do it, so we are just wasting disk space.

> Here's an alternative solution that permits slightly code simpler
> generator code, and thus avoids the need to know:
> 
> * Generate code for builtins exactly the same as for any other entities,
>   i.e. get rid of self._btin and the ifdeffery.
> 
> * If the program links just one generated schema, this just works.
> 
> * If the program links multiple generated schemata, the programmer has
>   to ensure their definitions get generated just once, and their
>   declarations are available everywhere anyway.  Straightforward method:
> 
>   - The programmer suppresses builtins *completely* for *all* schemata.
> The obvious way to suppress them is to filter them out in
> visit_needed().
> 
>   - Instead, he generates them once for the *empty* schema, with a
> well-known --prefix.
> 
>   - Suppressing builtins generates a suitable #include for the
> well-known .h with the builtin declarations.
> 
>   - Additionally link the .c containing the builtin definitions.

Reasonably nice ideas.  I'll play with them later, though, in the
interest of getting the existing patch queue flushed first.

> 
> Alternatively, trade some ease-of-use for the single schema case for
> ease-of-use for the multiple schemata case and fewer cases:
> 
> * The generators either generate for a schema, or they generate builtins.
> 
> * When they generate builtins, they always use well-known file names.
> 
> * When they generate for a schema, they always generate the #include for
>   the well-known builtin .h.  They never generate builtins.

I'd probably go with this variant (that is, set up the makefiles to run
once with -b to generate builtins, and multiple times (one per .json)
without -b to generate non-builtins.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Markus Armbruster]

Eric Blake  writes:

> On 11/11/2015 09:42 AM, Markus Armbruster wrote:
>> Eric Blake  writes:
>> 
>>> What's more meta than using qapi to define qapi? :)
>>>
>>> Convert qtype_code into a full-fledged[*] builtin qapi enum type,
>>> so that a subsequent patch can then use it as the discriminator
>>> type of qapi alternate types.  Doing so is easiest when renaming
>>> it to qapi conventions, as QTypeCode.
>> 
>> Out of curiosity: why does the rename make the conversion easier?
>
> It guarantees I found all affected instances.  (Although I guess the
> rename could be split to a separate patch from making it builtin).

Well, you have to find them only because you rename, don't you?

> It makes sure that if we later tighten rules about naming, we won't have
> to whitelist 'qtype_code' as an anomaly to our conventions.

Good point.

>> If we rename anyway, what about renaming to QType?  Hmm, we burned that
>> on a struct we use only internally in qobject/.  Oh well.
>
> Internal structs are often easy to rename.  So if we want to avoid the
> need for 'prefix', I could certainly try to achieve that (move internal
> QType out of the way, then rename qtype_code to QType, then make QType
> the builtin).  Looks like this one patch just became three :)

Not sure it's worth the bother; the patch is okay as it is.

QType is overkill.  Instead of

typedef struct QType {
qtype_code code;
void (*destroy)(struct QObject *);
} QType;

typedef struct QObject {
const QType *type;
size_t refcnt;
} QObject;

we could simply have

typedef struct QObject {
QTypeCode type;
size_t refcnt;
} QObject;

with an array mapping QTypeCode to destroy methods.  We're not going to
define additional types at run time.

Perhaps such a change would be actually worth the bother.

>>>Fortunately, there are not
>>> many places in the tree that were actually spelling the type name
>>> out, and the judicious use of 'prefix' in the qapi defintion
>> 
>> definition
>
> I've got to quit coding late at night - my rate of typos increases :)
>
>>> +++ b/docs/qapi-code-gen.txt
>>> @@ -163,6 +163,7 @@ The following types are predefined, and map to C as 
>>> follows:
>>> accepts size suffixes
>>>bool  bool   JSON true or false
>>>any   QObject *  any JSON value
>>> +  QTypeCode QTypeCode  JSON string of enum QTypeCode values
>> 
>> QTypeCode is currently used only internally, so the JSON values don't
>> matter.  I don't expect that to change.  However, we either enforce
>> internal use somehow, or document the JSON values.  Documenting them is
>> easier.
>> 
>> In short, your patch is fine.
>> 
>
>>> -
>>> -struct QObject;
>>> +#include "qapi-types.h"
>>>
>>>  typedef struct QType {
>>> -qtype_code code;
>>> +QTypeCode code;
>>>  void (*destroy)(struct QObject *);
>>>  } QType;
>>>
>>typedef struct QObject {
>>const QType *type;
>>size_t refcnt;
>>} QObject;
>> 
>> Note: typedef name QObject still defined here.
>
> Oh, I see what you're saying. Since qapi-types.h now has a forward
> declaration of the QObject typedef, this could be changed to just
>
> struct QObject {
> ...
> };
>
>>> +++ b/scripts/qapi-types.py
>>> @@ -233,8 +233,14 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>>>  self.defn += gen_type_cleanup(name)
>>>
>>>  def visit_enum_type(self, name, info, values, prefix):
>>> -self._fwdecl += gen_enum(name, values, prefix)
>>> -self._fwdefn += gen_enum_lookup(name, values, prefix)
>>> +# Special case for our lone builtin enum type
>>> +if name == 'QTypeCode':
>> 
>> Would "if not info" work?  Same in qapi-visit.py below.
>
> Feels a bit hacky, since we just recently added is_implicit() to hide
> (and then change) the 'if not info' check on objects.  Maybe an accessor
> is_builtin() makes more sense?  But yes, same approach to both client files.

QAPISchemaEntity methods like is_implicit() or a new is_builtin() can't
work here, because we lack the entity.

We have one in visit_needed(), and we use its is_implicit() to skip
implicit object types.  We could use entity.is_builtin() to skip (some)
builtins, and handle them elsewhere, but that doesn't feel like an
improvement over your code.

Let's take a step back and reconsider how we do builtins.

>> +self._btin += gen_enum(name, values, prefix)
>> +if do_builtins:
>> +self.defn += gen_enum_lookup(name, values, prefix)
>> +else:
>> +self._fwdecl += gen_enum(name, values, prefix)
>> +self._fwdefn += gen_enum_lookup(name, values, prefix)
>>
>>  def visit_array_type(self, name, info, element_type):
>>  if isinstance(element_type, QAPISchemaBuiltinType):

Linking generated code from multiple schemata that share names may fail,
because multiple 

Re: [Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Markus Armbruster]

Eric Blake  writes:

> What's more meta than using qapi to define qapi? :)
>
> Convert qtype_code into a full-fledged[*] builtin qapi enum type,
> so that a subsequent patch can then use it as the discriminator
> type of qapi alternate types.  Doing so is easiest when renaming
> it to qapi conventions, as QTypeCode.

Out of curiosity: why does the rename make the conversion easier?

If we rename anyway, what about renaming to QType?  Hmm, we burned that
on a struct we use only internally in qobject/.  Oh well.

>Fortunately, there are not
> many places in the tree that were actually spelling the type name
> out, and the judicious use of 'prefix' in the qapi defintion

definition

> avoids churn to the spelling of the enum constants.
>
> To avoid circular definitions, we have to flip the order of
> inclusion between "qobject.h" vs. "qapi-types.h".  Back in commit
> 28770e0, we had the latter include the former, so that we could
> use 'QObject *' for our implementation of 'any'.  But that usage
> also works with only a forward declaration, whereas the
> definition of QType requires QTypeCode to be a complete type.
>
> [*] The type has to be builtin, rather than declared in
> qapi/common.json, because we want to use it for alternates even
> when common.json is not included. But since it is the first
> builtin enum type, we have to add special cases to qapi-types
> and qapi-visit to only emit definitions once, even when two
> qapi files are being compiled into the same binary (the way we
> already handled builtin list types like 'intList').  We may
> need to revisit how multiple qapi files share common types,
> but that's a project for another day.
>
> Signed-off-by: Eric Blake 
>
> ---
> v11: new patch
> ---
>  block/qapi.c |  4 ++--
>  docs/qapi-code-gen.txt   |  1 +
>  include/hw/qdev-core.h   |  2 +-
>  include/qapi/qmp/qobject.h   | 19 +++
>  qobject/qdict.c  |  2 +-
>  scripts/qapi-types.py| 13 ++---
>  scripts/qapi-visit.py| 10 --
>  scripts/qapi.py  |  7 ++-
>  tests/qapi-schema/alternate-empty.out|  2 ++
>  tests/qapi-schema/comments.out   |  2 ++
>  tests/qapi-schema/empty.out  |  2 ++
>  tests/qapi-schema/event-case.out |  2 ++
>  tests/qapi-schema/flat-union-empty.out   |  2 ++
>  tests/qapi-schema/ident-with-escape.out  |  2 ++
>  tests/qapi-schema/include-relpath.out|  2 ++
>  tests/qapi-schema/include-repetition.out |  2 ++
>  tests/qapi-schema/include-simple.out |  2 ++
>  tests/qapi-schema/indented-expr.out  |  2 ++
>  tests/qapi-schema/qapi-schema-test.out   |  2 ++
>  tests/qapi-schema/union-clash-data.out   |  2 ++
>  tests/qapi-schema/union-empty.out|  2 ++
>  21 files changed, 58 insertions(+), 26 deletions(-)
>
> diff --git a/block/qapi.c b/block/qapi.c
> index ec0f513..4211f11 100644
> --- a/block/qapi.c
> +++ b/block/qapi.c
> @@ -539,7 +539,7 @@ static void dump_qlist(fprintf_function func_fprintf, 
> void *f, int indentation,
>  int i = 0;
>
>  for (entry = qlist_first(list); entry; entry = qlist_next(entry), i++) {
> -qtype_code type = qobject_type(entry->value);
> +QTypeCode type = qobject_type(entry->value);
>  bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
>  const char *format = composite ? "%*s[%i]:\n" : "%*s[%i]: ";
>
> @@ -557,7 +557,7 @@ static void dump_qdict(fprintf_function func_fprintf, 
> void *f, int indentation,
>  const QDictEntry *entry;
>
>  for (entry = qdict_first(dict); entry; entry = qdict_next(dict, entry)) {
> -qtype_code type = qobject_type(entry->value);
> +QTypeCode type = qobject_type(entry->value);
>  bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
>  const char *format = composite ? "%*s%s:\n" : "%*s%s: ";
>  char key[strlen(entry->key) + 1];
> diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
> index 54a6a7b..35301c5 100644
> --- a/docs/qapi-code-gen.txt
> +++ b/docs/qapi-code-gen.txt
> @@ -163,6 +163,7 @@ The following types are predefined, and map to C as 
> follows:
> accepts size suffixes
>bool  bool   JSON true or false
>any   QObject *  any JSON value
> +  QTypeCode QTypeCode  JSON string of enum QTypeCode values

QTypeCode is currently used only internally, so the JSON values don't
matter.  I don't expect that to change.  However, we either enforce
internal use somehow, or document the JSON values.  Documenting them is
easier.

In short, your patch is fine.

>
>
>  === Includes ===
> diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
> index 8057aed..8e7df8e 100644
> --- a/include/hw/qdev-core.h
> +++ b/include/hw/qdev-core.h
> @@ -239,7 +239,7 

Re: [Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Eric Blake]

On 11/11/2015 09:42 AM, Markus Armbruster wrote:
> Eric Blake  writes:
> 
>> What's more meta than using qapi to define qapi? :)
>>
>> Convert qtype_code into a full-fledged[*] builtin qapi enum type,
>> so that a subsequent patch can then use it as the discriminator
>> type of qapi alternate types.  Doing so is easiest when renaming
>> it to qapi conventions, as QTypeCode.
> 
> Out of curiosity: why does the rename make the conversion easier?

It guarantees I found all affected instances.  (Although I guess the
rename could be split to a separate patch from making it builtin).

It makes sure that if we later tighten rules about naming, we won't have
to whitelist 'qtype_code' as an anomaly to our conventions.

> 
> If we rename anyway, what about renaming to QType?  Hmm, we burned that
> on a struct we use only internally in qobject/.  Oh well.

Internal structs are often easy to rename.  So if we want to avoid the
need for 'prefix', I could certainly try to achieve that (move internal
QType out of the way, then rename qtype_code to QType, then make QType
the builtin).  Looks like this one patch just became three :)

> 
>>Fortunately, there are not
>> many places in the tree that were actually spelling the type name
>> out, and the judicious use of 'prefix' in the qapi defintion
> 
> definition

I've got to quit coding late at night - my rate of typos increases :)

>> +++ b/docs/qapi-code-gen.txt
>> @@ -163,6 +163,7 @@ The following types are predefined, and map to C as 
>> follows:
>> accepts size suffixes
>>bool  bool   JSON true or false
>>any   QObject *  any JSON value
>> +  QTypeCode QTypeCode  JSON string of enum QTypeCode values
> 
> QTypeCode is currently used only internally, so the JSON values don't
> matter.  I don't expect that to change.  However, we either enforce
> internal use somehow, or document the JSON values.  Documenting them is
> easier.
> 
> In short, your patch is fine.
> 

>> -
>> -struct QObject;
>> +#include "qapi-types.h"
>>
>>  typedef struct QType {
>> -qtype_code code;
>> +QTypeCode code;
>>  void (*destroy)(struct QObject *);
>>  } QType;
>>
>typedef struct QObject {
>const QType *type;
>size_t refcnt;
>} QObject;
> 
> Note: typedef name QObject still defined here.

Oh, I see what you're saying. Since qapi-types.h now has a forward
declaration of the QObject typedef, this could be changed to just

struct QObject {
...
};

>> +++ b/scripts/qapi-types.py
>> @@ -233,8 +233,14 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>>  self.defn += gen_type_cleanup(name)
>>
>>  def visit_enum_type(self, name, info, values, prefix):
>> -self._fwdecl += gen_enum(name, values, prefix)
>> -self._fwdefn += gen_enum_lookup(name, values, prefix)
>> +# Special case for our lone builtin enum type
>> +if name == 'QTypeCode':
> 
> Would "if not info" work?  Same in qapi-visit.py below.

Feels a bit hacky, since we just recently added is_implicit() to hide
(and then change) the 'if not info' check on objects.  Maybe an accessor
is_builtin() makes more sense?  But yes, same approach to both client files.


>> -#include "qapi/qmp/qobject.h"
>> +
>> +typedef struct QObject QObject;
> 
> Typedef name QObject now also defined here.  GCC accepts this silently
> without -Wpedantic, but other compilers might not.  Whether we care for
> such compilers or not, defining things in exactly one place is neater.
> 
> Possible fixes:
> 
> * Drop the typedef from qobject.h
> 
> * Don't add it to qapi-types.h, and use struct QObject there
> 

I favor dropping the second typedef.


>> +++ b/scripts/qapi.py
>> @@ -33,7 +33,7 @@ builtin_types = {
>>  'uint32':   'QTYPE_QINT',
>>  'uint64':   'QTYPE_QINT',
>>  'size': 'QTYPE_QINT',
>> -'any':  None,   # any qtype_code possible, actually
>> +'any':  None,   # any QTypeCode possible, actually
>>  }
>>
> 
> Should we list QTypeCode here?

Yeah, probably.  This array is only used by the ad hoc parser, and may
disappear later as we move more into check(), but we should be
consistent in the meantime.

> 
>>  # Whitelist of commands allowed to return a non-dictionary
>> @@ -1243,6 +1243,11 @@ class QAPISchema(object):
>>  self.the_empty_object_type = QAPISchemaObjectType(':empty', None, 
>> None,
>>[], None)
>>  self._def_entity(self.the_empty_object_type)
>> +self._def_entity(QAPISchemaEnumType('QTypeCode', None,
>> +['none', 'qnull', 'qint',
>> + 'qstring', 'qdict', 'qlist',
>> + 'qfloat', 'qbool'],
>> +'QTYPE'))
>>
>>  def _make_implicit_enum_type(self, name, info, values):
>>

[Qemu-devel] [PATCH v11 21/28] qapi: Convert qtype_code into qapi enum type

[Eric Blake]

What's more meta than using qapi to define qapi? :)

Convert qtype_code into a full-fledged[*] builtin qapi enum type,
so that a subsequent patch can then use it as the discriminator
type of qapi alternate types.  Doing so is easiest when renaming
it to qapi conventions, as QTypeCode.  Fortunately, there are not
many places in the tree that were actually spelling the type name
out, and the judicious use of 'prefix' in the qapi defintion
avoids churn to the spelling of the enum constants.

To avoid circular definitions, we have to flip the order of
inclusion between "qobject.h" vs. "qapi-types.h".  Back in commit
28770e0, we had the latter include the former, so that we could
use 'QObject *' for our implementation of 'any'.  But that usage
also works with only a forward declaration, whereas the
definition of QType requires QTypeCode to be a complete type.

[*] The type has to be builtin, rather than declared in
qapi/common.json, because we want to use it for alternates even
when common.json is not included. But since it is the first
builtin enum type, we have to add special cases to qapi-types
and qapi-visit to only emit definitions once, even when two
qapi files are being compiled into the same binary (the way we
already handled builtin list types like 'intList').  We may
need to revisit how multiple qapi files share common types,
but that's a project for another day.

Signed-off-by: Eric Blake 

---
v11: new patch
---
 block/qapi.c |  4 ++--
 docs/qapi-code-gen.txt   |  1 +
 include/hw/qdev-core.h   |  2 +-
 include/qapi/qmp/qobject.h   | 19 +++
 qobject/qdict.c  |  2 +-
 scripts/qapi-types.py| 13 ++---
 scripts/qapi-visit.py| 10 --
 scripts/qapi.py  |  7 ++-
 tests/qapi-schema/alternate-empty.out|  2 ++
 tests/qapi-schema/comments.out   |  2 ++
 tests/qapi-schema/empty.out  |  2 ++
 tests/qapi-schema/event-case.out |  2 ++
 tests/qapi-schema/flat-union-empty.out   |  2 ++
 tests/qapi-schema/ident-with-escape.out  |  2 ++
 tests/qapi-schema/include-relpath.out|  2 ++
 tests/qapi-schema/include-repetition.out |  2 ++
 tests/qapi-schema/include-simple.out |  2 ++
 tests/qapi-schema/indented-expr.out  |  2 ++
 tests/qapi-schema/qapi-schema-test.out   |  2 ++
 tests/qapi-schema/union-clash-data.out   |  2 ++
 tests/qapi-schema/union-empty.out|  2 ++
 21 files changed, 58 insertions(+), 26 deletions(-)

diff --git a/block/qapi.c b/block/qapi.c
index ec0f513..4211f11 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -539,7 +539,7 @@ static void dump_qlist(fprintf_function func_fprintf, void 
*f, int indentation,
 int i = 0;

 for (entry = qlist_first(list); entry; entry = qlist_next(entry), i++) {
-qtype_code type = qobject_type(entry->value);
+QTypeCode type = qobject_type(entry->value);
 bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
 const char *format = composite ? "%*s[%i]:\n" : "%*s[%i]: ";

@@ -557,7 +557,7 @@ static void dump_qdict(fprintf_function func_fprintf, void 
*f, int indentation,
 const QDictEntry *entry;

 for (entry = qdict_first(dict); entry; entry = qdict_next(dict, entry)) {
-qtype_code type = qobject_type(entry->value);
+QTypeCode type = qobject_type(entry->value);
 bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
 const char *format = composite ? "%*s%s:\n" : "%*s%s: ";
 char key[strlen(entry->key) + 1];
diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 54a6a7b..35301c5 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -163,6 +163,7 @@ The following types are predefined, and map to C as follows:
accepts size suffixes
   bool  bool   JSON true or false
   any   QObject *  any JSON value
+  QTypeCode QTypeCode  JSON string of enum QTypeCode values


 === Includes ===
diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 8057aed..8e7df8e 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -239,7 +239,7 @@ struct Property {
 PropertyInfo *info;
 int  offset;
 uint8_t  bitnr;
-qtype_code   qtype;
+QTypeCodeqtype;
 int64_t  defval;
 int  arrayoffset;
 PropertyInfo *arrayinfo;
diff --git a/include/qapi/qmp/qobject.h b/include/qapi/qmp/qobject.h
index 4b96ed5..8d6322b 100644
--- a/include/qapi/qmp/qobject.h
+++ b/include/qapi/qmp/qobject.h
@@ -34,23 +34,10 @@

 #include 
 #include 
-
-typedef enum {
-QTYPE_NONE,/* sentinel value, no QObject has this type code */
-QTYPE_QNULL,
-QTYPE_QINT,
-QTYPE_QSTRING,
-QTYPE_QDICT,
-QTYPE_QLIST,
-QTYPE_QFLOAT,
-QTYPE_QBOOL,
-QTYPE_MAX,
-} qtype_code;
-
-struct QObject;
+#include