Re: [Qemu-devel] [PATCH v2 6/9] qapi: Rewrite string-input-visitor
David Hildenbrand writes:
> On 20.11.18 21:58, Markus Armbruster wrote:
>> I think the title should be something like
>>
>> qapi: Rewrite string-input-visitor's integer and list parsing
>>
>> because you don't actually rewrite all of it.
>>
>> Eric Blake writes:
>>
>>> On 11/20/18 3:25 AM, David Hildenbrand wrote:
The input visitor has some problems right now, especially
- unsigned type "Range" is used to process signed ranges, resulting in
inconsistent behavior and ugly/magical code
- uint64_t are parsed like int64_t, so big uint64_t values are not
supported and error messages are misleading
- lists/ranges of int64_t are accepted although no list is parsed and
we should rather report an error
- lists/ranges are preparsed using int64_t, making it hard to
implement uint64_t values or uint64_t lists
- types that don't support lists don't bail out
- visiting beyond the end of a list is not handled properly
- we don't actually parse lists, we parse *sets*: members are sorted,
and duplicates eliminated
So let's rewrite it by getting rid of usage of the type "Range" and
properly supporting lists of int64_t and uint64_t (including ranges of
both types), fixing the above mentioned issues.
Lists of other types are not supported and will properly report an
error. Virtual walks are now supported.
Tests have to be fixed up:
- Two BUGs were hardcoded that are fixed now
- The string-input-visitor now actually returns a parsed list and not
an ordered set.
Please note that no users/callers have to be fixed up. Candiates using
>>>
>>> s/Candiates/Candidates/
>>>
visit_type_uint16List() and friends are:
- backends/hostmem.c:host_memory_backend_set_host_nodes()
-- Code can deal with dupilcates/unsorted lists
>>>
>>> s/dupilcates/duplicates/
>
> Thanks, both fixed.
>
>>
@@ -330,9 +381,10 @@ static void parse_type_null(Visitor *v, const char
*name, QNull **obj,
{
StringInputVisitor *siv = to_siv(v);
+assert(siv->lm == LM_NONE);
*obj = NULL;
-if (!siv->string || siv->string[0]) {
+if (siv->string[0]) {
>>>
>>> Why did this condition change?
>>
>> As far as I can tell, siv->string can't ever be null. Sticking the
>> change into this patch is perhaps debatable. I'm okay with it.
>
> Yes, we have an assertion when creating the visitor. Do you want me to
> pull this into a separate patch?
I'm okay with it as is. Up to you.
> (It made sense under the old patch subject ;) )
It did :)
>>> Reviewed-by: Eric Blake
>>
>> With the commit message improved once more:
>> Reviewed-by: Markus Armbruster
>>
> Thanks!
Re: [Qemu-devel] [PATCH v2 6/9] qapi: Rewrite string-input-visitor
On 20.11.18 21:58, Markus Armbruster wrote:
> I think the title should be something like
>
> qapi: Rewrite string-input-visitor's integer and list parsing
>
> because you don't actually rewrite all of it.
>
> Eric Blake writes:
>
>> On 11/20/18 3:25 AM, David Hildenbrand wrote:
>>> The input visitor has some problems right now, especially
>>> - unsigned type "Range" is used to process signed ranges, resulting in
>>>inconsistent behavior and ugly/magical code
>>> - uint64_t are parsed like int64_t, so big uint64_t values are not
>>>supported and error messages are misleading
>>> - lists/ranges of int64_t are accepted although no list is parsed and
>>>we should rather report an error
>>> - lists/ranges are preparsed using int64_t, making it hard to
>>>implement uint64_t values or uint64_t lists
>>> - types that don't support lists don't bail out
>>> - visiting beyond the end of a list is not handled properly
>>> - we don't actually parse lists, we parse *sets*: members are sorted,
>>>and duplicates eliminated
>>>
>>> So let's rewrite it by getting rid of usage of the type "Range" and
>>> properly supporting lists of int64_t and uint64_t (including ranges of
>>> both types), fixing the above mentioned issues.
>>>
>>> Lists of other types are not supported and will properly report an
>>> error. Virtual walks are now supported.
>>>
>>> Tests have to be fixed up:
>>> - Two BUGs were hardcoded that are fixed now
>>> - The string-input-visitor now actually returns a parsed list and not
>>>an ordered set.
>>>
>>> Please note that no users/callers have to be fixed up. Candiates using
>>
>> s/Candiates/Candidates/
>>
>>> visit_type_uint16List() and friends are:
>>> - backends/hostmem.c:host_memory_backend_set_host_nodes()
>>> -- Code can deal with dupilcates/unsorted lists
>>
>> s/dupilcates/duplicates/
Thanks, both fixed.
>
>>> @@ -330,9 +381,10 @@ static void parse_type_null(Visitor *v, const char
>>> *name, QNull **obj,
>>> {
>>> StringInputVisitor *siv = to_siv(v);
>>> +assert(siv->lm == LM_NONE);
>>> *obj = NULL;
>>> -if (!siv->string || siv->string[0]) {
>>> +if (siv->string[0]) {
>>
>> Why did this condition change?
>
> As far as I can tell, siv->string can't ever be null. Sticking the
> change into this patch is perhaps debatable. I'm okay with it.
Yes, we have an assertion when creating the visitor. Do you want me to
pull this into a separate patch?
(It made sense under the old patch subject ;) )
>
>> Reviewed-by: Eric Blake
>
> With the commit message improved once more:
> Reviewed-by: Markus Armbruster
>
Thanks!
--
Thanks,
David / dhildenb
Re: [Qemu-devel] [PATCH v2 6/9] qapi: Rewrite string-input-visitor
I think the title should be something like
qapi: Rewrite string-input-visitor's integer and list parsing
because you don't actually rewrite all of it.
Eric Blake writes:
> On 11/20/18 3:25 AM, David Hildenbrand wrote:
>> The input visitor has some problems right now, especially
>> - unsigned type "Range" is used to process signed ranges, resulting in
>>inconsistent behavior and ugly/magical code
>> - uint64_t are parsed like int64_t, so big uint64_t values are not
>>supported and error messages are misleading
>> - lists/ranges of int64_t are accepted although no list is parsed and
>>we should rather report an error
>> - lists/ranges are preparsed using int64_t, making it hard to
>>implement uint64_t values or uint64_t lists
>> - types that don't support lists don't bail out
>> - visiting beyond the end of a list is not handled properly
>> - we don't actually parse lists, we parse *sets*: members are sorted,
>>and duplicates eliminated
>>
>> So let's rewrite it by getting rid of usage of the type "Range" and
>> properly supporting lists of int64_t and uint64_t (including ranges of
>> both types), fixing the above mentioned issues.
>>
>> Lists of other types are not supported and will properly report an
>> error. Virtual walks are now supported.
>>
>> Tests have to be fixed up:
>> - Two BUGs were hardcoded that are fixed now
>> - The string-input-visitor now actually returns a parsed list and not
>>an ordered set.
>>
>> Please note that no users/callers have to be fixed up. Candiates using
>
> s/Candiates/Candidates/
>
>> visit_type_uint16List() and friends are:
>> - backends/hostmem.c:host_memory_backend_set_host_nodes()
>> -- Code can deal with dupilcates/unsorted lists
>
> s/dupilcates/duplicates/
>
>> - numa.c::query_memdev()
>> -- via object_property_get_uint16List(), the list will still be sorted
>> and without duplicates (via host_memory_backend_get_host_nodes())
>> - qapi-visit.c::visit_type_Memdev_members()
>> - qapi-visit.c::visit_type_NumaNodeOptions_members()
>> - qapi-visit.c::visit_type_RockerOfDpaGroup_members
>> - qapi-visit.c::visit_type_RxFilterInfo_members()
>> -- Not used with string-input-visitor.
>>
>> Signed-off-by: David Hildenbrand
>> ---
>> include/qapi/string-input-visitor.h | 4 +-
>> qapi/string-input-visitor.c | 405
>> tests/test-string-input-visitor.c | 18 +-
>> 3 files changed, 234 insertions(+), 193 deletions(-)
>>
>
>> struct StringInputVisitor
>> {
>> Visitor visitor;
>> -GList *ranges;
>> -GList *cur_range;
>> -int64_t cur;
>> +/* List parsing state */
>> +ListMode lm;
>> +RangeElement rangeNext;
>> +RangeElement rangeEnd;
>> +const char *unparsed_string;
>> +void *list;
>> +/* The original string to parse */
>> const char *string;
>> -void *list; /* Only needed for sanity checking the caller */
>> };
>>
>
> Makes sense.
>
>> @@ -179,88 +106,208 @@ static GenericList *next_list(Visitor *v, GenericList
>> *tail, size_t size)
>> static void check_list(Visitor *v, Error **errp)
>> {
>> const StringInputVisitor *siv = to_siv(v);
>> -Range *r;
>> -GList *cur_range;
>> -if (!siv->ranges || !siv->cur_range) {
>> +switch (siv->lm) {
>> +case LM_INT64_RANGE:
>> +case LM_UINT64_RANGE:
>> +case LM_UNPARSED:
>> +error_setg(errp, "Fewer list elements expected");
>
> Bike-shedding - I don't know if "Too many list elements supplied"
> would make the error any more legible.
See my review of PATCH RFC 3/6:
Hmm. qobject_input_check_list() reports "Only %u list elements expected
in %s" here. Looks unintuitive, until you remember how it's normally
used: to parse user input. The error gets reported when the parsing
didn't consume the whole list. Can only happen with a virtual walk.
And when it happens, the issue to report is "you provided a longer list
than I can accept". qobject_input_check_list() does exactly that. Your
message is less clear.
David didn't feel counting elements just for that was worthwhile here,
and I agreed.
>> static void parse_type_int64(Visitor *v, const char *name,
>> int64_t *obj,
>>Error **errp)
>> {
>> StringInputVisitor *siv = to_siv(v);
>> -
>> -if (parse_str(siv, name, errp) < 0) {
>> +int64_t val;
>> +
>> +switch (siv->lm) {
>> +case LM_NONE:
>> +/* just parse a simple int64, bail out if not completely consumed */
>> +if (qemu_strtoi64(siv->string, NULL, 0, &val)) {
>> +error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
>> + name ? name : "null", "int64");
>> +return;
>> +}
>> +*obj = val;
>> return;
>> +case LM_UNPARSED:
>> +if (try_parse_int64_list_entry(siv, obj)) {
>> +error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name ? name :
>> "null",
>> +
Re: [Qemu-devel] [PATCH v2 6/9] qapi: Rewrite string-input-visitor
On 11/20/18 3:25 AM, David Hildenbrand wrote:
The input visitor has some problems right now, especially
- unsigned type "Range" is used to process signed ranges, resulting in
inconsistent behavior and ugly/magical code
- uint64_t are parsed like int64_t, so big uint64_t values are not
supported and error messages are misleading
- lists/ranges of int64_t are accepted although no list is parsed and
we should rather report an error
- lists/ranges are preparsed using int64_t, making it hard to
implement uint64_t values or uint64_t lists
- types that don't support lists don't bail out
- visiting beyond the end of a list is not handled properly
- we don't actually parse lists, we parse *sets*: members are sorted,
and duplicates eliminated
So let's rewrite it by getting rid of usage of the type "Range" and
properly supporting lists of int64_t and uint64_t (including ranges of
both types), fixing the above mentioned issues.
Lists of other types are not supported and will properly report an
error. Virtual walks are now supported.
Tests have to be fixed up:
- Two BUGs were hardcoded that are fixed now
- The string-input-visitor now actually returns a parsed list and not
an ordered set.
Please note that no users/callers have to be fixed up. Candiates using
s/Candiates/Candidates/
visit_type_uint16List() and friends are:
- backends/hostmem.c:host_memory_backend_set_host_nodes()
-- Code can deal with dupilcates/unsorted lists
s/dupilcates/duplicates/
- numa.c::query_memdev()
-- via object_property_get_uint16List(), the list will still be sorted
and without duplicates (via host_memory_backend_get_host_nodes())
- qapi-visit.c::visit_type_Memdev_members()
- qapi-visit.c::visit_type_NumaNodeOptions_members()
- qapi-visit.c::visit_type_RockerOfDpaGroup_members
- qapi-visit.c::visit_type_RxFilterInfo_members()
-- Not used with string-input-visitor.
Signed-off-by: David Hildenbrand
---
include/qapi/string-input-visitor.h | 4 +-
qapi/string-input-visitor.c | 405
tests/test-string-input-visitor.c | 18 +-
3 files changed, 234 insertions(+), 193 deletions(-)
struct StringInputVisitor
{
Visitor visitor;
-GList *ranges;
-GList *cur_range;
-int64_t cur;
+/* List parsing state */
+ListMode lm;
+RangeElement rangeNext;
+RangeElement rangeEnd;
+const char *unparsed_string;
+void *list;
+/* The original string to parse */
const char *string;
-void *list; /* Only needed for sanity checking the caller */
};
Makes sense.
@@ -179,88 +106,208 @@ static GenericList *next_list(Visitor *v, GenericList
*tail, size_t size)
static void check_list(Visitor *v, Error **errp)
{
const StringInputVisitor *siv = to_siv(v);
-Range *r;
-GList *cur_range;
-if (!siv->ranges || !siv->cur_range) {
+switch (siv->lm) {
+case LM_INT64_RANGE:
+case LM_UINT64_RANGE:
+case LM_UNPARSED:
+error_setg(errp, "Fewer list elements expected");
Bike-shedding - I don't know if "Too many list elements supplied" would
make the error any more legible.
static void parse_type_int64(Visitor *v, const char *name, int64_t *obj,
Error **errp)
{
StringInputVisitor *siv = to_siv(v);
-
-if (parse_str(siv, name, errp) < 0) {
+int64_t val;
+
+switch (siv->lm) {
+case LM_NONE:
+/* just parse a simple int64, bail out if not completely consumed */
+if (qemu_strtoi64(siv->string, NULL, 0, &val)) {
+error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ name ? name : "null", "int64");
+return;
+}
+*obj = val;
return;
+case LM_UNPARSED:
+if (try_parse_int64_list_entry(siv, obj)) {
+error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name ? name :
"null",
+ "list of int64 values or ranges");
The error message might be a bit misleading for a range larger than 64k,
but that's not too bad.
+return;
+}
+assert(siv->lm == LM_INT64_RANGE);
+/* fall through */
+case LM_INT64_RANGE:
+/* return the next element in the range */
+assert(siv->rangeNext.i64 <= siv->rangeEnd.i64);
+*obj = siv->rangeNext.i64++;
+
+if (siv->rangeNext.i64 > siv->rangeEnd.i64 || *obj == INT64_MAX) {
I think our compiler options guarantee that we have sane signed
wraparound and thus this is a safe comparison on overflow; but if you
were to swap it so that the *obj == INT64_MAX check is performed first,
you wouldn't even have to debate about whether undefined C semantics are
being invoked.
+/* end of range, check if there is more to parse */
+siv->lm = siv->unparsed_string[0] ? LM_UNPARSED : LM_END;
+}
+return;
+case LM_END:
+error_setg(errp, "Fewer list elements expected");
Again, bi
[Qemu-devel] [PATCH v2 6/9] qapi: Rewrite string-input-visitor
The input visitor has some problems right now, especially
- unsigned type "Range" is used to process signed ranges, resulting in
inconsistent behavior and ugly/magical code
- uint64_t are parsed like int64_t, so big uint64_t values are not
supported and error messages are misleading
- lists/ranges of int64_t are accepted although no list is parsed and
we should rather report an error
- lists/ranges are preparsed using int64_t, making it hard to
implement uint64_t values or uint64_t lists
- types that don't support lists don't bail out
- visiting beyond the end of a list is not handled properly
- we don't actually parse lists, we parse *sets*: members are sorted,
and duplicates eliminated
So let's rewrite it by getting rid of usage of the type "Range" and
properly supporting lists of int64_t and uint64_t (including ranges of
both types), fixing the above mentioned issues.
Lists of other types are not supported and will properly report an
error. Virtual walks are now supported.
Tests have to be fixed up:
- Two BUGs were hardcoded that are fixed now
- The string-input-visitor now actually returns a parsed list and not
an ordered set.
Please note that no users/callers have to be fixed up. Candiates using
visit_type_uint16List() and friends are:
- backends/hostmem.c:host_memory_backend_set_host_nodes()
-- Code can deal with dupilcates/unsorted lists
- numa.c::query_memdev()
-- via object_property_get_uint16List(), the list will still be sorted
and without duplicates (via host_memory_backend_get_host_nodes())
- qapi-visit.c::visit_type_Memdev_members()
- qapi-visit.c::visit_type_NumaNodeOptions_members()
- qapi-visit.c::visit_type_RockerOfDpaGroup_members
- qapi-visit.c::visit_type_RxFilterInfo_members()
-- Not used with string-input-visitor.
Signed-off-by: David Hildenbrand
---
include/qapi/string-input-visitor.h | 4 +-
qapi/string-input-visitor.c | 405
tests/test-string-input-visitor.c | 18 +-
3 files changed, 234 insertions(+), 193 deletions(-)
diff --git a/include/qapi/string-input-visitor.h
b/include/qapi/string-input-visitor.h
index 33551340e3..921f3875b9 100644
--- a/include/qapi/string-input-visitor.h
+++ b/include/qapi/string-input-visitor.h
@@ -19,8 +19,8 @@ typedef struct StringInputVisitor StringInputVisitor;
/*
* The string input visitor does not implement support for visiting
- * QAPI structs, alternates, null, or arbitrary QTypes. It also
- * requires a non-null list argument to visit_start_list().
+ * QAPI structs, alternates, null, or arbitrary QTypes. Only flat lists
+ * of integers (except type "size") are supported.
*/
Visitor *string_input_visitor_new(const char *str);
diff --git a/qapi/string-input-visitor.c b/qapi/string-input-visitor.c
index b89c6c4e06..bd92080667 100644
--- a/qapi/string-input-visitor.c
+++ b/qapi/string-input-visitor.c
@@ -4,10 +4,10 @@
* Copyright Red Hat, Inc. 2012-2016
*
* Author: Paolo Bonzini
+ * David Hildenbrand
*
* This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
* See the COPYING.LIB file in the top-level directory.
- *
*/
#include "qemu/osdep.h"
@@ -18,21 +18,42 @@
#include "qapi/qmp/qerror.h"
#include "qapi/qmp/qnull.h"
#include "qemu/option.h"
-#include "qemu/queue.h"
-#include "qemu/range.h"
#include "qemu/cutils.h"
+typedef enum ListMode {
+/* no list parsing active / no list expected */
+LM_NONE,
+/* we have an unparsed string remaining */
+LM_UNPARSED,
+/* we have an unfinished int64 range */
+LM_INT64_RANGE,
+/* we have an unfinished uint64 range */
+LM_UINT64_RANGE,
+/* we have parsed the string completely and no range is remaining */
+LM_END,
+} ListMode;
+
+/* protect against DOS attacks, limit the amount of elements per range */
+#define RANGE_MAX_ELEMENTS 65536
+
+typedef union RangeElement {
+int64_t i64;
+uint64_t u64;
+} RangeElement;
struct StringInputVisitor
{
Visitor visitor;
-GList *ranges;
-GList *cur_range;
-int64_t cur;
+/* List parsing state */
+ListMode lm;
+RangeElement rangeNext;
+RangeElement rangeEnd;
+const char *unparsed_string;
+void *list;
+/* The original string to parse */
const char *string;
-void *list; /* Only needed for sanity checking the caller */
};
static StringInputVisitor *to_siv(Visitor *v)
@@ -40,136 +61,42 @@ static StringInputVisitor *to_siv(Visitor *v)
return container_of(v, StringInputVisitor, visitor);
}
-static void free_range(void *range, void *dummy)
-{
-g_free(range);
-}
-
-static int parse_str(StringInputVisitor *siv, const char *name, Error **errp)
-{
-char *str = (char *) siv->string;
-long long start, end;
-Range *cur;
-char *endptr;
-
-if (siv->ranges) {
-return 0;
-}
-
-if (!*str) {
-return 0;
-}
-
-do {
-errno = 0;
-start = strtoll(str, &endptr, 0);
-if (errno ==
