Re: [Qemu-devel] [PATCH 1/2] qemu-iotests: add _filter_qmp_events() for filtering out QMP events

[Sascha Silbe]

Dear Eric,

Eric Blake  writes:

>>> tr '\n' '\t' \
>>>   | sed -e
>>> 's/{\s*"timestamp":\s*{[^}]*},\s*"event":[^,}]*\(,\s*"data":\s*{[^}]*}\)\?\s*}\s*//g'
>>> \
>>>   | tr '\t' '\n'
>> 
>> Nice trick. Why didn't I come up with it? ;)
>
> Mishandles any event whose data includes nested dicts.  But a little bit
> more creativity can do it in two passes (the first modifies the stream
> to mark the start of an event, the second then does multiline matching
> to nuke the entire event):
>
> tr '\n' '\t' |
>   sed 's/^{\(\s*"timestamp":\s*{[^}]*},\s*"event":\)/{MARK\1/g' |
>   tr '\t' '\n' |
>   sed '/^{MARK/,/^}/d'

This is starting to get too complex for my taste. If
_filter_qmp_events() isn't generic enough to be used by other test
cases, I can make it internal to 067. Alternatively I can port 067 to
Python, which can handle JSON data types natively with no need for
post-processing hacks.

Sascha
-- 
Softwareentwicklung Sascha Silbe, Niederhofenstraße 5/1, 71229 Leonberg
https://se-silbe.de/
USt-IdNr. DE281696641

Re: [Qemu-devel] [PATCH 1/2] qemu-iotests: add _filter_qmp_events() for filtering out QMP events

[Max Reitz]

On 09.02.2016 14:23, Sascha Silbe wrote:
> The order of some QMP events may depend on the architecture being
> tested. Add support for filtering out QMP events so we can use a
> single reference output for all architecture when the test doesn't
> care about the events.
> 
> Signed-off-by: Sascha Silbe 
> ---
>  tests/qemu-iotests/common.filter | 6 ++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/tests/qemu-iotests/common.filter 
> b/tests/qemu-iotests/common.filter
> index 84b7434..b908aa2 100644
> --- a/tests/qemu-iotests/common.filter
> +++ b/tests/qemu-iotests/common.filter
> @@ -178,6 +178,12 @@ _filter_qmp()
>  -e 'QMP_VERSION'
>  }
>  
> +# remove QMP events from output
> +_filter_qmp_events()
> +{
> +sed -e '/^{\(.*, \)"event": ".*}$/ d'
> +}

There is a pretty good reason test 067 uses -qmp-pretty (as you yourself
say, the lines get pretty long otherwise, and if we have any change
within, the whole line needs to be changed). Using the following ugly
piece of code here instead, we would still be able to use it:

tr '\n' '\t' \
  | sed -e
's/{\s*"timestamp":\s*{[^}]*},\s*"event":[^,}]*\(,\s*"data":\s*{[^}]*}\)\?\s*}\s*//g'
\
  | tr '\t' '\n'

(I'm too lazy for multi-line sed, obviously; and this will break if the
data contains any JSON objects in turn.)

The correct way would be to actually parse the JSON (using perl, python
or whatever) and remove all the top-level objects containing an "event"
key, obviously... But that's probably too much.

I'm not strictly against just dropping -qmp-pretty in 067, but there is
a good reason it's there.

Max

> +
>  # replace driver-specific options in the "Formatting..." line
>  _filter_img_create()
>  {
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH 1/2] qemu-iotests: add _filter_qmp_events() for filtering out QMP events

[Sascha Silbe]

Dear Max,

Max Reitz  writes:

>> +# remove QMP events from output
>> +_filter_qmp_events()
>> +{
>> +sed -e '/^{\(.*, \)"event": ".*}$/ d'
>> +}
>
> There is a pretty good reason test 067 uses -qmp-pretty (as you yourself
> say, the lines get pretty long otherwise, and if we have any change
> within, the whole line needs to be changed).

Additionally, it's a lot easier to read when indented properly,
especially with the block info containing nested dicts.

> Using the following ugly
> piece of code here instead, we would still be able to use it:
>
> tr '\n' '\t' \
>   | sed -e
> 's/{\s*"timestamp":\s*{[^}]*},\s*"event":[^,}]*\(,\s*"data":\s*{[^}]*}\)\?\s*}\s*//g'
> \
>   | tr '\t' '\n'

Nice trick. Why didn't I come up with it? ;)

It definitely is a bit ugly, though. We can't just drop the entire line
(using "d") as the entire stream now is a single line. Matching
parenthesis pairs is context sensitive, so we can't just use regular
expressions to aggregate results into lines. And before I start
implementing a JSON indenter in awk, I'd rather rewrite the whole test
in Python. So if we stay with the shell test for now, we need something
like your incantation above. It's not perfect, but good enough for now
and I can't think of anything significantly simpler right now either.

Will test your version and send a v2. Thanks for the suggestion!

Sascha
-- 
Softwareentwicklung Sascha Silbe, Niederhofenstraße 5/1, 71229 Leonberg
https://se-silbe.de/
USt-IdNr. DE281696641

Re: [Qemu-devel] [PATCH 1/2] qemu-iotests: add _filter_qmp_events() for filtering out QMP events

[Eric Blake]

On 02/10/2016 11:52 AM, Sascha Silbe wrote:
> Dear Max,
> 
> Max Reitz  writes:
> 
>>> +# remove QMP events from output
>>> +_filter_qmp_events()
>>> +{
>>> +sed -e '/^{\(.*, \)"event": ".*}$/ d'
>>> +}
>>
>> There is a pretty good reason test 067 uses -qmp-pretty (as you yourself
>> say, the lines get pretty long otherwise, and if we have any change
>> within, the whole line needs to be changed).
> 
> Additionally, it's a lot easier to read when indented properly,
> especially with the block info containing nested dicts.
> 
>> Using the following ugly
>> piece of code here instead, we would still be able to use it:
>>
>> tr '\n' '\t' \
>>   | sed -e
>> 's/{\s*"timestamp":\s*{[^}]*},\s*"event":[^,}]*\(,\s*"data":\s*{[^}]*}\)\?\s*}\s*//g'
>> \
>>   | tr '\t' '\n'
> 
> Nice trick. Why didn't I come up with it? ;)

Mishandles any event whose data includes nested dicts.  But a little bit
more creativity can do it in two passes (the first modifies the stream
to mark the start of an event, the second then does multiline matching
to nuke the entire event):

tr '\n' '\t' |
  sed 's/^{\(\s*"timestamp":\s*{[^}]*},\s*"event":\)/{MARK\1/g' |
  tr '\t' '\n' |
  sed '/^{MARK/,/^}/d'


> It definitely is a bit ugly, though. We can't just drop the entire line
> (using "d") as the entire stream now is a single line. Matching
> parenthesis pairs is context sensitive, so we can't just use regular
> expressions to aggregate results into lines. And before I start
> implementing a JSON indenter in awk, I'd rather rewrite the whole test
> in Python. So if we stay with the shell test for now, we need something
> like your incantation above. It's not perfect, but good enough for now
> and I can't think of anything significantly simpler right now either.
> 
> Will test your version and send a v2. Thanks for the suggestion!
> 
> Sascha
> 

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH 1/2] qemu-iotests: add _filter_qmp_events() for filtering out QMP events

[Sascha Silbe]

The order of some QMP events may depend on the architecture being
tested. Add support for filtering out QMP events so we can use a
single reference output for all architecture when the test doesn't
care about the events.

Signed-off-by: Sascha Silbe 
---
 tests/qemu-iotests/common.filter | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 84b7434..b908aa2 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -178,6 +178,12 @@ _filter_qmp()
 -e 'QMP_VERSION'
 }
 
+# remove QMP events from output
+_filter_qmp_events()
+{
+sed -e '/^{\(.*, \)"event": ".*}$/ d'
+}
+
 # replace driver-specific options in the "Formatting..." line
 _filter_img_create()
 {
-- 
2.1.4