John Snow <js...@redhat.com> writes: > On 12/24/19 8:41 AM, Daniel P. Berrangé wrote: >>> * scripts/qmp/qmp-shell >>> >>> Half-hearted attempt at a human-friendly wrapper around the JSON >>> syntax. I have no use for this myself. >> I use this fairly often as its a useful debugging / experimentation >> / trouble shooting tool. There's similar ish functionality in >> virsh qemu-monitor-command. I think there's scope of a supported >> tool here that can talk to libvirt or a UNIX socket for doing >> QMP commands, with a friendlier syntax & pretty printing. >> > > qmp-shell is one of my go-to tools for working through bitmap workflows > where we don't have convenience commands yet, as some of the setups > required for fleecing et al involve quite a number of steps. > > I can copy-paste raw JSON into a socket, but personally I like seeing my > commands neatly organized in a format where I can visually reduce them > to their components at a glance. > > (What I mean is: It's hard to remember which QMP commands you've barfed > into a terminal because JSON is hard to read and looks very visually > repetitive.) > > I tried to rewrite qmp-shell late last year, actually. I wanted to write > a new REPL that was json-aware in some manner such that you could write > multi-line commands like this: > >> example-command arg={ > "hello": "world" > } > > This requires, sadly, a streamable JSON parser. Most JSON parsers built > into Python as-is simply take a file pointer and consume the entirety of > the rest of the stream -- they don't play very nice with incomplete > input or input that may have trailing data, e.g.: > >> example-command arg={ > "hello": "world" > } arg2={ > "oops!": "more json!" > }
QMP is in the same boat: it needs to process input that isn't necessarily full expressions (JSON-text in the RFC's grammar). Any conventional parser can be made streaming by turning it into a coroutine. This is probably the simplest solution for handwritten streaming LL parsers, because it permits recursive descent. In Python, I'd try a generator. Our actual solution for QMP predates coroutine support in QEMU, and is rather hamfisted: * Streaming lexer: it gets fed characters one at a time, and when its state machine says "token complete", it feeds the token to the "streamer". * "Streamer": gets fed tokens one at a time, buffers them up counting curly and square bracket nesting until the nesting is zero, then passes the buffered tokens to the parser. * Non-streaming parser: it gets fed a sequence of tokens that constitute a full expression. The best I can say about this is that it works. The streamer's token buffer eats a lot of memory compared to a real streaming parser, but in practice, it's a drop in the bucket. > Also, due to the nature of JSON as being a single discrete object and > never a stream of objects, no existing JSON parser really supports the > idea of ever seeing more than one object per buffer. That plainly sucks. > ...So I investigated writing a proper grammar for qmp-shell. Any parser must start with a proper grammar. If it doesn't, it's a toy, or a highway to madness. > Unfortunately, this basically means including the JSON grammar as a > subset of the shell grammar and writing your own parser for it entirely. Because qmp-shell is a half-hearted wrapper: we ran out of wrapping paper, so JSON sticks out left and right. Scrap and start over. > I looked into using Python's own lexer; but it's designed to lex > *python*, not *json*. I got a prototype lexer working for this purpose > under a grammar that I think reflects JSON, but I got that sinking > feeling that it was all more trouble than it was worth, and scrapped > working on it any further. Parsing JSON is pretty simple. Data point: QAPISchemaParser parses our weird derivative of JSON in 239 SLOC. > I did not find any other flex/yacc-like tools that seemed properly > idiomatic or otherwise heavily specialized. I gave up on the idea of > writing a new parser. While I recommend use of tools for parsing non-trivial grammars (you'll screw up, they won't), they're massive overkill for JSON. > I'd love to offer a nice robust QMP shell that is available for use by > end users, but the syntax of the shell will need some major considerations. Scrap and start over. [...]