Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas
On Thu, Feb 18, 2021 at 11:50 AM Andreea Florescu wrote: > On 2/17/21 1:20 PM, Stefan Hajnoczi wrote: > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and know > > the content is safe. > > > > > > > > Thanks, I have published the rust-vmm project ideas on the wiki: > > https://wiki.qemu.org/Google_Summer_of_Code_2021 > Thanks, Stefan! > > > > Please see Sergio's reply about virtio-consoile is libkrun. Maybe it > > affects the project idea? > I synced offline with Sergio. It seems I've misread his comment. > The code is already available in libkrun, and to port it to rust-vmm > will likely take just a couple of days. We explored the option of also > requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an > appropriate GSoC project, but we decided this is not a good idea since > it doesn't look like that feature is useful for the projects consuming > rust-vmm. It also adds complexity, and we would need to maintain that as > well. > > Since it would still be nice to have virtio-console in rust-vmm, I'll > just open an issue in vm-virtio and label it with "help wanted" so > people can pick it up. > Can we remove the virtio-console project from the list of GSoC ideas? Done. > Also, i...@amazon.com will like to help with mentoring the GSoC projects. > Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"? Done. Thanks for getting the rust-vmm mentors together! I'm looking forward to more Rust virtualization projects :). Stefan
Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas
On 2/17/21 1:20 PM, Stefan Hajnoczi wrote: > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > Thanks, I have published the rust-vmm project ideas on the wiki: > https://wiki.qemu.org/Google_Summer_of_Code_2021 Thanks, Stefan! > > Please see Sergio's reply about virtio-consoile is libkrun. Maybe it > affects the project idea? I synced offline with Sergio. It seems I've misread his comment. The code is already available in libkrun, and to port it to rust-vmm will likely take just a couple of days. We explored the option of also requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an appropriate GSoC project, but we decided this is not a good idea since it doesn't look like that feature is useful for the projects consuming rust-vmm. It also adds complexity, and we would need to maintain that as well. Since it would still be nice to have virtio-console in rust-vmm, I'll just open an issue in vm-virtio and label it with "help wanted" so people can pick it up. Can we remove the virtio-console project from the list of GSoC ideas? Also, i...@amazon.com will like to help with mentoring the GSoC projects. Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"? Sorry for the ping-pong, and thanks again for everything! Andreea Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005. 0x9B51563C1FA36782.asc Description: application/pgp-keys
Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas
Thanks, I have published the rust-vmm project ideas on the wiki: https://wiki.qemu.org/Google_Summer_of_Code_2021 Please see Sergio's reply about virtio-consoile is libkrun. Maybe it affects the project idea? Stefan
Re: Call for Google Summer of Code 2021 project ideas
On 2/15/21 6:05 AM, Paolo Bonzini wrote:
On 14/01/21 17:36, John Snow wrote:
The sane way to evade the language design problem is to use the existing
QMP language.
I wouldn't mind implementing this for version 0.1 -- just allow
copy-pasting JSON into the input bar -- it's a feature I wanted anyway.
I think the only way out of language design is to instead design a TUI
for inputting JSON. For example:
> * after typing the ' for a key you can autocomplete the next field,
using the TAB key similar to vi
* after typing the : the TUI tells you the field type
* after typing the ' for an enum, the TAB brings up a menu to pick an enum
* after typing the last character in a key or value you automatically
get a suggestion for what to type next (comma and next apostrophe after
a value, colon and possible apostrophe/bracket/brace for a key)
One idea that has worked for me in the past was to write a mockup that
shows what things are going to look like, with fake user interaction.
You would have something like
// {
keypress("{")
show_suggestion("'")
// '
keypress("'")
start_autocomplete_choices(["execute", "arguments"])
// TAB
await asyncio.sleep(1)
do_autocomplete()
// '
await asyncio.sleep(1)
keypress("'")
// string argument, propose ' for the value automatically
show_suggestion(": '")
// TAB
await asyncio.sleep(1)
do_autocomplete()
start_autocomplete_choices("query-status", "query-kvm") # many more
etc.
Then you plug in an incremental lexer, so that you can e.g. replace
show_autocomplete(": '")
with
lex_state(Lexer.AFTER_KEY); // this would come from the lexer
show_autocomplete("'") // this would come from the schema
And then again plug the incremental visitor to autocomplete on the
schema types.
Another advantage of this approach is that you also have a natural API
for the mocks, and thus it becomes easier to write testcases.
Paolo
Yes, I follow.
I quite like the VSCode auto-suggest behavior for writing JSON when you
have a schema, but that's baked into their borg-cube, I assume.
I wonder if there's any libraries or more re-usable pieces out there
that would allow us to write a nice interactive TUI for complex
structures in Python, or if we'd have to design and build it ground-up.
Changes how willing I am to embark upon the quest.
--js
Re: Call for Google Summer of Code 2021 project ideas
On 14/01/21 17:36, John Snow wrote:
The sane way to evade the language design problem is to use the existing
QMP language.
I wouldn't mind implementing this for version 0.1 -- just allow
copy-pasting JSON into the input bar -- it's a feature I wanted anyway.
I think the only way out of language design is to instead design a TUI
for inputting JSON. For example:
* after typing the ' for a key you can autocomplete the next field,
using the TAB key similar to vi
* after typing the : the TUI tells you the field type
* after typing the ' for an enum, the TAB brings up a menu to pick an enum
* after typing the last character in a key or value you automatically
get a suggestion for what to type next (comma and next apostrophe after
a value, colon and possible apostrophe/bracket/brace for a key)
One idea that has worked for me in the past was to write a mockup that
shows what things are going to look like, with fake user interaction.
You would have something like
// {
keypress("{")
show_suggestion("'")
// '
keypress("'")
start_autocomplete_choices(["execute", "arguments"])
// TAB
await asyncio.sleep(1)
do_autocomplete()
// '
await asyncio.sleep(1)
keypress("'")
// string argument, propose ' for the value automatically
show_suggestion(": '")
// TAB
await asyncio.sleep(1)
do_autocomplete()
start_autocomplete_choices("query-status", "query-kvm") # many more
etc.
Then you plug in an incremental lexer, so that you can e.g. replace
show_autocomplete(": '")
with
lex_state(Lexer.AFTER_KEY); // this would come from the lexer
show_autocomplete("'") // this would come from the schema
And then again plug the incremental visitor to autocomplete on the
schema types.
Another advantage of this approach is that you also have a natural API
for the mocks, and thus it becomes easier to write testcases.
Paolo
Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas
Hey Stefan, Thanks for taking care of organizing GSoC, and for allowing rust-vmm to also participate under the QEMU umbrella! I am a bit unsure of how can we propose projects related to rust-vmm. We did a bit of brainstorming in our team, and we came up with 3 project ideas. I'll just paste them below, but please let me know if we were supposed to propose them some other way. === Implement the Virtio Console device in Rust === '''Summary:''' Implement the basic emulation for the Virtio Console device in Rust Implement the basic functionality (excluding the optional features: VIRTIO_CONSOLE_F_SIZE, VIRTIO_CONSOLE_F_MULTIPORT, or VIRTIO_CONSOLE_F_EMERG_WRITE) of the Virtio Console Device, using the Virtio building blocks (queue implementations, VirtioDevice traits) defined in rust-vmm/vm-virtio. The virtio console device uses one virtio queue for transmitting data, and one virtio queue for receiving data. The implementation can be extended to also support a subset of the previously mentioned optional features. '''Links:''' * About rust-vmm: https://github.com/rust-vmm/community * rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio * virtio-console spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2550003 '''Details:''' * Skill level: intermediate * Language: Rust * Mentor: i...@amazon.com * Suggested by: fand...@amazon.com<mailto:fand...@amazon.com> === Mocking framework for Virtio Queues === '''Summary:''' Implement a mocking framework for virtio queues Paravirtualized devices (such as those defined by the Virtio standard) are used to provide high performance device emulation. Virtio drivers from a guest VM communicate with the device model using an efficient mechanism based on queues stored in a shared memory area that operate based on a protocol and message format defined by the standard. Various implementations of devices and other virtualization building blocks require mocking the contents that a driver would place into a Virtio queue for validation, testing, and evaluation purposes. This project aims to lay the foundations of a reusable framework for mocking the driver side of Virtio queue operation, that can be consumed by rust-vmm crates and other projects. At the basic level, this means providing a flexible and easy to use interface for users to set up the underlying memory areas and populate contents (as the driver would do) for the basic split queue format in a generic manner. This can further be extended for the packed format and with device-specific mocking capabilities. '''Links:''' * About rust-vmm: https://github.com/rust-vmm/community * Virtio queue spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-230005 Issue in rust-vmm about reusing the mocking logic: rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio '''Details:''' * Skill level: intermediate * Language: Rust * Mentor: aa...@amazon.com * Suggested by: aa...@amazon.com === Local running rust-vmm-ci === '''Summary:''' Run the rust-vmm-ci locally The rust-vmm-ci provides automation for uniformely running the tests on all rust-vmm repositories. It is built on top of Buildkite, and only allows running the tests in the Buildkite context. To run the same tests as in the CI locally, users need to manually copy the Buildkite pipeline steps. The scope of this project is to make it possible for the same tests to easily run locally. This project makes it easier to contribute to all rust-vmm repositories. In order for that to be possible, the following steps are required: - the Buildlkite pipeline is autogenerated from code instead of being a static list of tests to run. This also allows us to uniformely use the same container version for running all the tests (instead of manually modifying each step in the pipeline) - the code for autogenerating the Buildkite pipeline is reused for generating a Python script which can be run locally '''Links:''' * rust-vmm-ci: https://github.com/rust-vmm/rust-vmm-ci * Buildkite pipeline that currently runs the tests: https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml * About rust-vmm: https://github.com/rust-vmm/community * Buildkite documentation: https://buildkite.com/docs/tutorials/getting-started '''Details:''' * Skill level: intermediate * Language: Python * Mentor: fand...@amazon.com * Suggested by: fand...@amazon.com ?Thanks again! Andreea From: Stefan Hajnoczi Sent: Monday, January 11, 2021 1:47 PM To: qemu-devel; kvm; rust-...@lists.opendev.org; Alex Bennée; Alexander Graf; Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; Philippe Mathieu-Daudé Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project ideas C
Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas
ipeline that currently runs the tests: > https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml > * About rust-vmm: https://github.com/rust-vmm/community > * Buildkite documentation: > https://buildkite.com/docs/tutorials/getting-started > > '''Details:''' > * Skill level: intermediate > * Language: Python > * Mentor: fand...@amazon.com > * Suggested by: fand...@amazon.com > > > ?Thanks again! > > Andreea > > > From: Stefan Hajnoczi > Sent: Monday, January 11, 2021 1:47 PM > To: qemu-devel; kvm; rust-...@lists.opendev.org; Alex Bennée; Alexander Graf; > Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; > Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; > Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; > Philippe Mathieu-Daudé > Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project > ideas > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > Dear QEMU, KVM, and rust-vmm community, > QEMU will apply for Google Summer of Code > (https://summerofcode.withgoogle.com/) again this year. This internship > program offers paid, 10-week, remote work internships for > contributing to open source. QEMU can act as an umbrella organization > for KVM kernel and rust-vmm projects too. > > Please post project ideas on the QEMU wiki before February 14th: > https://wiki.qemu.org/Google_Summer_of_Code_2021 > > What's new this year: > * The number of internship hours has been halved to 175 hours over >10 weeks. Project ideas must be smaller to fit and students will have >more flexibility with their working hours. > * Eligibility has been expanded to include "licensed coding school or >similar type of program". > > Good project ideas are suitable for 175 hours (10 weeks half-day) work by a > competent programmer who is not yet familiar with the codebase. In > addition, they are: > * Well-defined - the scope is clear > * Self-contained - there are few dependencies > * Uncontroversial - they are acceptable to the community > * Incremental - they produce deliverables along the way > > Feel free to post ideas even if you are unable to mentor the project. > It doesn't hurt to share the idea! > > I will review project ideas and keep you up-to-date on QEMU's > acceptance into GSoC. > > For more background on QEMU internships, check out this video: > https://www.youtube.com/watch?v=xNVCX7YMUL8 > > Stefan > > ___ > Rust-vmm mailing list > rust-...@lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm > > > > Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar > Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in > Romania. Registration number J22/2621/2005. signature.asc Description: PGP signature
Re: Call for Google Summer of Code 2021 project ideas
On Thu, Jan 14, 2021 at 11:36:23AM -0500, John Snow wrote: > On 1/14/21 7:29 AM, Markus Armbruster wrote: [...] > So I see two possible options for "not inventing a language": > > 1. Raw QMP > 2. The existing qmp-shell syntax, warts and all. > > I don't see a tremendous problem with doing both; but we can start with raw > QMP. The existing qmp-shell syntax is at least definitely very easy to write > a new parser for, even if it's kind of ugly and insufficient. I still see > value in designing a new TUI with the old syntax. > > > The project then aims to build a tool that adds useful features over > > "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" > > UNIX-CONNECT:/path/to/socket". > > > > If it succeeds, you can still design and implement a "better" language, > > and let users choose the one they prefer. Or you could add features to > > help with typing QMP. > > > > > I don't > > > think it's a blocker to have someone work on the TUI and asynchronous > > > dispatch elements. I think even just keeping our current parsing but > > > adding some of the features outlined in the proposal would be a big > > > usability win. > > > > I don't feel this particular itch, but I'm certainly not objecting to > > anyone scratching. > > > > It's something I'd like to see so that I can walk non-QEMU devs through > interacting with QEMU at a low level for the purposes of debugging, > reproducing problems, prototyping features, etc. > > I use qmp-shell all the time for debugging things myself, I find it easier > to use than copy-pasting things directly into socat. I wouldn't mind the > shell getting a little smarter to help me out -- the ability to see async > events and reconnect on disconnect would already be a massive improvement to > *my* quality of life. As an infrequent user of `qmp-shell`, the async events stuff is really beneficial to me too. And, I'm happy to play the test guinea pig to give the patchs a spin. (I'm somewhat behind on the goings-on in this area, very slowly catching up.) > So much so that I spent a lot of time in December to write an async qmp > library O:-) Nice, I recall that you planned to use the 'asyncio' primitives from Python 3.6. -- /kashyap
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 04:55:30PM +, Daniel P. Berrangé wrote: > On Thu, Jan 14, 2021 at 04:49:17PM +, Stefan Hajnoczi wrote: > > On Thu, Jan 14, 2021 at 03:22:41PM +, Daniel P. Berrangé wrote: > > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow > > > > > > > > wrote: > > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, > > > > > > > > e.g. even > > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how > > > > > > > mitmproxy > > > > > > > works. > > > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > > > understand QMP traffic, but that's not something I know how to do > > > > > > > or would > > > > > > > be prepared to mentor.) > > > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are > > > > > > > using > > > > > > > production environment QEMU, how do we attach the QMP listener to > > > > > > > it? Or > > > > > > > does this idea require that we start QEMU in a specific fashion > > > > > > > with a > > > > > > > second debug socket that qmp-shell can connect to in order to > > > > > > > listen? > > > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that > > > > > > > libvirt connects > > > > > > > to ...? > > > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > > > Goals for this feature: > > > > > > > > > > > > 1. No manual steps required for setup. > > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > > >restarting QEMU. > > > > > > 3. Available to unprivileged users. > > > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU > > > > > > monitor > > > > > > command. Approaches that come to mind: > > > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > > >runtime. The debug-chardev receives both monitor input > > > > > > (commands) and > > > > > >output (responses and events). This does not allow MITM, rather > > > > > > it > > > > > >mirrors traffic. > > > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev > > > > > > and > > > > > >then use the existing chardev-change command to replace the > > > > > > monitor > > > > > >chardev with a chardev connected to qmp-shell. This inserts > > > > > > qmp-shell > > > > > >as a proxy between the QMP client and server. qmp-shell can > > > > > > remove > > > > > >itself again with another chardev-change command. This approach > > > > > >allows MITM. The downside is it assumes the QMP chardev is a file > > > > > >descriptor, so it won't work with all types of chardev. > > > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an > > > > > > origin > > > > > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > > >source chardev. The data flow is origin <-> monitoring sink <-> > > > > > >monitoring source <-> QMP monitor. qmp-shell creates the > > > > > > monitoring > > > > > >sink (for receiving incoming QMP commands) and monitoring source > > > > > >chardev (for forwarding QMP commands or MITM commands), and then > > > > > > it > > > > > >uses change-chardev to instantiate a chardev-proxy that directs > > > > > > the > > > > > >original libvirt chardev through the monitoring sink and source. > > > > > > > > > > > >This is the most complex but also completely contained within the > > > > > >QEMU chardev layer. > > > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command > > > > > > or an > > > > > > equivalent API to start/stop monitoring a running VM without manual > > > > > > setup steps. > > > > > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > > > the othe API for receiving QMP events. > > > > > > > > > > Essentially it just needs to be split into two layers. The upper > > > > > layer works in terms of individual QMP
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On 1/14/21 8:52 AM, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote:
2. Ability to watch QMP activity on a running QEMU process, e.g. even
when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
works.
(Actually, mitmproxy could theoretically be taught how to read and
understand QMP traffic, but that's not something I know how to do or would
be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using
production environment QEMU, how do we attach the QMP listener to it? Or
does this idea require that we start QEMU in a specific fashion with a
second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects
to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup.
2. Ability to start/stop monitoring traffic at runtime without
restarting QEMU.
3. Available to unprivileged users.
Excellent goals, and I agree completely.
I think the easiest way to achieve this is through a new QEMU monitor
command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at
runtime. The debug-chardev receives both monitor input (commands) and
output (responses and events). This does not allow MITM, rather it
mirrors traffic.
So you have a socket that relays I/O. I wonder if it needs to modify the
stream format to some extent to annotate directionality?
For now, directionality can be inferred, but maybe that's brittle.
(greeting messages, events and return statements are from the server;
negotiation and execute statements are from the client.)
Maybe if we used a hypothetical qmp-shell log format, we could add
timestamps here instead of relying on the client to produce them. This
might be interesting for analyzing race conditions and measuring
response delays as experienced by the server.
{"message": original_json_message_here, "direction": "in", "timestamp":
1610627721}
(Downside: JSON is still not a streaming message format, but I guess
it's one we already use all over the place anyway.)
2. Add a chardev-get-fd command that fetches the fd from a chardev and
then use the existing chardev-change command to replace the monitor
chardev with a chardev connected to qmp-shell. This inserts qmp-shell
as a proxy between the QMP client and server. qmp-shell can remove
itself again with another chardev-change command. This approach
allows MITM. The downside is it assumes the QMP chardev is a file
descriptor, so it won't work with all types of chardev.
It seems a little more prone to failure if the insertion/removal fails,
and has some downsides about which configurations it can inject into.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
source chardev, 2. a monitoring sink chardev, and 3. a monitoring
source chardev. The data flow is origin <-> monitoring sink <->
monitoring source <-> QMP monitor. qmp-shell creates the monitoring
sink (for receiving incoming QMP commands) and monitoring source
chardev (for forwarding QMP commands or MITM commands), and then it
uses change-chardev to instantiate a chardev-proxy that directs the
original libvirt chardev through the monitoring sink and source.
I'm not sure I understand the topology here, exactly. I could stand to
be a little more familiar with how chardevs are modeled in QEMU ...
This is the most complex but also completely contained within the
QEMU chardev layer.
In all these approaches qmp-shell uses virsh qemu-monitor-command or an
equivalent API to start/stop monitoring a running VM without manual
setup steps.
Gotcha. I think I am leaning towards the first suggestion, but maybe the
third one that I don't quite grasp yet is good too.
Stefan
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 05:14:28PM +, Stefan Hajnoczi wrote: > On Thu, Jan 14, 2021 at 04:55:30PM +, Daniel P. Berrangé wrote: > > On Thu, Jan 14, 2021 at 04:49:17PM +, Stefan Hajnoczi wrote: > > > On Thu, Jan 14, 2021 at 03:22:41PM +, Daniel P. Berrangé wrote: > > > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow > > > > > > > > > wrote: > > > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, > > > > > > > > > e.g. even > > > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how > > > > > > > > mitmproxy > > > > > > > > works. > > > > > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read > > > > > > > > and > > > > > > > > understand QMP traffic, but that's not something I know how to > > > > > > > > do or would > > > > > > > > be prepared to mentor.) > > > > > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are > > > > > > > > using > > > > > > > > production environment QEMU, how do we attach the QMP listener > > > > > > > > to it? Or > > > > > > > > does this idea require that we start QEMU in a specific fashion > > > > > > > > with a > > > > > > > > second debug socket that qmp-shell can connect to in order to > > > > > > > > listen? > > > > > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that > > > > > > > > libvirt connects > > > > > > > > to ...? > > > > > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > > > > > Goals for this feature: > > > > > > > > > > > > > > 1. No manual steps required for setup. > > > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > > > >restarting QEMU. > > > > > > > 3. Available to unprivileged users. > > > > > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU > > > > > > > monitor > > > > > > > command. Approaches that come to mind: > > > > > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it > > > > > > > at > > > > > > >runtime. The debug-chardev receives both monitor input > > > > > > > (commands) and > > > > > > >output (responses and events). This does not allow MITM, > > > > > > > rather it > > > > > > >mirrors traffic. > > > > > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a > > > > > > > chardev and > > > > > > >then use the existing chardev-change command to replace the > > > > > > > monitor > > > > > > >chardev with a chardev connected to qmp-shell. This inserts > > > > > > > qmp-shell > > > > > > >as a proxy between the QMP client and server. qmp-shell can > > > > > > > remove > > > > > > >itself again with another chardev-change command. This approach > > > > > > >allows MITM. The downside is it assumes the QMP chardev is a > > > > > > > file > > > > > > >descriptor, so it won't work with all types of chardev. > > > > > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an > > > > > > > origin > > > > > > >source chardev, 2. a monitoring sink chardev, and 3. a > > > > > > > monitoring > > > > > > >source chardev. The data flow is origin <-> monitoring sink <-> > > > > > > >monitoring source <-> QMP monitor. qmp-shell creates the > > > > > > > monitoring > > > > > > >sink (for receiving incoming QMP commands) and monitoring > > > > > > > source > > > > > > >chardev (for forwarding QMP commands or MITM commands), and > > > > > > > then it > > > > > > >uses change-chardev to instantiate a chardev-proxy that > > > > > > > directs the > > > > > > >original libvirt chardev through the monitoring sink and > > > > > > > source. > > > > > > > > > > > > > >This is the most complex but also completely contained within > > > > > > > the > > > > > > >QEMU chardev layer. > > > > > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command > > > > > > > or an > > > > > > > equivalent API to start/stop monitoring a running VM without > > > > > > > manual > > > > > > > setup steps. > > > > > > > > > > > > Why go to the trouble of adding
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 04:49:17PM +, Stefan Hajnoczi wrote: > On Thu, Jan 14, 2021 at 03:22:41PM +, Daniel P. Berrangé wrote: > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow > > > > > > > wrote: > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. > > > > > > > even > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how > > > > > > mitmproxy > > > > > > works. > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > > understand QMP traffic, but that's not something I know how to do > > > > > > or would > > > > > > be prepared to mentor.) > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are > > > > > > using > > > > > > production environment QEMU, how do we attach the QMP listener to > > > > > > it? Or > > > > > > does this idea require that we start QEMU in a specific fashion > > > > > > with a > > > > > > second debug socket that qmp-shell can connect to in order to > > > > > > listen? > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt > > > > > > connects > > > > > > to ...? > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > Goals for this feature: > > > > > > > > > > 1. No manual steps required for setup. > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > >restarting QEMU. > > > > > 3. Available to unprivileged users. > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > > command. Approaches that come to mind: > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > >runtime. The debug-chardev receives both monitor input (commands) > > > > > and > > > > >output (responses and events). This does not allow MITM, rather it > > > > >mirrors traffic. > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > > >then use the existing chardev-change command to replace the monitor > > > > >chardev with a chardev connected to qmp-shell. This inserts > > > > > qmp-shell > > > > >as a proxy between the QMP client and server. qmp-shell can remove > > > > >itself again with another chardev-change command. This approach > > > > >allows MITM. The downside is it assumes the QMP chardev is a file > > > > >descriptor, so it won't work with all types of chardev. > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an > > > > > origin > > > > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > >source chardev. The data flow is origin <-> monitoring sink <-> > > > > >monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > > >sink (for receiving incoming QMP commands) and monitoring source > > > > >chardev (for forwarding QMP commands or MITM commands), and then it > > > > >uses change-chardev to instantiate a chardev-proxy that directs the > > > > >original libvirt chardev through the monitoring sink and source. > > > > > > > > > >This is the most complex but also completely contained within the > > > > >QEMU chardev layer. > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or > > > > > an > > > > > equivalent API to start/stop monitoring a running VM without manual > > > > > setup steps. > > > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > > the othe API for receiving QMP events. > > > > > > > > Essentially it just needs to be split into two layers. The upper > > > > layer works in terms of individual QMP command/replies, and QMP > > > > events. The lower layer provides a transport that is either a > > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > > virDomainQemuMonitorCommand, and funnells back
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 03:22:41PM +, Daniel P. Berrangé wrote: > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. > > > > > > even > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how > > > > > mitmproxy > > > > > works. > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > understand QMP traffic, but that's not something I know how to do or > > > > > would > > > > > be prepared to mentor.) > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > > production environment QEMU, how do we attach the QMP listener to it? > > > > > Or > > > > > does this idea require that we start QEMU in a specific fashion with a > > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt > > > > > connects > > > > > to ...? > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > Goals for this feature: > > > > > > > > 1. No manual steps required for setup. > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > >restarting QEMU. > > > > 3. Available to unprivileged users. > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > command. Approaches that come to mind: > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > >runtime. The debug-chardev receives both monitor input (commands) and > > > >output (responses and events). This does not allow MITM, rather it > > > >mirrors traffic. > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > >then use the existing chardev-change command to replace the monitor > > > >chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > >as a proxy between the QMP client and server. qmp-shell can remove > > > >itself again with another chardev-change command. This approach > > > >allows MITM. The downside is it assumes the QMP chardev is a file > > > >descriptor, so it won't work with all types of chardev. > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > >source chardev. The data flow is origin <-> monitoring sink <-> > > > >monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > >sink (for receiving incoming QMP commands) and monitoring source > > > >chardev (for forwarding QMP commands or MITM commands), and then it > > > >uses change-chardev to instantiate a chardev-proxy that directs the > > > >original libvirt chardev through the monitoring sink and source. > > > > > > > >This is the most complex but also completely contained within the > > > >QEMU chardev layer. > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > Finally we get backing file chains for chardevs! :-) > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > > equivalent API to start/stop monitoring a running VM without manual > > > > setup steps. > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > the othe API for receiving QMP events. > > > > > > Essentially it just needs to be split into two layers. The upper > > > layer works in terms of individual QMP command/replies, and QMP > > > events. The lower layer provides a transport that is either a > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > virDomainQemuMonitorCommand, and funnells back the response. > > > > I think the idea was to show the QMP traffic that libvirt produces for > > other management applications, not for the QMP shell. These APIs > > probably don't allow this? > > FWIW if you want to monitor what libvirt is sending/receiving we have > a script for that that uses our systemtap probe points: > > >
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > works. > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > understand QMP traffic, but that's not something I know how to do or > > > > would > > > > be prepared to mentor.) > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > does this idea require that we start QEMU in a specific fashion with a > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt > > > > connects > > > > to ...? > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > Goals for this feature: > > > > > > 1. No manual steps required for setup. > > > 2. Ability to start/stop monitoring traffic at runtime without > > >restarting QEMU. > > > 3. Available to unprivileged users. > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > command. Approaches that come to mind: > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > >runtime. The debug-chardev receives both monitor input (commands) and > > >output (responses and events). This does not allow MITM, rather it > > >mirrors traffic. > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > >then use the existing chardev-change command to replace the monitor > > >chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > >as a proxy between the QMP client and server. qmp-shell can remove > > >itself again with another chardev-change command. This approach > > >allows MITM. The downside is it assumes the QMP chardev is a file > > >descriptor, so it won't work with all types of chardev. > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > >source chardev. The data flow is origin <-> monitoring sink <-> > > >monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > >sink (for receiving incoming QMP commands) and monitoring source > > >chardev (for forwarding QMP commands or MITM commands), and then it > > >uses change-chardev to instantiate a chardev-proxy that directs the > > >original libvirt chardev through the monitoring sink and source. > > > > > >This is the most complex but also completely contained within the > > >QEMU chardev layer. > > I have an idea for the QMP command name: chardev-snapshot-sync! > > Finally we get backing file chains for chardevs! :-) Help mom, I'm being made fun of on qemu-devel! :) > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > equivalent API to start/stop monitoring a running VM without manual > > > setup steps. > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > libvirt has. qmp-shell can just directly use the libvirt Python API > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > the othe API for receiving QMP events. > > > > Essentially it just needs to be split into two layers. The upper > > layer works in terms of individual QMP command/replies, and QMP > > events. The lower layer provides a transport that is either a > > UNIX socket, or is the libvirt QMP passthrough API. > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > a UNIX socket, accepts QMP commands and turns them into calls to > > virDomainQemuMonitorCommand, and funnells back the response. > > I think the idea was to show the QMP traffic that libvirt produces for > other management applications, not for the QMP shell. These APIs > probably don't allow this? Yes. Stefan signature.asc Description: PGP signature
Re: Call for Google Summer of Code 2021 project ideas
On 1/14/21 7:29 AM, Markus Armbruster wrote: John Snow writes: On 1/13/21 4:19 AM, Markus Armbruster wrote: John Snow writes: On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I have one that is probably way too ambitious, but requires a particular skillset that might be of good interest to a student that has some experience in the area already. The idea is for a TUI qmp-shell (maybe using urwid?) to create an irssi-like REPL interface for QMP. The idea would be to mimic the mitmproxy TUI interface (Check it out if you haven't!) All the ideas below are extremely tentative to give a basic gist of what I mean; exact layouts/hotkeys/etc are for the sake of explanation only. Essentially, I want an interface like this: --- | QMP Mode| |=| | | | Welcome to the Qemu Machine Protocol shell. | | Please type /help or Ctrl+H to see available commands. | | | | | | | |-| | > | --- commands are entered in the bottom and appear in a log window above, appearing most-recent last, like irssi works. As an example, let's say we issue block-dirty-bitmap-add: | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | (...syntax up for debate...! We want something easy to parse, but easy to type. For structured data, that's Hard.) "Hard" is a red flag for such a brief internship. We need to solve or somehow exclude this design problem before we hand the task to the student. [...] Yes, there is a reason I capitalized "Hard". :~) This task CAN be split out, though. For starters, we can use an extremely simplified parsing strategy while we work on the REPL and and UI elements. We can accept things of the form: command arg=value arg2=value and we can worry about how to input structured data later. Bzzzt! You just designed a language. The fact that it is obviously inadequate for the task does not make this not language design. Did *I* design a language, or did qmp-shell authors design a language? O:-) (Oh, I'm one of the authors. Well then.) Two outcomes: 1. The project fails for other reasons. The language doesn't matter. 2. The project succeeds, i.e. it becomes useful to people. Very quickly these people will demand you evolve your inadequate language into a less inadequate one, compatibly. Your evasion of the language design problem (because hard) has now added accidental constraints, making the design problem even harder. There's no reason that an interactive shell has to stay backwards compatible. Version 0.1 and 0.2 might offer different input paradigms. It's not nice, but it's fine for beta-level stuff. Still, I appreciate that "Beta! Might change!" is an oft-ignored warning. This isn't a library or API though, so I still think it's okay to take some liberty here. The sane way to evade the language design problem is to use the existing QMP language. I wouldn't mind implementing this for version 0.1 -- just allow copy-pasting JSON into the input bar -- it's a feature I wanted anyway. Some things I would like to see, but suspect are not reasonably possible to implement quickly: 1. A
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > works. > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > understand QMP traffic, but that's not something I know how to do or > > > > would > > > > be prepared to mentor.) > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > does this idea require that we start QEMU in a specific fashion with a > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt > > > > connects > > > > to ...? > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > Goals for this feature: > > > > > > 1. No manual steps required for setup. > > > 2. Ability to start/stop monitoring traffic at runtime without > > >restarting QEMU. > > > 3. Available to unprivileged users. > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > command. Approaches that come to mind: > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > >runtime. The debug-chardev receives both monitor input (commands) and > > >output (responses and events). This does not allow MITM, rather it > > >mirrors traffic. > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > >then use the existing chardev-change command to replace the monitor > > >chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > >as a proxy between the QMP client and server. qmp-shell can remove > > >itself again with another chardev-change command. This approach > > >allows MITM. The downside is it assumes the QMP chardev is a file > > >descriptor, so it won't work with all types of chardev. > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > >source chardev. The data flow is origin <-> monitoring sink <-> > > >monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > >sink (for receiving incoming QMP commands) and monitoring source > > >chardev (for forwarding QMP commands or MITM commands), and then it > > >uses change-chardev to instantiate a chardev-proxy that directs the > > >original libvirt chardev through the monitoring sink and source. > > > > > >This is the most complex but also completely contained within the > > >QEMU chardev layer. > > I have an idea for the QMP command name: chardev-snapshot-sync! > > Finally we get backing file chains for chardevs! :-) > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > equivalent API to start/stop monitoring a running VM without manual > > > setup steps. > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > libvirt has. qmp-shell can just directly use the libvirt Python API > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > the othe API for receiving QMP events. > > > > Essentially it just needs to be split into two layers. The upper > > layer works in terms of individual QMP command/replies, and QMP > > events. The lower layer provides a transport that is either a > > UNIX socket, or is the libvirt QMP passthrough API. > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > a UNIX socket, accepts QMP commands and turns them into calls to > > virDomainQemuMonitorCommand, and funnells back the response. > > I think the idea was to show the QMP traffic that libvirt produces for > other management applications, not for the QMP shell. These APIs > probably don't allow this? FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points: https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > works. > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > understand QMP traffic, but that's not something I know how to do or would > > > be prepared to mentor.) > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > does this idea require that we start QEMU in a specific fashion with a > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt > > > connects > > > to ...? > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control > > > > Goals for this feature: > > > > 1. No manual steps required for setup. > > 2. Ability to start/stop monitoring traffic at runtime without > >restarting QEMU. > > 3. Available to unprivileged users. > > > > I think the easiest way to achieve this is through a new QEMU monitor > > command. Approaches that come to mind: > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > >runtime. The debug-chardev receives both monitor input (commands) and > >output (responses and events). This does not allow MITM, rather it > >mirrors traffic. > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > >then use the existing chardev-change command to replace the monitor > >chardev with a chardev connected to qmp-shell. This inserts qmp-shell > >as a proxy between the QMP client and server. qmp-shell can remove > >itself again with another chardev-change command. This approach > >allows MITM. The downside is it assumes the QMP chardev is a file > >descriptor, so it won't work with all types of chardev. > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > >source chardev, 2. a monitoring sink chardev, and 3. a monitoring > >source chardev. The data flow is origin <-> monitoring sink <-> > >monitoring source <-> QMP monitor. qmp-shell creates the monitoring > >sink (for receiving incoming QMP commands) and monitoring source > >chardev (for forwarding QMP commands or MITM commands), and then it > >uses change-chardev to instantiate a chardev-proxy that directs the > >original libvirt chardev through the monitoring sink and source. > > > >This is the most complex but also completely contained within the > >QEMU chardev layer. I have an idea for the QMP command name: chardev-snapshot-sync! Finally we get backing file chains for chardevs! :-) > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > equivalent API to start/stop monitoring a running VM without manual > > setup steps. > > Why go to the trouble of adding more chardevs to a running QEMU that > libvirt has. qmp-shell can just directly use the libvirt Python API > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > the othe API for receiving QMP events. > > Essentially it just needs to be split into two layers. The upper > layer works in terms of individual QMP command/replies, and QMP > events. The lower layer provides a transport that is either a > UNIX socket, or is the libvirt QMP passthrough API. > > Or alternatively, provide a virt-qmp-shim command that listens on > a UNIX socket, accepts QMP commands and turns them into calls to > virDomainQemuMonitorCommand, and funnells back the response. I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this? Kevin
Re: Call for Google Summer of Code 2021 project ideas
Am 14.01.2021 um 13:29 hat Markus Armbruster geschrieben: > John Snow writes: > > > On 1/13/21 4:19 AM, Markus Armbruster wrote: > >> John Snow writes: > >> > >>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: > Dear QEMU, KVM, and rust-vmm community, > QEMU will apply for Google Summer of Code > (https://summerofcode.withgoogle.com/) again this year. This internship > program offers paid, 10-week, remote work internships for > contributing to open source. QEMU can act as an umbrella organization > for KVM kernel and rust-vmm projects too. > Please post project ideas on the QEMU wiki before February 14th: > https://wiki.qemu.org/Google_Summer_of_Code_2021 > What's new this year: > * The number of internship hours has been halved to 175 hours over > 10 weeks. Project ideas must be smaller to fit and students will > have > more flexibility with their working hours. > * Eligibility has been expanded to include "licensed coding school or > similar type of program". > Good project ideas are suitable for 175 hours (10 weeks half-day) > work by a > competent programmer who is not yet familiar with the codebase. In > addition, they are: > * Well-defined - the scope is clear > * Self-contained - there are few dependencies > * Uncontroversial - they are acceptable to the community > * Incremental - they produce deliverables along the way > Feel free to post ideas even if you are unable to mentor the > project. > It doesn't hurt to share the idea! > > >>> > >>> I have one that is probably way too ambitious, but requires a > >>> particular skillset that might be of good interest to a student that > >>> has some experience in the area already. > >>> > >>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an > >>> irssi-like REPL interface for QMP. The idea would be to mimic the > >>> mitmproxy TUI interface (Check it out if you haven't!) > >>> > >>> All the ideas below are extremely tentative to give a basic gist of > >>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation > >>> only. > >>> > >>> Essentially, I want an interface like this: > >>> > >>> --- > >>> | QMP Mode| > >>> |=| > >>> | | > >>> | Welcome to the Qemu Machine Protocol shell. | > >>> | Please type /help or Ctrl+H to see available commands. | > >>> | | > >>> | | > >>> | | > >>> |-| > >>> | > | > >>> --- > >>> > >>> commands are entered in the bottom and appear in a log window above, > >>> appearing most-recent last, like irssi works. > >>> > >>> As an example, let's say we issue block-dirty-bitmap-add: > >>> > >>> > >>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | > >>> > >>> > >>> (...syntax up for debate...! We want something easy to parse, but easy > >>> to type. For structured data, that's Hard.) > >> "Hard" is a red flag for such a brief internship. We need to solve > >> or > >> somehow exclude this design problem before we hand the task to the > >> student. > >> [...] > >> > > > > Yes, there is a reason I capitalized "Hard". :~) > > > > This task CAN be split out, though. For starters, we can use an > > extremely simplified parsing strategy while we work on the REPL and > > and UI elements. > > > > We can accept things of the form: > > command arg=value arg2=value > > > > and we can worry about how to input structured data later. > > Bzzzt! You just designed a language. The fact that it is obviously > inadequate for the task does not make this not language design. > > Two outcomes: > > 1. The project fails for other reasons. The language doesn't matter. > > 2. The project succeeds, i.e. it becomes useful to people. Very quickly > these people will demand you evolve your inadequate language into a less > inadequate one, compatibly. Your evasion of the language design problem > (because hard) has now added accidental constraints, making the design > problem even harder. We don't even promise compatibility for HMP, which is a part of the system emulator binary and has existed since I first used QEMU, which must have been around 0.6 times. If it's possible to simply say "no" to compatibility demands there, then surely an external shell with a similar target
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Thu, Jan 14, 2021 at 01:52:34PM +, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > when libvirt is directly connected to the monitor. > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > works. > > > > (Actually, mitmproxy could theoretically be taught how to read and > > understand QMP traffic, but that's not something I know how to do or would > > be prepared to mentor.) > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > production environment QEMU, how do we attach the QMP listener to it? Or > > does this idea require that we start QEMU in a specific fashion with a > > second debug socket that qmp-shell can connect to in order to listen? > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > to ...? > > Here is the QEMU command-line that libvirt uses on my F33 system: > > -chardev socket,id=charmonitor,fd=36,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control > > Goals for this feature: > > 1. No manual steps required for setup. > 2. Ability to start/stop monitoring traffic at runtime without >restarting QEMU. > 3. Available to unprivileged users. > > I think the easiest way to achieve this is through a new QEMU monitor > command. Approaches that come to mind: > > 1. Add a -mon debug-chardev property and a QMP command to set it at >runtime. The debug-chardev receives both monitor input (commands) and >output (responses and events). This does not allow MITM, rather it >mirrors traffic. > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and >then use the existing chardev-change command to replace the monitor >chardev with a chardev connected to qmp-shell. This inserts qmp-shell >as a proxy between the QMP client and server. qmp-shell can remove >itself again with another chardev-change command. This approach >allows MITM. The downside is it assumes the QMP chardev is a file >descriptor, so it won't work with all types of chardev. > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin >source chardev, 2. a monitoring sink chardev, and 3. a monitoring >source chardev. The data flow is origin <-> monitoring sink <-> >monitoring source <-> QMP monitor. qmp-shell creates the monitoring >sink (for receiving incoming QMP commands) and monitoring source >chardev (for forwarding QMP commands or MITM commands), and then it >uses change-chardev to instantiate a chardev-proxy that directs the >original libvirt chardev through the monitoring sink and source. > >This is the most complex but also completely contained within the >QEMU chardev layer. > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > equivalent API to start/stop monitoring a running VM without manual > setup steps. Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events. Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API. Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > when libvirt is directly connected to the monitor. > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > works. > > (Actually, mitmproxy could theoretically be taught how to read and > understand QMP traffic, but that's not something I know how to do or would > be prepared to mentor.) > > Is this possible to do in a post-hoc fashion? Let's say you are using > production environment QEMU, how do we attach the QMP listener to it? Or > does this idea require that we start QEMU in a specific fashion with a > second debug socket that qmp-shell can connect to in order to listen? > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > to ...? Here is the QEMU command-line that libvirt uses on my F33 system: -chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control Goals for this feature: 1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users. I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind: 1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic. 2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev. 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source. This is the most complex but also completely contained within the QEMU chardev layer. In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps. Stefan signature.asc Description: PGP signature
Re: Call for Google Summer of Code 2021 project ideas
John Snow writes: > On 1/13/21 4:19 AM, Markus Armbruster wrote: >> John Snow writes: >> >>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! >>> >>> I have one that is probably way too ambitious, but requires a >>> particular skillset that might be of good interest to a student that >>> has some experience in the area already. >>> >>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an >>> irssi-like REPL interface for QMP. The idea would be to mimic the >>> mitmproxy TUI interface (Check it out if you haven't!) >>> >>> All the ideas below are extremely tentative to give a basic gist of >>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation >>> only. >>> >>> Essentially, I want an interface like this: >>> >>> --- >>> | QMP Mode| >>> |=| >>> | | >>> | Welcome to the Qemu Machine Protocol shell. | >>> | Please type /help or Ctrl+H to see available commands. | >>> | | >>> | | >>> | | >>> |-| >>> | > | >>> --- >>> >>> commands are entered in the bottom and appear in a log window above, >>> appearing most-recent last, like irssi works. >>> >>> As an example, let's say we issue block-dirty-bitmap-add: >>> >>> >>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | >>> >>> >>> (...syntax up for debate...! We want something easy to parse, but easy >>> to type. For structured data, that's Hard.) >> "Hard" is a red flag for such a brief internship. We need to solve >> or >> somehow exclude this design problem before we hand the task to the >> student. >> [...] >> > > Yes, there is a reason I capitalized "Hard". :~) > > This task CAN be split out, though. For starters, we can use an > extremely simplified parsing strategy while we work on the REPL and > and UI elements. > > We can accept things of the form: > command arg=value arg2=value > > and we can worry about how to input structured data later. Bzzzt! You just designed a language. The fact that it is obviously inadequate for the task does not make this not language design. Two outcomes: 1. The project fails for other reasons. The language doesn't matter. 2. The project succeeds, i.e. it becomes useful to people. Very quickly these people will demand you evolve your inadequate language into a less inadequate one, compatibly. Your evasion of the language design problem (because hard) has now added accidental constraints, making the design problem even harder. The sane way to evade the language design problem is to use the existing QMP language. The project then aims to build a tool that adds useful features over "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" UNIX-CONNECT:/path/to/socket". If it succeeds, you can still design and implement a "better" language, and let users choose the one they prefer. Or you could add features to help with typing QMP. >I don't > think it's a blocker to have someone work on the TUI and asynchronous >
Re: Call for Google Summer of Code 2021 project ideas
On 1/13/21 4:19 AM, Markus Armbruster wrote: John Snow writes: On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I have one that is probably way too ambitious, but requires a particular skillset that might be of good interest to a student that has some experience in the area already. The idea is for a TUI qmp-shell (maybe using urwid?) to create an irssi-like REPL interface for QMP. The idea would be to mimic the mitmproxy TUI interface (Check it out if you haven't!) All the ideas below are extremely tentative to give a basic gist of what I mean; exact layouts/hotkeys/etc are for the sake of explanation only. Essentially, I want an interface like this: --- | QMP Mode| |=| | | | Welcome to the Qemu Machine Protocol shell. | | Please type /help or Ctrl+H to see available commands. | | | | | | | |-| | > | --- commands are entered in the bottom and appear in a log window above, appearing most-recent last, like irssi works. As an example, let's say we issue block-dirty-bitmap-add: | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | (...syntax up for debate...! We want something easy to parse, but easy to type. For structured data, that's Hard.) "Hard" is a red flag for such a brief internship. We need to solve or somehow exclude this design problem before we hand the task to the student. [...] Yes, there is a reason I capitalized "Hard". :~) This task CAN be split out, though. For starters, we can use an extremely simplified parsing strategy while we work on the REPL and and UI elements. We can accept things of the form: command arg=value arg2=value and we can worry about how to input structured data later. I don't think it's a blocker to have someone work on the TUI and asynchronous dispatch elements. I think even just keeping our current parsing but adding some of the features outlined in the proposal would be a big usability win. --js
qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: I have one that is probably way too ambitious, but requires a particular skillset that might be of good interest to a student that has some experience in the area already. The idea is for a TUI qmp-shell (maybe using urwid?) to create an irssi-like REPL interface for QMP. The idea would be to mimic the mitmproxy TUI interface (Check it out if you haven't!) Great, I think this project idea lends itself to an incremental milestones. How far it gets will depend on the intern and we'll be able to merge useful patches regardless of how far they take it. Yeah. I wrote a lot, but I think a lot of the desires and goals can actually be split out. You can start with just the REPL mode; no bells or whistles. Just type commands, issue them, and have the log pane populate with the raw JSON as a starting point. That'd be useful enough already. From there, the bells and whistles that make it truly shine can be added. Two more ideas: 1. Ability to load libvirt log files for offline viewing. This could be a major use case for this tool because the raw libvirt logs can be hard to read today. Yeah, that would be excellent. (Especially because I have such a hard time understanding libvirt-ese; seeing the QMP log in the same tool I use to communicate directly with QEMU would be an excellent debugging boon.) mitmproxy has a similar feature where packet captures can be saved to file and loaded again later for analysis. A similar thing here would be nice. I mentioned wanting to be able to save sessions for later viewing in my proposal; which likely means developing a QMP log format. It's unlikely we'll want to use the libvirt log format here, so we'd need to develop a reader that can parse libvirt logs; but a writer is not likely important. 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor. That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works. (Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.) Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen? ... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...? Stefan
Re: Call for Google Summer of Code 2021 project ideas
John Snow writes: > On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: >> Dear QEMU, KVM, and rust-vmm community, >> QEMU will apply for Google Summer of Code >> (https://summerofcode.withgoogle.com/) again this year. This internship >> program offers paid, 10-week, remote work internships for >> contributing to open source. QEMU can act as an umbrella organization >> for KVM kernel and rust-vmm projects too. >> Please post project ideas on the QEMU wiki before February 14th: >> https://wiki.qemu.org/Google_Summer_of_Code_2021 >> What's new this year: >> * The number of internship hours has been halved to 175 hours over >> 10 weeks. Project ideas must be smaller to fit and students will have >> more flexibility with their working hours. >> * Eligibility has been expanded to include "licensed coding school or >> similar type of program". >> Good project ideas are suitable for 175 hours (10 weeks half-day) >> work by a >> competent programmer who is not yet familiar with the codebase. In >> addition, they are: >> * Well-defined - the scope is clear >> * Self-contained - there are few dependencies >> * Uncontroversial - they are acceptable to the community >> * Incremental - they produce deliverables along the way >> Feel free to post ideas even if you are unable to mentor the >> project. >> It doesn't hurt to share the idea! >> > > I have one that is probably way too ambitious, but requires a > particular skillset that might be of good interest to a student that > has some experience in the area already. > > The idea is for a TUI qmp-shell (maybe using urwid?) to create an > irssi-like REPL interface for QMP. The idea would be to mimic the > mitmproxy TUI interface (Check it out if you haven't!) > > All the ideas below are extremely tentative to give a basic gist of > what I mean; exact layouts/hotkeys/etc are for the sake of explanation > only. > > Essentially, I want an interface like this: > > --- > | QMP Mode| > |=| > | | > | Welcome to the Qemu Machine Protocol shell. | > | Please type /help or Ctrl+H to see available commands. | > | | > | | > | | > |-| > | > | > --- > > commands are entered in the bottom and appear in a log window above, > appearing most-recent last, like irssi works. > > As an example, let's say we issue block-dirty-bitmap-add: > > > | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | > > > (...syntax up for debate...! We want something easy to parse, but easy > to type. For structured data, that's Hard.) "Hard" is a red flag for such a brief internship. We need to solve or somehow exclude this design problem before we hand the task to the student. [...]
Re: Call for Google Summer of Code 2021 project ideas
On Tue, Jan 12, 2021 at 9:10 PM John Snow wrote: > I have one that is probably way too ambitious, but requires a particular > skillset that might be of good interest to a student that has some > experience in the area already. > > The idea is for a TUI qmp-shell (maybe using urwid?) to create an > irssi-like REPL interface for QMP. The idea would be to mimic the > mitmproxy TUI interface (Check it out if you haven't!) Great, I think this project idea lends itself to an incremental milestones. How far it gets will depend on the intern and we'll be able to merge useful patches regardless of how far they take it. Two more ideas: 1. Ability to load libvirt log files for offline viewing. This could be a major use case for this tool because the raw libvirt logs can be hard to read today. 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor. Stefan
Re: Call for Google Summer of Code 2021 project ideas
On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I have one that is probably way too ambitious, but requires a particular skillset that might be of good interest to a student that has some experience in the area already. The idea is for a TUI qmp-shell (maybe using urwid?) to create an irssi-like REPL interface for QMP. The idea would be to mimic the mitmproxy TUI interface (Check it out if you haven't!) All the ideas below are extremely tentative to give a basic gist of what I mean; exact layouts/hotkeys/etc are for the sake of explanation only. Essentially, I want an interface like this: --- | QMP Mode| |=| | | | Welcome to the Qemu Machine Protocol shell. | | Please type /help or Ctrl+H to see available commands. | | | | | | | |-| | > | --- commands are entered in the bottom and appear in a log window above, appearing most-recent last, like irssi works. As an example, let's say we issue block-dirty-bitmap-add: | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | (...syntax up for debate...! We want something easy to parse, but easy to type. For structured data, that's Hard.) we press [Enter] to submit the command, and the history now shows: - | QMP Mode | |===| | | | | | | | [11:28] block-dirty-bitmap-add [Pending ...] | |---| | > | - After a few moments, the command resolves: | QMP Mode | |==| | | | | | | | [11:28] block-dirty-bitmap-add [OK] | | >| The basic unit of history here is either an RPC call/response pair or an asynchronous QMP event. Clicking on the history pane or pressing Alt+↑ or Alt+↓ (Mimics irssi keys for switching buffers, let's say it's user-configurable) to change focus to the history pane: | Command History Mode | |==| | | | | | | | [11:28] block-dirty-bitmap-add [OK] | ---| | Use ↑ or ↓ to select a command, press [Enter] to inspect | Using the arrow keys, you can highlight the history item and
Call for Google Summer of Code 2021 project ideas
Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I will review project ideas and keep you up-to-date on QEMU's acceptance into GSoC. For more background on QEMU internships, check out this video: https://www.youtube.com/watch?v=xNVCX7YMUL8 Stefan
