On Wed, Oct 08, 2025 at 08:35:39AM +0200, Paolo Bonzini wrote: > [People in Cc are a mix of Python people, tracing people, and people > who followed the recent AI discussions. - Paolo] > > This series adds type annotations to tracetool. While useful on its own, > it also served as an experiment in whether AI tools could be useful and > appropriate for mechanical code transformations that may not involve > copyrightable expression. > > In this version, the types were added mostly with the RightTyper tool > (https://github.com/RightTyper/RightTyper), which uses profiling to detect > the types of arguments and return types at run time. However, because > adding type annotations is such a narrow and verifiable task, I also developed > a parallel version using an LLM, to provide some data on topics such as: > > - how much choice/creativity is there in writing type annotations? > Is it closer to writing functional code or to refactoring? > > - how does AI output for such mechanical transformations compare to other > automated tools, whose output is known not to be copyrightable? > > - what is the kind of time saving that the tool can provide? > > - how effective is an LLM for this task? Is the required human help a > pain to provide, or can the human keep the fun part for itself? > > While the version submitted here does not use any LLM-generated content > in compliance with QEMU's policy, the first version was developed using > an LLM tool (Claude Code) with a very simple prompt: > > Add type annotations to tracetool.py and all .py files in > tracetool/*.py. Use new-style annotations, for example list[str] > instead of typing.List[str]. When done, use mypy to find any > issues. > > The results were surprisingly good for such a simple-minded test, and > while there were some issues they turned out to be mostly due to dead > code that confused the LLM. Thus, I removed the dead code (this is > already in qemu.git) and then did the actual experiment, running > both the RightTyper script (which is detailed in patch 4) and the LLM > in separate branches. The remaining errors from the LLM were also > reasonably easy to fix, so I did that manually: > > tracetool/__init__.py:239: error: Need type annotation for "_args" > (hint: "_args: list[<type>] = ...") [var-annotated] > tracetool/__init__.py:272: error: Argument 1 to "Arguments" has > incompatible type "list[tuple[str, str]]"; expected "list[tuple[str, str] | > Arguments]" [arg-type] > tracetool/__init__.py:579: error: Incompatible types in assignment > (expression has type "str | None", variable has type "None") [assignment] > tracetool/__init__.py:580: error: Incompatible types in assignment > (expression has type "str | None", variable has type "None") [assignment] > > After reviewing the changes, I followed up with another prompt for mechanical > changes: > > Change "backend: Any" to "backend: Wrapper" in all tracetool/format/ > files > > Honestly, I could have done this part in less time without any help; I > was just curious to see if the LLM would also remove the unused import. > Not only it didn't; this prompt didn't even touch most of the files so > I did the change, ran isort and called it a day. > > Comparing the results from RightTyper and AI, both tools benefit from human > help: the dead code removal which I mentioned, the small cleanups in patch 1, > the final manual fixes for the remaining (mostly trivial) errors. But at > least in this case, AI is a winner: > > - it left basically no unannotated code: fixing the above errors was enough > to pass "mypy --strict", unlike RightTyper which needed a little more work > due to its profiling-based nature and a few other limitations (see patch 5). > > - most importantly, trying various other tools that didn't work, as well as > figuring out how to use RightTyper, took a couple hours more. Surprising > as it was, I could not find any static type inferencing tool for Python; > neither pytype nor pyre worked for me. This is also why I think this > is not apples to oranges, but a fair comparison between AI-based and > regular tooling. > > I want to highlight "in this case". I had other experiments where the > LLM's output was all but awful, and I wasted time debugging code that > I should have thrown away immediately! > > After the diffstat, you can find a diff from this series to the version > based on Claude Code. It's impossible to be 100% objective but, > besides being functionally equivalent, I don't think either would be > identifiable as written by an LLM, by a person, by a tool+human combo, > or even by a good type inferencing tool (if it existed). > > Based on this experience, my answer to the copyrightability question is > that, for this kind of narrow request, the output of AI can be treated as > the output of an imperfect tool, rather than as creative content potentially > tainted by the training material. Of course this is one data point and > is intended as an experiment rather than a policy recommendation. > > Paolo > > > > Paolo Bonzini (6): > tracetool: rename variable with conflicting types > tracetool: apply isort and add check > tracetool: "import annotations" > tracetool: add type annotations > tracetool: complete typing annotations > tracetool: add typing checks to "make -C python check" > > python/tests/tracetool-isort.sh | 4 + > python/tests/tracetool-mypy.sh | 5 ++ > scripts/tracetool.py | 12 +-- > scripts/tracetool/__init__.py | 84 ++++++++++---------- > scripts/tracetool/backend/__init__.py | 21 ++--- > scripts/tracetool/backend/dtrace.py | 19 ++--- > scripts/tracetool/backend/ftrace.py | 13 +-- > scripts/tracetool/backend/log.py | 13 +-- > scripts/tracetool/backend/simple.py | 19 ++--- > scripts/tracetool/backend/syslog.py | 13 +-- > scripts/tracetool/backend/ust.py | 11 +-- > scripts/tracetool/format/__init__.py | 9 ++- > scripts/tracetool/format/c.py | 7 +- > scripts/tracetool/format/d.py | 7 +- > scripts/tracetool/format/h.py | 7 +- > scripts/tracetool/format/log_stap.py | 12 +-- > scripts/tracetool/format/rs.py | 7 +- > scripts/tracetool/format/simpletrace_stap.py | 7 +- > scripts/tracetool/format/stap.py | 10 ++- > scripts/tracetool/format/ust_events_c.py | 7 +- > scripts/tracetool/format/ust_events_h.py | 7 +- > 21 files changed, 173 insertions(+), 121 deletions(-) > create mode 100755 python/tests/tracetool-isort.sh > create mode 100755 python/tests/tracetool-mypy.sh > -- > 2.51.0 >
Discussion on AI policy (does not affect these patches) and future cleanups aside, this looks good to go. Thanks, applied to my tracing tree: https://gitlab.com/stefanha/qemu/commits/tracing Stefan
signature.asc
Description: PGP signature
