Re: DebGPT: how LLM can help debian development? demo available.
Hi! I've noticed that the most recent LLMs are actually very good at finding information, summarizing and giving code examples about Debian already without extra training. One just needs to double check that the answer was correct to not fall for when they are simply making up plausible sounding stuff. However, checking if an answer is correct is much faster than figuring it out something from scratch. Anyone can test it for themselves at https://chat.lmsys.org/ - no registration required, just participate in the research by reading the replies from two LLMs and telling the leaderboard which reply you think was better. Knowing there is so much repetitive petty work involved in Debian package maintenance that drains a of energy both from current and aspiring maintainers I have been skimming the README at https://salsa.debian.org/deeplearning-team/debgpt with high hopes for DebGPT to evolve over time. It would be great if there was a LLM with enough logic and context that it could do things like Lintian-brush (or even go all the way like Janitor and start filing MRs on Salsa for humans to review/finalize).
Re: DebGPT: how LLM can help debian development? demo available.
Am Wed, Jan 03, 2024 at 01:06:25PM +0100 schrieb Andrey Rakhmatullin: > On Wed, Jan 03, 2024 at 11:33:06AM +0200, Andrius Merkys wrote: > > On 2024-01-03 11:12, Andrey Rakhmatullin wrote: > > > On Wed, Jan 03, 2024 at 09:58:33AM +0200, Andrius Merkys wrote: > > > > To me the most time consuming task in Debian recently is the Python > > > > transitions. I wonder whether DebGPT could help with them. Maybe there > > > > are > > > > other, non-Debian-specific GPTs for this task, but I would prefer a > > > > Debian > > > > one. > > > As "transitions" is too broad, can you list actual problems you spend time > > > on for them? > > > > Mostly failing tests, and mostly due to API changes between subsequent > > Python 3.x versions. > So the solution is either find a patch in the upstream repo (committed or > proposed in issues/PRs) or write one yourself. Not sure what can AI help > here with. Well, may be people replacing 'assertEquals' by the new name 'assertEqual'[1] could talk with some GPT first how much work all their downstreams might have due to this kind of estetic changes to find patches? In the last year I've seen quite some changes which do not obviously serve any better purpose than fitting some esthetics pattern breaking some API. (The s/assertEquals/assertEqual/ one is not the only example.) Kind regards Andreas. [1] https://salsa.debian.org/python-team/packages/pypandoc/-/blob/debian/master/debian/patches/0005-Python3.12.patch?ref_type=heads -- http://fam-tille.de
Re: DebGPT: how LLM can help debian development? demo available.
debgpt v0.4.90 has been uploaded to NEW, targeting at unstable. This tool is still under development, new features will be added later. Usage examples can be found in debgpt(1) or README.md . They are the same file. During my (limited number) of experiments when developing this tool, LLMs are good at: 1. $ debgpt git commit (many recent git commits in debgpt repo are generated using LLM) This is the one I like most so far. This will likely become a part of my git workflow. 2. summarizing arbitrary texts in pretty format 3. explaining arbitrary code files 4. fortune, e.g. $ debgpt fortune :joke 5. writing boring boiler plate code (such as matplotlib) LLMs are bad at: 1. logic and deduction for example, it does not always tell the relation between "SyntaxError: print 'hello world'" and an upstream PR named "initial python3 support. (Maybe chain of thoguht can improve this but I postponed it) 2. generating unix-format patches Let it give you the edited files directly, instead of the diff. This tool is not specific to Debian distribution. Adding additional text sources like Arch Wiki, Gentoo Policy/Wiki is very easy and planned as future feature. Or you can load these web pages through `debgpt --cmd 'curl '`. Support for dealing very long documents will be further investigated in future releases. On 1/3/24 20:07, Mo Zhou wrote: I have implemented the OpenAI API frontend, with streaming to terminal enabled. On 1/2/24 17:07, M. Zhou wrote: Following what has been discussed in d-project in an offtopic subthread, I prepared some demo on imagined use cases to leverage LLMs to help debian development. https://salsa.debian.org/deeplearning-team/debgpt
Re: DebGPT: how LLM can help debian development? demo available.
I have implemented the OpenAI API frontend, with streaming to terminal enabled. Just export your OPENAI_API_KEY to environment if you have one, and specify `-F openai` in the debgpt command line. It work work without the self-hosted LLM inference backend. That means the command `debgpt none -i -F openai` falls back to a general terminal client to ChatGPT. I believe adding this frontend allows more people to try LLMs with debian-specific tasks conveniently. Since the openai frontend allows people to use this tool without pytorch and a bunch of deep learning frameworks, I plan to upload debgpt to experimental/non-free, following python-openai (unstable/non-free). Packages in debian archive should be enough to run the openai frontend, but not for the self-hosted LLM backend. The following are my replies to previous comments: On 1/2/24 17:49, Jeremy Stanley wrote: but one of the most useful ways I've seen LLM leveraged is large open source projects feeding in their corpus of complex documentation, and then providing users with a human language interaction prompt where they can ask questions and get conversational responses based on that documentation. A continuously trained LLM backing a sort of "search" function for all of www.d.o/doc (and maybe also wiki.d.o) could be cool. Yes. So the `debgpt policy ...` and `debgpt devref` are two examples on letting LLM read long documents and answer questions. The problem is that the full document is too long, while the supported context length is just 4k ~ 16k tokens for openai api, or 8k tokens for self-hosted mistral7b. Through merely prompt engineering, of course we cannot feed the whole policy document to the context. Because solely section 4.9.1 is already overlength against the typical chatgpt model gpt-3.5-turbo. That's why the interface is designed to feed a specific section of long document. That said, with more works, I think it should be possible to feed the LLM the table of contents first, and let it choose a section it wants to refer based on your question. On 1/3/24 02:58, Andrius Merkys wrote: I find this pretty impressive. Thanks a lot for working on it. Thanks. I had fun experimenting with this. To me the most time consuming task in Debian recently is the Python transitions. I wonder whether DebGPT could help with them. Maybe there are other, non-Debian-specific GPTs for this task, but I would prefer a Debian one. This is in fact not Debian specific. Essentially the current status of this project is almost a prompt generator, automatically gathering information about the task you specified in the command line, and send all the information to LLM. It is not different from using the web-based ChatGPT, by copy and pasting the same information before asking the question to LLM. But if the requirement is relatively loose -- it can be seen as Debian specific. Particularly for ChatGPT, there is a debian-specific system prompt in debgpt/frontend.py, which asks ChatGPT to play the role of a Debian developer when responding. On 1/3/24 04:33, Andrius Merkys wrote: Mostly failing tests, and mostly due to API changes between subsequent Python 3.x versions. My idea is to extract the failure from the BTS, and append the "breaking changes" section of the cpython changelog, and see what suggestion the LLM can provide. I do not expect perfect bug fixing suggestions but it should be able to conclude something. The corresponding cli may look like this: $ debgpt bts --id BUG_NUMBER --pychange 3.11-3.12 free -i # to be implemented Or possily pull the list of recent github upstream issues list, and let LLM figure out which upstream bug or pull request is most relevant. $ debgpt bts --id BUG_NUMBER --upstream_issues free -i # to be implemented The upstream issues webpage can be found in the Control/metadata file, and the source package name can be typically found in the bugs page. This automatic process should be able to save people some time. On 1/3/24 07:06, Andrey Rakhmatullin wrote: So the solution is either find a patch in the upstream repo (committed or proposed in issues/PRs) or write one yourself. Not sure what can AI help here with. We can ask LLMs to suggest a fix to the bug. Or let LLM to check whether the recent upstream issues list contains a title which might be relevant to the content of the debian bug report. For instance, the LLM can tell `Syntax error: , print "hello world"` is most relevant to an upstream pull request named, e.g. `[pull-request] initial python3 support`, and directly give you the pull request link. (you may have to retry a couple of times to make it think correctly though). I made up a test sample about this at: examples/e4d7fc9f-7469-4cad-959d-373b89498663-initmsg.txt Complicated inference tasks are what we hope LLMs can do very well eventually. Grasping the semantics of natural language is something traditional software can never do. On 1/3/24 03:09, PICCA
Re: DebGPT: how LLM can help debian development? demo available.
On Wed, Jan 03, 2024 at 11:33:06AM +0200, Andrius Merkys wrote: > On 2024-01-03 11:12, Andrey Rakhmatullin wrote: > > On Wed, Jan 03, 2024 at 09:58:33AM +0200, Andrius Merkys wrote: > > > To me the most time consuming task in Debian recently is the Python > > > transitions. I wonder whether DebGPT could help with them. Maybe there are > > > other, non-Debian-specific GPTs for this task, but I would prefer a Debian > > > one. > > As "transitions" is too broad, can you list actual problems you spend time > > on for them? > > Mostly failing tests, and mostly due to API changes between subsequent > Python 3.x versions. So the solution is either find a patch in the upstream repo (committed or proposed in issues/PRs) or write one yourself. Not sure what can AI help here with.
Re: DebGPT: how LLM can help debian development? demo available.
On 2024-01-03 11:12, Andrey Rakhmatullin wrote: On Wed, Jan 03, 2024 at 09:58:33AM +0200, Andrius Merkys wrote: To me the most time consuming task in Debian recently is the Python transitions. I wonder whether DebGPT could help with them. Maybe there are other, non-Debian-specific GPTs for this task, but I would prefer a Debian one. As "transitions" is too broad, can you list actual problems you spend time on for them? Mostly failing tests, and mostly due to API changes between subsequent Python 3.x versions. Best, Andrius
Re: DebGPT: how LLM can help debian development? demo available.
On Wed, Jan 03, 2024 at 09:58:33AM +0200, Andrius Merkys wrote: > To me the most time consuming task in Debian recently is the Python > transitions. I wonder whether DebGPT could help with them. Maybe there are > other, non-Debian-specific GPTs for this task, but I would prefer a Debian > one. As "transitions" is too broad, can you list actual problems you spend time on for them?
Re: DebGPT: how LLM can help debian development? demo available.
> Installation and setup guide can be found in docs/. Is it planed to package transformers in Debian instead of using conda/mamba venv for this installation ? * It would be great to help with the Debian patch workflow. - upstream status - find upstream bug equivalent to a Debian bug report. - prepare bug report for upstream. - propose improved patch description. * doing request on codesearch.net Cheers Frederic
Re: DebGPT: how LLM can help debian development? demo available.
Hi, On 2024-01-03 00:07, M. Zhou wrote: Following what has been discussed in d-project in an offtopic subthread, I prepared some demo on imagined use cases to leverage LLMs to help debian development. https://salsa.debian.org/deeplearning-team/debgpt I find this pretty impressive. Thanks a lot for working on it. [...] You can also tell me more ideas on how we can interact with LLM for debian-specific tasks. It is generally not difficult to implement. The difficulty stems from the hardware capacity, and hence the context length. Thus, the client program has to fetch the most-relevant information regarding the task. To me the most time consuming task in Debian recently is the Python transitions. I wonder whether DebGPT could help with them. Maybe there are other, non-Debian-specific GPTs for this task, but I would prefer a Debian one. Andrius
Re: DebGPT: how LLM can help debian development? demo available.
On 2024-01-02 17:07:57 -0500 (-0500), M. Zhou wrote: [...] > You can also tell me more ideas on how we can interact with LLM > for debian-specific tasks. It is generally not difficult to > implement. The difficulty stems from the hardware capacity, and > hence the context length. Thus, the client program has to fetch > the most-relevant information regarding the task. [...] I consider myself an "AI skeptic" (probably due to watching the Terminator films at an impressionable age), but one of the most useful ways I've seen LLM leveraged is large open source projects feeding in their corpus of complex documentation, and then providing users with a human language interaction prompt where they can ask questions and get conversational responses based on that documentation. A continuously trained LLM backing a sort of "search" function for all of www.d.o/doc (and maybe also wiki.d.o) could be cool. -- Jeremy Stanley signature.asc Description: PGP signature
DebGPT: how LLM can help debian development? demo available.
Hi folks, Following what has been discussed in d-project in an offtopic subthread, I prepared some demo on imagined use cases to leverage LLMs to help debian development. https://salsa.debian.org/deeplearning-team/debgpt To run the demo, the least requirement is a CUDA GPU with > 6GB memory. You can run it on CPU of course, but that will require > 64GB RAM, and it may take > 20 minutes to give you a reply (I tested this using Xeon Gold 6140). A longer context will require more memory. If you cannot run the demo, I also provided a couple of example sessions. You can use `reply.py` to replay my llm session to figure out how it works. Installation and setup guide can be found in docs/. First start the LLM inference backend: $ debgpt backend --device cuda --precision 4bit Then you can launch the frontend to interact with it. The complete list of potential use cases are listed in demo.sh . I have recorded my session as an example for every single command inside. The following are some selected examples use cases: (the results are not always perfect. You can ask LLM to retry though) 1. let LLM read policy section 4.9.1 and implement "nocheck" support in pytorch/debian/rules command: debgpt x -f examples/pytorch/debian/rules --policy 4.9.1 free -i replay: python3 replay.py examples/84d5a49c-8436-4970-9955-d14592ef1de1.json 2. let LLM add armhf, and delete kfreebsd-amd64 from archlist in pytorch/debian/control command: debgpt x -f examples/pytorch/debian/control free -i replay: python3 replay.py examples/e98f8167-be4d-4c27-bc49-ac4b5411258f.json 3. I always forget which distribution should I target when uploading to stable. Is it bookworm? bookworm-pu? bookworm-updates? bookworm-proposed-updates? We let llm read devref section 5.1 and let it answer the question command: debgpt devref -s 5.5 free -i replay: python3 replay.py examples/6bc35248-ffe7-4bc3-93a2-0298cf45dbae.json 4. Let LLM explain the difference among proposals in vote/2023/vote_002 . command: debgpt vote -s 2023/vote_002 diff replay: python3 replay.py examples/bab71c6f-1102-41ed-831b-897c80e3acfb.json Note, this might be sensitive. I added a big red warning in the program if you ask LLM about vote questions. Do not let LLM affect your vote. 5. Mimic licensecheck. The licensecheck perl implementation is based on regex. It has a small knowledge base, and does not work when the text is very noisy. command: debgpt file -f debgpt/llm.py licensecheck -i replay: python3 replay.py examples/c7e40063-003e-4b04-b481-27943d1ad93f.json 6. My email is too long and you dont want to read it. LLM can summarize it. command: debgpt ml -u 'https://lists.debian.org/debian-project/2023/12/msg00029.html' summary -i replay: python3 replay.py examples/95e9759b-1b67-49d4-854a-2dedfff07640.json 7. General chat with llm without any additional information. command: debgpt none -i replay: python3 replay.py examples/da737d4c-2e93-4962-a685-2a0396d7affb.json The core idea of all those sub functionalities are the same. Just gather some task-specific information. And send them together to the LLM. I felt the state-of-the-art LLMs are much better than that in a few months ago. I'll leave it to the community to evaluate how LLM can help debian development, as well as how useful it is, and how reliable it is. You can also tell me more ideas on how we can interact with LLM for debian-specific tasks. It is generally not difficult to implement. The difficulty stems from the hardware capacity, and hence the context length. Thus, the client program has to fetch the most-relevant information regarding the task. How do you think?
