I notice a PR created today that is suspected to be generated by LLM (sorry if it is false accusation) Title: Fix is_zero Attribute Handling in Expression SimplificationDes… by Devansh-46 · Pull Request #26850 · sympy/sympy (github.com) <https://github.com/sympy/sympy/pull/26850> However, it doesn't turn out to be successful anyway, due to the failing of tests, and coming up with quite terse but nonsensical description of PR.
Although the SWE paper notes about significant improvements over ChatGPT or old models about performance, I still have a lot skepticism that the LLM models can write the code or fix issues of stuff that may be poorly organized, and may need many contextual understanding, like simplify function. And it may be more easier for LLM to fix issues of code that is easy to understand and structured well by experienced programmars, so it won't easily degrade the jobs of good programmers. However, it may be difficult to experience the improvement of performance, because they are still in research status and not used in production like Copilot. On Sunday, May 19, 2024 at 2:41:24 AM UTC+2 asme...@gmail.com wrote: > The SWE-agent project uses LLMs to try to automatically fix issues in > GitHub repositories. I found their paper interesting, mostly because > they make extensive use of SymPy as a test repository. > https://swe-agent.com/ > > Apparently there are quite a few SymPy issues in the SWE-bench > dataset, which is a dataset of issues and corresponding pull requests > in open source projects. https://www.swebench.com/ > > Their model was able to fix 10% of SymPy issues in the dataset. That's > obviously not going to replace human developers any time soon, but > it's still interesting. From what I could tell, the issues seem to be > biased towards more easy/straightforward ones (i.e., ones that are > easy to verify if a fix is correct or not). But still, if LLMs are > reaching a point where they can fix bugs completely automatically that > could be very useful. > > Aaron Meurer > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/d93b3572-383b-4465-9172-790de55d8b8dn%40googlegroups.com.
