Jean Louis <[email protected]> writes:
> But do you really have that issue in the Org mode project?
>
> In Org mode, contributors are not average person, they are special for
> using Org mode and participating on the mailing list. I am not
> monitoring those contributions, that is why we should be practical and
> see if the problem really applies in Org mode community, or maybe not.
>
> Real practical problem solving is appreciated.
>
> It gives me pause when generalizations are applied to the Org mode
> community, potentially to the detriment of users.
>
> I just believe that contributors to Org mode are not reckless as given
> in those generalized statements.
There are sometimes lower-quality contributions, but not often.
But I suspect that part of the reason is the barrier of entry
(email+patch workflow).
LLMs objectively lower the barrier of entry. More so in more mainstream
workflows like in web pull requests (agent APIs and MCP servers make the
barriers even lower there), but also for patch workflows.
So, I do anticipate less experienced contributors to submit patches to
Org as a result of LLM adoption.
More contributions are generally a good thing. However, they are only
good when the patch quality is good. Or, alternatively (that's
important!), when the less experienced contributors are ready to learn,
and later come back with more, better-quality patches.
LLMs create problems here. In *some* (not all, mind you) cases, users do
not need to learn anything and simply hand over a task to LLMs, still
getting something that is working (or seem to be working). However, even
the best LLMs often fail to be accurate when the task description does
not include enough details. So, we can end up with (1) patch that is of
low quality; (2) user not learning much along the way.
Without LLMs, I could simply ask to fix things in the patch, and the
user will have to learn by practice. With LLMs, my instructions risk
simply being fed to LLM - that will get the patch improved, but
contributor's understanding of Org code will not improve much.
In order for such LLM contributions to be useful (not just for the patch
in question, but also for building the contributor community), we need
people to *build understanding* of the code. And that requires learning
effort. But how to encourage such learning?
The above is not a theoretical question. This discussion was triggered
by LLM-generated patch. And we had a couple LLM-assisted patches in the
past as well.
There are several options:
1. Reject LLM contributions, so that we simply do not bother with the
problem. That's ok for now, since we do not have *many* LLM
contributions, but LLM adoption is increasing, so, sooner or later,
this kind of policy will become an obstacle. Part of the community
(that support LLM adoption) will also be disappointed.
2. Reject LLM contributions for now, but reconsider later. It may be an
option, but we are just postponing this discussion then. Unless we
expect additional qualitative changes in the LLM performance in the
near future, that will not be wise.
3. Accept contributions only from regular contributors. They are already
familiar with Org code, so we are not missing as much on
learning. Still, some risk that LLM generation can lower the usual
quality controls on the submitter side - potentially more patches,
but harder reviews. New contributors will experience increasingly
high barrier of entry (especially as LLM adoption rates grow, and
*new* people would be reluctant to contribute when LLM use is
prohibited).
4. Accept LLM contributions for everyone, but introduce some guidelines
for LLM usage. Here, the idea is to (1) encourage people to use the
best LLM practices, possibly even adding Org-specific instructions
for LLMs; (2) make sure that people still learn themselves on high
level without completely handing off the work, including
architecture, to LLMs. *This is the option I am leaning towards so far*
5. Do not check, treat all the patches equally. That's roughly what we
do now (except that we are prohibited from accepting patches that are
declared to be written by LLMs as instructed by Emacs upstream).
But, as I wrote above, this approach may hurt building the *regular*
contributor community - people familiar with the project who can
become future maintainers.
--
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
