3. Prompt injection. Gitlab comments are an untrusted
input source and LLMs fundamentally cannot discriminate
between instructions and data. You can imagine prompt
injection attacks where hidden text in a gitlab comment,
attached bug reproducer, link etc. gets the model to do
various destructive things, like spamming the gitlab,
editing/deleting your prior comments, mining bitcoin on the
CI runners, attempting to inject malicious code into the
compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have
to apply some safeguards in the review but it's still running on _my_
system. Regarding the "untrusted input source": accounts already need
to be verified. Recent models are also increasingly resistant to
prompt injection.
I would advise against giving any LLM system "write access"
to the gitlab, where it can autonoumously post things, push
commits, trigger pipelines and so on without a "human in the
loop".
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an
undifferentiated input stream, and while the harnessnes try
to impose some structure on it, it comes through things
approximating xml tags, json etc. However, tool output can
produce these "structured tags" just as easily and this gets
injected into the context stream with no escaping/validation
etc so the model has no chance of differentiating between
"legitimate" instructions like `<system-reminders>` etc
that actually come from the harness and those that come
from running arbitrary commands etc. I have a few proof of
concepts of this I can show you off-thread, you don't even
have to be too clever about it.
_______________________________________________
ghc-devs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
