"Peter N. M. Hansteen" <pe...@bsdly.net> writes:
Here's the story of my asking it to write a PF.conf -
https://nxdomain.no/~peter/chatgpt_writes_pf.conf.html or with
nicer formatting and trackers
https://bsdly.blogspot.com/2023/06/i-asked-chatgpt-to-write-pfconf-to-spec.html
so in this context, near totally useless, likely due to
insufficient volume of actually useful configurations in the
data it was trained on.
Yes, i was thinking that might be one reason ChatGPT is, mm, 'not
great', in terms of OpenBSD content. Another reason might be that,
since only the two most recent OpenBSD releases are supported,
there can be changes in the base system which result in
e.g. tutorials becoming out-of-date more quickly. Which is, of
course, why the first port of call should always be the man pages
and the FAQ. :-)
Still, even on topics where there's ample information available,
'AI' can get it very wrong, as in the case of an AI-generated
article on CNET about compound interest:
https://gizmodo.com/cnet-ai-chatgpt-news-robot-1849996151
But hey, it doesn't really matter whether factual information is
or isn't available: just make up data! "ChatGPT Created a Fake
Dataset With Skewed Results - Findings raise concerns about the
ease of generating false medical evidence",
https://www.medpagetoday.com/special-reports/features/107247
Finally, there's a paper relevant to many OpenBSD user's
interests: "Do Users Write More Insecure Code with AI
Assistants?", https://arxiv.org/abs/2211.03622.
Overall, we find that participants who had access to an AI
assistant based on OpenAI's codex-davinci-002 model wrote
significantly less secure code than those without
access. Additionally, participants with access to an AI
assistant were more likely to believe they wrote secure code
than those without access to the AI assistant. Furthermore, we
find that participants who trusted the AI less and engaged more
with the language and format of their prompts (e.g. re-phrasing,
adjusting temperature) provided code with fewer security
vulnerabilities.
Alexis.
