Ben Goertzel wrote about an attack on their crypto currency network that stole about $2 million, apparently by AI agents attacking some AWS infrastructure code they wrote in 2018, finding many small vulnerabilities that gave them access to a bridge between block chains and allowing them to mint tokens. https://bengoertzel.substack.com/p/averting-the-cybersecurity-apocalypse
In the rest of the article, he talks about how to avert a cyber security apocalypse. In his previous article a few days ago he wrote about his hybrid architecture where he inserted Hyperon agent Omega into the middle layers of an open weight transformer network. In this article he talks about Omega Purple, a team of 3 agents. A red team attacks your own software to find vulnerabilities, a blue team fixes them, and the purple team manages them so they don't grade their own work. He also discusses agents to assist in a formal verification stack from high level natural language down to machine level, each proving that the code matches the specification. I agree that software developers should be using AI to red team their software, just like they have been using other automated hacking tools like nmap since the 1990's. Software vulnerabilities will never go away because of Rice's theorem, but as long as both sides have access to the same tools, I don't think the problem will get any worse. It's just a matter of spending as much time and money to defend your system as your attackers are willing to spend. Rice's theorem, why formal verification won't work, is an extension of the halting theorem, which says that any non trivial property of programs is undecideable. The question of whether a program will give an error or allow remote execution can be converted to a halting problem by writing a wrapper that halts one way but not the other. The halting problem is actually an easier to understand version of Gödel's incompleteness theorem, which says that in any mathematical system, there are true theorems that cannot be proven true, for example, "program P halts". Suppose you had a function that did that. Then you could write a wrapper that halts if P runs forever and halts otherwise. What would the wrapper answer about a copy of itself? -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3a29522fad15bbb7-Mabda8bcea67cb0a507b11854 Delivery options: https://agi.topicbox.com/groups/agi/subscription
