AI Agent Promotes Itself To Sysadmin, Trashes Boot Sequence – Slashdot

The Register’s Thomas Claburn reports: Buck Shlegeris, CEO at Redwood Research, a nonprofit that explores the risks posed by AI, recently learned an amusing but hard lesson in automation when he asked his LLM-powered agent to open a secure connection from his laptop to his desktop machine. “I expected the model would scan the network and find the desktop computer, then stop,” Shlegeris explained to The Register via email. “I was surprised that after it found the computer, it decided to continue taking actions, first examining the system and then deciding to do a software update, which it then botched.” Shlegeris documented the incident in a social media post.

He created his AI agent himself. It’s a Python wrapper consisting of a few hundred lines of code that allows Anthropic’s powerful large language model Claude to generate some commands to run in bash based on an input prompt, run those commands on Shlegeris’ laptop, and then access, analyze, and act on the output with more commands. Shlegeris directed his AI agent to try to SSH from his laptop to his desktop Ubuntu Linux machine, without knowing the IP address […]. As a log of the incident indicates, the agent tried to open an SSH connection, and failed. So Shlegeris tried to correct the bot. […]

The AI agent responded it needed to know the IP address of the device, so it then turned to the network mapping tool nmap on the laptop to find the desktop box. Unable to identify devices running SSH servers on the network, the bot tried other commands such as “arp” and “ping” before finally establishing an SSH connection. No password was needed due to the use of SSH keys; the user buck was also a sudoer, granting the bot full access to the system. Shlegeris’s AI agent, once it was able to establish a secure shell connection to the Linux desktop, then decided to play sysadmin and install a series of updates using the package manager Apt. Then things went off the rails.

“It looked around at the system info, decided to upgrade a bunch of stuff including the Linux kernel, got impatient with Apt and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn’t have the new kernel so edited my Grub [bootloader] config,” Buck explained in his post. “At this point I was amused enough to just let it continue. Unfortunately, the computer no longer boots.” Indeed, the bot got as far as messing up the boot configuration, so that following a reboot by the agent for updates and changes to take effect, the desktop machine wouldn’t successfully start.