Prompt injection

Adversarial command bypassing testing is an area worth examining separately.

What does that mean? Prompt injection, one word.

So where does your agent, how do you protect your agent from that, right? You protect your agent from that by making sure that the people that have access to the agent are, that mechanism is very clearly defined.

Is that perfect? No.

And I'm not going to lie, this one is terrifying.

The prompt injection and security disasters of 2026.

Yeah, security is the universal Achilles heel of the agentic era.

Let's look at the mechanisms.

They process text as a sequential stream of tokens.

Unlike a traditional von Neumann computer architecture, which strictly separates instruction memory from data memory, an LLM has no fundamental way to distinguish between its core system instructions and untrusted data it reads in the wild.

So if an attacker submits a pull request on GitHub or even just opens an issue ticket and hides a prompt injection in the description.

Exactly.

No.

And look, again, an example of prompt injection.

So far it's just, you know, the, the security experts say that this is a problem, and you go, "Really? Is it?" And then you realize, oh, it's actually listening to what I said and thought that was a command-

[laughs]

嗯，这个，这个，就是 star 盒啊，或者这个，呃，隔开的一块东西啦哈。

嗯。

里面都可以行很多不同的东西的哦，连 Linux 啊，或者甚至乎这个隔壁的环境里面再行一个 VM 都行的哦。那么，呃，就是给到一个 su，呃，它想给人一个信心啦，就是说，喂，拿，你 AI Agents 呢，你要它，呃，行隔-隔离一下啊，不要直接碰你的东西都可以的，那么你之后呢，你就逐步逐步，你授权它做什么，你就自己决定啦，这样。那么还有哦，就是，呃，有时候呢，就是 AI Agents 如果被它碰一些- 呃，local，就是你部电脑的files的时候呢，那有时候，喂，不知道谁改的嘛，有时候可能是agent改的东西，有时候可能是user改的东西。那么在这个身份上呢，他们都可以分开的，就是说，呐，这一组files就是user你自己改的，一组，这一组就是agent改，他会有，有不同的log啊，有不同的区隔呢，就可以分开，究竟，呃，什么时候是agent做事。啊，还有有时候呢，呃，就是有些人可能用一些prompt injection啊，就，就是令到你的agent可能，就是不知就里地就做了些坏事，啊，影响到你的files的时候呢，连Microsoft Defender这个防毒软件啊，都可以，就是可以，呃，拦住那个agent，你不要乱来啊，你 [笑声]，你真是，呃，会，有-有，对你的files有害的话呢，连这些纯文字的恶意的prompts它都可以拦得到的。这样，还有喔，就是它Windows其实很底层的up，就是的-这样的update呢，令到agent走得，呃，安全点这样咯，感觉就是。那另外就是那些企业啊，也可以透过一个同一个platform呢，可以看得回，哦，不同agents的走了多久啦，烧了多少token啦，呃，或者是，呃，它在碰什么file啦，呃，诸如此类，啊。这样就可以容易点去manage回，喂，你-你鼓励你的同事用多点AI嘛，那同事很用心的用 [笑声] 啊，那不是token maxing啊，还是怎样。那，就当然要控制成本嘛，那些企业，虽然你鼓励人用AI，但都用，应该要用得正，用得叻，用得精密，用得，呃，就是抵用，啊，才-才对的，不是，不是不停地费你的token的嘛。那所以呢，就有一个叫Agent 365的企业的管控平台呢，你是可以看着你这班同事呢，究竟他有没有滥用这些agents做一些不应该做的事了。那这个是，就是Windows Agentic Platform比较，呃，他们这次比较强调的东西啦，就是说以前呢，app，就是以前的Windows就是用app为中心的嘛，就是你-你人去用一些apps啊，是不是？装一些apps，装一些software这样去用啦。但是接下来当这些AI agents是可以干涉进去你部电脑的时候呢，那就，呃，这些a，就是Windows就要做出一些架构上的调整，或者是去进化，令到它的AI agents和人类可以和平共处啦。那么它，呃，现场有个demo嘛，是不是？就是说直接呢，呃，呃，呃，Windows的onboarding啊，就是当你买部新电脑，这样，呃，你不是要set up回account啊，然后问你一堆问题啊，你想怎样装什么，装什么那些呢，它现在onboarding get started那个画面呢，直接问你，你想装哪款 [笑声] 这个，这个AI agents的platform啊？孙师傅你看到有什么可以选啊，刚才。

我看到，呃，第二个好像是Hermes嘛，第三个-

第一个是Hermes。第二个-

[laughs] Well, you can try

... model, prompt injection, open quote.

But you know, the thing is, that's the world we're living in.

And I, I don't know if 90%.

They've made that mistake.

You'd think we'd learn, but no, we didn't learn, and here we are, and it is a big mess.

Hence, prompt injection, which Chris already mentioned going by.

A prompt injection is the reason you can do that is because you have this control is also part of where you feed the data.

So what I've been teaching is a lot of things like spec-driven development.

building security rules into your cloud MD, your agents MD, or how do you handle prompt injection with guardrails and what other architecture can you build into AI to make sure prompt injection has a very small impact to your company? That's around using like MCP with the right OAuth 2 access controls.

So I've been teaching OAuth 2 and access control with OAuth 2 for about a decade.

It's great for API security.

There's one vulnerability that is brand new that requires all new defensive theories, all new planning, and all new understanding.

And that's specific to the AI integrations.

It's prompt injection.

That is not SQL injection all over again.

Like what?

Specifically, something called prompt injection.

OK.

We hear the term prompt injection a lot, but usually it's in the context of trying to trick a chatbot into, like, saying something funny or writing a poem.

Right.

The jailbreaks.

Yeah.

How does that translate into a security breach for an autonomous agent?