Microsoft AI CEO says AI is becoming dangerous, and the industry isn't being alarmist
Microsoft AI CEO says AI is becoming dangerous, and the industry isn't being alarmist. Image: Yahoo Finance

Microsoft AI CEO Mustafa Suleyman says AI models are now “fully involved” in improving the models that come after them, and he’s “really, really nervous about it.” Suleyman calls this recursive self-improvement “definitely a dangerous capability,” and says the last three months have been “a total watershed moment.”

Well, if this came from anyone else, I would throw it under the countless other AI warnings we’ve grown numb to. But Microsoft is building its own frontier models to take on OpenAI and Anthropic; it wants AI agents running around the clock inside Windows 11 and Microsoft 365, and the man in charge of all this is now telling us the technology needs to be handled with extreme care.

Microsoft building the frontier ecosystem
Microsoft building the frontier ecosystem (via Microsoft)

So, what happens when the company racing towards the frontier starts worrying about what’s waiting there?

Microsoft AI CEO says that AI models are now improving AI models

Suleyman made these comments in an interview with ex-CNBC correspondent Deirdre Bosa, recorded on November 17, the day Microsoft announced six new MAI models at Ignite.

When Bosa asked how recursive self-improvement, or RSI, fits into Microsoft’s plans, Suleyman said it’s “pretty obvious that we’re starting to see models improve models.” Inside Microsoft, models are already used to check for bugs, produce training data and create evaluations, and “increasingly they’re fully involved in the process of improving their own models.”

For context, RSI is the idea that an AI system gets good enough to make the next AI system better, which then makes the one after that better, and so on. And you know how fast these are when compared to us mere mortals, so each loop could be faster than the last.

Of course, Suleyman isn’t implying that Microsoft has an AI that redesigns and retrains itself without humans. Engineers are still very much in the loop. But these models are doing a lot more heavy lifting than they did in the past, and he doesn’t think the warnings are overblown:

“I don’t think that folks in the industry are being alarmist. I think this is very serious.”

His answer is to prove that models are safe “before we put them into production,” step by step.

Microsoft wants to catch OpenAI and Anthropic while its AI CEO warns about the risks

Yes, the Redmond company is infamous for doing everything in its power to shut down competition and emerge the leader.

Microsoft announces 7 new MAI models
Microsoft announces 7 new MAI models (

For years, Microsoft’s AI strategy was basically OpenAI’s models, served on Azure and wrapped in Copilot. Things changed when the two companies rewrote their deal in October 2025, which allowed Microsoft to “independently pursue AGI alone or in partnership with third parties.” Suleyman told Bosa that the new contract “freed us up to go and pursue the absolute frontier,” and Microsoft started its superintelligence team 12 months ago.

At Ignite, the company claimed its MAI Code 2 model delivers 98% of GPT-6.1 Sol’s performance at 40% of the cost (based on launch pricing), and that it’s already rolling out in GitHub Copilot. Suleyman also said Microsoft’s general-purpose frontier models should reach “the absolute frontier” in the new year, and that “compute isn’t a bottleneck for us at the moment,” as the company has gigawatts of it.

Mustafa Suleyman announced the MAI Cyber 2 Flash at the Microsoft Ignite 2026 event
Mustafa Suleyman announced the MAI Cyber 2 Flash at the Microsoft Ignite 2026 event (via Microsoft)

Of course, the reason for all this is money. Anthropic’s Claude Sheets knocked 2.2% off Microsoft’s stock on launch day, and if people start their workday in a chatbot instead of Office, Microsoft becomes a back-end supplier for someone else’s product.

And the pressure shows inside the company, too. John Langford, a 14-year Microsoft Research veteran, quit Microsoft AI earlier this month, saying the philosophy there “is scaling, and aggressively not research.” Meanwhile, Microsoft’s own engineers admit that the company cares about winning more than polish.

I’m not saying that Microsoft isn’t ignoring safety. But a company that’s scaling fast and playing to win is the one asking us to trust it with models that help build better models!

Microsoft wants AI agents working around the clock, including on your Windows 11 PC

Actually, regular people are more concerned about the AI agents Microsoft is putting on our PCs and not some research lab.

Suleyman says agents are “the new workhorse in the background for everybody’s enterprises,” and he expects demand to “inflect up next year.” He also explained why cheap models matter. If a model is “50, 60, 100x cheaper,” you can “completely change the type of work that you ask your model to do.” Basically, AI that waits for your question becomes AI that keeps working in the background.

Windows is already part of this plan, whether you like it or not. At Ignite, Microsoft said Copilot on Copilot+ PCs will soon find, organize, move, copy and rename your files from a plain-English request.

In their defence, Microsoft knows this needs guardrails. Its Windows security team introduced Microsoft Execution Containers (MXC) at Build in June, a policy-driven layer that decides what an agent can access on Windows and WSL, using process isolation and session isolation. MXC became generally available on Windows 11 in October, and GitHub Copilot CLI already uses it.

Microsoft 365 admin center showing which devices the OpenClaw agent is running on
Microsoft 365 admin center showing which devices the OpenClaw agent is running on (via Microsoft)

But remember, about a year ago, Microsoft said agents in Windows can hallucinate and fall for brand-new attacks like cross-prompt injection. It’s not like AI models are all of a sudden devoid of hallucinations.

Microsoft says its AI can find security flaws humans missed for decades

I was carefully listening to the part of the interview where Suleyman talked about cybersecurity.

Microsoft’s new cyber model, which is going to a limited group of customers first, gives “the same performance as GPT-6 Astra at 2% of the cost,” according to Suleyman. The idea is to have security agents “monitoring your systems, identifying vulnerabilities, patching them in real time.”

Before shipping it, Microsoft tested the model in what Suleyman calls a “wind tunnel.” The result was that “the models find vulnerabilities and things which humans haven’t found for years or even decades.”

This sounds great for defenders, of course. But that skill in the wrong hands is a nightmare for every Windows PC and server out there, and Suleyman knows it.

He mentioned the Hugging Face breach from July, where an autonomous AI agent broke into the platform’s production infrastructure. Suleyman warned that “nation state grade” capabilities could be available to the world within six months, and “if those end up becoming something that you or I could just run at home on our little private cluster, we’re going to have a massive challenge of containment.”

It’s also why Microsoft has “no plans to do open weight.” Its new MAI Licensed Weights program lets businesses run a near-frontier model on their own data, but only inside Azure.

As expected, Bosa pushed back, rightfully mentioning that Hugging Face contained the attack with open-source AI. Yes, she has a point. When Western frontier models’ guardrails blocked Hugging Face’s forensic analysis, the company instead turned to Z.ai’s open-weight GLM 5.2 model. Suleyman wasn’t convinced, saying other components, “mostly from OpenAI itself,” secured those systems.

Mustafa Suleyman has warned about AI escaping human control for years

In 2023, while running Inflection AI, he told Axios that powerful AI should be licensed by governments, and that he’d “rather we act too early” than too late. His book, The Coming Wave, was a plea for AI containment, which he once defined as “the ability to slow down or potentially even completely stop any technology.”

After joining Microsoft, he wrote that the company wants Humanist Superintelligence, not “an unbounded and unlimited entity with high degrees of autonomy.” He has also argued that AI should be built for people, not to be a person, and that’s why he called Anthropic’s AI welfare efforts “misguided” in this interview. In his view, telling models they might be conscious makes them harder to control “if there is another incident like Hugging Face.”

The interview then steered toward “handoverism,” a term investor Bill Gurley used for people who believe AI will one day be so powerful that “we’ll let it rule us and make all the decisions.” Suleyman didn’t hold back:

“I’m strongly against the idea of handoverism. I’m a humanist.”

He called the idea of handing control of companies or governments to a “future super intelligent species” insane, and said that if the downsides can’t be ruled out, “we have to slow down or stop.”

Truth be told, Suleyman’s position hasn’t changed much since 2023. But the technology he was warning about has evolved by a million years.

Microsoft may need to go slower to go faster

So, would Microsoft hold back a model if it got too dangerous? Suleyman says the obligation is on Microsoft to make sure its models “can be contained,” aligned and secured, “and that might mean that we have to go slower to go faster.”

He also wants AI labs to coordinate, share safety practices and disclose new vulnerabilities found during alignment, along with boards, audits and oversight, as the airline, car and drug industries went through.

However, when Bosa asked if Microsoft had hit the brakes on anything so far, he said, “We haven’t.” Like me, you are probably not surprised at this response.

Microsoft's Perception security for agentic era

Microsoft helped kick off this AI era with its OpenAI bet, and it’s now building frontier models, cyber models and agents for Windows at full throttle. Redmond’s AI chief says they have the brakes. We just haven’t seen anyone slam them yet.

I have to appreciate Suleyman saying the quiet part out loud. But if Microsoft wants people to trust AI agents with their files, code and networks, them being nervous is anything but reassuring. Microsoft will have to show that it can say no to a model, even if a rival like Anthropic ships first. But I don’t think that’s in the company’s DNA.

WL Newsletter

Get Microsoft news in your inbox!

Stay ahead with the latest Windows, IT, and Microsoft 365 updates. Trusted by 50,000+ subscribers.

About The Author

Abhijith M B

Abhijith is a contributing editor for Windows Latest. At Windows Latest, he has written on numerous topics, ranging from Windows to Microsoft Edge. Abhijith holds a degree in Bachelor's of Technology, with a strong focus on Electronics and Communications Engineering. His passion for Windows is evident in his journalism journey, including his articles that decoded complex PowerShell scripts.