Every time Microsoft launched a major AI feature this year, I couldn’t help but feel more skeptical about the company’s new direction. Here’s Microsoft, a notoriously conservative and slow-moving giant, reshaping its products around artificial intelligence not long after most people learned generative AI existed. The last time it made such a dramatic shift we got Windows 8, a failed attempt at making its flagship OS tablet and touchscreen friendly.

Now, the company is bringing AI right into the heart of Windows and I’m left wondering: Is Microsoft jumping into artificial intelligence to actually make its products better? Or is it just trying to stake a claim as an AI innovator and pray that the technology actually lives up to the hype? At this point, it’s genuinely hard to tell.

As the Zune, WebTV and Windows Phone have shown, Microsoft isn’t so great at timing. Its products often either land too early to be useful (as in the case of the sluggish WebTV), or arrive far too late to make an impact (like the genuinely great Zune HD). But when the company unveiled its AI-powered Bing Chat earlier this year, it was perfectly positioned to coast on the success of ChatGPT, which by then had reportedly reached 100 million users in just two months. According to UBS analysts, that would have made ChatGPT the fastest growing consumer application in history. What better time to mate the power of generative AI with one of its notoriously beleaguered products? Microsoft had nothing to lose.

After investing a total of $13 billion in ChatGPT-maker OpenAI (and acquiring a 49 percent stake in the process), Microsoft was probably eager to show off its shiny new toy ahead of Google and others. The introduction of Bing Chat officially kicked off Microsoft’s year of AI: Copilot launched on Edge, Microsoft 365 products like Word and Powerpoint and eventually made its way to Windows 11. Even more surprising, the company recently announced that Copilot is coming to Windows 10 — a sign that it wants AI features in front of as many people as possible. (Windows 11 reportedly accounts for 26 percent of Windows installations, while Windows 10 still has 69 percent. By targeting both platforms Copilot could potentially reach up to 1.4 billion users.)

There’s no doubt that Copilot makes a great first impression. Type in a few words (or speak them aloud), and it returns with direct answers to your questions, like a whip-smart assistant. There are no ads to wade through, and you only have to engage with additional links if you want. It’s a glimpse at a world beyond search engines, one where AI could help guide us through an increasingly chaotic media landscape. Microsoft’s Copilots can also help out in specific applications: In Edge it can summarize the webpage you’re looking at; it can help to transcribe and generate action points in Teams Meetings; and it can help unearth hard to find settings in Windows (for example, you could just type “How do I turn on Night Mode?” to flip that on).

But Copilot’s confident veneer hides the fact that it often makes errors and can occasionally misunderstand your questions entirely. It’s far less responsive than using a typical search engine, as there’s a lot of opaque AI processing happening in the background. And in my testing, it also crashes more often than you’d think, which requires a “reboot” of your session (but at least it doesn’t flash a blue screen like Windows).

In an effort to temper our expectations, Microsoft has a helpful note emblazoned atop Bing’s AI chat: “Bing is powered by AI, so surprises and mistakes are possible. Please share feedback so we can improve!” Microsoft appears to show a bit of humility here by acknowledging that its AI chat isn’t perfect, and it’s trying to earn some brownie points by saying it’s listening to your feedback. Mostly, though, that warning serves as a way out for Microsoft. It can tout Copilot’s ability to write essays for you and hold vaguely realistic conversations, but the minute it screws up, the company can just say, “It’s just a beta, LOL!”

The big test for Microsoft’s Copilots and other generative AI tools comes down to one thing: trust. Can a user trust that it’ll deliver the relevant information when it asks a question? Can we be sure Copilot will even understand our query correctly?

Aaron Woodman, Microsoft’s VP of Windows Marketing, tells us that trust will ultimately come down to users “kicking the tires” for themselves and seeing how well Copilot performs. “I think that type of organic growth is one that we’re going to see over time,” he said in an interview with Engadget at the Windows Copilot launch in September. “And I bet it’ll be explosive because the value is there, and I think customers will see that very quickly.”

Woodman also believes that users will understand that Copilot won’t always be perfect, especially during these early days. “I weirdly think we’re probably more empathetic with people and understand where they’re at with growth than we are with technology,” he said. “I think the best thing that we can do is honestly own that, be transparent about it. At some level, every conversation we’re in, we’re trying to lean into [that] this is a growth process. We want to make sure you understand reference materials. I think people will understand that we’re trying to accelerate bringing [new] technology to them.”

I’ve been using Microsoft’s AI solutions since Bing Chat launched earlier this year, and while it’s helpful for simple tasks, like creating a specification table comparing two products, it hasn’t exactly changed the way I work. Microsoft also had to seriously restrict Bing Chat’s capabilities early on after it started arguing with users and issuing disturbing responses. In Windows 11, Copilot can sometimes help me find settings like dark mode, but it can’t always pull up the controls within the Copilot pane, and sometimes it just sends me to general settings menus if it can’t figure out what I’m asking for.

More recently, I’ve had disappointing conversations with Bing when I asked if it was a good time to buy a Nintendo Switch (it took some prodding for it to bring up rumors of a potential Switch follow-up coming next year), and its ability to answer questions around images is still less useful than Google’s image search.

When I took a photo of my kid’s baby monitor and asked “What is this?,” Bing was aware of its function, but it got the actual model and manufacturer wrong. That query also took five seconds to complete. The Google Image Search took half a second and correctly identified it as the Eufy Space Monitor. Score one for traditional search (and yes, I know it’s also powered by its own set of computer vision models).

We can look to Microsoft’s Github Copilot, which launched in November 2021, as one way users can learn to work with AI. It’s mainly meant to serve as a partner alongside an experienced programmer: It’ll look out for potential issues and it can even whip up some simple code.

According to developer Aidan Tilgner, Github Copilot can be genuinely useful for coders, so long as you keep your expectations in check. In the paper “GitHub Copilot AI pair programming: Asset or Liability?” authors Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C.Desmarais, and Zhen Ming Jiang found Github Copilot similarly useful, but note “it can also become a liability if it is used by novices, those who may not be familiar with the problem context and correct coding methods.”

“Copilot suggests solutions that might be buggy and difficult to understand, which may be accepted as correct solutions by novices,” the authors add. “Adding such buggy and complex code into software projects can highly impact their quality.”

By leaning so much on Copilots in the future, Microsoft may also be tying itself too closely to OpenAI, a young company that recently went through one of the most volatile weekends in Silicon Valley history. OpenAI’s board fired CEO Sam Altman, but after a significant amount of internal pressure (and some cajoling from Microsoft CEO Satya Nadella), it ultimately re-hired him a few days later. If OpenAI goes through another tumultuous event, it won’t just be Microsoft’s $13 billion investment in danger: It’ll be the company’s future plans for practically all of its products.

According to Windows Central, Microsoft’s next major Windows update, “Hudson Valley,” may arrive next year with a slew of AI enhancements in tow. That includes the ability to analyze content being displayed in video chats, an improved Copilot that can remember everything you’ve done on your PC, and better system-wide search. Some features may also require CPUs with NPUs, like AMD’s last batch of chips and Intel’s new Core Ultra hardware. That’s similar to the Windows Studio Effects features like background blurring and auto-framing, which also require NPUs.

The one constant around AI these days is that everything is changing quickly. Since I started writing this piece, Microsoft announced Copilot would be upgraded with the more powerful GPT-4 Turbo and Dall-E 3 models, which will make them even more capable. Perhaps Microsoft and OpenAI will eventually be able to fix all of the issues I’ve seen with Copilot so far, and ultimately deliver a transformative AI tool that’s easily available to everyone. But I also hoped for the best when it came to the company’s dual-screen Duo and Neo plans, and all I got in return was disappointment.