The longstanding sales pitch to the public for AI development has been imagining a world where menial jobs and tasks are automated, freeing up time and energy to focus on the things in life we enjoy. The inaugural class of agentic personal assistants is giving us a glimpse at what handing real-life tasks to AI can look like, and what it can cost when it inevitably goes wrong.
AI chatbots hallucinating false information is a well-documented problem, and another risk of a technology in its infancy having cultural prominence. Now, raise the stakes further: agents are hallucinating and misinterpreting instructions to take real-world actions in the name of their users.
But what about when agentic personal assistants work as intended? The description of a workless future with universal basic income is an attractive sell, if you don’t examine it critically and wonder why the industry’s richest figureheads would want to transform society into a post-capitalist utopia for the benefit of the working class. For the sake of engaging with the idea in good faith, a well-functioning AI assistant managing life’s mundanity sounds appealing. Many people already use them, deciding the reward outweighs the risk.
First of all, the risk still wins sometimes. My colleague Ariella wrote last week about Meta’s Muse, and the many ways it has already misused access to its users’ private information, data, and communications.
Muse Is Meta’s Latest Non-Consensual Surveillance Tool
Last week, while attempting some of the most pathetic stand-up comedy I’ve ever seen, Mark Zuckerberg stood on stage in a “Building Is My Love Language” shirt to walk us through Meta’s new product announcements. Among them is Muse, Meta’s personalized AI agent that embeds itself in all of your data—from email and text messages to your calendar and other…
It’s a glaring problem with Muse, but not unique to it. The executive of an AI-powered virtual clinic warned others to “never get your AI to order flowers” for your fiancé, after the card arrived with a watermark from Wajo AI’s “Fo” assistant taking responsibility for sending them. People discussed using AI to run their fantasy football league, and others defended it with a snarky tone. Elon Musk has been advertising X’s Grok Bot as a financial advisor for months now, and advocating for giving Grok access to your banking information. This week, the CEO of a software company claimed his network of Grok Bots, which he’d given access to his bank accounts and nicknamed his “Chief Financial Officer,” posted the balance of his bank account to his company-wide Slack. Less than an hour later, Elon advertised Grok Bot from his account again, referencing the first post with a wink.
Second: When the agents work as intended, your interaction with your own plans are minimized for the sake of convenience. If a near-future’s normal is AI agents we text with instructions — to book us a flight, make reservations at a restaurant, find time to catch up with a friend by coordinating with their agent — we lose personal discernment and connection to gain convenience. It could also reshape and centralize how people interact with the internet, and hand an imperfect technology, and the companies designing them, the ability to make decisions and form experiences on your behalf.
A friend of mine who works for a major tech company in San Francisco was telling me Instinct is being used and discussed more and more in her circles (she does not use it, and jokingly covers her ears and sings when AI personal assistants are discussed). Unlike Muse, Instinct does not have an interface; you text it from your phone number, mimicking an interaction with a human personal assistant. Its CEO said in an interview the human-emanating vibe was a deeply intentional choice.
“It’s easiest to literally pretend you have a superhuman person with a phone, a computer, an email address, who can call you - like you’re dealing with a person, and that’s it,” CEO Noah Shinn explained, shortly after his company was valued at $10 billion one month after launch. “That makes you wonder how you think about everyone in the world having one of these, having an Instinct, having an agent, and how that will reorder things.”
Convenience always has a cost. Even if we overlook the privacy concerns and inevitable errors of AI personal assistants, we sacrifice our right to be arbiters of our own experiences. They can get menial tasks out of the way, if you trust them enough to do so, but like one CEO’s fiancé discovered, these agents can also reveal which human experiences its user isn’t interested in performing. Instinct can scan every item in your closet and plan outfits for you. It can plan a date in collaboration with your significant other’s Instinct bot. It can plan your entire vacation — choose the place, buy your flight, book your hotel and restaurant reservations, suggest activities. It can select, order, and send your friends birthday gifts.
These seem like the things we were promised more time to do ourselves in the absence of work, once the AI utopia arrives. Right now, with zero reason to believe working for income will be phased out of society, the opposite seems more realistic: AI agents suited for replacing our role in self expression, our interpersonal relationships, and the structure of our free time.
Mostly, if the way we interact with the internet becomes deeply centralized in one company’s infrastructure and algorithm rather than many — by design, these assistants reduce your interactions with other platforms and websites to save time — we hand over to an intermediary the time we’d used to consume information that developed our personal tastes, judgements, and experiences. What’s left to do without that except for… work?
Other things I’m paying attention to this week:
Speaking of the promise of AI utopia, Anthropic whistleblower Jacob Coxon advocated for it on Monday’s Daily Show. He regurgitates the same promises of entirely automated production and blue-collar jobs, high-paying universal basic income reallocated from automation’s dividends, and said he believes the global consensus is in favor of an AI utopian future without purpose-driven work. Stewart responded to applause, “I don’t think that’s the majority. I think they’d get absolutely f***ing crushed.”
It’s tech week in San Francisco, reminiscent of the AI World’s Fair earlier this year: billed as gathering the “top founders, funds, and companies around the globe” for exclusive events and parties with long lines out the door. Los Angeles will have its own iteration next week.
Pope Leo XIV’s X account continues a motif of examining how the prevalence of AI and technology in today’s society affects humans spiritually.