Is That Photo AI Or Not? A Little-Known “Credential” Network May Tell Us
“Content provenance” verifies the origins of photos and videos. A new report calls out how this could actually adversely impact the privacy of journalists, and proposes preemptive solutions.
These days, reality is murky. We assume one photo is AI when it’s real; we believe another photo is real when it’s in fact AI. We question whether Mitch McConnell is alive, and then when a “proof-of-life” photo comes out, we question whether that is real.
And it’s not just something journalists have to grapple with, either—just today, I read a story about a man who rented a house for the weekend, only to get an AI-generated image from the homeowner claiming $6,000 of tabletop damage. (No doubt, such a con has sadly probably worked in the past.)
One system has been helping the layperson identify where images are coming from for years—an attempt to combat misinformation and disinformation. Called C2PA (short for the Coalition for Content Provenance and Authenticity), information—like images and videos—can be “credentialed” through verification of technologists, software companies, and hardware companies. A Canon or Leica camera, for example, can sign off on the fact that that camera with a certain serial number took the photo. An editing platform like Pixel Studio can then sign off on its role in altering the image. All of this can be communicated to social dissemination platforms, like Instagram or TikTok, where the information is shared.
Tech companies like OpenAI, Meta, YouTube, LinkedIn, and TikTok are part of the coalition to participate in this network—as well as phone manufacturers, editing platforms, camera manufacturers and some AI companies. It’s not completely fool-proof, of course—in part because some organizations don’t participate in the coalition (Anthropic and Apple are not, and neither is X, after the Elon Musk takeover…naturally.) Plus, while the metadata attached to images is true, C2PA isn’t necessarily saying the content itself is true.
In theory and in practice, C2PA has been an important tool in tracing the origins of digital information. But as with all innovations, there may be unintended consequences—and that’s why human rights nonprofit Witness published a report calling out the scenarios in which a beneficial tool could be weaponized to dox journalists or human rights defenders.
I spoke earlier today with Jacobo Castellanos, a trust and safety researcher at Witness. He told me that while C2PA is critically important, work needs to be done to ensure that journalists, photographers, and human rights advocates involved in sensitive information dissemination are not being traced if they don’t want to be.
While content provenance inherently cannot allow for identity to be included, different organizations wanting to legitimize their information with identity attachment—including a news organization, let’s say—might want to have an extension that allows for their journalists’ names to be attached. That individual journalist, however, might want to separate their name for safety reasons.
The report calls out an example scenario of a hypothetical journalist in this conundrum named “Amara” — someone who wants to share information while also protecting herself:
“She strips the personal information from the manifest using the redaction function in her editing software, and publishes the footage anonymously through a smaller, unlicensed platform that does not require credential verification. She cannot use the outlet she normally works with. That outlet operates under a platform distribution license that requires credentials on the state trust list. Publishing without her credential is not an option.
The economic consequences are immediate. The outlet that wanted the story no longer has it. The relationship with that editor — built over years of reliable work — is strained. The fees she would have earned are gone. Content published on unlicensed platforms is easier to dismiss:
the government’s communications apparatus is practiced at labeling inconvenient journalism as unverified or fabricated. Without her credential and without the institutional backing of a licensed outlet, the footage is more vulnerable to exactly that framing.
She publishes anyway, because the alternative is not publishing at all. She thinks that is the end of it, but what Amara does not know is that the manifest, stripped of her name, still carries the fingerprint of her working practice: the device identifier from her camera, the editing software signature, the specific sequence of adjustments she applies, with minor variations, to almost everything she shoots.”
Someone outside of a news organization could also be susceptible, the report calls out—such as “a human rights documenter who redacts their name from a manifest before publishing, unaware that a watermark embedded in the image still points to the original signed file.” Even a war photographer who is very careful to trace his tracks in documenting military operations can be traced back to his camera and a mobile application—even if his organization and personal identity are anonymous.
These are hypotheticals, reiterates Castellanos, not actual problems occurring in real time yet. But he still urges regulators to become familiar with the technical architecture of C2PA, and proposes an increased presence of independent, public-interest watchdogs to ensure that this origin-tracing power is not too centralized in the hands of one state government or tech company.
What else we’re paying attention to…
A few months ago, a speaker came to a panel conversation with an unusual tact: she sat on stage without uttering a word. The speaker was a woman named Sarah Wynn-Williams, a former policy director at Meta, then-Facebook. Meta sued her for breaching her non-disparagement agreement. She has since sued them for violating her free speech, claiming she was “under duress” when she signed the agreement.
Meta is using AI to target employees who have disabilities, according to a lawsuit. The company relied on factors such as productivity and AI token usage, which disadvantaged people who missed work because of medical conditions.
When AI is a member of your family: how one single mother, seemingly primed by a dependency on Amazon’s Alexa, developed a relationship with an AI named Sapphire.
Passengers are falling asleep in Waymo vehicles leading to frantic 911 calls.
Nearly two hundred economists, Nobel laureates, and executives have signed on to a statement warning of the harms of AI, titled “We Must Act Now.” Signatories include Jack Clark, a co-founder of Anthropic; Eric Schmidt, the former chief executive of Google; and Vinod Khosla, a prominent venture capitalist; however there were no specific policy proposals included in the letter.
Even Vogue thinks Meta’s smart glasses have a surveillance problem. From Dr. Sarah Saska, who we interviewed a few months ago about Meta’s 2016 “trend”: “The immediate risks are covert and sexualized recording, filming people in vulnerable moments, capturing children, harassment, stalking, and footage that can fuel extortion or deepfakes. The ‘creep’ framing is provocative, and it points to something real … But I worry that it shrinks the problem to one creep on a train taking a picture, when the structural danger is always-on cameras becoming ordinary, human workers reviewing footage, and data pooling in an ecosystem owned by one company.”
New York is the first state to impose a one-year ban on the development of data centers, specifically those using 50 megawatts or more.
Nurses at Kaiser, the largest private employer in the state of California, are protesting the hospital system’s emphasis on AI to evaluate its performance. One commenter on a post about the story writes: “As a bedside RN, what strikes me is that Kaiser inverted the correct design. The nurse should be the judge of the AI’s output, not the other way around. When an algorithm scores a nurse’s empathy, professional judgment becomes something to defend against a machine. When a nurse evaluates what an algorithm produces, judgment stays where it belongs.”