Synthetic content though, is just part of the problem. The disruption of AI also means new opportunities for automated distribution through coordinated inauthentic behavior, making industrial-level deception through bots accessible to a wider set of actors and allowing those who were already present in the space to make the most out of their considerable skills for mass manipulation.

Crucially, as AI lowered access barriers, the companies behind the most popular social media companies have also made it even easier for information integrity threats to thrive on their platforms. In a system where the visibility of user-generated content is dictated by engagement-oriented algorithms, Big Tech is constantly rewarding the kind of content that is most likely to be untrue and doing so in different ways. This article will show insights into AI threats to information integrity, such as deepfakes on TikTok or customer fraud on Meta, with a final assessment of what is the real impact made by civil society organizations (CSOs), who try to fight it.

AI & Monetization: How TikTok Funds Attacks on Information Integrity

Not everyone, but many of the people who dedicate their lives to producing and disseminating falsehoods online do it for the money. Those “creators” sit at one of the angles of the attention economy: they produce content that is so attention-grabbing, so outrageous, so incredible that it is often made up. The more engaging it is, the more it will be rewarded by the other two points in the triangle: users and platforms.

In terms of user consumption, disinformation is like any other kind of engaging fiction: surprising, exciting, enraging. This intentional deception that makes it dangerous, as it easily seduces the public. AI is the perfect tool in the hands of serial purveyors of false, sensational stories because they know what works in getting attention online; AI just gives them the ability to create their messages and distribute them more quickly and efficiently, with a level of technical quality that makes the material indistinguishable from genuine content.

Platforms, on the other hand, reward anything that is marked as engaging and proceed to give it even more visibility. It is what is known as “algorithmic amplification”: the more users interact with something, the more prone the algorithm is to show it to even more users. Why? Because the commercial interest of the platform lies in the users spending more time on its service. That is also why platforms reward the users producing precisely that kind of content.

A few months ago, at Fundación Maldita.es we started noticing something weird: super realistic AI-generated videos of political protests were popping up on TikTok feeds all over the world. It piqued our curiosity, and we ended up with a list of 550 accounts that had posted more than 5,800 videos referring to 18 different countries. The videos had generated more than 89 million views, but the most interesting part came when we talked directly with some of the people behind them.

They certainly did not seem to feel bad about misleading people, but they were crystal clear about something even more important: they did not have a political agenda. Their only goal was for their accounts, often several at the same time, to reach the magic number of 10,000 followers. Why? Because it was what they needed to qualify for TikTok’s “creator rewards” program through which the platform pays the influencers. Political disinformation, in this case, was just a tool. They had just found out through trial and error that there was nothing that TikTok’s algorithm would amplify more. The people we talked to had tried other stuff: pets, babies, you name it. However, in their quest to reach 10,000 followers, it was the emotional and false political content that worked best.

In essence, the financial interests of both TikTok and these users are in alignment. Highly emotional or surprising content is sure to attract attention, and attention means money: for the creators, through TikTok’s own monetization program; for TikTok, through content that makes users spend more time on the platform, leading to, for example, clicking on ads. AI just makes the whole thing easier. TikTok’s policies prohibit AI‑generated content that is “misleading about matters of public importance”, but those policies are simply not applied.

AI and Information Integrity: Unreliable Systems

The financial incentives provided by the biggest digital platforms don’t stop at creator reward programs AI has also completely supercharged the already thriving scamming industry in most of those online spaces: falsifications of products are more realistic and easier to do than ever, with credible brand and store impersonation just one click away. The platforms where the scams are taking place are just unprepared for this kind of challenge.

In Fundación Maldita.es, we routinely find massive networks dedicated to consumer fraud on the major platforms. Recently, we documented how in the span of 3 months, 170 supposedly “verified” Meta accounts posted over 67,000 of different scams, in Europe alone. The accounts advertising the frauds had been either “stolen” from their legitimate owners through hacking or, even worse, had been “verified” as being the genuine profiles of different public figures who had nothing to do with them.

Again, Meta had policies in place that should have prevented this from happening but, similarly to TikTok, they were not enforced. It is also not a problem with only these two platforms: we have also documented how YouTube finances climate misinformation against its own policies or how a massive scam campaign on X reached 76 million impressions. None of those successful manipulation schemes would have been possible without the use of AI.

AI and the Real Impact

As the wider adoption of AI creates immense challenges for information integrity, AI tools must also be used to address those challenges and document the harm. Moreover, organizations and individuals who want more accountability need to change their mindset to evolve from “uncovering” AI-facilitated abuses to finding ways to actually rectify them.

As most global tech companies scale back in social responsibility, researchers of online harms and advocates for their mitigation must carefully consider which strategies have a better chance of actually improving things for users: regulatory oversight, strategic litigation, public pressure campaigns, or development of alternative tools and online spaces.

This is a process Fundación Maldita.es has gone through. In all of the investigations mentioned in this article, we documented thoroughly all the available evidence, not only to provide examples of malpractice for those reading our analyses, but also to build consistent datasets that can meet the standard of evidence in court or to provide solid footing for regulatory action.

We’ve learnt that in the AI era, the “impact” of our investigations cannot be simply measured by the number of engaged users, referrals to our findings, or reporters who reach out to follow up on them. Our impact (and that of anyone interested in a safer information integrity environment) needs to be considered in the light of its contribution to change in the attitudes, rules, and actions of the handful of dominant AI tools tech companies.

Some will engage with the democratic authorities or independent regulators; others will use the evidence obtained from the investigation to raise public awareness, and lead community organizing that puts pressure on these companies. One thing is clear: in this new era, the real impact doesn’t result only from criticizing, but also from building a strategy for change.

Using their platforms and experience, civil society organizations can fight for this change and put pressure on social media platforms, such as TikTok and Meta, to modify their strategies towards AI deepfakes as well as their compensation and false advertising policies.

Your Feedback Matters

What did you think of this text? Take 30 seconds to share your feedback and help us create meaningful content for civil society!

Want to learn more about the reports? Have a look!

About The Author

Carlos Hernández-Echevarría is a journalist with 15 years of experience in television as a reporter, correspondent, and program manager. He is a member of the permanent task force of the Code of Practice on Disinformation and the EDMO working group on disinformation. He holds a degree in Journalism from Universidad San Pablo CEU and a Master's in Elections and Campaign Management as a Fulbright Fellow at Fordham University.

Disclaimers

This piece of resources has been created as part of the AI for Social Change project within TechSoup's Digital Activism Program, with support from Google.org.

AI tools are evolving rapidly, and while we do our best to ensure the validity of the content we provide, sometimes some elements may no longer be up to date. If you notice that a piece of information is outdated, please let us know at content@techsoup.org.

This content was created with AI assistance and has been reviewed and edited by Carlos Hernández-Echevarría.