Why dashboards aren't enough anymore (and what AI agents unlock instead)

In the previous article, I described how the role of a Data Analyst is changing over time and how tedious tasks that previously occupied an unhealthy amount of time in an analyst's day are now being automated. The focus of the article was on what is the essence and actual scope of Data Analysts' work, but it did not dive into detail on what can be automated and how to leverage AI beyond these routine tasks, and give it real autonomy to work as a colleague (or an intern, if you will, or want to still have enough control).

Olga Berezovsky, in her Anticipating 2026 piece in the Data Analysis Journal, puts the shift simply:

The job becomes powering decisions for both humans and AI. Success is measured by the number of decisions you can make that are fast, causal, and safe to automate.

Benn Stancil on his Substack is even more direct:

"Have a drink with a person who works in technology, and they will eventually tell you how they feel about their job: They are stuck in a white collar salt mine. The majority of their day is spent slogging through grunt work that is, if not beneath them, beneath their potential. They were hired to do higher impact work; more strategic work; more valuable work."

And then, specifically about analysts:

"Analysts will say they want to be looking for strategic insights, not making yet another dashboard."

 So what work is there left for analysts beyond dashboards (that everyone can now pull with ease using AI)?

The dashboard problem

A sharp observation from one of the posts I came across:

This pattern isn't new, as dashboards have been multiplying faster than they've been read for the better part of a decade. What's new is that the conditions behind it are finally changing. Until recently, dashboards were the only way for a non-analyst to get a look at the data, and if a producer wanted to see retention by acquisition channel, somebody had to build it for them. If the CFO wanted to see it by cohort revenue, somebody had to build it again. Each new ask created another dashboard that would get looked at once or twice and then quietly abandoned.

In the AI era, that bottleneck disappears, and anyone with access to a well-structured data model and a decent AI assistant can generate the view they want in minutes. Which means the role of the data team shifts from the people who build dashboards to the people who decide what's worth measuring, define how it should be measured, and own the system that lets everyone else explore it safely.

That's a much more interesting job. And it's also a much harder one.

Did dashboards ever really work?

A bold question - I know. But for as long as data teams have existed, trust has been the quiet problem underneath everything they produce. A stakeholder opens a chart, sees a number, and has to decide whether to act on it. Nobody has time to audit the SQL that generated it, re-derive the metric definition, or trace back whether the event schema changed last quarter. So trust ends up being granted by proxy: the chart looks reasonable, the dashboard looks well-made, the person who built it has a good reputation, and that has to be enough.

Dashboards never really solved this, and they mostly moved the problem out of sight. A clean chart in a familiar dashboard looks like an answer, which makes it easy to stop asking where the answer came from. Benn makes this point sharply in a separate piece:

There's no way to know if a chart "works" unless you understand how it was built; the validity of the external product depends on the validity of its internals. Is the source data reliable? Was the math done correctly? Is every formula correctly encoded? We can't "test" a chart, or verify that it's correct on its own merits. Instead, we can only judge it by proxy: Do its conclusions sound plausible enough? Does it look like it was carefully made? Who made it? And then we have to choose if we trust it, or not.

That same problem applies to AI-generated analysis, and arguably more so. A chart in a dashboard at least sits inside a system someone built deliberately. An AI agent spinning up an answer on the fly has none of that context unless someone built it in. Which brings us to the actual job the data team now has.

What data analysts and other data roles are now supposed to do

Once stakeholders can generate their own views, the data team’s value shifts upward. The work moves away from execution and toward judgment, ownership, and infrastructure.

The first place the hours go is into framing. When a producer asks “is the new onboarding working?”, that’s not actually a data question yet. Somebody has to decide whether “working” means D1 retention, tutorial completion, first-session length, first-purchase rate, or some combination. AI will answer whichever version it’s given but picking the right one is a human call, and it shapes everything downstream.

Close behind framing comes ownership of what the metrics even mean. Definitions drift as the game evolves, and nobody notices unless someone’s paying attention. What counts as an “active” player when a new game mode ships? When PvP launches, does “session” still mean the same thing? Agents don’t catch definition drift. Analysts do, and when they don’t, the next board update gets built on numbers that look right but aren’t.

That ownership extends into the data model itself. Agents need a well-defined semantic layer with clear relationships, clean naming, and documented business logic to produce reliable answers. The better the foundation, the better everything the AI does on top of it. The reverse holds too: agents will confidently query garbage data and produce confident garbage answers, which makes data quality more valuable.

Experimentation design is the other obvious place where the hours go. A/B interpretation is table stakes now, and agents handle most of it. The judgment calls happen earlier: what to test, when, how long, which cohorts to exclude, whether a result will hold at scale. The leverage grows when analysts stop running tests one at a time and start building the experimentation infrastructure that lets producers, designers, and UA managers run their own tests safely.

Then there’s the newer category of work that didn’t really exist a year ago: validating what AI generates. When anyone can produce an answer in minutes, the question of which answers to trust becomes the whole game. Someone still has to catch the join that joined the wrong tables, the cohort that was defined slightly off, the correlation that’s actually noise. This is the new quality assurance layer, and the people who are good at it are becoming the most valuable members of the team.

Right alongside sits the role of pushing back on requests that would mislead. The CMO wants to see conversion by day of week. The analyst’s job is to say “the sample size won’t support that cut, let me show you the version that will.” Agents build whatever is asked. Analysts save the business from asking the wrong thing in the first place.

And finally, there’s the layer of context that no warehouse contains. An App Store policy changed. A competitor shipped something that pulled your audience’s attention for two weeks. A seasonal pattern isn’t really seasonal, it’s a one-time Reddit mention. Agents only see the data. Analysts bring in what they read on Slack, what they heard at the all-hands, what a colleague mentioned over lunch, which is often the thing that explains the number.

The through-line: everything above requires judgment, context, or taste that AI doesn’t have. The mechanical layer being automated frees up the analyst to spend their day on work where their human experience is the actual input. The old week was 80% execution and 20% judgment. The new one flips that ratio.

Where this is already happening

If you want to see what this actually looks like in practice, the best current reference is Simon Willison's recent conversation on the Talking Postgres podcast about AI for data engineers. He walks through specific examples of what the current generation of AI tools can do when given real access to a database, including spinning up a 150-line SQL query with regular expressions in minutes (a question that would have cost him an hour or two of work, and therefore wouldn't have been worth asking at all), and running agents in a loop that investigate a database, propose their own queries, and build a visualization of what they find.

Simon's framing of the shift is the one I'd land on:

AI makes me more ambitious in both my project and my side project because all of these things where you'd normally think, how long would it take me to do that? That's just not worth it. The time for those kinds of tasks reduces so much.

That's the full picture of what dashboards never gave you. Dashboards exist for questions people asked often enough to justify building them. All the questions that weren't frequent enough to warrant a chart went unanswered. AI agents with real data access close that gap. For a game studio, this is the difference between answering only the questions that fit into a weekly review cycle, and answering the questions that come up in the middle of a LiveOps event when the ops team actually needs to know.

Of course, giving an AI agent that kind of access to your game data raises an obvious question of its own: how do you do it safely? That's a separate conversation, and one my colleague Francesco Perrone, who leads security and compliance at GameAnalytics, just wrote about. If you're thinking about putting any of this into production, read his piece next.