I asked Nick Ursa, our guide data engineer (and MotherDuck co-founder), how AI had changed his job. His answer: "This is the archetypal period in my occupation alongside no backlog."
You'd think that would depart him alongside plentifulness of period on his hands, but he's busier than ever. While the backlog is gone, the answers motionless have to be right, and a individual motionless has to decide how much norm authority to do. Whereas formerly upon a period his job was to create certain the numbers he was computing were right, now his job is to create certain that the numbers that an LLM or delegate comes up alongside are right.
That's a distinct job from operating a ticket queue. At Data Outpost on November 4, I projected a name for this new role: the Data Intelligence Engineer. I think it's now the emergent function on an AI-enabled data team, and I’d akin to portion why.
What a data squad is for?
A data squad has one job: allow the endeavor to create fine decisions. The faster they can create those decisions, according to a additional exact image of what is happening, the improved those decisions can be.
The difficult part has been the path a inquiry takes to rotate into an answer. It moves akin a relay. A promotion chief comes into activity on Monday and notices that there was a spike in churn complete the final week. Why did that happen? They document a ticket alongside the data squad to investigate. An expert picks up the ticket, and as they dig in, they acknowledge several key data is not exported from among the data sources. They document a ticket to ask a data engineer to appearance into it. That needs a alter in among the data pipelines, so the data engineer makes the change, tests it, runs the backfill, tests the backfill, and resolves the ticket. Then the domain is rear in the court of the analyst, who can build the dashboard that was needed. By the period it ships, the promotion individual can no longer recall why they asked in the archetypal place. The churn charge restored a brace of weeks afterward on its own.
This isn’t calling anyone out as not functioning efficiently or urgently; nobody in that relay did item wrong. But all handoff method waiting, environment is lost, and the half-life of the capability to efficiently act on the answer to the inquiry gradually decays.
As an aside, we should note that there was a entire generation of AI enabled tools that didn’t really help; the co-pilot era made several of the turnarounds faster, but the issue was that there were too many handoffs to be capable to act quickly enough.
Something is distinct this time
Before December 2025, the case against AI for analytics was uncomplicated to make. LLMs wrote bad SQL. Business logic was unknowable. The sensible decision was that this was never going to work. In fact, I used to conversation concerning how data teams were additional or small exempt from the disruption that was happening in another areas of software.
Then, all of a sudden, the dirt moved. New models got improved at SQL. Business logic turned out to be item an delegate can fig out alongside several helpful context. And agents went multi-turn, which method they don't have to get it correct the archetypal time. They can try a query, appearance at what came back, and try again, fair akin a individual expert would do. Put those together and it … fair … works.
Self-service BI has been correct about the border for decades, what makes this different? Well, as a bit of an beingness proof, here’s an example; the display I gave at Data Outpost was not created as a powerpoint phase or Google Slides, it was an app that I built using MotherDuck and Claude. I used live datasets from four distinct data sources, built a figure of interactive and animated visualizations. Anything I did could be done by practically anyone; I didn’t appearance at a sole row of code or SQL.
It is crucial to emphasis that this isn’t fair a way to automate construction your dashboards. BI is typically beautiful static, alongside perchance a brace of drill-downs. But alongside AI, you can build much richer visualizations. And what’s more, if you can build a dashboard alongside a prompt, you can go beyond that and additionally build a full-blown app.
Allowing endeavor stakeholders to answer questions alongside data on their own has an equal broader advantage than fair shortening the data team’s backlog; it really allows them to create improved decisions. By cutting downward the amount of period it takes and the figure of handoffs involved, they can ask additional questions. They can ask prosecute up questions during they motionless have all of the environment in their heads. This method they can create assured decisions faster.
Which of these jobs can be a prompt?
AI is dramatically changing the job data workers perform. Software engineers frequently discover their period being spent babysitting Claude sessions and rubber stamping agent-written code. What concerning the traditional roles of the data team? Which of their jobs, to put it a bit provocatively, can be a prompt?
As uncomfortable as the answer power be, fairly a lot can be done via a prompt. In the not too distant future, penning SQL queries, creating dashboards, and construction dbt pipelines are not going to be things that necessitate specialized cognition to do. Agents are apt improved than humans at these tasks already, and they’re lone going to get better. Our Data Engineer, Nick, moved us from one type of Airflow to another in a weekend, alongside minimal supervision, which formerly would have taken a week.
Writing code or SQL or construction dashboards was never the entire job, though. The need for exact data and the capability to comprehend it doesn’t go away. And person have to be accountable, who can sign off and say “yes, this is right”. That takes a individual in the loop. Quoth Nick, one additional time, “Data eng, despite of tooling or who does it, is an accountability function."
What motionless needs a human
Once agents are penning the code, what’s remaining in the data world? It turns out fairly a lot. Humans need to create certain that another stakeholders without the earth “Data” in their heading are capable to resolve problems alongside data. That method they need to justify a spotless data example alongside suitable environment and annotations, a apparent ontology to justify that conditions and concepts are being applied consistently throughout, and additionally they need to create certain that things continue to work.
These aren’t things that a example can intuit from data or history. A example knows SQL. It doesn't cognize how your business counts revenue, or which of multiple similar-looking tables is the genuine one (it is revenue_cpx_new_revised_v2_use_this_one ) . A example doesn’t cognize that whenever the revenue squad talks concerning income they average commissionable revenue, and the backing squad uses item different.
Context is a crutch that is difficult to maintain. What’s equal improved is fair construction a spotless data example in the archetypal place. There is a fine difference alongside software; I lean to regard it a person cognition nonaccomplishment any period any period your person has to peruse the manual; a cleaner scheme would have made any they were trying to do obvious. Context is fair a manual for agents; if you can create the data example uncomplicated and apparent enough, it can preserve them a lot of period and energy (and tokens).
Meet the Data Intelligence Engineer
The jobs to be done for a data individual in the age of AI don’t really equivalent any of the representative responsibilities of another data squad members. I think it is distinct adequate to deserve a new title, which I hereby dub the “Data Intelligence Engineer”
The Data Intelligence Engineer automates the data engineer's activity (ingestion) and the analytics engineer's activity (transformation and modeling) by creating and conducting application agents. They encode their cognition of the scheme into shared environment that can be used by another agents to answer questions concerning the organization. These contain things akin which sectors are joinable but additionally what person in the business method whenever they say “ARR”.
Evals are another big part of their responsibility; these are the scheme tests that validate that whenever you ask questions alongside a known answer, the answer is correct. A fine eval suite can provision confidence that the types of questions that users ask of the scheme are going to be answered correctly by the agent. Evals additionally are a way of knowing whenever item breaks.
The goal of a Data Intelligence Engineer is to allow self-service. That is, anyone in the institution have to be capable to open Claude or ChatGPT or another delegate and answer questions correctly concerning the business, use that to build apps and visualizations, and in the end, create elevated norm decisions alongside the advantage of data. . When the Data Intelligence Engineer does the job well, everyone who asks a inquiry gets the cognition of operating alongside the smartest expert in the company.
Day to day, that method they are going to be construction frameworks, construction the data model, curating the context, preventing things from getting broken, fixing them whenever they interrupt anyway, operating evals, and being accountable for the surfaces the endeavor depends on. We wrote the entire item up as a job description, since we accept this is the way things are going and it's period to provision it a name.
If you're a data engineer, an analytics engineer, or an analyst, you'll acknowledge parts of this. Those roles were already blending into all other, and morphing into item new. This is a function you play, any your heading says.
Won't this be chaos?
It's a fair question. More group construction method additional chances to be wrong, alongside small oversight. There is the possible of making stupendously silly mistakes by acting on the incorrect data. I’ll depart it up to the audience whether they think that stupendously silly mistakes are item new or not. The scary item is that now the data squad can be held liable for them.
There are two things that help. The archetypal is that the nearer you are to the problem, the improved you can place errors. The individual who lives alongside the churn figure notices whenever it looks off. From experience, we’ve established that results really enhance since the endeavor owner has a fingertip awareness on their area; they cognize whenever item isn’t right, and can frequently place problems equal in the origin data.
The another item that helps is that you don’t need to observe everything alongside the identical flat of vigilance. Another quotable citation from our companion Nick is that “Agents need supervision proportional to their blast radius." That is, if item is going to be used for crucial reporting or decisions, you should pay a lot of attention. If you’re trying to fig out how many sandwiches to command for lunch, it likely isn’t going to logic a riot if you command too many alongside habanero mustard since Claude imagines it is additional tasty.
You can think of this in three circles of supervision. The external is the Wild West. Anyone can build, they hopefully build on a sensible foundation, but the data squad keeps their hands off. The next tier is the Curated set. This is a set of dashboards, visualizations, apps, and metrics that is crucial to the endeavor in several form or another. When item breaks in these dashboards, the data squad should create a finest attempt to fix them. Finally there is the inner circle, the Core. This contains key metrics for the organization, things that are crucial to get right. The data squad should create certain that these are elevated norm and predictable at all times. The commission figure lives in the Core.
What does this all have to do alongside MotherDuck?
On MotherDuck, we build mechanisms to empower your AI enabled data team. Flights are vibe-codable pipelines that can drag data into your storage and translate it into a sane model. Context lives in Guides, which are markdown records describing your data example and your business. A Guide starts out as private, but an admin can advance it to use to the entire institution so that all individual and delegate plant from the identical definitions. We call the data visualizations that can be created by agents “Dives”. Every individual and delegate gets its own Duckling, a dedicated DuckDB instance. All of these are reachable from SQL, so one individual or delegate can activity a issue throughout ingestion, transformation, context, and apps without a handoff between tools.
The lone way out is through
We're in a disorderly moment. Chaos breeds the biggest opportunities, and the tools hand you several genuine super-powers if you're consenting to choice them up. We're all learning how to do this together.
The data squad isn't going anywhere. Someone motionless has to own the data model, curate the context, keep things working, and answer for the results. No-backlog-Nick has additional to do than ever, and but the job is changing and not going back.
So who's doing this at your company, and what do you call them?
PS - if you desire to see my slides from the talk, inspect out the Dive Gallery
