How to Measure a Marketing Engineer: KPIs Built on What Buyers Did

How do you measure a marketing engineer's work?

The monthly report says 4,000 emails went out. The chart points up and to the right. Everyone nods.

Nobody asks the only question that matters. How many of those people wanted to buy?

Sending is the easiest number in marketing to grow. Point an AI agent at a list and it sends as many as you let it.

That is not output. That is noise with a send button.

Most lists of marketing engineer KPIs count motion: tasks automated, posts shipped, dashboards built. We grade what buyers did instead.

The term is older than the job title. Professors Gary Lilien and Arvind Rangaswamy named marketing engineering for turning data and judgment into tools that improve marketing decisions. Today's marketing engineer also builds the systems: the pipelines, the AI agents, the tests.

So we grade the work like any decision tool. Did buyers act differently because it exists?

Marketing engineer KPIs: grade what buyers did, not what the system did

Every system a marketing engineer builds throws off two kinds of numbers. Activity counts what the system did: emails sent, ads launched, pages rewritten. Signal counts what a buyer did because of it: replied, booked, bought, came back.

Andy Grove drew this line in High Output Management. You judge a salesperson by the orders, he wrote, not by the calls. He called orders output and calls activity.

One question sorts any number. Could it go up while sales go down? If yes, it is activity.

The Lean Analytics authors add a second test. What will you do differently because this number moved? No answer means the number is decoration.

Big companies fall for this too. Intuit founder Scott Cook admits in Competing Against Luck that his teams measured whatever "rolled off our servers", and missed whether customers were better off. Amazon, the same book notes, counts orders delivered, not orders shipped.

A number that can only go up is a pat on the back. It is not a grade.

Find the signal with the clone test

Not sure what the signal is? Run a thought experiment from How to Measure Anything in Cybersecurity Risk. Picture two identical companies, and give only one of them the system.

What would you see that is different? Write down every answer you could actually observe.

For an outbound email system: more replies from accounts that fit, more first calls booked, calls booked sooner. For a content system: buyers who name a post on the sales call.

The same book defines measurement as a drop in uncertainty, based on what you observe. It warns that activity, like who attended a security training, is not the result.

Swap the training for "prompts written" and the warning fits AI work too. That part is our reading.

Cannot name one thing you would see? Then the system has no job yet. Fix that before you build.

Rewrite a vanity count until pushing it moves sales

A vanity count is a number that looks like progress but cannot change a decision. Amazon learned the cost the hard way. Its retail teams were once graded on new product pages created, so they added thousands of items.

Sales did not rise. The cost of holding inventory did.

The fix, told in Working Backwards, took several rewrites. Pages became page views, then views of products in stock, then views of products in stock and ready for two-day shipping.

At every step the team asked one question. If we push this number, will the result we want follow? That is the whole trick.

We run the same rewrite on outbound. Emails sent becomes replies, then replies from accounts that fit a written profile, then fit replies that booked a first call.

Write the final definition in one sentence before anyone counts. Two people who know your market should read the same reply and reach the same verdict. If they do not, the definition is not finished.

Pair every number with the one it can break

Any number you grade will get pushed. Grove's answer was pairing: measure the effect and the counter-effect together. Inventory goes with shortages, and vouchers processed go with errors found.

The authors of Trustworthy Online Controlled Experiments sort numbers into three kinds. A goal number is what you finally care about. A driver number moves faster and predicts the goal.

A guardrail number is what you refuse to break. They also keep diagnostic numbers, used only to dig into a problem. That is where activity counts belong.

Skip the guardrail and you get the fast-food manager from the same book. He hit a perfect score on wasted chicken by cooking only after customers ordered.

He won the award. The long waits put the restaurant out of business.

Great number. Dead restaurant.

For outbound, the table below fills in all four. Volume is a clue, never the grade.

Health: a system that does not run scores zero

A great signal from a system that ran twice is luck. So every system gets a health line under its signal.

Software teams already have a model. In the survey research behind Accelerate, Nicole Forsgren and her co-authors settled on four delivery measures. They track how long a change takes to go live, how often teams ship, how fast they recover from a failure, and how often a change breaks something.

The DORA metrics guide still teaches the same four.

Their high performers were faster and more stable at the same time. Speed or safety? They got both.

Our reading for marketing systems is in the table below: idea-to-live time, launches per month, days to fix a broken funnel, and the share of launches that needed a fix.

We add one line the software measures do not have: approval rate. That is the share of drafts a person approves without rewriting.

Say the approver rewrites 60 percent of drafts in week two and 15 percent in week eight. The system is learning. A flat rate means corrections are not being written into the agent's job spec; how to evaluate AI agents covers the tests.

Count hours saved the way a finance person would

Hours saved is the number most likely to be padded. Keep it anyway. Just give it rules.

Count only work someone already did, timed before the build. Take minutes per task and how often it ran over two weeks. Time the same task after launch.

Hours for work nobody did before are not saved. They are new work. They go under signal, if anywhere.

Google's site reliability engineers call the work being removed toil: manual, repetitive, automatable work that grows with the service. They aim to keep it under half of each engineer's time.

Turn that rule on the marketing engineer's own week. If more than half goes to keeping old systems alive, the role has slid into upkeep. System two never gets built.

Do not grade them on how busy they look, either. Queueing theory says that as a team nears 100 percent busy, wait times climb toward infinity. A marketing engineer with no slack fixes nothing fast.

Put the running cost on the same line as the hours. The chart below shows why.

Why one blended score hides the truth

Sooner or later someone wants one number. Blend the signals into a single score, the thinking goes, and grade on that.

Amazon tried it. Teams got weighted formulas, like 50 percent of one number plus 30 of another plus 20 of a third.

They argued about the weights. Nobody could say what a rising line meant. Amazon went back to separate numbers, each with its own goal.

Every number is a stand-in for an idea. Keep several on the card and a broken one stands out.

Economists named what happens next. Goodhart's law, in its common form: once a number becomes the target, it stops being a good measure.

The Lean Analytics authors tell of a sales chief who paid bonuses on deals in the pipeline, not deals closed. The team chased the money. Cleaning out the junk leads took two quarters.

How to turn a marketing engineer into a spam cannon in four easy steps. Step one: pay them on emails sent. Step two: put replies in the bonus, with no definition of a reply.

Step three: drop the unsubscribe line because it looks bad in the deck. Step four: act surprised when your domain lands in spam. Congratulations, you built a damn efficient way to annoy people.

Every story in the table below hit its number. That was the problem.

When should you judge the results? After the lag, against a written forecast

Results trail the work. Grove once rated a manager superior after a superb year. The next year, the unit fell apart.

He decided the good numbers came from work done much earlier. He called them light from distant stars, and put the lag at about a year.

So write the forecast before the build. Our rule: by day 30, the signal has a one-sentence definition, a baseline from real data, and a written range for day 90.

The textbook that named the field grades models the same way. It reports one forecasting model tested on 44 new products: predicted market share averaged 7.77, actual share 7.16, with a 0.95 correlation. A forecast you can check is a forecast you can trust.

At the call, read health first, as the grid below shows. Health red with a flat signal means it never ran enough to judge: fix it and give it four more weeks.

Health green with a flat signal means it ran fine and buyers did not care. That is a kill.

Can you pin revenue on one person? No, and the card does not try. It asks whether one system's signal moved against its own baseline, and whether the next step in the funnel moved with it.

Grove's rule still applies, though. Rate the builder no higher than the output of the systems they run. The full first-quarter plan is in the first 90 days with a marketing engineer.

The Signal Card: seven lines, one screen, every Monday

The Signal Card is one card per system, read every Monday in five minutes. It has seven lines: signal, guardrail, health, hours and cost, clues, a decision, and the next test.

Two reads matter each week. Did the signal move against its baseline? Did last week's change move the number it aimed at?

If the signal fell, open the clues line. That is the only time activity earns a look. Steady sends with fewer fit replies point at the message, and fewer sends point at the system.

Where does the signal come from? For paid ads, the CRM sends booked calls back to the platform, as in server-side conversion tracking. For AI answers, it is a fixed set of buyer questions, rerun the same way every week.

Cards for every system roll up into one weekly memo in the growth cockpit. Selling on calls? Put the rates from book-a-call math under the card.

The signal should show up in those rates within a quarter. If it does not, it was further from revenue than it looked.

Build your Signal Card with AI

The prompt below builds a Signal Card for one system. Paste it into your preferred AI and answer seven questions. It shows its reasoning and never invents a number.

This week: first, run the clone test on your busiest system. Second, rewrite its main count until pushing it would move sales. Third, set the baseline and write the day-90 forecast.

Not sure which system deserves a card first? The marketing engineering audit finds it.

Want the card built in from day one? That is how our marketing engineering work runs.

Back to the 4,000 emails. The question was never how many went out. It was how many people wanted to buy.

Count the fish, not the casts.