Product: Inference Insights, formerly HawkEye, brought model signals and analysis into a shared investigation for recommendation and ranking models across several Meta products.
Scope: One-quarter alpha → extensible internal platform, approximately 2021–2023+.
My role: Solo founding product designer across strategy, research, information architecture, interaction design, and validation.
Impact: Meta reported “order-of-magnitude improvements in time spent debugging production issues” during its first two years. (specific metrics can not be published)
Example screen of Inference Insights (Details changed or abstracted for NDA)
Central design challenge
How might we help on-call engineers trace a production failure across the ML pipeline—without manually correlating evidence across 10–15 disconnected tools or relying on a senior expert to guide the investigation?
02 Problem reframing
The evident problem was tool fragmentation
Before Inference Insights, a model incident brought model owners, data on-calls, and infrastructure engineers into a shared document. They pasted screenshots and links from custom notebooks, dashboards, and command-line tools—often ten to fifteen per investigation. With no shared timeline or terminology, evidence had to be correlated by hand.
Investigation fragmentation
The process lived largely in expert memory. When an incident arrived overnight or on a weekend, a new on-call could inherit a war room with no map.
The deeper problem was an absent investigation model
Foundational interviews and usage analysis, reinforced by ongoing feedback and product data, revealed that consolidating dashboards would not solve the deeper issue. Engineers lacked a coherent investigation model that preserved context across tools and handoffs, and made the path from symptom to cause understandable to every responder.
They needed to trace a symptom through the model version, input signals, upstream data, and training process while retaining direct access for experts. The work became a shared operating model across teams, not another observability surface.
Expertise was unevenly distributed
Experienced on-calls carried the tribal knowledge that helped them move quickly; newer responders had to reconstruct it during an incident. The platform needed to meet both where they were:
Experienced on-calls needed speed: direct access to the right evidence under pressure.
Making expert reasoning visible without slowing experts down or forcing nonlinear investigations into a rigid flow became the design tension.
Newer on-calls needed guidance: enough context to progress without waiting for an expert.
03 Building alignment
Working as the connective layer
Within Meta’s Prediction Robustness program, I was the only designer dedicated to the platform. I connected the core product team with the on-call users, model owners, infrastructure partners, design peers, and leaders whose decisions shaped the investigation experience.
Collaboration model
My role was not simply to collect requirements from those groups. It was to turn competing local needs into a product direction the teams could share: what should be standardized, what needed flexibility, and which complexity belonged in the platform rather than with the user.
Research as a continuous operating loop
Research moved through Understand ⇆ Ideate ⇆ Validate ⇆ Iterate as a continuous loop. Before launch, interviews and existing-tool usage mapped how investigations unfolded and where engineers lost context. I turned those findings into competing concepts, then tested clickable prototypes with the same users.
After launch, product usage and direct feedback fed the next cycle, shaping immediate fixes and later releases.
Qualitative: More on-calls than expected needed help interpreting insights, not just finding the right metrics or results. That finding led to more inline guidance.
“Now the flow and interface is intuitive enough that doesn’t require too much domain knowledge to use.”
— ML engineer and ongoing customer partner, 2023
Quantitative: Unusually long time on a screen or step prompted a friction check, not a claim of engagement.
04 Designing the system
1
Hypothesis validation
- Identified focused analysis phase to test (Feature upstream traceability)
- Rolled out alpha to users in under 1 quarter
- Validated core value
- Secured leadership backing /Unlocked formal investment
2
Guided investigation
- Transparent, traceable investigation stages/ visual investigation flow
- Investigation guidance
- 1-2 investigators are able to complete investigation in hours instead of days.
3
Automated analysis
- Minutes to root cause for many use-cases
- Unlocked deeper layers of investigation ie. model parameters/ model internal states (MIS)
- Complex cases remain reviewable
4
Platform + self-service
- Formalized into a modular component based platform
- Self-serviceable/ customizable workflows, user onboarded metrics and analyzers
- Broader adoption
- Fewer bespoke customer solutions
05 Scaling outcomes
In its public engineering account, Meta reported that HawkEye helped cut production debugging time by an order of magnitude in its first two years. It also reported a several-orders-of-magnitude reduction in the time from triage to isolating relevant serving inputs. The toolkit supported recommendation and ranking models across multiple Meta products, while guided exploration let less experienced responders investigate complex problems with less reliance on specialists.
The fourth outcome was organizational. A shared investigation model reduced reliance on tribal knowledge, while self-service patterns let partner teams bring in new needs without routing every addition through the founding group. Over time, the platform no longer needed a standing dedicated team. It scaled by absorbing work the team would otherwise have had to keep doing.
Business impact
Eight-figure annual impact*
Shorter investigations meant model regressions cost less before they were resolved.
Debugging velocity
Order-of-magnitude improvements*
Critical investigations condensed from days to hours
Root cause isolation
Days of searching, cut to a fraction*
Engineers could trace a bad model to the inputs behind it far faster.
Guided triage
One investigation, two ways through
Guidance made complex triage less expert-dependent.
Self-serve @ scale
Removed the “founding team” bottleneck
New capabilities and workflows no longer required a standing team.
* Published outcomes: Meta Engineering, “AI debugging at Meta with HawkEye” (2023). Business impact and fourth panel: Robin’s account. Illustrations are not to scale; exact internal metrics cannot be shared.
06 Learnings
I knew the design worked when a major incident was resolved in Inference Insights while I was off-grid on a camping trip. No one needed the platform’s designer to explain how it worked. The workflow stood on its own.
A platform can succeed by making its creators less necessary: success is whether it preserves enough context, guidance, and flexibility for others to operate without the founding team.
Observability must explain, not just report: teams need to connect a production symptom to the model, inputs, and upstream conditions behind it.
The highest-leverage design work lives between interfaces: the shared mental model, and the boundary between guidance and direct access, mattered more than any single screen.
Design for contribution, not just consumption: defining the rules, defaults, and quality bar let partner teams extend the system without waiting on the core team.