Skip to content
← Notes
3 min readoperations

What actually happens when a sourcing run goes sideways

A places_hunter query came back thin for two hours before anyone noticed, and the fix was smaller than the alert.

The first sign was not an error. It was an absence. Solara Pro’s Tuesday sourcing run finished on schedule, logged a clean exit code, and handed back nineteen prospects instead of the low hundreds a normal run for their territory produces. Nothing crashed. Nothing retried. The run just quietly did less work than it should have, and every downstream system treated nineteen as a legitimate answer because, technically, it was one.

This is the failure mode that matters most in autonomous sourcing: not the crash, which pages someone immediately, but the degraded-and-still-green run. A query that returns zero rows looks broken. A query that returns a plausible but wrong number looks like a slow week.

We caught it because someone was looking, not because anything alerted. Connor happened to open the sourcing dashboard two hours later to check on an unrelated tenant setting and noticed the prospect count for that day looked off against the trailing average. That is not a process. That is luck wearing a process costume, and it is exactly the gap a places_hunter yield-floor check exists to close now.

The actual cause, once we traced it, was almost embarrassing: a geography filter upstream had been narrowed by a config change meant for a different territory entirely, and it silently applied to the wrong query shape. The API call was well-formed. The credentials were fine. The rate limiter never fired. It asked a narrower question than intended and got a narrower answer, and every layer between the query and the dashboard passed that narrower answer along as if nothing had gone wrong — because from each layer’s point of view, nothing had.

The fix took four minutes once we found it: revert the filter, re-queue the run, done. Finding it took most of an afternoon, because we were looking for a bug and what we had was a bad assumption wearing correct syntax.

What changed afterward was not the query logic. It was the check around it. We added a floor: if a sourcing run for a given territory returns fewer prospects than some fraction of its trailing seven-day average, the run gets flagged for review before its output is treated as final, instead of flowing straight into scoring and outreach queues. Low output is not automatically wrong — some weeks a market really is thinner — but low output that nobody looked at is a risk we do not take anymore.

The uncomfortable part of writing this down is admitting how normal the failure was. There was no bad actor, no external outage, no exotic race condition. A config value meant for one territory leaked into a query for another, and every system involved was behaving exactly as designed. That is the actual shape of most production incidents in a system like this: not the system breaking its rules, but the rules being quietly wrong for one case nobody wrote a check for yet. The job, most weeks, is finding the next one of those before a dashboard has to.