A founder we talked to last year had already burned four months and a meaningful chunk of a seed round on a mobile app that still crashed on login. The agency’s proposal deck had been beautiful. Their case studies had logos everywhere. Nobody on the founder’s side had asked a single technical question before signing, because nobody on the founder’s side was technical enough to know which questions mattered.

You don’t need a CS degree to catch a bad dev shop before you pay them. You need about fifteen minutes and six questions that force them to be specific instead of persuasive. Salespeople and account managers stumble on these. Engineers who actually build things answer in one or two sentences without reaching for a slide.

The questions, and what the two answers sound like

Ask these in a first call, before any contract talk. Don’t announce that it’s a test — just ask them plainly, the way you’d ask about timeline or price.

  • “Who specifically will be writing the code, and can I talk to them directly during the project?” Red flag: “our team” or “our delivery unit,” vague and plural, with all communication routed through a project manager. Good sign: a name, and a yes to direct access — even informal access like a shared Slack channel — without hesitation.
  • “Show me an app you’ve shipped that’s live in the App Store or Play Store right now, not a demo build.” Red flag: they show you a Figma prototype or a staging link and call it “shipped.” Good sign: they hand you a real store listing, install it in front of you, and can describe one specific technical decision they made in it and why.
  • “What’s the biggest thing that went wrong on a past project, and how did you find out about it?” Red flag: “nothing really, we’re pretty thorough” — nobody who’s shipped real software has a clean record, and claiming one means they’re either new or not being honest. Good sign: a specific failure (a crash rate that spiked after a release, a third-party API that changed under them) and a specific detection method, not just “the client told us.”
  • “If you were building this, what would you build first, and what would you deliberately leave out?” Red flag: they agree to build everything in your spec exactly as written, no pushback. Good sign: they push back on scope and explain the sequencing — this reveals whether they’ve actually thought about your product or are just pricing out a document.
  • “How do you handle security for [the specific sensitive thing in your product — payments, user data, AI features]?” Red flag: “we follow best practices,” said with no elaboration. Good sign: a concrete mechanism — how API keys are stored, how they’d prevent a prompt injection if you’re using an LLM feature, what they log versus what they encrypt.
  • “What happens to the codebase and credentials if we stop working together?” Red flag: hedging, or “we’d need to discuss that.” Good sign: an immediate, unambiguous answer that you own the repo and all accounts from day one.

Why these six and not a longer checklist

Each question targets a different way dev shops fail founders. The first two catch the classic bait-and-switch, where the senior people who pitch you disappear and junior contractors you never meet do the actual work. The third and fourth catch the shops that are optimized for sounding agreeable rather than for building something that survives contact with real users. The last two catch the shops that quietly hold your project hostage — either through knowledge locked in one person’s head, or through infrastructure and credentials you never actually control.

None of these questions require you to understand code. They require the other side to be specific, and specificity is the one thing a shop that hasn’t actually built anything can’t fake convincingly for more than a sentence or two.

What to do if the answers are vague

Vague isn’t automatically disqualifying — some agencies are genuinely good at delivery even if their first-call answers are polished rather than technical, because a founder or salesperson is doing the talking instead of an engineer. The fix is simple: ask to get the actual engineer on the next call, and ask these same six questions again. If that request gets deflected, or if the answers change and get vaguer under a second pass, that’s your real signal. A shop confident in its own work will happily put an engineer in front of you. One that isn’t will find reasons not to.

We built orithLabs around the assumption that founders will ask exactly these kinds of questions, because we’ve been on the other side of a stalled build ourselves. Apps like Crumb Count and Kompete exist as answers you can install and poke at, not slides you have to take on faith, and the two of us who wrote the code are the ones who’ll be on your call. If you’re mid-evaluation and want a second technical opinion on a proposal you’ve already received, that’s a conversation we’re happy to have even if it doesn’t turn into a project.