Marketplace @knack vendor-evaluation

Vendor Evaluation

Weighted scorecard + five-area diligence (security, reliability, pricing, exit, roadmap) + recommendation memo. References from real customers, not vendor-supplied.

v0.1.0 by @knack (Knack) vendor-evaluation

Install with the knack CLI: knack pull @knack/vendor-evaluation — then it runs in Claude Code, Cursor, Codex, or any agent that reads the open Anthropic Skills format.

A scorecard-driven evaluation that surfaces the load-bearing trade-offs, not just feature checkmarks. Optimized for B2B software but generalizes to most services with a contract.

When to use

  • Choosing between 2–4 finalist vendors after a longer list
  • Re-evaluating an incumbent (renewal time)
  • Writing the recommendation memo for a stakeholder who didn't sit in the demos

Skip for: free open-source picks (lighter-weight decision), one-off services under $5k/year (the evaluation costs more than the contract), commodity vendors where switching is cheap.

Workflow

  1. Define what "good" looks like before scoring. Write down the 4–6 criteria that actually matter for THIS purchase. Most vendor scorecards online have 20 criteria; that diffuses signal. Pick what's load-bearing: e.g., for an observability tool, often it's (a) ingest cost at our volume, (b) on-call surface, (c) integration with our existing alert routes, (d) data retention policy.

  2. Weight the criteria. Not all equal. The top criterion should be ~2x the bottom one. Force-rank if needed.

  3. Score each vendor on a 1–5 against each criterion, with one-line evidence.

    • 5 = clearly best in class on this criterion
    • 3 = adequate
    • 1 = blocks the deal

    Without the evidence line, scores are vibes. Real vendor scorecards have a paper trail.

  4. Diligence the finalist on five concerns (the standard list):

    • Security: SOC2 / ISO27001 status, recent pen-test date, how they handle a customer data breach
    • Reliability: SLAs in the actual contract (not the marketing page), last 3 major incidents and their RCAs
    • Pricing: total cost of ownership at OUR scale (not their starter tier), what triggers a tier change, what's negotiable
    • Exit cost: data export format, retention of data post-cancellation, migration paths (real ones, not the migration "ebook")
    • Roadmap commitment: a specific feature that matters to us; is it on their roadmap with a date or "we're considering it"?

  5. The reference call. Two customers, NOT the ones the vendor offered. Ask: what surprised them in year 1, what would they have negotiated differently, what's still broken.

  6. Write the memo for one specific reader. Identify the decision-maker. Open with the recommendation in one sentence. Then: the 2–3 things that drove the call. Then: the meaningful runner-up and what would flip the decision.

Memo shape

# Vendor recommendation: <name>

**Recommendation**: Sign with <vendor> for <contract scope>, $<amount>/year.
**Decision-maker**: <name>. **Decision needed by**: <date>.

## What drove the call
1. <criterion + score + evidence>
2. <criterion + score + evidence>
3. <criterion + score + evidence>

## Runner-up: <vendor 2>
<2–3 sentences. What would flip the decision in 12 months?>

## Risks accepted
- <risk + mitigation if any>
- <risk + mitigation if any>

## Negotiated terms vs. list
- <term + what we got>

## Reference calls
- <Customer 1, role>: <one-line takeaway>
- <Customer 2, role>: <one-line takeaway>

Definition of done

  • Criteria explicitly named + weighted before scores were written
  • Each criterion has a one-line evidence per vendor (not just a number)
  • All five diligence areas covered (security, reliability, pricing, exit, roadmap)
  • Two reference calls completed and quoted
  • Memo opens with the recommendation in one sentence
  • Decision-maker named, decision deadline named

Gotchas

  1. Don't pick the criteria during scoring. That's how bias leaks in — you find yourself weighting criteria where your preferred vendor happens to score high. Lock criteria + weights BEFORE running the scorecard.

  2. The demo is theater. Vendor demos are rehearsed on tooth-paste-perfect data. Insist on a trial against YOUR data, even if it's a sandbox snapshot. Demos lie; trials don't.

  3. Pricing pages lie. The published price is the starter. At enterprise scale you'll renegotiate, but at small scale you'll often pay MORE than the published tier (per-seat creep, per-event creep). Ask for a TCO projection at your 18-month scale, not your current scale.

  4. "It integrates with X" needs to be tested. Their integration page lists 200 things. Half of those are stub-quality. Test the 2–3 you actually depend on.

  5. References from the vendor are pre-selected. Find your own — LinkedIn for someone in a similar-shaped company using the tool, ask them quietly. Vendor-supplied references will be uniformly positive.

  6. Beware the "we just shipped that" reply. When you ask about a missing feature and the rep says "we just shipped that" or "that's in beta now," ask to see it in production with a customer. Half of "just shipped" is "the engineer just merged the PR."

  7. Switching cost is a real number. Estimate it. If switching off Vendor A costs 6 months of two engineers' time, that goes in the math against Vendor B's savings.