Reviews
Part of How should I compare AI content operations tools without vendor spin?
What an AI content operations review methodology means for buyers
How to structure an AI content operations review methodology: what gets assessed, what does not, and how to separate vendor claims from independent evidence.
What to take away
For AI content operations, a review methodology turns broad supplier claims into checks a buyer can repeat and explain. Apply the same four assessment areas to the workflow, the model and the service around it; record what was tested and what remains unverified.
- Enforcement pressure has changed the evidence bar: the ASA publishes a list of non-compliant online advertisers, so vendor claims about reach and compliance now need checking rather than accepting.
- A workable methodology scores four thingswhat the tool does, what evidence supports the claim, what your team must supply, and what happens when it fails.
- Vendor material is treated as a claim, never as evidence. Independent sources, your own trials and editorial judgement sit in separate columns.
- Nothing in a desk-based review counts as hands-on testing. Say so plainly in the write-up.
- Published methods let two buyers reach similar conclusions from the same facts, which matters when procurement asks why a supplier scored lower.
Why the bar moved
The Advertising Standards Authority (ASA) publishes its Non-compliant online advertisers list, naming advertisers that have not come into line. It is relevant when a content workflow produces advertising: check the claims and disclosures the supplier says its AI can support, but do not treat the list as evidence of a tool's accuracy or quality.
Crawler claims deserve the same treatment, but only when a supplier says its tool reads or verifies Google crawlers. Check that claim against Google's own guidance on verifying requests from Google crawlers; it is not evidence that an AI model produces accurate content. If the vendor cannot show the mechanism, mark the crawler claim unverified.
What a review methodology covers
A methodology is the written rule for how you gather, weigh and record evidence. It is not a scorecard on its own. Write the rules first, then apply them to every supplier so the comparison holds together.
Applying the four areas to AI workflows
- Capability:test the tool on representative content tasks. Check factual accuracy, unsupported statements or hallucinations, and whether outputs follow the team's editorial and brand requirements. Record the prompts and outputs used in any trial.
- Evidence quality:ask for model documentation, available model cards and AI audit reports. Check what is disclosed about training-data provenance, evaluation methods and known limitations; distinguish documents from the supplier from independent evidence and your own test results.
- Team dependency:record what people must provide or check, including prompts, source material, human review, integrations and training. Note whether the workflow relies on specialist staff to correct or approve outputs.
- Failure and exit:check how the service handles inaccurate or unsafe outputs, model or service changes, access to logs, data deletion and export, fallback arrangements and contract exit. Record any part the supplier will not confirm in writing.
Four assessment areas
- Capabilitydoes the tool do the job
- Evidencewhat supports the claim
- Dependencywhat your team must provide
- Failuresupport terms, exit routes, data handling
The wider reviews and comparison methods article sets out how these areas fit alongside pricing and contract checks, which is worth reading before you write your own scoring rules.
Separating claims from evidence
Every statement in a review should carry a label. Vendor claim, independent evidence, or editorial judgement. Three labels, three columns, no mixing.
Three labels for every statement
Vendor claim
- What it is
- Supplier's own numbers
- Examples
- Case studies, accuracy %
- How to treat
- Mark unverified
Independent evidence
- What it is
- Outside checkable sources
- Examples
- Regulators, platform docs
- How to treat
- Cite the source
Editorial judgement
- What it is
- Your reasoning on fit
- Examples
- Fit for your team
- How to treat
- Label as judgement
Vendor claims include case study numbers, accuracy percentages and integration lists. Record them, attribute them, and mark them unverified unless something outside the vendor supports them.
Independent evidence includes regulator publications, platform documentation, published standards and your own documented trials. For an AI workflow, add model documentation, available model cards, independent AI audit reports and the platform policies on AI-generated content or advertising that apply to the work. A general reference such as the NIST AI Risk Management Framework can help organise risk questions, but it is not UK law. Training records also count. A supplier that can point to certified staff, such as those holding LinkedIn Ads certifications, has a checkable credential rather than a slogan; that credential is relevant only where the service includes LinkedIn advertising.
Editorial judgement is your reasoning about fit. Label it as judgement so a colleague can challenge it without reopening the whole file.
What was assessed and what was not
State the boundary in the first paragraph of any review. Assessed: published documentation, pricing pages, support terms, security statements, regulator records and any trial your team ran. For an AI workflow, also record the model and version reviewed, the prompts and content used, and any disclosed training-data provenance or evaluation results. Not assessed: internal roadmaps, undocumented performance, staff conduct, or anything the vendor would not put in writing. Where personal data is involved, assess the supplier's data handling and contractual terms against the UK GDPR and Data Protection Act 2018, using relevant Information Commissioner's Office (ICO) guidance.
This matters because buyers often read a review as a test. If nobody logged into the product and ran a task, the review is desk-based. Say that. A selection checklist helps you record which items you verified and which you simply read.
| Assessment area | Evidence accepted | Typical label |
|---|---|---|
| Capability | Product documentation, your trial notes | Vendor claim or independent |
| Evidence quality | Regulator pages, platform docs, certifications | Independent |
| Team dependency | Onboarding guides, support responses | Vendor claim |
| Failure and exit | Contract terms, data policy, references | Independent or judgement |
Scoring without false precision
Use four named bands per area rather than a hundred-point scale: Strong means relevant evidence supports the claim; Adequate means it supports most of the claim but leaves a stated gap; Weak means it is partial, inconsistent or contradicted; Not evidenced means there is no usable support and receives zero. Apply the definitions consistently and explain the evidence behind each band.
Set the relative weight of the areas before scoring and publish the rationale. A content team with no in-house engineers may weight dependency heavily; a regulated buyer may weight failure and exit terms heavily. Explain how those priorities reflect the workflow and its risks, and do not present an unweighted total as a weighted score.
Where evidence is missing, use the Not evidenced band and score zero rather than guessing. A missing answer is itself a finding, and it belongs in the summary.
Worked example
Suppose a supplier says its drafting tool checks factual accuracy. Record that as a vendor claim, then ask for the method, model documentation and evaluation evidence. Without a documented trial or other usable support, mark capability and evidence quality Not evidenced rather than treating the claim as proven. If your team runs a trial and finds unsupported statements, record the prompts and outputs; use those results to explain a Weak capability rating. Then assess team dependency and failure and exit separately, including who reviews outputs and what happens to content and data when the service ends.
Common questions
Does a desk-based review count as testing?
No. Desk-based means you assessed documents and public sources only. If your team ran a trial, describe the trial separately and say who ran it and when.
How do you handle a vendor that refuses to answer?
Record the refusal, the date and the question. Score the area as unevidenced and note it in the summary, because silence on security or data handling is a material risk.
Can one methodology cover tools and services?
Yes, if the four areas stay the same. For an AI service, include the people who prepare prompts, review outputs and manage model or workflow changes. Assess team structure and training records as evidence of dependency, rather than rewriting the method.
How often should the methodology change?
Review it when a regulator or platform changes its published rules, when the supplier changes the model or workflow being assessed, or when a scoring band produces repeated arguments. Otherwise leave it alone so comparisons stay consistent across buying cycles.
