Data Readiness
Find out early whether the customer's data can support the system, before the build depends on it.
Data Readiness
Data readiness asks five questions about every source the use case needs:
Most failed AI projects fail one of these, and most of those failures were discoverable in the first two weeks. Readiness is not a gate you pass once; it is a rating per source that feeds straight back into the scope.
Why it matters
Teams often discover in week six that the critical field is free text, the export needs a three-month approval, or half the documents are scanned images. By then the plan, the budget and the sponsor's expectations all assume data that is not there.
AI systems are unusually sensitive to data problems. A dashboard with 10% missing values is still mostly useful; a retrieval system that cannot see the latest policy version gives confidently wrong answers. Duplicates and outdated versions in a document store are a common cause of "hallucinations" that are really the system faithfully quoting the wrong document.
Checking early turns a late surprise into an early scoping decision, and gives you evidence for conversations about timelines.
How to do it
List every data source the use case needs, including the ones that "just exist".
For each, record the owner, format, volume, freshness and access route, and how long access takes to approve.
Ask for a real sample in week one, not a description: a few hundred records or documents, as close to production as policy allows.
Profile the sample: completeness of key fields, duplicates, contradictions between sources, outdated versions, scanned or image-only content, and language mix.
Confirm the legal basis and permissions with the data owner, and check whether document-level permissions must be enforced in the system.
Write a readiness rating per source (ready, ready with work, not ready) with the gaps that change scope, and send it to the sponsor.
Profile a real sample in minutes
import pandas as pd
def profile(df: pd.DataFrame, key_fields: list[str], text_field: str | None = None,
date_field: str | None = None, id_field: str | None = None) -> dict:
"""Quick readiness profile of a real data sample."""
report = {"rows": len(df)}
report["missing_pct"] = {f: round(df[f].isna().mean() * 100, 1) for f in key_fields}
if id_field:
report["duplicate_ids"] = int(df[id_field].duplicated().sum())
if text_field:
text = df[text_field].fillna("")
report["empty_text_pct"] = round((text.str.strip() == "").mean() * 100, 1)
report["short_text_pct"] = round((text.str.len() < 40).mean() * 100, 1) # often scans or stubs
if date_field:
dates = pd.to_datetime(df[date_field], errors="coerce")
report["date_range"] = (str(dates.min().date()), str(dates.max().date()))
report["unparseable_dates"] = int(dates.isna().sum())
return report
# sample = pd.read_csv("tickets_sample.csv")
# print(profile(sample, key_fields=["category", "resolution"], text_field="description",
# date_field="created_at", id_field="ticket_id"))Worked example: "we have two years of tickets"
The support team says two years of tickets exist. The week-one sample tells a different story:
The readiness rating becomes "ready with work": request a database export through IT (three weeks), use only tickets after March for training categories, and treat resolutions as unreliable. The scope changes before any code is written, and the sponsor hears it in week one instead of week six.
In a real engagement
Always ask for a sample in week one. Descriptions of data are optimistic; samples are honest.
For retrieval systems, check document permissions and version control as part of readiness, not later. If the customer's document store has no clear "current version", the system will need a rule for choosing one, and that rule belongs in the scope.
Ask how data changes after go-live: who edits it, how often, and how the system will hear about it. A system built on a one-off export quietly goes stale.
Record access approvals as a dependency on the plan with a named owner on the customer's side. They are the most common source of delay in enterprise AI projects.
Challenge
You need two years of support tickets. The team says they exist, but the export tool only returns 90 days. List three options and the trade-offs of each.
Checklist
Production checklist
0 of 8 checked
Knowledge check
What is the most reliable way to judge data quality early?
Complete
You have finished “Data Readiness”. Continue to the next chapter or practise in the FDE simulator.
Mark this chapter as complete
Track your progress and unlock the next chapter.