Quick answer: Evaluating a real-world data vendor comes down to whether the dataset fits your specific research question, not how large it is. Check these eleven things before you shortlist anyone:
Each point measures a different aspect of data quality, usability, and research readiness factors that matter far more than headline patient counts.
Real-world data is now a core input for research, regulatory submissions, and health economics work. But most vendor pitches lead with one number: total patients, total records, total data points.
A dataset covering millions of patients is useless for a treatment sequencing study if it holds one year of history per person. Clean structured fields are useless if your outcome only ever appears in a clinical note. The question is never "how big is it." It is "does this contain what I need, at the quality I need, for the population I care about."
Use the checklist below as a scorecard. Send it to vendors before the first call.
Buying on volume. Headline counts are the easiest number to inflate and the least informative. A "patient" might have a decade of records or a single visit.
Skipping feasibility. Teams sign, then find their cohort has 400 patients instead of 4,000.
Treating compliance as a checkbox. "HIPAA-compliant" is a claim, not a document.
What to ask: What are the original sources, and are they contracted directly or acquired through intermediaries?
Why it matters: Data assembled through intermediaries loses traceability. Direct relationships with EHR, revenue cycle, and registry sources mean the vendor can explain how any given field came to exist.
What a good answer looks like: Named source types, direct provider relationships, and a stated provider count. Vagueness here is the biggest warning sign in the process.
What to ask: How much of the data comes from ambulatory settings, and are independent practices included?
Why it matters: Chronic disease management, medication titration, and routine follow-up all happen outside the hospital. Health-system-only datasets miss the patient journey between admissions.
What a good answer looks like: A setting-level breakdown, with clarity on whether small and independent practices are represented.
What to ask: Which therapeutic areas are densely represented, not just present? What is the geographic spread?
Why it matters: A dataset that technically contains cardiology patients is very different from one built on cardiology-dense practices. Density determines whether your cohort is large enough to analyze.
What a good answer looks like: Named specialties with real depth, aggregate de-identified population characteristics, and honesty about thin areas.
What to ask: How many years of continuous history are available, and for what share of patients?
Why it matters: Progression studies, treatment sequencing, and long-term safety work all need depth rather than breadth. This is the most skipped and most decisive criterion.
What a good answer looks like: A stated range with the proportion of patients reaching it. A single maximum figure is not an answer.
What to ask: Are clinical notes, medical reports, and scanned documents available, and how are they processed?
Why it matters: Disease staging, symptom severity, and reasons for discontinuation usually live in free text. Structured-only datasets cannot answer those questions.
What a good answer looks like: Both layers available, with unstructured content de-identified on a project-specific basis rather than sold as a fixed extract.
What to ask: Is data from different source platforms curated to a single standard? How current is it?
Why it matters: Data pulled from many disparate systems without a unified standard shifts the cleaning burden onto your analysts, which is where most study time disappears.
What a good answer looks like: One documented curation standard applied across all sources, plus a stated update rhythm.
What to ask: Does the dataset combine clinical, utilization, and patient-reported layers? Can it link to external sources?
Why it matters: EHR data shows what was documented in care. Claims show what was billed. Patient-reported outcomes show why. Any single layer leaves a blind spot.
What a good answer looks like: Clinical, claims, and PRO layers composited into one patient view, with external linkage options described.
What to ask: How well populated are the exact fields my study depends on?
Why it matters: A field can exist in the schema and be populated for a small minority of patients. Overall completeness figures tell you nothing about your variables.
What a good answer looks like: A field-level assessment for your named variables, provided during scoping rather than after signature.
What to ask: Which method was applied, safe harbor or expert determination? Who determined it, and for what scope?
Why it matters: The method decides whether your legal team can approve the license and whether a journal or regulator will accept the resulting evidence. All research data should be de-identified and analyzed at aggregate or cohort level, but the paperwork is what makes that verifiable.
What a good answer looks like: A named method, expert determination where clinical documents are involved, and documentation you can forward to compliance.
What to ask: Will you assess my cohort against my inclusion and exclusion criteria before I commit budget?
Why it matters: The most protective question on the list. It tells you whether the population exists at usable scale before any money moves.
What a good answer looks like: Yes, returned as aggregate de-identified counts, scoped to your program rather than a generic capability deck.
What to ask: What are the publication rights, vendor review rights, usage scope, derived-data terms, and renewal pricing?
Why it matters: Academic and regulatory work depends on publication freedom. Restrictive review clauses can make a strong dataset unusable for peer-reviewed research.
What a good answer looks like: Publication freedom in writing, bounded usage scope, and defined treatment of derived datasets.
Provenance not described in writing
Headline counts offered, population breakdowns withheld
Specialty coverage claimed but never quantified
Longitudinal depth given only as a maximum
No field-level completeness review during scoping
No cohort assessment before signature
Compliance claimed verbally with no documentation
Publication subject to vendor approval
Pricing quoted without defined usage scope
Any two of these together is usually enough to stop.
The FDA's Advancing Real-World Evidence Program, established in 2022, invites sponsors to meet with the Agency before protocol development or study initiation when planning research built on real-world evidence. Proposals describe the pivotal study, which means the data source is already part of the conversation at that point.
For buyers, the message is simple. Questions about data provenance, completeness, longitudinal depth, and de-identification should be answered before selecting a vendor, not after.
The takeaway: a large dataset is not enough. Regulators expect data to be fit for purpose, and your vendor evaluation should follow the same principle
Score each of the eleven points from 0 to 3.
0 = no answer or refused
1 = vague or verbal only
2 = documented and adequate
3 = documented, quantified, and strong
Weight the points that matter most for your question. Long-term safety work should weight longitudinal depth and data currency. Trial feasibility should weight specialty density and feasibility scoping. Academic work should weight publication rights and curation standard.
Anything scoring 0 or 1 on provenance, completeness, compliance documentation, or feasibility scoping should not proceed to contract, no matter how well it scores elsewhere.
Most bad real-world data purchases are not caused by bad vendors. They are caused by evaluations that stopped at the headline number.
These eleven points add a few weeks. They save far more in rework and re-scoping.
Sidus Insights works with biopharma sponsors, providers, payers, and academic institutions, matching de-identified ambulatory real-world data to specific research questions across cardiology, oncology, OB/GYN, behavioral health, primary care, gastroenterology, and pediatrics. If you are working through an evaluation, get in touch with the Sidus team.