When a data team receives a synthetic dataset and asks “can we trust this?” the answer is: it depends on how and what you want to use it for. BlueGen’s evaluation report exists to make that question answerable with evidence across three independent dimensions. The scores and examples in this blog are drawn from a real evaluation run on a labour economics earnings dataset, intended to show what the report produces rather than to represent expected performance levels for any given dataset or configuration, as this varies for each use-case.
This blog walks through what each section of that report actually measures, how to read the scores correctly, and why the privacy section covers significantly more ground than most teams expect.
Jump-to:
Why the Evaluation Report Exists
Generating synthetic data without measuring its quality is like shipping software without testing it. The risk is not theoretical. A dataset that looks statistically plausible but breaks the relationships your models depend on will produce wrong answers. A dataset that preserves those relationships but memorises individual records is a privacy liability. BlueGen’s evaluation report catches both failure modes at once because it runs entirely on-premise, against the same data environment where your original records live. No data leaves your infrastructure during synthesis or evaluation. The report is generated as part of the synthesis pipeline, not as an afterthought.
BlueGen measures three independent quality dimensions: resemblance, utility, and privacy. Each can pass or fail independently. A dataset can score perfectly on resemblance and still fail on privacy. That independence matters because it prevents a strong score in one area from masking a weakness in another. BlueGen enforces this separation by applying differential privacy during the generation process itself, rather than adding noise after the fact, so privacy is built into the data rather than bolted on afterwards.
Resemblance: Does It Look Like the Real Data?
Resemblance answers whether the statistical distributions in the synthetic dataset match the original dataset. BlueGen measures this at three levels. The Resemblance KPI (91.02 in this run) is their average. High resemblance confirms statistical alignment. It does not confirm that a model trained on synthetic data will perform equally. That is what utility measures.
Univariate Similarity: 95.0
Univariate similarity compares each column individually using Jensen-Shannon divergence. The report generates side-by-side histograms for every variable.
In the earnings dataset example, the wage distribution in the synthetic data looks nearly identical to the real one. Same for gender split, education levels, occupation category, and every other column. If your team is exploring pay gap analysis or descriptive workforce statistics, this score tells you the synthetic data will return the same headline figures.
Perfect column-level fidelity is possible but rare. The remaining 5% gap reflects small distributional differences in edge-of-range values. For example, the wage histogram shows slightly more synthetic workers clustering near the median than in the real data. For internal analytics, this difference is negligible. For regulatory reporting that depends on tail distributions, you should verify specific columns before use.
Bivariate Similarity: 98.0
Steps up to pairs of columns. The report computes a full correlation matrix for both datasets and measures the difference. The upper triangle absolute difference of 0.0147 means correlations between any two variables deviate by less than 1.5 percentage points on average.
In the earnings dataset example, the relationship between education and wage behaves the same in synthetic as in real. If your analysis depends on how two variables interact. For example, modelling how gender and occupation jointly predict salary. This score confirms those interactions are intact in the synthetic version. Preserving these variable relationships is critical for use cases like energy demand forecasting, where EDF relies on intact correlations between consumption, temperature, and heating type in BlueGen’s synthetic smart meter data.
On the other hand, it would be considered a failure if bivariate similarity dropped below 85. You would see correlation matrices diverge meaningfully. A model trained on that synthetic data might find that education no longer predicts wage, or that gender correlates with industry in a way the real data does not support. That would produce biased model outputs, even though individual column histograms might look fine.
Multivariate Similarity: 80.0
This is the most demanding test. A binary classifier is trained to distinguish real records from synthetic records. If it cannot tell them apart, the score approaches 100. A score of 80 means the classifier retains some separating ability when looking at all columns simultaneously. It can still identify patterns that are slightly more ‘synthetic’ than real.
In high fidelity applications, a report should aim for >85 in multivariate similarity. When used for internal analytics, model training and exploratory work, typically, 80 is sufficient. As the utility scores below confirm. However, if your use-case requires the synthetic data to be statistically the same as real data across all variable combinations simultaneously, then a multivariate similarity below 85 introduces risk that structural patterns are off. For example, if you are producing synthetic data to share with an academic partner who will run structural equation models or causal inference analyses. The classifier is essentially detecting that risk.
Fidelity and Diversity
A fourth metric uses a nearest-neighbour approach following Alaa et al. (2022). The three sub-scores address distinct questions:
| Metric | Score | What it means for your team |
|---|---|---|
| Fidelity | 95.16% | 95% of synthetic records are realistic, they fall within the typical range of the real data. For an HR earnings dataset, this means synthetic worker profiles are plausible: an employee with 12 years of education and 8 years of experience earning $2,800/week exists in realistic proportion to the rest of the dataset. |
| Diversity | 57.72% | The synthetic dataset covers about 58% of the full variety present in the real data. Rare worker profiles, for example, female executives in blue-collar industries, or workers with very short tenures and very high wages, are underrepresented. If your model needs to perform well on edge cases or minority groups, this gap matters. Consider retraining with diversity-boosting settings. In BlueGen, you can address this by enabling augmentation mode during synthesis, which oversamples underrepresented subgroups to increase coverage of rare profiles. |
| Generalization | 37.82% | Only 38% of synthetic records are genuinely new, they sit between real data points rather than sitting directly on top of them. This connects to privacy: records that are too close to a single real record risk exposing that individual. A low generalization score signals the generator is interpolating rather than creating truly novel profiles. The privacy section quantifies this risk concretely. |
Utility: Can You Actually Use It?
Resemblance says the data looks right. Utility says it works right. The Utility KPI (100 in this run) trains models on both datasets and compares performance on held-out real data. BlueGen’s validation report also runs use-case-specific analysis. For this dataset, XGBoost classification models were trained on both and tested on real holdout data:
| Metric | Real model | Synthetic model | Difference |
|---|---|---|---|
| F1 | 0.9330 | 0.9088 | -2.6% |
| Precision | 0.9241 | 0.8972 | -2.9% |
| Recall | 0.9437 | 0.9227 | -2.2% |
| ROC AUC | 0.9809 | 0.9733 | -0.8% |
For regression, the synthetic model explained 58.6% of wage variation (R²) versus 67.5% from the real data. Continuous, right-skewed variables like wage are harder to synthesise. If your use case depends on regression accuracy, validate against your own error thresholds.
This could mean that you could train your classification model, for example, predicting whether an employee will fall into a high-wage bracket, entirely on synthetic data and expect decisions within 3% of the real-data model.
For most production pipelines, that gap is negligible. If false positives or false negatives carry specific costs (for example, in credit decisioning or hiring algorithms), validate the specific metric that matters most against your own tolerance threshold.
Privacy: Is It Safe to Share?
Privacy is not a single number. BlueGen’s validation report runs seven different analyses covering identity disclosure, attribute disclosure, and membership disclosure. The framework follows the Article 29 Working Party’s guidance on anonymisation (Article 29 Working Party, 2014), with attack simulations based on Giomi et al. (2023). Thresholds are set at 9/100, following recommendations from the European Medicines Agency and the Personal Data Protection Commission Singapore.
Unlike platforms that measure privacy as a post-hoc statistic, BlueGen integrates differential privacy directly into its generative models. The differential privacy parameters, which controls the mathematical privacy guarantee, is configured before synthesis begins. This means the privacy boundary is enforced during training, not estimated after the fact. The evaluation report then verifies that the resulting dataset stays within the bounds that the differential privacy configuration was designed to enforce.
Re-identification Risks: All Clear
Inference Risk: 1/100
Can an adversary who knows someone’s age and occupation guess their wage from the synthetic data? Barely. Highest per-column risk: wage at 6.04, well within the threshold of 9.
A higher score for wage would mean that knowing someone works in white-collar sector with 12 years of education gives an attacker an advantage in guessing their salary from the synthetic data. In an HR context, that is a serious disclosure risk: competitors, journalists, or disgruntled employees who obtain the synthetic dataset could reconstruct approximate salary bands for identifiable individuals. A failed inference score on a column like ‘wage’ would block external sharing and require retraining with stronger privacy noise on that specific column.
Linkability Risk: 0/100
Can the synthetic data bridge two external datasets to re-link individuals? No. Synthetic and control attack scores were identical.
Linkability failure is one of the most dangerous outcomes in data sharing. Imagine your synthetic HR dataset is released to a research consortium. A member of that consortium has a second dataset, like a payroll register or a benefits dataset with overlapping attributes. A failed linkability score means they could use your synthetic data as a bridge to reconnect rows across both datasets, effectively reconstructing real individuals from two supposedly separate sources. This is the mechanism behind many high-profile re-identification attacks. A failure would immediately block any external release.
Singling Out Risk: 0/100
Can unique attribute combinations in the synthetic data isolate a single real person, for example, ‘the only female blue-collar worker with 18 years of education in our dataset’? No additional risk above the control.
Singling out does not require knowing a person’s name. It requires identifying that only one record in the dataset matches a specific combination of attributes. Under GDPR, singling out a person from a dataset is itself a re-identification event, even if no direct identifier is present. A failure on this metric would mean the synthetic data has preserved rare real-world attribute combinations so faithfully that specific individuals become identifiable by exclusion. That is a legal exposure, not just a technical flag.
Where This Report Flags Action
Exact Duplicates: 128 (3.07%)
These are synthetic rows identical to real individuals. Each one is a direct re-identification risk if the dataset is shared externally. When used for internal analytics and model training, they may be acceptable depending on your data governance policy. For any external release, each duplicate is legally equivalent to sharing a real record.
BlueGen’s evaluation detects exact duplicates. To resolve this, you can either filter the duplicates or retrain the synthesis with stricter privacy settings, then re-run the evaluation.
NNDR: all five quantile thresholds flagged (p = 0.0)
The Nearest Neighbor Distance Ratio measures how close synthetic records sit to real records relative to how close real records sit to each other. All five quantile tests fail, meaning synthetic records systematically cluster closer to real records than the real data’s own internal spacing.
The NNDR flags a subtler problem than exact copying. Consider this hypothetical synthetic worker profile: 40 years old, white-collar, 12 years of education, $3,200 per week. No single field is an exact match to any real record, so the duplicate filter passes it cleanly. But across all 14 dimensions simultaneously, this synthetic profile sits statistically closer to one specific real worker than real workers typically sit to each other. Someone who already knows that person’s age, occupation, and education level could use the synthetic record to infer their wage with reasonable confidence, without ever touching the original dataset.
To resolve NNDR flags, BlueGen’s team can adjust the privacy and utility trade-off settings in your synthesis configuration (typically by increasing privacy protection), then re-run synthesis and evaluation to verify the NNDR tests pass.
The Privacy-Utility Trade-Off
Privacy and utility are in tension by design. Fewer training iterations produce more private data but lower utility. More iterations produce higher utility but risk overfitting, the generator starts memorising individual records rather than learning the underlying distribution. BlueGen sets privacy thresholds first based on your sharing context, then optimises utility within those constraints.
| Use case | Status | Action required |
|---|---|---|
| Internal analytics, model training, feature engineering | Ready now | None. All re-identification metrics pass. Classification within 3% of real-data baselines. |
| External sharing (research partners, vendors, regulators) | Not yet cleared | Filter exact duplicates + increase privacy constraint level in BlueGen synthesis settings. Re-run evaluation before release. |
Internal analytics and model training are ready now. All re-identification metrics pass, classification is within 3% of real-data baselines. External sharing is not yet cleared. A more privacy-constrained retrain would likely clear the flags but widen the F1 gap to roughly 8-10%. The report makes that position visible so you can make an informed decision.
What Does a Passing Report Look Like?
Fully Passing Report
A fully passing report confirms three things simultaneously: the synthetic data statistically resembles the original dataset, preserves predictive structure for downstream models, and constrains re-identification risks according to European data protection guidance.
In practice, a full pass means that your data governance team has documented evidence that the synthetic dataset meets anonymisation standards. Your data scientists can proceed to model training without provisioning access to the original sensitive data. Your legal team has a report they can point to if asked to demonstrate privacy-by-design compliance.
Partial Pass (Like This One)
A partial pass, where resemblance and utility clear but privacy flags remain, is a gated release scenario. The dataset is can be used for internal work, and/or sharing with contracted third parties. External release is blocked until remediation is complete. The report specifies exactly which tests failed and why, giving your team a concrete remediation path rather than a vague ‘not approved.’
No Pass Report
A full privacy failure, where inference, linkability, or singling out scores exceed 9/100. This means the synthetic data cannot be used in any external context and should be treated with the same access controls as the original.
In all cases, synthetic data remains safer than using original or production data, as it breaks the one-to-one relationship with the source records. In a limited number of instances, individuals were generated that bear a close resemblance to records in the original dataset. It is not possible to identify who these individuals are within the full dataset.
A high resemblance score on a failed report does not make it safer. Statistical plausibility and privacy safety are measured independently, and both need to pass before external release.
Explore The Earnings Report!
Click here to download a sample validation report to see the full set of scores, histograms, and privacy analyses. Or book a demo and we will run through it with your use case in mind.
