The Right Data, Not Just Less Data: Measuring What Survives Dataset Distillation

Written by Yue Cao and Mohsen Zardadi

In our previous article on dataset distillation, we explored an ambitious idea: replace a large training dataset with a much smaller synthetic dataset that retains enough useful information to train an effective artificial intelligence model.

That promise is attractive for organizations facing high data-storage costs, limited computing capacity, slow experimentation cycles, or restrictions on moving large datasets. But it also introduces a quality-control problem. A compact dataset is valuable only if it preserves the information the model actually needs.

Recent collaborative research involving TerraSense Analytics examines this problem from a new angle. Instead of judging a distilled dataset mainly by how closely it resembles the original dataset overall, the study asks which visual patterns have been retained, how those patterns are combined, and whether the combination remains useful for learning.

A smaller dataset is not automatically a better dataset

To look inside a compact dataset, our research uses a concept called visual tokenization. This process maps an image onto a sequence of entries drawn from a fixed catalogue of visual patterns. You can think of these entries as visual words. They are recurring units that appear across many images in different combinations and at different scales.

The value of this approach is highly practical. Once images become sequences drawn from a finite vocabulary, their contents can easily be counted. It becomes possible to ask which patterns appear, how often they show up, and in what mix. The result provides hard numbers rather than vague impressions. This is incredibly difficult to do with the dense mathematical representations normally used to compare datasets.  

Consider airplanes in ordinary photographs versus airplanes in satellite imagery. They differ enormously in viewpoint, background, scale, and style. Judged only by their overall distribution, these two collections look almost unrelated. However, if viewed pattern by pattern, they clearly share a set of frequently used visual words that capture the essence of an airplane. That gap is exactly why a traditional overall comparison can be misleading.

Three questions for evaluating a compact dataset

The proposed assessment combines three measurements. Each catches a different way a distilled image can be useful or lacking.

1. Is the composition typical for the class?

The first measurement compares an image’s token distribution with the average token distribution of its class. A closer match suggests that the image contains a more typical combination of patterns for that category. This property is called contextual fit and is measured using Jensen–Shannon divergence (JSD).

2. Are class-relevant patterns present?

The second measurement examines how much of the image is represented by tokens that are especially associated with its class in the original dataset. A low value may occur when the target object is small, weakly represented, or absent. The paper calls this categorical presence and measures it through a class-token coverage rate (COV).

The three measurements are combined into a structural score. The score is not a universal certificate of dataset quality. In the experiments, however, the combined measurements tracked validation accuracy more closely than any one measurement alone, and balanced token composition showed the strongest relationship with performance.

3. Does the image contain a balanced range of patterns?

The third measurement checks whether a few tokens dominate the image or whether token usage is more balanced. Heavy repetition can indicate limited visual variety. More balanced token use is treated as a sign of richer composition. The study measures this concentration using the Herfindahl–Hirschman Index (HHI).

Each row shows one of the three measurements, with high values on the left and low values on the right. Top row (JSD): a high value means the image departs from the typical composition of its category, visible here as distorted shapes and mismatched parts. Lower is better. Middle row (COV): a high value means the category's defining content is well represented. On the left the animal is clear and complete; on the right it is small, faint, or absent. Higher is better. Bottom row (HHI): a high value means a few patterns dominate the image, visible here as the same pose and framing repeating across samples. Lower is better.

What the experiments actually showed

The researchers first analyzed compact datasets created by several existing dataset-distillation approaches. They calculated the token statistics of each dataset and compared them with the validation accuracy of classifiers trained on those datasets. On ImageWoof, a benchmark containing ten visually similar dog-breed classes, the combined structural score correlated closely with the reported validation results.

The study then tested whether the same relationship appeared elsewhere. On ImageNette, the relative importance of the three measurements remained similar, although the fitted numerical weights changed. On EuroSAT remote-sensing imagery, a score developed using ImageWoof did not predict exact accuracy, but it increased alongside model accuracy over successive stages of one distillation process. This supports using the score as a relative progress indicator in that experiment, not as a universal accuracy estimator for every new domain.

A careful reading of the evidence

• The score was evaluated primarily on image-classification benchmarks.

• It provided useful relative signals in the tested settings, but it did not produce exact accuracy predictions across a large domain shift.

• The results show association and experimental usefulness; they do not establish that one token statistic is a universal cause of model performance.

From evaluating datasets to generating better candidates

The research goes beyond measurement. It also uses the structural score to guide a diffusion-based dataset-distillation method called Token-Guided Dataset Distillation, or TGDD.

TGDD groups real images according to their token composition. Within each group, it ranks candidate images using the structural score and selects a limited number of high-scoring examples as anchors. The anchor representations are then combined to guide a pretrained diffusion model in generating the compact synthetic dataset.

The practical idea is straightforward: do not guide generation with every available example or with only a mathematical cluster centre. Instead, use a selected set of examples that collectively appear representative, compositionally rich, and class-relevant according to the proposed measurements.

Across the reported classification benchmarks, TGDD was competitive with, and often stronger than, the evaluated baselines. On ImageWoof, it produced the highest result in nearly all reported configurations.

Why this matters to business and program leaders

A more disciplined way to screen synthetic datasets

Comparing candidate datasets normally requires training and evaluating models on each one. The proposed approach creates the possibility of screening compact datasets using structural measurements before committing to every full evaluation run. The paper does not quantify enterprise-wide savings, but an earlier screening step could help teams focus expensive experiments on stronger candidates.

Better use of tight data budgets

When only a small number of images can be retained per category, redundancy becomes costly. A dataset filled with similar poses, backgrounds, or textures may waste its limited capacity. Token-based analysis offers a way to ask whether the compact set contains varied compositions and preserves class-relevant patterns rather than merely looking realistic.

More accountable synthetic-data generation

Generative models can produce large volumes of visually convincing data, but visual quality alone does not establish training value. A measurable selection process can make the creation of synthetic datasets easier to audit: teams can document why certain real examples were chosen as guidance anchors and which structural properties were considered.

Potential relevance to specialized imagery

The EuroSAT experiment is relevant to organizations working with remote sensing because it shows that the score can track improvement outside the natural-image benchmark on which its coefficients were developed. However, this is an initial result from one dataset and one distillation trajectory. Further evidence would be needed before applying the score as a decision threshold in operational remote-sensing programs.

What this research does not yet establish

This research has been accepted at ECCV 2026. You can read the full paper here. The conclusions should be interpreted within the scope of the reported experiments. The main evaluations involve image classification, not object detection, and segmentation. Those tasks contain spatial and temporal requirements that are not captured by classification accuracy alone.

The next question is not simply how much data we can remove

Dataset distillation began with a powerful compression question: how can a large training set be replaced by a much smaller one? This research points to the next stage of the conversation: how can we understand what remains inside that compact dataset?

By examining images as compositions of discrete visual patterns, the structural score offers a more detailed view of representativeness, variety, and class-relevant content. The TGDD experiments further suggest that the same measurements can be used not only to evaluate compact datasets, but also to guide their generation.

For organizations building AI under computing, storage, transmission, or data-access constraints, the business lesson is simple: reducing data volume is not the final objective. The objective is to preserve the right training information and to measure that preservation carefully.

Less data creates value only when the remaining data still teaches the model what matters.