seqme.metrics.Precision

class seqme.metrics.Precision(n_neighbors, reference, embedder, *, batch_size=256, device='cpu', strict=True, name='Precision')

Evaluates how realistic synthetic samples are compared to reference data.

Computes the Improved Precision metric [1], which measures the realism of synthetic samples by assessing how closely each sample aligns with the reference data in the embedding space. This metric quantifies fidelity, i.e., the degree to which synthetic samples resemble the true data distribution. To achieve this, the reference manifold is approximated using nearest-neighbor balls.

Its value ranges from 0 to 1, representing the fraction of synthetic samples that lie on the reference manifold and are therefore considered realistic.

References

[1] Kynkäänniemi et al., “Improved precision and recall metric for assessing generative models”, NeurIPS 2019

(https://arxiv.org/abs/1904.06991)

__init__(n_neighbors, reference, embedder, *, batch_size=256, device='cpu', strict=True, name='Precision')

Initialize the Improved Precision metric.

Constructs the reference manifold using the provided sequences and prepares the metric for evaluation. The reference manifold is approximated using nearest-neighbor balls, with radii determined by the specified number of neighbors.

Parameters:

• n_neighbors (int) – Number of nearest neighbors used to define the radii of the nearest-neighbor balls. More neighbors result in larger radii.

• reference (list[str]) – List of reference sequences used to build the reference manifold.

• embedder (Callable[[list[str]], ndarray]) – Function mapping sequences to embeddings.

• batch_size (int) – Number of samples per batch when computing distances.

• device (str) – Compute device, e.g., "cpu" or "cuda".

• strict (bool) – If True, enforces an equal number of evaluation and reference samples.

• name (str) – Metric name.

Raises:

• ValueError – If n_neighbors < 1.

• ValueError – If reference contains fewer than 1 sequence after embedding.

__call__(sequences)

Compute the Improved Precision of the sequences.

Parameters:

sequences (list[str]) – Sequences to evaluate.

Returns:

Improved Precision.

Return type:

MetricResult

Methods

__init__(n_neighbors, reference, embedder, *)	Initialize the Improved Precision metric.
__call__(sequences)	Compute the Improved Precision of the sequences.

Attributes

name	Name of the metric.
objective	Whether lower or higher scores indicate better performance.