cluster_selection_persistence does not prevent low-persistence clusters from appearing

[notluquis]

I’m observing that even when I set:

import hdbscan

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=XX,
    min_samples=YY,
    cluster_selection_persistence=0.2,
    algorithm='best',
    core_dist_n_jobs=-1,
)
labels = clusterer.fit_predict(X)

some of the final clusters have a measured persistence (cluster_persistence_) below 0.2. In other words, branches with low persistence still survive and become clusters.

Steps to reproduce
1. Prepare any dataset X.
2. Run:

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=50,
    min_samples=10,
    cluster_selection_persistence=0.2,
    algorithm='best',
    core_dist_n_jobs=-1,
)
labels = clusterer.fit_predict(X)
print(clusterer.cluster_persistence_)
# observe values < 0.2

What I’ve tried
1. Default persistence flag

HDBSCAN(..., cluster_selection_persistence=0.2)

→ low-persistence clusters still appear in labels_, cluster_persistence_, and in the condensed tree.

2.	Manual patch in _tree_to_labels

I tried to insert a filter immediately after condensation:

def _tree_to_labels(...):
    condensed = condense_tree(single_linkage_tree, min_cluster_size)
-   if cluster_selection_persistence > 0.0 and condensed.shape[0] > 0:
-       condensed = simplify_hierarchy(condensed, cluster_selection_persistence)
+   # attempt to filter low-persistence branches too early
+   if cluster_selection_persistence > 0.0 and condensed.size > 0:
+       condensed = condensed[condensed["lambda_val"] >= cluster_selection_persistence]
+   else:
+       condensed = simplify_hierarchy(condensed, cluster_selection_persistence)

After reinstalling/reloading, behavior did not change—branches with lambda_val < 0.2 still become clusters.

Expected behavior / Request behavior

I would expect that setting cluster_selection_persistence=0.2 would prevent any cluster whose true persistence is below 0.2 from appearing in the final labels or tree.

Actual behavior

Clusters with cluster_persistence_ < 0.2 still appear, and their low persistence values are included in clusterer.cluster_persistence_ and shown in the condensed tree.

Questions

1. How exactly does cluster_selection_persistence interact with the EOM/leaf selection?
2. At what point is persistence applied, and why are low-persistence branches still surviving?
3. What is the intended mechanism to ensure that any branch below the persistence threshold is treated as noise (label –1)?

Any pointers to the relevant code paths or suggestions for how to enforce this behavior would be greatly appreciated.

[JelmerBot]

Good find! It looks like cluster_persistence_ is a bit of a misnomer. The documentation describes that attribute as:

It is implemented at

hdbscan/hdbscan/_hdbscan_tree.pyx

Line 604 in 8ecf239

result[n] = stability[c] / (cluster_size * max_lambda)

and divides the cluster's stability by its maximum possible stability. The resulting value relates to persistence, but is different from what cluster_selection_persistence uses to simplify the condensed tree. So, it cannot be used to verify the cluster_selection_persistence parameter.

Setting a cluster_selection_persistence parameter ensures that all leaves in the condensed tree meet the following threshold:
(max_density - min_density) >= cluster_selection_persistence. Leaves that do not meet this condition are pruned from the condensed tree before clusters are selected.
This code snippet can be used to check that the condition holds:

from hdbscan._hdbscan_tree import get_cluster_tree_leaves

raw_tree = clusterer._condensed_tree
for leaf in get_cluster_tree_leaves(raw_tree[raw_tree["child_size"] > 1]):
    min_density = raw_tree['lambda_val'][raw_tree["child"] == leaf][0]
    max_density = raw_tree['lambda_val'][raw_tree["parent"] == leaf].max()
    print(f"{leaf}: {max_density - min_density :.2f}")