Benchmarking protein sequence and structure search methods for remote homology detection

30-second read

An independent benchmark evaluates 14 methods across five scenarios for remote-homology retrieval, residue alignment, and efficiency.

Central question

Across fold, function, multidomain architecture, intrinsic disorder, and predicted-structure confidence, which search paradigm excels where—and is there one universal winner?

Intuition

Rather than compressing everything into one score, the study compares 14 sequence, structure-alignment, and representation methods under a common top-10 protocol across five biological scenarios, each with task-appropriate truth and metrics.

Why it matters

This is the newest independent panoramic calibration: DALI leads residue-level lDDT alignment, with GTalign/TM-align close, yet no search method wins every task. FAST is absent, so it contextualizes modern candidates without directly adjudicating FAST.

Prerequisites

• Understand that retrieval ranking, residue alignment, and functional consistency are different layers.
• Know what annotations CATH, GO, TED, DisProt, and AlphaFold pLDDT provide.
• Be able to read F1max, sensitivity before first false positive, lDDT, and wall time.

paper-specific guide · plain → technical → input → output → source

Method walkthrough

1. 01 · Construct five complementary scenarios

   Each method faces clean domains, low-sequence-identity functional search, multidomain proteins, disordered proteins, and predicted structures of varying confidence, avoiding a winner declared only on favorable data.

   Technical reading: Data derive from CATH-S20, AFDB SwissProt plus GO, TED domain decomposition, DisProt, and pLDDT-stratified AFDB. Datasets are redundancy-reduced with CD-HIT at 20% identity and scenario-specific query/target sets are fixed.

   Input: Public structure, sequence, domain, function, disorder, and confidence resources.

   Output: Five benchmark datasets with distinct biological challenges.

   Boundary: Some SwissProt-derived data may overlap deep-model training corpora, so leakage cannot be fully excluded.

   PDF pp. 5–6, Fig. 2; PDF pp. 24–26, Methods: Benchmark datasets

2. 02 · Run 14 methods under a common top-10 protocol

   Each tool returns as many hits as possible, ranks them by its native score, and contributes its top ten; if fewer than ten exist, fixed-seed random nonhits fill the list for consistent metric shapes.

   Technical reading: Four structure aligners, six representation models, and four sequence methods use public versions with default/recommended parameters. PfamClan filtering remains for CATH in PLMSearch/ERAST but is disabled in reference-free scenarios to reduce external-annotation confounding.

   Input: Scenario-specific query/target sets and 14 tools.

   Output: A standardized top-10 ranked list and runtime for each query.

   Boundary: Random padding penalizes strict filters; native scores and search completeness are not fully equivalent.

   PDF pp. 6 and 29–33, Tables 1–2 and Evaluation protocol

3. 03 · Score and interpret each task separately

   Fold evaluation asks what is recovered before the first error, function uses GO, multidomain structure uses local distances, and disorder/pLDDT test robustness; only then are patterns summarized across plots.

   Technical reading: CATH uses sensitivityFP1 and per-query F1max; function/disorder use Wang GO-MF semantic similarity; structural similarity uses modified lDDT, taking the maximum lDDT across DALI/TM-align/GTalign/Foldseek alignments for each query-hit and averaging top ten.

   Input: Top-10 hits, annotations, and required residue alignments.

   Output: Scenario-specific retrieval, functional, structural, and efficiency results.

   Boundary: Taking the maximum lDDT over four aligners is a method-agnostic upper envelope, not an alignment each retrieval tool itself necessarily produces.

   PDF pp. 26–28, Evaluation tasks and metrics

Key result

DALI leads residue-level lDDT, with GTalign/TM-align close; no method is a single winner across every task.

Evidence-guided deep reading

Paper facts, project readings, and teaching models are labelled separately.

paper-fact

The independent residue-alignment result

To compute top-hit lDDT, the authors run DALI, TM-align, GTalign, and Foldseek on the same query-hit pairs. DALI consistently yields the highest lDDT, TM-align and GTalign are comparable, and Foldseek is lower, consistent with its emphasis on fast retrieval.

This is an independent residue-level comparison, but still not RMSD plus aligned length, and FAST is absent. It supports DALI as a strong current alignment reference without fabricating head-to-head numbers versus FAST.

Source locator: PDF pp. 11–12, Fig. 5 and Additional-file Fig. S1 discussion

paper-fact

Different tasks have different winners

GTalign/TM-align are consistently strong for CATH fold classification; local DALI/Foldseek better capture functional consistency and multidomain local structure; representation methods are more robust under low pLDDT and partial disorder and search quickly.

Figure 9 is about applicability boundaries, not a row winner. Averaging five metric types would silently decide for users how function, geometry, and speed should be weighted.

Source locator: PDF pp. 18–20, Fig. 9 and Discussion: Applicability boundaries

project-reading

Boundaries of this evidence

The manuscript was accepted on 2026-07-08 and is not copyedited, so wording and pagination may change. The authors also note model-training exposure, GO annotation imbalance, semantic-metric choice, and reuse bias in constructed datasets.

FAST is not among the 14 methods. The paper is most useful for choosing DALI/GTalign/TM-align/Foldseek and a five-scenario protocol for a future FAST re-benchmark—not for equating a modern benchmark leader with proven superiority over FAST.

Source locator: PDF p. 1, manuscript metadata; PDF pp. 22–24, limitations and Conclusions; PDF p. 6, Table 1

Study design and evaluation

Data and samples

CATH-S20 with 15,043 domains; SwissProt function set with 1,000 queries/7,735 targets; multidomain 100/4,573 and TED-decomposed 204/11,041; DisProt 534/6,955; and 100 queries per pLDDT tier.

Baselines

• Structure: GTalign, TM-align, DALI, and Foldseek.
• Representation: GraSR, TMvec, PLMSearch, DHR, FoldExplorer, ERAST; sequence: BLAST, Diamond, jackhmmer, MMseqs2.

Metrics

SensitivityFP1 / F1max

Measure early retrieval before the first error and the best precision–recall balance over a ranked list.
Boundary: Depend on CATH labels rather than coordinate geometry.

Modified lDDT

Compares aligned local inter-residue distances at 0.5/1/2/4 Å and penalizes fragmented alignments.
Boundary: Taking the maximum over four aligners creates an upper envelope and is not RMSD.

GO semantic similarity

Evaluates top-10 functional coherence using Wang graph similarity and best-match average.
Boundary: GO coverage is uneven and especially misses context-dependent IDP functions.

Reported result

There is no universal winner: global structural alignment leads fold tasks, local structural alignment leads several function/local-geometry settings, and representations lead some robustness and efficiency settings; DALI gives the best residue-level lDDT.

PDF pp. 5–19, Figs. 2–9; PDF pp. 24–33, Methods

teaching-model · not a reported experiment

Teaching example (not a reported experiment)

Why the same method reverses rank across scenarios

Suppose a query has two domains and a target shares only one catalytic domain, with a different overall domain arrangement.

1. A global aligner pays a penalty for the unshared domain in the whole-protein superposition, lowering its rank.
2. A local aligner can isolate the catalytic domain, obtaining high local lDDT and better GO functional consistency.
3. After TED domain decomposition, a representation method also avoids dilution by the other domain; the paper observes gains of 3–32.1% across methods.

Takeaway: Rank changes can come from changing the unit of the problem, not from an algorithm suddenly becoming smarter.

independent-context

Evidence boundary versus FAST

This is valuable independent context, but FAST is absent and no result can be read as a head-to-head victory over FAST.

Lawful source and access

46 pages · SHA-256 07040af0aa0a09e67e22bc9182ea285b7ef947452ab237a170a14c17974a205f

Accepted 2026-07-08 and not yet copyedited; pagination may change in the version of record.

Springer accepted-manuscript PDF

Limits and misreadings

• This is an uncopyedited accepted manuscript; wording and pagination may change.

Source locator map

1. PDF pp. 5–6, Fig. 2 and Table 1
2. PDF pp. 11–12, Fig. 5 and lDDT alignment comparison
3. PDF pp. 18–20, Fig. 9 and Discussion
4. PDF pp. 22–24, limitations and Conclusions
5. PDF pp. 24–33, Methods and Tables 1–2

Check understanding

1. Which aligner has the best residue-level lDDT in this study?

   Answer: DALI; TM-align and GTalign are close, with Foldseek lower.

   This compares alignments on the same query-hit pairs, not overall retrieval winners.

2. Why do the authors take the maximum lDDT across four alignments for each pair?

   Answer: To make top-hit evaluation less dependent on any one alignment strategy.

   This improves method agnosticism but creates an upper envelope, not each tool's own alignment output.

3. Can this paper directly declare DALI or GTalign universally better than FAST?

   Answer: No. FAST was not tested and no head-to-head RMSD-plus-coverage result exists.

   It is independent modern context, not a direct FAST comparison.

Completion task: Choose a method class for same-fold search, functional-motif search, and low-confidence predicted-structure search; justify each from Fig. 9 and state one limitation.

Paper-specific glossary

sensitivity before first FP

The fraction of true positives retrieved before the first incorrect hit.

lDDT

A score measuring whether local residue–residue distances are preserved within several tolerances.

pLDDT

AlphaFold's per-residue confidence score, stratified here by protein-average value.

applicability boundary

The data, task, and metric regime in which a method's advantage holds, rather than one overall rank.