Paper Title Similarity Deduplication Demo

Group duplicate papers by title similarity and choose the most authoritative version as the canonical record

Scenario

Scientific agents often retrieve preprints, published versions, and PDF or web duplicates of the same paper. They need to merge these into one canonical record to avoid repeated citations.

Estimated calls

~10-20 API calls per deduplication task

Tools

agentic-searchmeta-searchcontent

Pipeline

agentic-search→ 候选文献列表→ meta-search 按 DOI/标题聚合→ 去重合并→ canonical pack

Input example

Input query:
"AlphaFold2 protein structure prediction"
Need: group duplicate records and select the canonical version.

Output example

Canonical pack:
- canonical_title: Highly accurate protein structure prediction with AlphaFold
- versions: Nature article, preprint, PDF mirror
- selected_reason: peer-reviewed journal version with highest authority

Agent Prompt example

You are a paper deduplication assistant. Retrieve candidate papers, group similar titles, compare metadata, and select the most authoritative canonical record.

Implementation steps

Step 1: Set up the environment

Configure token and dependencies

!pip install httpx
import os
os.environ["SCIVERSE_API_TOKEN"] = "sv-your-token-here"  # 替换为你的真实值

Step 2: Retrieve candidate papers semantically

Search for candidate papers related to the topic

import os
import asyncio
import httpx
from collections import defaultdict

BASE = "https://api.sciverse.space"
TOKEN = os.environ["SCIVERSE_API_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}"}

async def search_candidates(query: str, top_k: int = 50):
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.post(
            f"{BASE}/agentic-search", headers=HEADERS,
            json={"query": query, "top_k": top_k}
        )
        resp.raise_for_status()
        return (resp.json().get("hits") or [])

hits = await search_candidates("Attention Is All You Need transformer")
print(f"Raw hits: {len(hits)}")

Step 3: Group by title similarity

Normalize titles and cluster highly similar records

from difflib import SequenceMatcher

def title_similarity(a: str, b: str) -> float:
    return SequenceMatcher(None, a.lower(), b.lower()).ratio()

def cluster_by_title(hits, threshold=0.85):
    clusters = []
    used = set()
    for i, h in enumerate(hits):
        if i in used:
            continue
        group = [h]
        used.add(i)
        for j in range(i + 1, len(hits)):
            if j in used:
                continue
            if title_similarity(h["title"], hits[j]["title"]) >= threshold:
                group.append(hits[j])
                used.add(j)
        clusters.append(group)
    return clusters

clusters = cluster_by_title(hits)
print(f"Clustered into {len(clusters)} unique papers")
for c in clusters[:3]:
    print(f"  [{len(c)} versions] {c[0]['title'][:60]}")

Step 4: Confirm the published version and generate a canonical pack

Prefer peer-reviewed or authoritative versions and output version metadata

async def find_primary(cluster):
    """\u4ece\u4e00\u7ec4\u7248\u672c\u4e2d\u627e\u5230\u6700\u6743\u5a01\u7684\u6b63\u5f0f\u53d1\u8868\u7248"""
    # \u4f18\u5148\u7ea7: \u6709 DOI > \u6709 venue > arXiv
    best = cluster[0]
    for item in cluster:
        # \u7528 meta-search \u67e5\u8be2\u66f4\u591a\u5143\u6570\u636e
        async with httpx.AsyncClient(timeout=30) as client:
            resp = await client.post(
                f"{BASE}/meta-search", headers=HEADERS,
                json={
                    "query": item["title"],
                    "filters": [],
                    "page": 1, "page_size": 1
                }
            )
            if resp.status_code == 200:
                results = resp.json().get("results", [])
                if results and results[0].get("doi"):
                    best = item
                    break
    return {
        "title": best["title"],
        "primary_doc_id": best["doc_id"],
        "versions": [{"doc_id": v["doc_id"], "title": v["title"]} for v in cluster]
    }

async def build_canonical_pack(clusters):
    pack = []
    for cluster in clusters[:10]:
        canonical = await find_primary(cluster)
        pack.append(canonical)
    return pack

pack = await build_canonical_pack(clusters)
print(f"Canonical pack: {len(pack)} unique papers")

Notes

Title similarity is a heuristic, not a perfect deduplication method.
Preprints and final published versions may have different titles.
Keep all version metadata so users can audit the selection.
Do not discard records unless the evidence for duplication is strong.

FAQ

为什么需要论文去重?

同一论文可能存在预印本、正式发表版或重复元数据，需要聚合后再引用。

主记录如何选择?

优先 DOI、正式期刊版本、年份、引用数和元数据完整度。

去重会影响引用吗?

会，建议引用主记录，同时保留其他版本作为关联记录。

如何处理标题相似但不是同一篇?

需要结合 DOI、作者、年份、摘要和 venue 判断，不能只靠标题相似度。

Next steps

Need an API key?

Create one in Console > Tokens.The same API key works for enabled Sciverse, DianShi, and Skills capabilities, with starter quota available according to account permissions.

Open console