Paper Title Similarity Deduplication Demo
AgentMetadataIntermediate
Paper Title Similarity Deduplication Demo
Group duplicate papers by title similarity and choose the most authoritative version as the canonical record
Scenario
Scientific agents often retrieve preprints, published versions, and PDF or web duplicates of the same paper. They need to merge these into one canonical record to avoid repeated citations.
Estimated calls
~10-20 API calls per deduplication task
Tools
agentic-searchmeta-searchcontentPipeline
agentic-search→ 候选文献列表→ meta-search 按 DOI/标题聚合→ 去重合并→ canonical pack
Input example
Input query: "AlphaFold2 protein structure prediction" Need: group duplicate records and select the canonical version.
Output example
Canonical pack: - canonical_title: Highly accurate protein structure prediction with AlphaFold - versions: Nature article, preprint, PDF mirror - selected_reason: peer-reviewed journal version with highest authority
Agent Prompt example
You are a paper deduplication assistant. Retrieve candidate papers, group similar titles, compare metadata, and select the most authoritative canonical record.Implementation steps
Step 1: Set up the environment
Configure token and dependencies
!pip install httpx
import os
os.environ["SCIVERSE_API_TOKEN"] = "sv-your-token-here" # 替换为你的真实值
Step 2: Retrieve candidate papers semantically
Search for candidate papers related to the topic
import os
import asyncio
import httpx
from collections import defaultdict
BASE = "https://api.sciverse.space"
TOKEN = os.environ["SCIVERSE_API_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}"}
async def search_candidates(query: str, top_k: int = 50):
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.post(
f"{BASE}/agentic-search", headers=HEADERS,
json={"query": query, "top_k": top_k}
)
resp.raise_for_status()
return (resp.json().get("hits") or [])
hits = await search_candidates("Attention Is All You Need transformer")
print(f"Raw hits: {len(hits)}")
Step 3: Group by title similarity
Normalize titles and cluster highly similar records
from difflib import SequenceMatcher
def title_similarity(a: str, b: str) -> float:
return SequenceMatcher(None, a.lower(), b.lower()).ratio()
def cluster_by_title(hits, threshold=0.85):
clusters = []
used = set()
for i, h in enumerate(hits):
if i in used:
continue
group = [h]
used.add(i)
for j in range(i + 1, len(hits)):
if j in used:
continue
if title_similarity(h["title"], hits[j]["title"]) >= threshold:
group.append(hits[j])
used.add(j)
clusters.append(group)
return clusters
clusters = cluster_by_title(hits)
print(f"Clustered into {len(clusters)} unique papers")
for c in clusters[:3]:
print(f" [{len(c)} versions] {c[0]['title'][:60]}")
Step 4: Confirm the published version and generate a canonical pack
Prefer peer-reviewed or authoritative versions and output version metadata
async def find_primary(cluster):
"""\u4ece\u4e00\u7ec4\u7248\u672c\u4e2d\u627e\u5230\u6700\u6743\u5a01\u7684\u6b63\u5f0f\u53d1\u8868\u7248"""
# \u4f18\u5148\u7ea7: \u6709 DOI > \u6709 venue > arXiv
best = cluster[0]
for item in cluster:
# \u7528 meta-search \u67e5\u8be2\u66f4\u591a\u5143\u6570\u636e
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.post(
f"{BASE}/meta-search", headers=HEADERS,
json={
"query": item["title"],
"filters": [],
"page": 1, "page_size": 1
}
)
if resp.status_code == 200:
results = resp.json().get("results", [])
if results and results[0].get("doi"):
best = item
break
return {
"title": best["title"],
"primary_doc_id": best["doc_id"],
"versions": [{"doc_id": v["doc_id"], "title": v["title"]} for v in cluster]
}
async def build_canonical_pack(clusters):
pack = []
for cluster in clusters[:10]:
canonical = await find_primary(cluster)
pack.append(canonical)
return pack
pack = await build_canonical_pack(clusters)
print(f"Canonical pack: {len(pack)} unique papers")
Notes
- Title similarity is a heuristic, not a perfect deduplication method.
- Preprints and final published versions may have different titles.
- Keep all version metadata so users can audit the selection.
- Do not discard records unless the evidence for duplication is strong.
FAQ
为什么需要论文去重?
同一论文可能存在预印本、正式发表版或重复元数据,需要聚合后再引用。
主记录如何选择?
优先 DOI、正式期刊版本、年份、引用数和元数据完整度。
去重会影响引用吗?
会,建议引用主记录,同时保留其他版本作为关联记录。
如何处理标题相似但不是同一篇?
需要结合 DOI、作者、年份、摘要和 venue 判断,不能只靠标题相似度。
Next steps
Need an API key?
Create one in Console > Tokens.The same API key works for enabled Sciverse, DianShi, and Skills capabilities, with starter quota available according to account permissions.