Build a Paper Figure Extraction and Analysis Demo with Sciverse
MultimodalRetrievalAgentAdvanced
Build a Paper Figure Extraction and Analysis Demo with Sciverse
Retrieve papers, locate figure paths in Markdown, download figures, and analyze them with a multimodal model
Scenario
Researchers need to extract specific figures from papers and analyze them. The workflow retrieves relevant papers, locates figure assets, downloads them, and passes them to a multimodal model.
Estimated calls
~8-20 API calls
Tools
agentic-searchcontentresourcePipeline
agentic-search(论文主题)→ content(提取图表路径)→ resource(下载图片)→ 多模态 LLM 分析
Input example
Task: "Find papers that show battery electrolyte interface microscopy images and summarize what the figures demonstrate."
Output example
Figure analysis: - fig1.png: SEM image showing dense interfacial layer... - fig2.png: cycling performance comparison... Sources include doc_id and figure paths.
Agent Prompt example
You are a multimodal paper analysis assistant. Retrieve relevant papers, locate figure resources, download figures, and ask a vision model to extract figure-level evidence.Implementation steps
Step 1: Set up the environment
Configure token, HTTP client, and optional vision model credentials
!pip install httpx anthropic
import os
os.environ["SCIVERSE_API_TOKEN"] = "sv-your-token-here" # 替换为你的真实值
import os
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..." # 替换为你的真实值
Step 2: Retrieve papers and extract figure paths
Search relevant papers and parse Markdown resource paths from content
import os
import re
import asyncio
import httpx
from pathlib import Path
BASE = "https://api.sciverse.space"
TOKEN = os.environ["SCIVERSE_API_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}"}
async def search_papers(query: str, top_k: int = 10):
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.post(
f"{BASE}/agentic-search", headers=HEADERS,
json={"query": query, "top_k": top_k}
)
resp.raise_for_status()
return (resp.json().get("hits") or [])
async def get_figures_from_doc(doc_id: str):
"""读取全文并提取图表路径"""
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.get(
f"{BASE}/content", headers=HEADERS,
params={"doc_id": doc_id, "offset": 0, "limit": 4000}
)
resp.raise_for_status()
text = resp.json()["text"] # 注意:字段是 text
figure_paths = re.findall(r'!\\[.*?\\]\\((.*?)\\)', text)
return figure_paths
async def main():
hits = await search_papers("AlphaFold2 protein structure prediction accuracy")
print(f"Found {len(hits)} relevant papers")
# 对 top 3 论文提取图表
all_figures = []
for hit in hits[:3]:
paths = await get_figures_from_doc(hit["doc_id"])
print(f" {hit['title'][:50]}: {len(paths)} figures")
all_figures.extend([(hit["doc_id"], p) for p in paths])
return all_figures
all_figures = await main()
Step 3: Download figures and analyze with a multimodal model
Fetch figure files and send them to a multimodal model for analysis
import base64
from anthropic import Anthropic
async def download_figure(file_name: str, save_dir: str = "./figures"):
Path(save_dir).mkdir(exist_ok=True)
async with httpx.AsyncClient(timeout=60) as client:
resp = await client.get(
f"{BASE}/resource", headers=HEADERS,
params={"file_name": file_name} # 参数是 file_name
)
resp.raise_for_status()
local = f"{save_dir}/{file_name.split('/')[-1]}"
Path(local).write_bytes(resp.content)
return local
def analyze_figure(image_path: str, question: str) -> str:
"""用多模态 LLM 分析图表"""
client = Anthropic()
with open(image_path, "rb") as f:
img_data = base64.b64encode(f.read()).decode()
msg = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=2048,
messages=[{
"role": "user",
"content": [
{"type": "image", "source": {
"type": "base64", "media_type": "image/png", "data": img_data
}},
{"type": "text", "text": question}
]
}]
)
return msg.content[0].text
async def main():
if all_figures:
doc_id, path = all_figures[0]
try:
local = await download_figure(path)
analysis = analyze_figure(local, "请描述这张图表的主要发现,提取关键数值。")
print(f"\
Figure from {doc_id}:\
{analysis}")
except httpx.HTTPStatusError as e:
print(f"Download failed: {e.response.status_code}")
await main()
Notes
- Only analyze figures actually returned by the resource endpoint.
- Keep figure path and doc_id with every analysis result.
- Vision model output should still be checked against captions and text.
- Some papers may include low-resolution or missing figures.
- Cache binary assets for repeated analysis.
FAQ
适合什么场景?
适合需要查找论文图、表、流程图、实验结果图的多模态科研任务。
和 download-figures 有什么区别?
本案例从语义问题开始检索论文,再定位图表;download-figures 更偏给定论文后的资源下载。
需要哪些 API?
通常先用 agentic-search 找论文,再用 content 定位路径,最后用 resource 获取资源。
Sciverse 是否直接解释图片?
不直接解释图片,Sciverse 提供图表资源,上层多模态模型负责理解。
Next steps
Need an API key?
Create one in Console > Tokens.The same API key works for enabled Sciverse, DianShi, and Skills capabilities, with starter quota available according to account permissions.