Build a Paper Figure Extraction and Analysis Demo with Sciverse
MultimodalRetrievalAgentAdvanced

Build a Paper Figure Extraction and Analysis Demo with Sciverse

Retrieve papers, locate figure paths in Markdown, download figures, and analyze them with a multimodal model

View on GitHub
Scenario
Researchers need to extract specific figures from papers and analyze them. The workflow retrieves relevant papers, locates figure assets, downloads them, and passes them to a multimodal model.
Estimated calls
~8-20 API calls
Tools
agentic-searchcontentresource
Pipeline
agentic-search(论文主题)→ content(提取图表路径)→ resource(下载图片)→ 多模态 LLM 分析

Input example

Task:
"Find papers that show battery electrolyte interface microscopy images and summarize what the figures demonstrate."

Output example

Figure analysis:
- fig1.png: SEM image showing dense interfacial layer...
- fig2.png: cycling performance comparison...
Sources include doc_id and figure paths.

Agent Prompt example

You are a multimodal paper analysis assistant. Retrieve relevant papers, locate figure resources, download figures, and ask a vision model to extract figure-level evidence.

Implementation steps

Step 1: Set up the environment

Configure token, HTTP client, and optional vision model credentials

!pip install httpx anthropic
import os
os.environ["SCIVERSE_API_TOKEN"] = "sv-your-token-here"  # 替换为你的真实值
import os
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."  # 替换为你的真实值

Step 2: Retrieve papers and extract figure paths

Search relevant papers and parse Markdown resource paths from content

import os
import re
import asyncio
import httpx
from pathlib import Path

BASE = "https://api.sciverse.space"
TOKEN = os.environ["SCIVERSE_API_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}"}

async def search_papers(query: str, top_k: int = 10):
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.post(
            f"{BASE}/agentic-search", headers=HEADERS,
            json={"query": query, "top_k": top_k}
        )
        resp.raise_for_status()
        return (resp.json().get("hits") or [])

async def get_figures_from_doc(doc_id: str):
    """读取全文并提取图表路径"""
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.get(
            f"{BASE}/content", headers=HEADERS,
            params={"doc_id": doc_id, "offset": 0, "limit": 4000}
        )
        resp.raise_for_status()
        text = resp.json()["text"]  # 注意：字段是 text
        figure_paths = re.findall(r'!\\[.*?\\]\\((.*?)\\)', text)
        return figure_paths

async def main():
    hits = await search_papers("AlphaFold2 protein structure prediction accuracy")
    print(f"Found {len(hits)} relevant papers")
    # 对 top 3 论文提取图表
    all_figures = []
    for hit in hits[:3]:
        paths = await get_figures_from_doc(hit["doc_id"])
        print(f"  {hit['title'][:50]}: {len(paths)} figures")
        all_figures.extend([(hit["doc_id"], p) for p in paths])
    return all_figures

all_figures = await main()

Step 3: Download figures and analyze with a multimodal model

Fetch figure files and send them to a multimodal model for analysis

import base64
from anthropic import Anthropic

async def download_figure(file_name: str, save_dir: str = "./figures"):
    Path(save_dir).mkdir(exist_ok=True)
    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.get(
            f"{BASE}/resource", headers=HEADERS,
            params={"file_name": file_name}  # 参数是 file_name
        )
        resp.raise_for_status()
        local = f"{save_dir}/{file_name.split('/')[-1]}"
        Path(local).write_bytes(resp.content)
        return local

def analyze_figure(image_path: str, question: str) -> str:
    """用多模态 LLM 分析图表"""
    client = Anthropic()
    with open(image_path, "rb") as f:
        img_data = base64.b64encode(f.read()).decode()

    msg = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {
                    "type": "base64", "media_type": "image/png", "data": img_data
                }},
                {"type": "text", "text": question}
            ]
        }]
    )
    return msg.content[0].text

async def main():
    if all_figures:
        doc_id, path = all_figures[0]
        try:
            local = await download_figure(path)
            analysis = analyze_figure(local, "请描述这张图表的主要发现，提取关键数值。")
            print(f"\
Figure from {doc_id}:\
{analysis}")
        except httpx.HTTPStatusError as e:
            print(f"Download failed: {e.response.status_code}")

await main()

Notes

  • Only analyze figures actually returned by the resource endpoint.
  • Keep figure path and doc_id with every analysis result.
  • Vision model output should still be checked against captions and text.
  • Some papers may include low-resolution or missing figures.
  • Cache binary assets for repeated analysis.

FAQ

适合什么场景?

适合需要查找论文图、表、流程图、实验结果图的多模态科研任务。

和 download-figures 有什么区别?

本案例从语义问题开始检索论文，再定位图表；download-figures 更偏给定论文后的资源下载。

需要哪些 API?

通常先用 agentic-search 找论文，再用 content 定位路径，最后用 resource 获取资源。

Sciverse 是否直接解释图片?

不直接解释图片，Sciverse 提供图表资源，上层多模态模型负责理解。

Next steps

Need an API key?

Create one in Console > Tokens.The same API key works for enabled Sciverse, DianShi, and Skills capabilities, with starter quota available according to account permissions.

Open console