Skip to content

README.md

Latest commit

 

History

History
366 lines (275 loc) · 10.7 KB

README.md

File metadata and controls

366 lines (275 loc) · 10.7 KB
Raw

Diffbot Python Library

Python client library for Diffbot APIs.

Installation

Install the standalone CLI binary for agentic use:

curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh

If you prefer, the full Python library can also be installed with pip:

python3 -m pip install diffbot-python

For local development:

pip install -e ".[dev]"

Usage

Authentication

The CLI and the library can share a single credential. The token always has to be passed to the client explicitly, but resolve_token() gives you the same lookup the CLI uses, in this order:

  1. An explicit token passed to resolve_token(token).
  2. The DIFFBOT_API_TOKEN environment variable.
  3. A DIFFBOT_API_TOKEN=... line in ~/.diffbot/credentials.

Set it once and it works for both the CLI and your scripts. Either export it:

export DIFFBOT_API_TOKEN=<TOKEN>

…or write it to the shared credentials file (handy for keeping it out of your shell environment):

mkdir -p ~/.diffbot
printf 'DIFFBOT_API_TOKEN=%s\n' '<TOKEN>' > ~/.diffbot/credentials
chmod 600 ~/.diffbot/credentials

With either in place, resolve the token and pass it to the client:

from diffbot import Diffbot, resolve_token

db = Diffbot(token=resolve_token())  # from env var or ~/.diffbot/credentials
data = db.extract("https://www.example.com")

Client configuration

Diffbot and DiffbotAsync take the same keyword arguments — the client is the single place to configure the SDK.

Argument Default Used by
token — (required) all
timeout 30.0 (seconds) all
transport httpx default all
analyze_url https://api.diffbot.com/v3 extract
crawler_url https://api.diffbot.com/v3/crawl crawl, crawl_list_jobs, crawl_get_job, crawl_delete_job
llm_url https://llm.diffbot.com/rag/v1/chat/completions ask, ask_json
web_search_url https://llm.diffbot.com/api/v1/web_search web_search
nlp_url https://nl.diffbot.com/v1/ entities
dql_url https://kg.diffbot.com/kg/v3/dql dql, dql_parallel
ontology_url https://kg.diffbot.com/kg/ontology dql_fetch_ontology, dql_refresh_ontology 
from diffbot import Diffbot

db = Diffbot(
    token="YOUR_TOKEN",
    timeout=60.0,
    dql_url="http://localhost:8080/kg/v3/dql",
)

analyze_url is a base the SDK appends /{api} to, and crawler_url is used as given for job management and with /data appended for crawl results. The rest are complete endpoints, used as given.

Passing transport replaces the httpx transport, which is the hook for retries, proxies, or mocking in tests. Diffbot takes an httpx.BaseTransport, DiffbotAsync an httpx.AsyncBaseTransport:

import httpx
from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN", transport=httpx.HTTPTransport(retries=3))

Extract structured content

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
data = db.extract("https://www.example.com")

Ask Diffbot LLM

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
    print(chunk, end="")

Structured output

Pass a JSON Schema to constrain the answer. The model is held to the schema by a grammar during decoding, so the result parses reliably even though the answer is retrieved live from the web.

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
schema = {
    "type": "object",
    "properties": {
        "country": {"type": "string"},
        "capital": {"type": "string"},
    },
    "required": ["country", "capital"],
}
answer = db.ask_json([{"role": "user", "content": "What's the capital of France?"}], schema)
print(answer["capital"])

Omit the schema to let the model choose the shape, or use ask with response_format to stream a constrained answer:

from diffbot import Diffbot, json_schema_format

db = Diffbot(token="YOUR_TOKEN")
answer = db.ask_json([{"role": "user", "content": "What's the capital of France?"}])

for chunk in db.ask(
    [{"role": "user", "content": "What's the capital of France?"}],
    response_format=json_schema_format(schema),
):
    print(chunk, end="")

Avoid response_format={"type": "json_object"}. The endpoint accepts it, but applies no grammar to it — the RAG loop's internal tool call is itself a JSON object, so it can be returned as the final answer. This is reproducible whenever the request includes a system message. ask_json therefore defaults to a permissive JSON Schema instead, and raises ValidationError if it ever sees a tool call come back.

Crawl a site for structured content

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
for event in db.crawl("https://www.example.com", hops=1):
    print(event)

Query the Knowledge Graph

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
results = db.dql('type:Organization name:"Diffbot"')

Web Search

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
results = db.web_search("diffbot knowledge graph")
for r in results["search_results"]:
    print(r["score"], r["title"], r["pageUrl"])
    print(r["content"])

Entities (NLP)

from diffbot import Diffbot

db = Diffbot(token="YOUR_TOKEN")
result = db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
for entity in result["entities"]:
    print(entity["name"], entity.get("type"), entity.get("id"))
print("sentiment:", result.get("sentiment"))

Async Usage

Extract structured content

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        data = await db.extract("https://www.example.com")
        print(data)

asyncio.run(main())

Ask Diffbot LLM

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        async for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
            print(chunk, end="")

asyncio.run(main())

Structured output

import asyncio
from diffbot import DiffbotAsync

schema = {
    "type": "object",
    "properties": {
        "country": {"type": "string"},
        "capital": {"type": "string"},
    },
    "required": ["country", "capital"],
}

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        answer = await db.ask_json(
            [{"role": "user", "content": "What's the capital of France?"}], schema
        )
        print(answer["capital"])

asyncio.run(main())

Crawl a site for structured content

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        async for event in db.crawl("https://www.example.com", hops=1):
            print(event)

asyncio.run(main())

Query the Knowledge Graph

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        results = await db.dql('type:Organization name:"Diffbot"')
        print(results)

asyncio.run(main())

Web Search

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        results = await db.web_search("diffbot knowledge graph")
        for r in results["search_results"]:
            print(r["score"], r["title"], r["pageUrl"])
            print(r["content"])

asyncio.run(main())

Entities (NLP)

import asyncio
from diffbot import DiffbotAsync

async def main():
    async with DiffbotAsync(token="YOUR_TOKEN") as db:
        result = await db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
        for entity in result["entities"]:
            print(entity["name"], entity.get("type"), entity.get("id"))
        print("sentiment:", result.get("sentiment"))

asyncio.run(main())

CLI

This library also includes a CLI exposed as the db command.

To make db available from anywhere, install it as an isolated tool with uv:

uv tool install .

This drops a db executable into ~/.local/bin (ensure it is on your PATH). Use --force to reinstall or upgrade after changes, or --editable to have source edits take effect immediately. Alternatively, a plain pip install . (or pip install -e .) also installs the db entry point into the active environment.

Standalone binary

Every release also ships a self-contained db binary for Linux (x86_64 and aarch64) and macOS (Apple Silicon) as a Python-free option. The installer detects your platform, verifies the SHA256 checksum, and installs (or upgrades) db into ~/.local/bin:

curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh

Pin a specific release or install location with flags (or the DB_VERSION / DB_INSTALL_DIR environment variables); re-running the installer upgrades an existing install in place:

curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh -s -- --version v0.2.1 --bin-dir ~/bin

How to use

export DIFFBOT_API_TOKEN=your-token-here

db extract https://www.example.com
db ask "What's the capital of France?"
db ask "What's the capital of France?" --json
db ask "What's the capital of France?" --schema capital.json
db crawl https://www.example.com --hops 1
db crawl-list-jobs
db crawl-delete-job crawl-1234567890
db web-search "diffbot knowledge graph"
db web-search "diffbot knowledge graph" -n 5 -f json
db entities "Apple CEO Tim Cook announced record quarterly earnings."
db entities "Apple CEO Tim Cook announced record quarterly earnings." -f dql

How to use with an agent

Once installed, this library will work alongside diffbot-skills to enable your agent full access to structuring web knowledge with Diffbot. Diffbot Agent Skills even unlocks some additional skills like crafting DQL from natural language.

diffbot-skills will pick up or install this library automatically.

Tests

Run the mock test suite:

python -m pytest

Run live integration tests against the real API (requires a valid token). The token is resolved the same way as everywhere else — the DIFFBOT_API_TOKEN environment variable or ~/.diffbot/credentials:

DIFFBOT_API_TOKEN=your_token python -m pytest -m live