Anysite CLI
Command-line toolkit for web data extraction, batch processing, dataset pipelines, and analytics
What is Anysite CLI?
Anysite CLI is a powerful command-line toolkit that gives you full control over web data — from extraction to analysis to database loading. It supports 3,600+ endpoints across 600+ data sources including LinkedIn, Instagram, Twitter/X, Reddit, YouTube, ProductHunt, Google, GitHub, Amazon, SEC EDGAR, and more.
GitHub Repository
Source code, issues, and contributions — MIT licensed
Single API Calls
Make instant API requests with flexible output formats and field filtering
Batch Processing
Process thousands of inputs in parallel with rate limiting and error handling
Dataset Pipelines
Build declarative YAML workflows with chained dependencies and scheduling
Database Integration
Load data into SQLite, PostgreSQL, or ClickHouse with auto-schema and diff-sync
Database Discovery
Auto-discover database structure and browse saved catalogs
LLM Analysis
Classify, summarize, enrich, and deduplicate data using AI models
SQL Querying
Query collected datasets with DuckDB SQL and generate analytics
Agent Protocol
Structured JSON output, exit codes, and discovery for AI agents
Agent Examples
AI agents for automated data collection and executive search
Quick Start
Get up and running in 60 seconds:
Install the CLI
```bash
curl -fsSL https://raw.githubusercontent.com/anysiteio/anysite-cli/main/install.sh | bash
```
Or via pip: `pip install anysite-cli` — see [Installation](/docs/cli/installation) for all methods.
Configure your API key
Get your API key from [app.anysite.io](https://app.anysite.io) and configure it:
```bash
anysite config set api_key YOUR_API_KEY
```
Update the schema cache
```bash
anysite schema update
```
Make your first request
```bash
anysite api /api/linkedin/user user=satyanadella
```
Key Capabilities
| Feature | Description |
|---|---|
| 3,600+ Endpoints | Access data from LinkedIn, Instagram, Twitter/X, Reddit, YouTube, ProductHunt, Google, GitHub, Amazon, and more |
| Flexible Output | JSON, JSONL, CSV, and table formats with field selection and exclusion |
| Batch Processing | Parallel execution with rate limiting and configurable error handling |
| Dataset Pipelines | YAML-based multi-source workflows with dependency chains |
| Database Loading | Auto-schema creation, upsert, and diff-sync for SQLite, PostgreSQL, and ClickHouse |
| LLM Analysis | Classify, summarize, enrich, generate, match, and deduplicate records |
| Incremental Collection | Resume from where you left off with cursor tracking |
| Database Discovery | Auto-discover database structure, LLM-enriched catalogs |
| Agent Protocol | Auto-JSON in pipes, structured errors, exit codes, discovery payload |
| Scheduling | Cron-based automation with webhook notifications |
How It Fits in the Anysite Ecosystem
REST API
Direct HTTP API for programmatic integration into your applications
MCP Server
AI agent integration via Model Context Protocol for Claude, Cursor, and other tools
n8n Nodes
Visual workflow automation with drag-and-drop nodes
Claude Code Skill
Want AI to build and run pipelines for you? Install the anysite-cli skill for Claude Code:
/plugin marketplace add https://github.com/anysiteio/agent-skills
/plugin install anysite-cli@anysite-skills
The skill gives Claude Code full knowledge of all anysite commands — describe what data you need in natural language and let the agent handle the rest. See Installation for details.
Anysite CLI is ideal when you need programmatic control, reproducible pipelines, or want to integrate data extraction into scripts and CI/CD workflows.