Batch Processing

Process thousands of inputs in parallel with rate limiting and error handling

Overview

Batch processing runs the same API endpoint against multiple inputs — from a file or stdin — with parallel execution, rate limiting, and configurable error handling.

Basic Syntax

anysite api <endpoint> --from-file <file> --input-key <key> [OPTIONS]

Input Sources

From File

# users.txt — one value per line
satyanadella
jeffweiner08
billgates

anysite api /api/linkedin/user --from-file users.txt --input-key user

From CSV/JSON Files

The CLI auto-detects the format from the file extension:

# From CSV (uses the column matching --input-key)
anysite api /api/linkedin/company --from-file companies.csv --input-key company

# From JSONL (uses the field matching --input-key)
anysite api /api/linkedin/user --from-file users.jsonl --input-key user

From Stdin

cat urls.txt | anysite api /api/linkedin/user --input-key user --stdin

# Chain with other CLI commands
anysite api /api/linkedin/search/users keywords="CTO" count=50 -q --format jsonl | \
  jq -r '.urn.value' | \
  anysite api /api/linkedin/user --input-key user --stdin

Parallel Execution

anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5

Higher parallelism increases throughput but also API token consumption. Start with 3-5 workers and adjust based on your plan limits.

--delay adds a fixed pause (in seconds) between requests, but it only applies when --parallel is left at its default of 1 (sequential mode) — it has no effect once --parallel is greater than 1.

Rate Limiting

anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --rate-limit "10/s"

Rate limit formats:

• 10/s — 10 requests per second
• 100/m — 100 requests per minute
• 1000/h — 1000 requests per hour

Error Handling

anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --on-error skip

Strategy	Behavior
stop	Stop the entire batch on the first error (default)
skip	Print a warning, skip the failed input, and continue with the rest
retry	Accepted by the flag, but no retry logic runs for it in this version — a failed input is silently dropped from the results without stopping the batch

Progress and Statistics

# Real-time progress bar
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --progress

# Summary statistics after completion (total processed, succeeded, failed, elapsed time)
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --stats

Output Options

All output format options work with batch processing:

# Save batch results to CSV
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --format csv --output results.csv

# JSONL for streaming
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --format jsonl --output results.jsonl

# Quiet mode for piping
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 -q --format jsonl | anysite db insert mydb --table profiles --stdin

Complete Example

anysite api /api/linkedin/user \
  --from-file linkedin_urls.txt \
  --input-key user \
  --parallel 5 \
  --rate-limit "10/s" \
  --on-error skip \
  --progress \
  --stats \
  --format csv \
  --output enriched_profiles.csv

Options Reference

Option	Description
--from-file	Input file path (TXT, CSV, JSON, JSONL)
--input-key	Parameter name to map each input value to
--stdin	Read inputs from stdin instead of a file
--parallel, -j	Number of concurrent workers (default: 1)
--delay	Delay between requests in seconds — sequential mode only (default: 0)
--rate-limit	Maximum request rate (e.g., 10/s, 100/m, 1000/h)
--on-error	Error handling: stop, skip, retry (default: stop)
--progress / --no-progress	Show or hide the real-time progress bar
--stats	Display batch statistics after completion
--append	Append to an existing output file instead of overwriting it

Next Steps

Dataset Pipelines

Build multi-source workflows with dependency chains and scheduling

Database Operations

Load batch results directly into databases