Government API Access Guide
Most government data portals provide REST APIs for programmatic access. This guide covers API key registration, rate limiting practices, pagination patterns, and data format handling across major government data providers.
API Key Registration
Most US federal APIs require a free API key for authenticated access. Keys typically provide higher rate limits and access to additional endpoints. Registration is usually free and instant.
Common registration steps:
- Visit the provider's API documentation or developer portal
- Provide name and email address (sometimes organization name)
- Receive API key via email (usually within minutes)
- Include key in requests as a query parameter or HTTP header
Authentication methods vary by provider:
- Query parameter:
?api_key=YOUR_KEY(Census, BEA, FRED) - Request header:
token: YOUR_TOKEN(NOAA CDO) - POST body:
{"registrationkey":"YOUR_KEY"}(BLS v2)
Rate Limiting
Government APIs enforce rate limits to ensure fair access. When rate limited, the API returns HTTP 429 (Too Many Requests). Best practices:
- Implement exponential backoff with jitter when receiving 429 responses
- Cache responses locally to avoid redundant requests
- Use bulk download for large datasets instead of paginating through the API
- Respect the Retry-After header when present
- Keep requests sequential rather than parallel for stricter rate limits
Bulk Download vs. API
Use the API when:
- You need a specific subset of data
- You need the latest data in real-time
- Your application makes on-demand queries
- Dataset size is manageable (under 10,000 records)
Use bulk download when:
- You need the entire dataset or large portions
- You are building a local analytical database
- Rate limits would make API access impractical
- You need historical data spanning many years
Pagination Patterns
Offset/Limit (most common)
Use offset and limit (or start/rows) parameters. Increment offset by limit for each page. Used by CKAN, FRED, NOAA.
Page/Per-Page
Use page and per_page parameters. Simpler but less precise for large datasets. Used by World Bank API.
Date/Geography Bounded
No pagination. Instead, constrain by date range and geographic area to control result size. Used by BLS, EPA, and some Census endpoints.
Data Formats
JSON
Default format for most government APIs. Lightweight, easy to parse, widely supported. Most APIs return JSON by default.
CSV
Common for bulk downloads and tabular data. Easy to import into spreadsheets and databases. Watch for encoding issues (UTF-8 BOM).
XML
Older format still supported by Census, BEA, and SDMX-based APIs (OECD, UN). More verbose but well-suited for hierarchical data.
GeoJSON
Standard for geographic data. Used by Census TIGER/Line, EPA facility locations, and spatial data portals. Renderable by mapping libraries.
Provider API Comparison
| Provider | Registration | Rate Limit | Pagination | Formats | Bulk Download |
|---|---|---|---|---|---|
| Census Bureau | Free API key via api.census.gov/data/key_signup.html | 500 requests/day (without key), higher with key | Offset-based (get parameter) | JSON, XML | Yes (FTP) |
| BLS | Optional (v2 API requires registration at bls.gov) | 25 queries/day (v1), 500/day (v2 with key) | None (specify date ranges) | JSON | Yes (flat files via FTP) |
| BEA | Free API key via apps.bea.gov/API/signup | 100 requests/minute, 100 MB/minute | None (request specific tables/years) | JSON, XML | Yes (interactive tables) |
| FRED | Free API key via fredaccount.stlouisfed.org | 120 requests/minute | Offset/limit parameters | JSON, XML | Yes (bulk data downloads) |
| Data.gov (CKAN) | Not required for read access | Varies by underlying provider | rows/start parameters (Solr-style) | JSON (API), varies (datasets) | Yes (direct file download) |
| EPA AQS | Email-based registration at aqs.epa.gov | Not published; reasonable use expected | None (date/geography bounded) | JSON, CSV | Yes (annual summary files) |
| NOAA CDO | Free token via ncdc.noaa.gov/cdo-web/token | 5 requests/second, 1,000/day | Offset/limit parameters | JSON, CSV | Yes (bulk order system) |
| World Bank | Not required | Not enforced (fair use) | page/per_page parameters | JSON, XML, CSV | Yes (bulk download portal) |