Skip to main content

Web Scraping API

Extract data from any website with anti-bot protection bypass, global proxy rotation, CAPTCHA solving, and 99%+ success rates. Perfect for SEO tools, price monitoring, lead generation, and data intelligence.

🚀 Key Features​

  • Anti-Bot Bypass - Cloudflare, DataDome, advanced bot detection
  • CAPTCHA Solving - Automatic CAPTCHA bypass with machine learning
  • Global Proxy Network - 8 countries with residential and datacenter IPs
  • JavaScript Rendering - Full browser automation for dynamic content
  • Custom Fingerprints - Randomized user agents and browser signatures
  • Domain Protection - Intelligent blocking of restricted domains
  • Fast Response - Average response time under 1-6 seconds
  • Credit-Based Pricing - Pay only for successful requests

📋 Endpoint​

POST requests only - All web scraping requests must use the POST method:

POST https://scrapingapi.qoest.com/v1/web

🔑 Authentication​

All requests must include your API token in the Authorization header using Bearer authentication:

Authorization: Bearer YOUR_API_TOKEN

📊 Parameters​

ParameterRequiredTypeDescription
urlYesstringTarget URL to scrape (must be valid URL format)
request_methodNostringHTTP method: GET, POST, PUT, DELETE (default: GET)
request_headersNoobjectCustom headers as key-value pairs
request_bodyNoobjectRequest body data (required for POST requests)
proxy_countryNostringProxy country code: it, nl, fr, de, us, ca, gb, au
bypass_captchaNostringEnable CAPTCHA bypass: true or false (cannot be used with request_method, request_headers, request_body, proxy_country)

💰 Pricing​

Monthly Subscription Tiers​

PlanPriceCreditsCost per Credit
Tier 1$10/month10,000 credits$0.001
Tier 2$50/month55,000 credits$0.0009
Tier 3$100/month115,000 credits$0.00087
Tier 4$500/month600,000 credits$0.00083
Tier 5$1,000/month1,250,000 credits$0.0008

Credit Usage by Feature​

FeatureCredits Required
Basic Request1 credit
Proxy Country2 credits
CAPTCHA Bypass3 credits

Usage Examples​

  • Tier 1 ($10): 10,000 basic requests OR 5,000 proxy requests OR 3,333 CAPTCHA bypass requests
  • Tier 2 ($50): 55,000 basic requests OR 27,500 proxy requests OR 18,333 CAPTCHA bypass requests
  • Tier 3 ($100): 115,000 basic requests OR 57,500 proxy requests OR 38,333 CAPTCHA bypass requests

📝 Examples​

Basic Website Scraping (1 Credit)​

curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Custom Headers for User Agent Rotation​

curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://target-website.com/data",
    "request_headers": {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
      "Accept": "application/json",
      "Referer": "https://google.com"
    }
  }'

POST Request with Form Data​

curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://forms.example.com/submit",
    "request_method": "POST",
    "request_body": {
      "search_query": "web scraping tools",
      "category": "technology"
    },
    "request_headers": {
      "Content-Type": "application/json"
    }
  }'

API Scraping with Custom Headers and Request Body​

Scrape external APIs by proxying requests through the web scraping API with custom headers and request body parameters:

curl --location 'https://scrapingapi.qoest.com/v1/web' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_TOKEN' \
--data '{
  "url": "https://scrapingapi.qoest.com/v1/reddit",
  "request_method": "POST",
  "request_headers": {
    "authorization": "Bearer your-token",
    "Content-Type": "application/json"
  },
  "request_body": {
    "query": "upwork job alerts",
    "type": "posts"
  }
}'

Response Example:

{
  "response": "{\"response\":[{\"title\":\"Has anyone else started getting tons of New Job alerts all of a sudden?\",\"url\":\"https://www.reddit.com/r/Upwork/comments/1m1krnj/has_anyone_else_started_getting_tons_of_new_job/\",\"time\":\"2mo ago\",\"created_at\":\"2025-07-16T18:40:25.839Z\",\"subreddit\":\"Upwork\",\"votes\":\"1\",\"comments\":\"10\"},{\"title\":\"Job Alerts Suck\",\"url\":\"https://www.reddit.com/r/Upwork/comments/1hv0g9n/job_alerts_suck/\",\"time\":\"8mo ago\",\"created_at\":\"2025-01-06T14:37:56.682Z\",\"subreddit\":\"Upwork\",\"votes\":\"15\",\"comments\":\"33\"}]}"
}

Proxy Country Rotation (2 Credits)​

curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://geo-restricted-site.com",
    "proxy_country": "us"
  }'

CAPTCHA Bypass for Protected Sites (3 Credits)​

curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://protected-website.com",
    "bypass_captcha": "true"
  }'

📤 Response​

Success Response (200)​

{
  "response": "<!DOCTYPE html>\n<html>\n<head><title>Example</title></head>\n<body><h1>Example Domain</h1></body>\n</html>"
}

Error Responses​

Validation Error (422)​

{
  "message": "The given data was invalid.",
  "errors": {
    "url": ["The url field must be a valid URL."],
    "request_method": ["The selected request method is invalid."],
    "request_body": ["The request body field is required when request method is POST."]
  }
}

Insufficient Credits (403)​

{
  "message": "You do not have enough credits to perform this action."
}

Domain Blocked (403)​

{
  "message": "Domain not supported."
}

Authentication Required (401)​

{
  "message": "Unauthenticated."
}

⚠️ Validation Rules​

URL Requirements​

  • Must be a valid URL format (http:// or https://)
  • Domain restrictions apply to social media and search engines

HTTP Methods​

  • Only POST requests are allowed
  • request_method parameter supports: GET, POST, PUT, DELETE (for target site)
  • POST method to target sites requires request_body parameter

Headers & Body​

  • request_headers: Must be an object/array format
  • request_body: Must be an object/array format (required for POST)

Proxy Countries​

  • Available: it, nl, fr, de, us, ca, gb, au
  • Cannot be combined with bypass_captcha

CAPTCHA Bypass​

  • Valid values: true, false
  • Cannot be used with proxy_country parameter
  • Uses specialized infrastructure for protected sites

Restricted Domains​

The following domains are blocked for specialized APIs:

  • google.com (use Google Search API)
  • upwork.com (use Upwork Jobs API)
  • facebook.com, twitter.com, instagram.com, linkedin.com (use Social Media API)
  • youtube.com, tiktok.com, pinterest.com, reddit.com (use Social Media API)

🚨 Common Issues​

  • Domain Blocked: Use specialized APIs for social media, search engines
  • Insufficient Credits: Check credit balance before requests
  • Validation Errors: Ensure proper parameter formats and required fields
  • CAPTCHA + Proxy: Cannot combine both features in single request
  • JavaScript Sites: Automatically handled by browser automation
  • Rate Limiting: Implement delays between requests for better success rates

🎯 SEO & Business Use Cases​

SEO Competitor Analysis & SERP Tracking​

Monitor competitor websites, extract meta tags, analyze content strategies, and track SEO performance.

# Scrape competitor website for SEO analysis
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor-website.com",
    "request_headers": {
      "User-Agent": "SEO-Tool-Bot/1.0"
    }
  }'

E-commerce Price Intelligence & Product Monitoring​

Track competitor pricing, product availability, reviews, and market trends for e-commerce optimization.

# Monitor product prices across multiple regions
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://marketplace.com/product/12345",
    "proxy_country": "us"
  }'

Lead Generation & B2B Contact Scraping​

Extract business contacts, email addresses, company information, and sales leads from directories and websites.

# Scrape business directory for lead generation
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://business-directory.com/companies?industry=technology",
    "request_headers": {
      "Accept": "text/html,application/xhtml+xml"
    }
  }'

Content Marketing & Blog Monitoring​

Monitor industry blogs, news sites, and content platforms for content marketing insights and trend analysis.

# Monitor industry blogs for content ideas
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://industry-blog.com/latest-articles",
    "bypass_captcha": "true"
  }'

Real Estate Market Research & Property Data​

Scrape property listings, rental prices, market analytics, and real estate trends for investment decisions.

# Scrape property listings with geo-targeting
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://property-site.com/listings/california",
    "proxy_country": "us"
  }'

Market Research & Data Intelligence​

Gather market data, consumer insights, and business intelligence from various online sources.

# Research market trends and consumer data
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://market-research-site.com/reports",
    "request_method": "POST",
    "request_body": {
      "industry": "technology",
      "region": "north-america"
    }
  }'

E-learning & Course Content Aggregation​

Monitor online courses, educational content, and training materials for competitive analysis.

# Scrape course platforms for content analysis
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://learning-platform.com/courses/web-development",
    "request_headers": {
      "Accept-Language": "en-US,en;q=0.9"
    }
  }'

Financial Data & Investment Research​

Extract financial news, stock information, and investment data for trading and analysis platforms.

# Scrape financial news for investment insights
curl "https://scrapingapi.qoest.com/v1/web" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://financial-news.com/market-updates",
    "proxy_country": "gb"
  }'

📚 Related APIs​