Web Scraping Overview

POST /v1/web

Use this endpoint for general-purpose page fetching when the target is not one of the dedicated supported platforms like Google, YouTube, Reddit, X, or Upwork.

#Request Example

curl -X POST /v1/web \
  -H "Authorization: Bearer your-token" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "method": "GET"
  }'

#Request Body

{
  "url": "https://example.com",
  "method": "GET"
}

#What It Supports

• simple GET requests
• custom HTTP methods
• custom request headers
• JSON or form-style request bodies
• geo-targeted proxy routing via proxy_country
• captcha bypass mode via Bright Data

#What It Does Not Support

The generic web endpoint blocks these domains because they have dedicated product flows:

• google.com
• upwork.com
• facebook.com
• twitter.com
• instagram.com
• linkedin.com
• youtube.com
• tiktok.com
• pinterest.com
• reddit.com

Blocked domains return:

{
  "message": "Domain not supported."
}