PhantomJS is Dead: Using ScrapingBee API for Javascript Rendering

PhantomJS stopped being maintained in March 2018. Not deprecated, not soft-retired — the lead maintainer posted a note saying he was stepping down and recommended everyone migrate to Chrome Headless, and that was it. The project has had no meaningful commits since. If you’re still running it in a scraping pipeline, you’re running a headless browser built on a 2013 WebKit engine that modern websites actively detect and block.

The replacement for JavaScript rendering in scraping workflows is either a self-managed headless Chrome setup (Puppeteer or Playwright) or a managed rendering API that handles the infrastructure for you. ScrapingBee is the one I migrated to and the one I’d recommend to most teams. It takes a URL, renders it with a real Chrome instance, and returns the fully rendered HTML — without you managing browser instances, proxies, or headless Chrome’s increasingly demanding infrastructure requirements.

This post covers why PhantomJS failed, what ScrapingBee does differently, and the exact migration I ran on a Python scraping pipeline that was returning incomplete data on 31% of target pages. There’s also a straight comparison against the self-hosted alternatives at the end.

Why PhantomJS Actually Died?

The official reason was maintainer burnout. Vitaly Slobodin’s March 2018 post was honest about it — he’d been carrying the project largely alone, and when Google shipped headless Chrome, the main reason PhantomJS existed effectively disappeared. He said so directly and recommended people move on.

The technical reasons it became unusable are more specific than “it’s old.” PhantomJS runs on JavaScriptCore — WebKit’s JavaScript engine — rather than V8, which powers Chrome. The version it ships with is equivalent to a 2013-era browser. That means no native ES6 support in the rendering environment, no Promises without a polyfill, no async/await, no arrow functions if the target site isn’t transpiling to ES5. Sites that bundle with webpack and target modern browsers will often just fail to initialize in PhantomJS, returning a half-rendered page or nothing at all.

The other problem is detection. PhantomJS injects window.callPhantom into every page it loads — a global variable that bot-detection scripts check for explicitly. Its navigator.userAgent string identifies it by name. Its browser fingerprint fails a dozen checks that modern anti-bot systems run automatically. Sites that don’t actively hate scrapers will still block PhantomJS just because it trips every automated fingerprinting heuristic they run.

If you’re diagnosing a PhantomJS setup that’s failing quietly, the tell is sparse records rather than empty ones. The page loads — the static HTML comes back — but the JavaScript that populates the dynamic content never ran. The result looks like real data until you compare it to what the page actually shows in a real browser.

What JavaScript Rendering Actually Means for Scraping

A standard HTTP GET request fetches the HTML the server sends. For a React, Vue, or Angular app — or any page that loads content via XHR after the initial paint — that HTML is mostly a shell. The content that users see, and the content you want to scrape, is populated by JavaScript running in the browser after the page loads.

JavaScript rendering means spinning up a browser engine, loading the page in it, waiting for the JavaScript to execute and the DOM to settle, and then reading the HTML from the browser’s current state rather than from the server’s initial response. PhantomJS did this with an old WebKit engine. Headless Chrome does it with the same engine your target pages were built and tested against. The difference in success rate is not subtle.

The complication with headless Chrome is that running it in production is non-trivial. Chrome consumes significant memory per instance, crashes under concurrent load if not carefully managed, and requires a proxy layer if you’re scraping at any meaningful volume without getting IP-blocked. ScrapingBee’s job is to manage all of that so you don’t have to.

ScrapingBee Basics: The API That Replaced My PhantomJS Setup

ScrapingBee’s API is a single endpoint. You make a GET request to https://app.scrapingbee.com/api/v1/ with your API key and the URL you want to scrape. It returns the rendered HTML. That’s the whole thing at its simplest.

The free tier gives you 1,000 API credits to start, where one standard request costs one credit and a JavaScript-rendered request costs five. The Starter plan at $49/month gives you 150,000 credits. At my current usage of around 8,200 credits per month, Starter covers everything without thinking about it.

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = (
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
)

driver = webdriver.PhantomJS(desired_capabilities=caps)
driver.set_page_load_timeout(30)

try:
    driver.get("https://example.com/product-list")
    html = driver.page_source  # Often incomplete — JS may not have run
finally:
    driver.quit()

# Problems:
# - PhantomJS binary must be installed and on PATH
# - No proxy rotation — your IP gets blocked
# - window.callPhantom detected by anti-bot systems
# - ES6+ JS on the target page silently fails to execute

import requests
import os

SCRAPINGBEE_KEY = os.environ["SCRAPINGBEE_API_KEY"]

def scrape(url: str, wait_for: str = None, wait_ms: int = 0) -> str:
    params = {
        "api_key": SCRAPINGBEE_KEY,
        "url": url,
        "render_js": "true",
    }
    if wait_for:
        params["wait_for"] = wait_for   # CSS selector to wait for
    if wait_ms:
        params["wait"] = wait_ms        # Fixed wait in milliseconds

    response = requests.get(
        "https://app.scrapingbee.com/api/v1/",
        params=params,
        timeout=60
    )
    response.raise_for_status()
    return response.text

# Usage
html = scrape(
    "https://example.com/product-list",
    wait_for=".product-card",   # Wait until products are in the DOM
)

# No binary to install. No proxy management. Real Chrome rendering.

The migration from the PhantomJS version to the ScrapingBee version took about 40 minutes, most of which was adding the wait_for selectors for pages where I needed to wait for specific DOM elements rather than just the page load event. The actual API call structure is simpler than the Selenium setup it replaced.

Beyond Basic Rendering: ScrapingBee’s Useful Parameters

The basic render-and-return workflow handles most use cases. A few parameters that cover the rest of the common situations:

Executing custom JavaScript before scraping

Some pages require interaction before the content you want is visible — dismissing a cookie banner, clicking a “load more” button, or triggering a state change. The js_snippet parameter lets you pass JavaScript that runs after the page loads, before the HTML is returned. It takes a base64-encoded string.

import requests, base64, os

SCRAPINGBEE_KEY = os.environ["SCRAPINGBEE_API_KEY"]

# JavaScript to run after page load
js_code = """
const banner = document.querySelector('#cookie-banner .accept-btn');
if (banner) banner.click();
"""
js_b64 = base64.b64encode(js_code.encode()).decode()

response = requests.get(
    "https://app.scrapingbee.com/api/v1/",
    params={
        "api_key": SCRAPINGBEE_KEY,
        "url": "https://example.com/pricing",
        "render_js": "true",
        "js_snippet": js_b64,
        "wait_for": ".pricing-table",  # Wait for table to appear after click
    },
    timeout=60
)

html = response.text

Taking screenshots

Add screenshot=true to get a base64-encoded PNG of the rendered page. Useful for visual verification that the page is rendering correctly, or for archiving visual state alongside the scraped data. screenshot_full_page=true captures the entire scrollable page rather than just the viewport.

import requests, base64, os
from pathlib import Path

SCRAPINGBEE_KEY = os.environ["SCRAPINGBEE_API_KEY"]

response = requests.get(
    "https://app.scrapingbee.com/api/v1/",
    params={
        "api_key": SCRAPINGBEE_KEY,
        "url": "https://example.com",
        "render_js": "true",
        "screenshot": "true",
        "screenshot_full_page": "true",
    },
    timeout=60
)

# Response body is the base64 PNG when screenshot=true
img_data = base64.b64decode(response.content)
Path("screenshot.png").write_bytes(img_data)

Premium proxies for heavily protected pages

Standard ScrapingBee requests route through their rotating proxy pool. For sites with aggressive bot detection — major e-commerce platforms, travel aggregators, financial data sites — the premium_proxy=true parameter routes through residential proxies instead. Each premium proxy request costs 10–25 credits rather than 5. I use this selectively for the 15–20% of my target pages where the standard proxy pool gets challenged.

When to Self-Host Instead: Puppeteer and Playwright

ScrapingBee is the right choice when you want to start scraping quickly, don’t want to manage infrastructure, and your volume is under a few hundred thousand requests per month. Above that threshold, or if you need persistent browser sessions (maintaining login state across multiple page visits), self-hosting headless Chrome with Puppeteer or Playwright makes more economic sense.

The cost math at scale: a $6 DigitalOcean droplet running a single Puppeteer instance handles roughly 200–400 renders per hour depending on page complexity. Add a commercial proxy service (Bright Data, Oxylabs) and you’re looking at $100–$300/month for a setup that handles 50,000–100,000 requests/month. ScrapingBee’s Startup plan at $149/month gives you 500,000 credits — roughly 100,000 JS-rendered requests. Above that volume, the math increasingly favors self-hosting, but the operational overhead favors ScrapingBee right up until the cost difference becomes impossible to ignore.

Playwright has mostly replaced Puppeteer for new self-hosted setups. It supports Chromium, Firefox, and WebKit in the same API, the auto-wait behavior (waiting for elements to be actionable before interacting) is better designed than Puppeteer’s, and the TypeScript support is cleaner. If you’re building a self-hosted rendering setup in 2026, start with Playwright rather than Puppeteer unless you have an existing Puppeteer codebase worth preserving.

const { chromium } = require('playwright');

async function scrape(url, waitForSelector) {
  const browser = await chromium.launch({
    headless: true,
    args: ['--proxy-server=http://your-proxy:port'] // Add your proxy
  });

  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...'
  });

  const page = await context.newPage();

  try {
    await page.goto(url, { waitUntil: 'networkidle' });

    if (waitForSelector) {
      await page.waitForSelector(waitForSelector, { timeout: 15000 });
    }

    return await page.content();
  } finally {
    await browser.close(); // Critical — memory leak if omitted
  }
}

// What you're now responsible for vs ScrapingBee:
// - Browser binary installation and updates
// - Proxy pool management and rotation
// - Memory management (browsers leak if not closed cleanly)
// - Concurrency limits (one browser per process, use a pool library)
// - CAPTCHA handling (requires a separate service integration)

Comparison: ScrapingBee vs Self-Hosted vs Splash

Splash deserves a note. It’s an open-source JavaScript rendering service from the Scrapy team that runs in Docker and exposes an HTTP API similar to ScrapingBee. The JavaScript rendering engine is WebKit — which, yes, puts it closer to the PhantomJS problem than to headless Chrome. For most modern sites Splash handles fine, but if PhantomJS was failing you because of modern JS, Splash may fail you for the same reason. If you’re going the self-hosted route, Playwright is the more future-proof choice.

FAQ

Can ScrapingBee handle pages that require login?

Not directly — each ScrapingBee request is stateless, meaning it starts a fresh browser context with no cookies or session state from a previous request. For pages that require authentication, you have two options: pass cookies directly via the cookies parameter (extract them from an authenticated session and pass them as a header string), or use a self-hosted Playwright setup that can maintain a persistent browser context across requests. The cookie approach works for simple session-based auth; anything requiring OAuth flows or two-factor is better handled by Playwright.

How do I know if a page needs JavaScript rendering or not?

The fastest check: open your browser’s developer tools, go to Network, and reload the page with JavaScript disabled (you can do this in Chrome under Settings → Preferences → JavaScript). If the content you want to scrape is present in the initial HTML load, you don’t need JS rendering — a standard requests.get() will work fine and cost less. If the content is missing or the page shows a loading spinner, you need rendering. You can also view the raw HTML that a plain HTTP GET returns by looking at the first document response in the Network tab and comparing it to the rendered page source.

What happened to Selenium with PhantomJS? Can I just swap to Selenium with Chrome instead?

Yes, and it works. Selenium with ChromeDriver is a legitimate self-hosted replacement for Selenium with PhantomJS — same Selenium API, real Chrome under the hood. The limitation compared to ScrapingBee is everything you’re now responsible for: Chrome binary management, chromedriver version compatibility (this breaks constantly when Chrome auto-updates), proxy configuration, and any CAPTCHA handling. Selenium also has slower startup times per session than Playwright, which uses a more efficient browser connection protocol (CDP). If you’re already deeply invested in a Selenium codebase, the Selenium + ChromeDriver swap is the lowest-friction migration. For anything new, Playwright is worth the switch.

Does ScrapingBee’s rotating proxy pool get blocked by major sites?

The standard proxy pool gets challenged on some high-traffic targets — large e-commerce platforms, travel sites with aggressive bot detection, major financial data sources. That’s what the premium_proxy=true parameter is for: it routes through residential IPs rather than datacenter proxies, which pass most fingerprinting checks that block datacenter traffic. Expect premium proxy requests to cost 10–25 credits rather than 5. For a small number of high-value target pages, the additional cost is worth it. For bulk scraping across many targets, profile which pages need premium proxies and set the parameter selectively rather than on all requests.

How does ScrapingBee handle sites with infinite scroll?

You need to trigger the scroll behavior via the js_snippet parameter. The standard approach is to scroll the page to the bottom, wait for new content to load, and repeat. Since js_snippet runs once rather than in a loop, you’ll either write the scroll loop in the snippet itself or make multiple sequential API calls — each scrolling further and collecting the incremental content. For infinite scroll pages with a large number of items, multiple sequential requests is usually cleaner to implement and debug than a single complex JavaScript snippet.

Is there a Python SDK or do I have to use raw requests?

There’s an official Python SDK. Install it with pip install scrapingbee and use the ScrapingBeeClient class, which wraps the API calls and handles parameter encoding. The SDK is thin — it doesn’t abstract away the API parameters, it just makes the HTTP calls cleaner. For most use cases I use the raw requests approach shown earlier because I want explicit control over the parameters and the SDK doesn’t add enough to justify the dependency. Either approach works fine.

What’s the average response time for a JavaScript-rendered ScrapingBee request?

In my usage, between 2.1 and 3.8 seconds for most pages without a wait_for selector, and 3–6 seconds with a wait. The range is large because it depends heavily on the target page’s load time and how much JavaScript it runs. Pages that fire multiple API calls before displaying content take longer than pages that render from initial HTML. ScrapingBee has a 70-second timeout on their end; your client-side request should set a timeout of at least 60 seconds to avoid cutting off legitimate slow renders. Budget accordingly if you’re doing time-sensitive scraping.