Debugging Bot Traffic

This guide covers the most common issues you may encounter with PerfectSearch and how to resolve them. Use the checklist approach for each problem: verify the symptom, check the most likely cause, and apply the fix.

Why are bots seeing empty pages?

Empty pages typically mean the middleware is not matching the request path, no snapshot exists for that path, or your origin server is not accessible to the rendering worker. Work through the following checklist to identify and fix the root cause. Start with the bot view tool to confirm what bots actually see.

1. Check the bot view tool — Go to /tools/bot-view and enter the URL. If the tool shows empty content, the issue is confirmed.
2. Check middleware matcher — Verify that your middleware.ts matcher pattern covers the path in question. Common mistake: excluding paths that start with a specific prefix.
3. Check snapshot status — In the dashboard, navigate to your site and open the Snapshots tab. Search for the path. If no snapshot exists, the page has not been rendered yet.
4. Verify origin is up — The rendering worker must be able to reach your origin URL. Check that your origin returns a 200 status and the page loads correctly in a normal browser.

Why is my cache hit rate low?

A low cache hit rate means bots are frequently requesting pages that are not in the snapshot cache. Common causes include a short time-to-live setting, a large number of unique URL paths with query parameter variations, or a recent cache purge. Address each cause to improve your hit rate over time.

• Short TTL — Increase your cache TTL in dashboard settings. The default is 24 hours, which works well for most sites.
• URL parameter variations — Check if bots are appending tracking parameters (utm_source, fbclid, etc.) that create unique cache keys. Configure parameter stripping in your middleware or proxy settings.
• Recent cache purge — After a full cache purge, the hit rate drops to zero and rebuilds as bots revisit pages.
• Pre-warm the cache — Use the POST /v1/snapshot/queue API to queue your most important pages for rendering.

How do I check if middleware is working?

PerfectSearch adds an X-PerfectSearch-Status response header to every request it processes. If this header is missing from a response, the middleware is not matching that path. Verify your matcher pattern covers the URL, and ensure the middleware file is located at the project root next to your next.config.ts.

# Check for PerfectSearch headers with a bot User-Agent
curl -I -H "User-Agent: Googlebot" https://yoursite.com/some-page

Look for these headers in the response:

• x-perfectsearch: hit — snapshot was served from cache
• x-perfectsearch: miss — page was queued for rendering
• No header — middleware is not matching this path

How do I fix DNS errors?

DNS proxy mode can surface several common errors during setup. Each error has a specific cause and fix. Work through the list below based on the error you are seeing. Most DNS issues resolve within minutes once the configuration is corrected.

• ERR_TOO_MANY_REDIRECTS — Your origin server is redirecting back to the proxy domain, creating an infinite loop. Fix this by setting the origin URL in your dashboard to the non-proxied domain (e.g., your .vercel.app or .netlify.app URL).
• 526 SSL Error— The TLS certificate is still being provisioned. Wait 30 to 60 seconds and try again. Certificates are issued automatically via Let's Encrypt on the first request.
• Domain not found — The domain has not been added to PerfectSearch. Add your domain in the dashboard first, then configure the CNAME record.

How do I fix Vercel deployment protection issues?

If Vercel is showing a “Verifying your browser...” challenge page to bots, you need to configure a deployment protection bypass secret. This allows PerfectSearch proxy traffic to reach your Vercel origin without being blocked by the browser verification challenge. See the full step-by-step instructions in the dedicated guide.

Vercel Deployment Protection Bypass guide →

How do I check if snapshots exist for a path?

There are two ways to check whether a snapshot exists for a specific path. In the dashboard, navigate to your site, open the Snapshots tab, and search by path. You can also use the API by sending a GET request to /v1/snapshot with your site ID and the path as query parameters. A 200 response means a snapshot exists; a 404 means it does not.

curl -H "Authorization: Bearer ps_live_your_key" \
  "https://search.perfectline.io/api/v1/snapshot?site_id=site_abc123&path=/products/shoes"