Access Control

PerfectSearch Access Control lets you define fine-grained rules that determine how individual crawlers interact with specific sections of your site. Create rules that match a crawler and a path pattern, then apply an action — allow, block, or rate-limit. Rules are evaluated by priority so you always have predictable, deterministic behavior.

How do access control rules work?

Each access control rule matches a combination of a crawler identifier and a URL path pattern, then applies an action: allow, block, or rate-limit. Rules are evaluated in priority order from highest to lowest, and the first matching rule wins. If no rule matches a request, the default behavior is to allow the crawler through.

A rule consists of four fields:

• Crawler — The bot to match. Can be a specific crawler name (e.g., GPTBot) or all to match every crawler.
• Path pattern — The URL path or paths the rule applies to. Supports exact matches, single wildcards, and multi-segment wildcards.
• Action — What to do when the rule matches: allow, block, or rate-limit.
• Priority — A numeric value that determines evaluation order. Higher numbers are evaluated first.

When a bot request arrives, PerfectSearch checks all rules for the site in descending priority order. The first rule whose crawler and path pattern both match the incoming request determines the outcome. This means you can create broad catch-all rules at low priority and override them with specific rules at higher priority.

What pattern matching syntax is supported?

Path patterns use a simple wildcard syntax with four forms: exact paths, single-segment wildcards, multi-segment wildcards, and the site-wide catch-all. Patterns are matched against the URL path of the incoming request, starting from the root. All patterns are case-sensitive and must begin with a forward slash.

Crawler patterns accept either an exact crawler name (e.g., GPTBot, ClaudeBot) or the keyword all to match every known crawler. Crawler names are case-sensitive and must match the exact User-Agent identifier listed on the AI Crawlers page.

What actions can I apply to crawlers?

PerfectSearch supports three actions for access control rules: allow, block, and rate-limit. Each action determines what response the matched crawler receives when it requests a page that falls within the rule's path pattern. Actions take effect immediately after a rule is created or modified.

• Allow — The crawler receives the pre-rendered snapshot normally, just like any other bot request. This is the default behavior when no rule matches, so you typically only create explicit allow rules when using a default-deny strategy or when you need to override a broader block rule with a higher-priority allow for specific paths.
• Block — The crawler receives a 403 Forbidden response with an empty body. The request is logged in your analytics but no snapshot is served. Blocked requests do not count toward your monthly snapshot serve quota.
• Rate-limit — The crawler is allowed through up to a specified number of requests per minute. Once the limit is exceeded, additional requests receive a 429 Too Many Requests response with a Retry-After header. Rate limiting is useful for crawlers that make excessive requests without fully blocking them. Scale+

How does rule priority work?

Each rule has a numeric priority value. Rules with higher priority numbers are evaluated first. When a request arrives, PerfectSearch checks rules from highest to lowest priority and applies the first rule that matches both the crawler and path pattern. If no rule matches, the default action is allow.

Priority values can be any integer. PerfectSearch does not enforce uniqueness — if two rules share the same priority and both match, the one created most recently takes precedence. However, for clarity, it is best practice to use unique priority values.

Here is an example showing how priority ordering resolves conflicts:

In this configuration, GPTBot can access /blog/** pages (priority 100 matches first) but is blocked everywhere else (priority 50 catches remaining GPTBot requests). All other crawlers are allowed site-wide (priority 10 is the catch-all).

What access control features are available on each plan?

Access control capabilities vary by plan. Trial and Starter plans do not include access control rules. The Growth plan supports site-wide rules using the catch-all path pattern. The Scale plan unlocks the full feature set including granular path patterns and rate-limiting actions.

How do I create an access control rule?

To create a rule, go to your site in the PerfectSearch dashboard, open the Access Control tab, and click the “Add Rule” button. Fill in the crawler, path pattern, action, and priority, then save. The rule takes effect immediately for all subsequent bot requests.

Step-by-step:

1. Navigate to Dashboard and select your site.
2. Open the Access Control tab.
3. Click Add Rule in the top-right corner.
4. Select a Crawler from the dropdown — choose a specific crawler name or “All crawlers” to match everything.
5. Enter a Path pattern (e.g., /blog/** or /**).
6. Choose an Action: Allow, Block, or Rate-limit (Scale plan only).
7. Set a Priority number. Higher numbers are evaluated first.
8. Click Save. The rule is active immediately.

You can also use the API to manage rules programmatically. See the API Reference for details.

What are rule presets?

Rule presets are pre-configured sets of access control rules for common use cases. Applying a preset creates multiple rules at once, saving you from configuring each crawler individually. Presets are available on the Growth plan and above and can be applied from the Access Control tab.

• block-training — Blocks all training crawlers while leaving retrieval crawlers and Applebot-Extended allowed.
• allow-retrieval-only — Creates explicit allow rules for retrieval crawlers and a default block for everything else.
• block-all — Blocks all twelve tracked AI crawlers. No AI system can access your content. Use this for a complete AI opt-out.

To apply a preset, open the Access Control tab, click the “Presets” dropdown, and select the preset you want. Review the rules that will be created, then confirm. Preset rules are standard access control rules — you can edit or delete them individually after they are created.