How to Get Traffic From Perplexity: Crawler Access First, Content Second

When a brand asks why Perplexity never cites them, the first thing we look at isn't their content. It's their access logs, and whether PerplexityBot has successfully fetched anything at all in the last month. Often it hasn't, and nobody on that team decided it shouldn't. A hosting default decided it.

That's the unglamorous shape of this problem. Most advice about getting traffic from Perplexity is advice about writing answer-shaped content, which does matter, but it matters third. Eligibility comes first, and eligibility is a crawler-access question that lives in your robots.txt and your firewall rules rather than in your content calendar. We check it as part of the technical SEO and organic search work we run for brands whose DTC site carries real revenue, and if you'd rather somebody else ran the check, book a call and we'll pull it with you.

How do you get traffic from Perplexity?

Three things have to be true, in this order: Perplexity's crawler can read your pages, your pages answer a question somebody actually asks it, and your answer is specific enough that citing you beats citing a competitor. Most brands start at the third one and wonder why nothing moves.

The first condition is where nearly all the recoverable loss sits, because it's binary. A page that can't be fetched has a zero percent chance of being cited no matter how good it is. The other two are gradients. This one is a switch.

Perplexity runs two agents, and they don't follow the same rules

This is the detail that catches technical teams out, and it's documented publicly. Perplexity operates two separate user agents that do different jobs.

Agent	What Perplexity says it does	Obeys robots.txt?	What blocking it costs you
`PerplexityBot`	"designed to surface and link websites in search results on Perplexity", and "not used to crawl content for AI foundation models"	Perplexity's docs recommend allowing it in your robots.txt	Discovery. You're not in the pool of sources considered when an answer gets assembled.
`Perplexity-User`	Supports user actions. It visits a page to answer a question and link to it, and is "not used for web crawling or to collect content for training AI foundation models"	No. "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."	Live fetches. Somebody who points Perplexity straight at your URL gets nothing back.

Those descriptions are quoted from Perplexity's own crawler documentation, which is the only source on this subject worth trusting without a second opinion.

The asymmetry is the part to sit with. PerplexityBot builds the index that gets searched when somebody asks a question. Perplexity-User fetches a page in the moment, because a person pointed at it. So a Disallow aimed at the crawler doesn't make you invisible. It makes you undiscoverable while leaving you reachable to anyone who already has your link, which is the wrong half of the deal. You lose the citation that would have introduced you to a buyer who'd never heard of you, and you keep serving the ones who already know your name.

Why your store may already be blocking it

Almost nobody sits down and decides to block AI crawlers. It happens three ways, and all three are defaults rather than decisions.

Your CDN did it at signup

On 1 July 2025, Cloudflare made blocking AI crawlers the default for new domains and moved to a permission-based model, where new customers get asked at registration whether to allow them. If your store moved onto Cloudflare after that date and nobody answered that question deliberately, the answer on file is probably no. Existing customers can change the setting whenever they like, which is the good news: this is a checkbox, not a migration.

Your bot rules did it by shape

WAF and bot-mitigation rules written to stop scrapers catch AI crawlers too, because at the request level they look identical. Datacenter IP, no JavaScript execution, steady sequential hits against product URLs. We've seen perfectly good content sitting behind a challenge page that returns a 403 to anything without a browser fingerprint, and the brand's SEO team had no visibility into that layer at all. Robots.txt is a request. The edge is an enforcement point, and they're usually owned by different people.

Your robots.txt belongs to the platform, not to you

Shopify generates a robots.txt for every store. Per Shopify's own documentation, that generated file "works for most shops" and the template you'd use to override it "isn't included in any themes by default." Changing it means adding a robots.txt.liquid template, which replaces Shopify's generated output entirely.

Shopify's warning on that page is the real risk of the edit: the rendered template output doesn't always match the generated file exactly. Save your current /robots.txt before you publish and diff the two. It's easy to allow one agent and silently drop a rule that was doing something useful.

How to check it this week

Fetch your live robots.txt and search it for PerplexityBot, Perplexity-User, and any blanket User-agent: * with a Disallow: / underneath. Read the file your server returns, not the one in your repo.
Grep 30 days of access logs for both agent strings, and look at status codes rather than hit counts. A wall of 403s is a block. Zero lines is worse, because it means the request never reached your origin at all.
Verify the hits are genuine. Perplexity publishes the IP addresses each agent uses as JSON, at perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json, and its docs tell you to match on both the user-agent string and the IP. The crawler list is short, a dozen or so addresses, and it carries a generation timestamp because it gets rebuilt. Pull it fresh instead of pasting it into a firewall rule and forgetting about it.
Check the CDN and WAF layer on its own. A permissive robots.txt proves nothing if the edge is challenging the request before your application sees it.
Re-check 24 hours later. No crawler picks up a robots.txt change instantly, and treating a same-day log read as the verdict is how teams conclude a fix didn't work.

Which questions actually send traffic

Search Console won't help you here, and that's worth saying plainly because it's the instrument most teams reach for. Perplexity citations don't appear in it. You're working without the query report you're used to, which changes where the question list comes from.

Three sources we trust more than a keyword tool for this:

Ask it yourself. Run twenty real buying questions in your category through Perplexity and write down who gets cited. That's your competitive set for this channel, and it's frequently not the same set that outranks you on Google.
Your sales calls. The questions a prospect asks in the first ten minutes are the questions they asked an assistant the week before. They're phrased as questions already, which is the format that retrieves well.
Forum and community threads. Long-tail, specific, and phrased the way a person types when they don't expect ten blue links back.

Notice what all three have in common: they produce questions, not keywords. A keyword list optimized for Google's head terms is the wrong input for a system that retrieves passages to answer a sentence.

What earns the citation, once you're eligible

Being honest about what's knowable here is more useful than sounding certain. Perplexity hasn't published its source-selection method. The ranking weights circulating in SEO posts, the ones that tell you domain authority counts for fifteen percent or that nine in ten cited pages answer within the first hundred words, are vendor analyses rather than Perplexity documentation, and several of them contradict each other. Treat them as hypotheses somebody else hasn't tested properly.

What's defensible is what follows from how a retrieval-and-summarize system has to work:

Answer in the first two sentences under the heading. A summarizer extracts. It doesn't read to the end of your narrative build-up and reward you for the payoff.
One question per page, or per clearly labelled section. A page covering nine loosely related things is a weaker retrieval match than nine pages that each answer one thing.
Be the primary source for a number. Your own test results, your own pricing, your own methodology. A citation exists to show where a fact came from, and a fact you aggregated from three other sites points at those sites instead.
Date the page and keep it accurate. Freshness is one of the few tiebreakers entirely within your control.
Write for somebody who will never click. Most won't, and the citation is still a brand impression in front of a buyer at the research stage. The minority who do click arrive further down the funnel than almost anything else sends you.

That last point is the one brands argue with and the one we'd defend hardest. It's also the same discipline that works in classic organic search: win the specific question rather than the broad category term. On Amazon that's how Sparkle reached page one on 40+ keywords covering 500,000+ monthly searches, which was A9 and not AI search, and the transferable part isn't the tactic. It's the decision to go after the precise query somebody typed instead of the term with the biggest number next to it.

What this won't do

Perplexity isn't replacing your Google traffic this year, and a projection that says otherwise is selling you something. For the ecommerce brands we look at, AI-assistant referrals are still a small share of sessions.

They're also the share you'll measure worst. Sessions arriving from an AI answer frequently carry no referrer, so GA4 files them as Direct, and every figure you report is a floor rather than a total. We wrote up what GA4 can and can't attribute to ChatGPT and Perplexity separately, because fixing the measurement and fixing the eligibility are two different jobs. Do them in the wrong order and you can't tell whether the second one worked.

How we'd sequence it

Week one is the access audit above, and nothing else, because everything after it is conditional on it. Week two is choosing the five questions in your category you want to be the cited answer for, then auditing whether any page you already own answers one of them inside its first two sentences. Most brands find two or three pages that have the right answer buried at 60% depth, and re-cutting those is cheaper than writing anything new. Week three onward is the writing.

That's about a quarter of work for a normal catalog, and it belongs inside ordinary technical SEO rather than as a separate AI-search workstream. Our website optimization and SEO engagements start around $3,000 a month and this sits within them. The access audit on its own is small enough that we'll just do it on a call.

Check whether you're blocked before you write a single new word. If PerplexityBot can't fetch the page, the page doesn't exist. Book a call and we'll read the logs with you.