Will AI crawlers scrape the CV on your website?

What did Meta's new crawling terms actually change?

Meta added a crawling paragraph to the commercial terms taking effect on 30 October 2026, then stripped out the wording about using crawled content to train its products — but the permission to crawl websites tied to a business account stayed in. The original text covered domains, web pages and web addresses connected to a business, and listed developing, improving, researching and training Meta's products among the purposes. After legal scholars flagged how broadly that could be read, Meta edited the clause so the remaining uses point at advertising and business products, said the revision doesn't change how its crawlers behave, and noted that site owners can still set preferences through robots.txt.

The timeline is the interesting part. A notice landed in Meta's business tools at the start of October, the crawling clause wasn't widely noticed until mid-month, and the pushback produced an edit within roughly two days. Nobody signed anything along the way — continuing to use the platform for business was the agreement, and the published guidance was effectively "if you don't accept this, stop using our products for business." A paragraph most people never open decided whether their website could be accessed, indexed, cached and re-crawled by automated software. The only remaining lever was a plain text file at the root of a domain that crawlers honour by choice.

You're almost certainly not a news publisher. But if your CV lives on a personal site, a portfolio page, a shared workspace, a document link you pasted into a forum, or a profile set to fully public, the same machinery reaches you — without the lawyers or the licensing negotiations. Your employment history, phone number, referees' names and home city are just text on a page, and a crawler doesn't tell the difference between an investigative feature and a CV. That turns an abstract debate into a concrete job-search decision, and it's the same decision behind which personal details belong on a CV at all.

Will AI crawlers really scrape the CV on your website?

If your CV page is publicly reachable and you've never written crawl rules for it, assume it has already been fetched — probably more than once, by more than one company. AI crawlers work the way search crawlers always have: they follow links, collect the text, and store it. Training crawlers like GPTBot, ClaudeBot and CCBot identify themselves by name, and Google exposes a separate robots.txt token, Google-Extended, purely to opt content out of its generative models. Answering crawlers are a different group — the ones that fetch a page because a user asked a question right now and the assistant needs a citation.

Here's where candidates get it wrong: robots.txt is a published preference, not a permission system. The major labs state they respect it, and the well-behaved crawlers do, but there's no technical enforcement — plenty of less reputable bots read the file and ignore it. Blocking training crawlers also doesn't remove anything already collected in earlier passes. And a disallow rule only stops fetching; it doesn't remove a URL from search results. A page that's blocked from crawling can still be listed from inbound links alone, with no description, because the crawler never got close enough to read your noindex tag.

Shared links deserve their own warning. A document set to "anyone with the link" isn't automatically indexed, but one public mention of that URL — in a forum post, a job-board profile, a tweet, a page someone else publishes — creates the crawlable path, and "publish to web" hands it over outright. PDFs offer no shelter either: a text-layer PDF is fully readable to any crawler and parses almost as cleanly as HTML. The practical conclusion is unglamorous. Treat anything you can reach without logging in as published to everyone, permanently, including systems that don't exist yet.

Which CV details should you never publish on a public page?

Never publish your full street address, your national ID number, your date of birth, your referees' names and phone numbers, or scans of signatures, licences and certificates. None of them help a recruiter judge whether you can do the job, and every one of them is a field a fraudster needs. Consumer-protection agencies have warned for years about the same pattern: a message claiming someone "spotted your CV online," followed by a request for an ID number or bank details for "onboarding." Scraped CVs are the raw material. A public page simply automates the collection step for them.

Referee details are the part candidates underestimate, and it's the one I'd push hardest on. Publishing a former manager's mobile number isn't your privacy to spend — it's theirs, and it hands cold-callers a verified, named contact with a confirmed employment link to you. Keep referees out of the document entirely until an employer asks, then warn the people first. The same logic applies to anything that looks like a document image. A photographed licence or a scanned signature is reusable by whoever downloads it, while the credential number alone tells a qualified recruiter everything they actually need.

Trimming also has to survive the machine on the employer's side. Strip the wrong things — or hide your contact details inside a header, a text box or an image — and the parser that reads your file returns a half-empty profile. Keep your name, a professional email and a single phone line as ordinary body text at the top, and keep your dates in a consistent format. If you want a field-by-field version of this cut list, the longer breakdown of what to leave off a CV and what to keep covers the edge cases, including bilingual and veteran CVs.

How do you keep one CV for crawlers and one for employers?

Write two versions of the same CV: a public profile built to be found, and a private file built to be read. The public one carries your name, current title, city and country, a four-line summary, employers with dates, headline achievements and a dedicated email address. The private one adds everything a hiring decision genuinely needs — full contact details, exact dates, project specifics, client names you're allowed to mention, referees on request. Same career, two levels of disclosure. The public page does discovery work while you sleep; the file you attach does the convincing once a named person at a named company has asked for it.

Build the public version to be skimmed by both humans and assistants, because recruiters increasingly arrive via an AI answer rather than a search result. Use real headings instead of styled divs, keep it single-column, put the role title before the company, and write achievements with numbers attached — "cut onboarding time from 11 days to 4" survives summarisation in a way "responsible for process improvement" never will. Skip the clever layout. Two-column templates, icons standing in for labels and text inside images all degrade badly when any automated reader, crawler or parser, tries to extract structure from them.

Keeping two documents in sync is where good intentions die, so make the private file the master and generate the public one from it. Rewrite once, then export the trimmed variant — that way a promotion gets added in one place, not two. An AI CV builder that produces ATS-friendly exports is well suited to this: paste the full version in, refine it by chat, and produce a tailored file per application in PDF or Word, with Hebrew CVs written in Hebrew and exported right to left. What you publish openly should always be the shorter, blunter cousin of what you send.

How do you check what your CV already leaks before publishing?

Audit it the way a stranger would. Search your own name in quotes, then your phone number, then your street name, and see what comes back — if a crawler found your CV page, a search engine almost certainly lists it, and a cached copy may still show details you've since deleted. Ask an AI assistant what it knows about you professionally; the answer reveals what's been collected far faster than reading any policy. Then open your public CV page in a private browsing window to confirm what's genuinely visible without a login.

If you want to limit future collection, do it in the right order. Add a robots.txt group naming the training crawlers you'd rather avoid — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended — and leave the answering crawlers alone so recruiters' assistants can still cite you. For anything you want out of search entirely, leave the URL crawlable and serve a noindex tag instead, because a page blocked in robots.txt never gets read closely enough for the tag to register. Expect partial results. Robots.txt is forward-looking and voluntary, so the real control is what you choose to publish.

Then check how the version you actually send performs, because privacy edits and parsing often pull in opposite directions. A free CV analysis returns a score out of 100, five category scores covering experience, tech stack, impact and ownership, clarity and structure, and ATS compatibility, plus a visual layout analysis showing how the file reads once a machine has pulled it apart. It won't make a privacy judgement for you — that stays your call. What it will tell you is whether the trimmed, header-free, single-column file you just built still parses cleanly enough to reach a human.

Frequently asked questions