Learning Hub · Intermediate · 15 minutes
How to Write a robots.txt
Direct crawlers correctly — and avoid the classic self-inflicted wounds — a practical, no-jargon walkthrough with the classic mistake flagged before you make it.
The short answer
Goal: direct crawlers correctly — and avoid the classic self-inflicted wounds. Time needed: 15 minutes. Difficulty: intermediate.
The walkthrough below covers each step, the mistake most people make, and the shortcut worth knowing. On HostingAlly plans, several of the steps are handled for you — we note where.
Intermediate
Difficulty
15 minutes
Time needed
5
Steps
24/7
Help if you're stuck
You don't need to be technical for this — the walkthrough is written for first-timers, tested against our own platform, and honest about which parts are genuinely fiddly versus merely unfamiliar.
One promise before starting: nothing in this guide is irreversible. Where a step could bite, we say so and give the undo.
The shape of the job
The whole job breaks into clear stages: know what it is and isn't, start from the near-empty default, never block what should rank, reference the sitemap and test before trusting.
None of the stages requires code or a terminal unless the guide explicitly says so — and where it does, the exact commands are given. The full step-by-step sits below; the surrounding sections cover the context that makes it stick.
Where this goes wrong (and how to not)
Disallow: / left over from development — one character blocking the entire site, invisible on the page itself, discovered when traffic flatlines; the launch checklist item that pays for the whole checklist.
It's worth internalising because it's not a rare edge case — it's the single most common way this task generates a support ticket. Knowing it in advance converts the whole job from risky to routine.
One habit that makes this easier forever
Don't block CSS and JS paths — Google renders pages fully, and blocked assets make your pages render broken in its eyes, taxing rankings for a rule that gained nothing.
Small habits like this are the real difference between people who find hosting easy and people who find it stressful — the tools are identical; the workflow isn't.
How HostingAlly makes this easier
We've automated the steps that don't deserve your time: certificates issue and renew themselves, the installer handles application setup, daily backups cover the what-if, and per-site settings live in a panel instead of config files. The guide above covers what's left — the part that's actually about your site.
And when a step misbehaves anyway, support answers around the clock — with the actual fix, not a knowledge-base link and a shrug. Half our best guides started as patterns in the tickets.
The platform this guide assumes
Every walkthrough in our Learning Hub is tested on the platform we actually run — cPanel, LiteSpeed, NVMe, one-click installs — so instructions match your screen instead of gesturing at it.
Support is answered by real people around the clock — including the questions other hosts call out of scope.
- Step-by-step, tested as written
- The gotcha flagged before you hit it
- Automation covers the boring steps
- 24/7 support if you get stuck
Why HostingAlly
What you get with every plan
No jargon tax
Terms are explained in place or linked to the glossary — nothing assumes you already know.
Help on standby
Stuck at step three at midnight? Support answers around the clock, mid-guide included.
Works as written
Every step tested on our own platform — no 'your host may vary' hand-waving.
Automation where it belongs
SSL, backups and installs run themselves here — the guide covers only what's genuinely yours to do.
The undo is always named
Where a step could bite, the guide says so and gives the reversal.
Written from real tickets
Our guides come from the support desk — the gotchas are the ones people actually hit.
Getting Started
How to get going
- 1
Know what it is and isn't
Crawl guidance, not access control — robots.txt requests polite crawlers stay out of paths; it hides nothing from browsers or bad actors.
- 2
Start from the near-empty default
Most sites need almost nothing — allow everything, point at the sitemap, perhaps exclude genuine crawl-waste paths (internal search results, cart URLs).
- 3
Never block what should rank
Disallowed paths can't be crawled — pages you want indexed must stay reachable; the file's power is precisely why its mistakes are expensive.
- 4
Reference the sitemap
The Sitemap: line hands crawlers the map in the same file they check first — one line, standing invitation.
- 5
Test before trusting
Search Console's robots tester (and a direct read at /robots.txt) confirms the file says what you meant — syntax errors fail silently toward over-blocking.
Included
What's covered on our plans
- Free SSL certificate on every plan, renewed automatically
- LiteSpeed server-level caching
- cPanel — the industry-standard control panel
- One-click installer for WordPress and 400+ apps
- 99.9% uptime commitment, monitored around the clock
- Daily automatic backups with self-service restores
- No setup fees, ever
- NVMe SSD storage as standard
- Per-site PHP version selection
- Free website migration handled by our team
FAQ
Your questions, answered
Can robots.txt hide private content?
No — it's a public file politely requesting avoidance, and it actually advertises the paths it names. Real privacy needs authentication (directory passwords, application logins); robots.txt is traffic management, not security.
Do I even need a robots.txt?
Sites work without one (absence means allow-all) — but a minimal file with a sitemap reference is worth having, and having it means controlling it rather than leaving the URL to chance.
Which control panel do you provide?
cPanel — the industry standard, which means every tutorial on the internet matches your screen, your backups restore anywhere cPanel runs, and skills learned here transfer for life. Plesk and DirectAdmin options exist on specific plans for those who prefer them.
What happens if I outgrow my plan?
You upgrade in place — plan changes apply from your client area without migrations or downtime, and our range runs from small shared plans through VPS to dedicated servers, so growth is an account change rather than a new-host project.
Will you move my existing website for free?
Yes — open a ticket with your current host's access details and we handle the whole move: files, databases, email and configuration, verified by you before DNS switches. The old site keeps serving until the new copy takes over, so visitors never see a gap.
Do you include email with hosting?
Yes — mailboxes on your own domain come with every hosting plan, with webmail, IMAP/POP/SMTP access and spam filtering. Standalone email hosting also exists for domains whose websites live elsewhere.
Where is your company based?
HostingAlly is a trading name of Bohzo Ltd, a company registered in England and Wales — a real, verifiable business with published terms under English law, which is worth checking about any host before you hand them your domain.
Keep exploring
How to Set Custom Error Pages
Turn dead ends into branded, useful detours — beginner level, 30 minutes.
How to Create a Subdomain
Spin up blog.yourdomain or app.yourdomain in minutes — beginner level, 5 minutes.
VPS Hosting
KVM virtual servers with root access, DDoS protection and flat pricing.
Web Hosting
NVMe cPanel hosting with free SSL, migration and a free first-year domain.
Your site deserves better hosting.
From your first site to a fleet of servers, the upgrade path is an account change, not a migration.
See Hosting Plans