XML Sitemaps and Robots.txt: Roles, Limits and Diagnosis
An XML sitemap tells search engines which URLs your site considers important. robots.txt tells compliant crawlers which paths they may fetch. They are complementary signals, not indexing controls: a submitted URL still needs to be accessible and useful, while a disallowed URL is not automatically private or absent from search.
XML sitemap and robots.txt do different jobs
For the French query cluster, the editorial plan records “sitemap XML” at volume 60, KD 86 and traffic potential 350, and “robots.txt” at volume 10. This is an Ahrefs Morocco estimate, 22 October 2026. It is not a prediction of rankings, visits, enquiries, or revenue.
A sitemap is a structured list of pages and, where relevant, media and localised versions that a site wants a search engine to discover. Google says it can improve crawling on large, new, or complex properties. It is only a hint: Google does not guarantee that every listed URL will be crawled or indexed. Useful internal links remain a core discovery path Google Search Central — Learn about sitemaps.
robots.txt is a public text file at the root of an origin, such as https://example.ma/robots.txt. Its User-agent, Allow, and Disallow rules manage crawler access. Google describes its main purpose as crawl-traffic management, not keeping HTML pages out of Google Search Google Search Central — Introduction to robots.txt.
The practical distinction is simple: a sitemap offers URLs for discovery; robots.txt limits fetching for selected crawlers and paths. Normally, do not list a canonical, important URL in a sitemap while blocking it from crawling. The crawler may then be unable to assess the page and its on-page signals.
Choose the right measure for the intended URL state
Start by defining the purpose of each URL family, rather than copying a generic robots file. Is it a page to discover, a low-value filtered view, a public page that should not be in search, or private material?
| Situation | Appropriate focus | Limitation to remember |
|---|---|---|
| Valuable service or editorial page | Canonical, indexable URL in sitemap plus descriptive internal links | A sitemap does not promise indexing or rankings |
| Internal-search, session, or filter URLs | Architecture and carefully reviewed crawl rules | robots.txt alone does not remove an existing result |
| Public page to keep out of search | An accessible noindex, or removal where appropriate | Blocking it first can stop the crawler reading noindex |
| Client, staging, or confidential area | Authentication or server-side access control | robots.txt is not a security control |
A disallowed HTML URL can still be discovered through links elsewhere and may appear as a URL-only result. The IETF standard is explicit that the Robots Exclusion Protocol is not access authorisation RFC 9309. Never place a sensitive path in a public robots file and treat that as protection.
Build a sitemap that reflects the intended site
Select URLs before generating the file
Treat the sitemap as a curated inventory, not every address a server can return. For each candidate, check the HTTP response, intended HTTPS host, canonical target, indexability, redirect status, and visitor value. Exclude tests, redirects, duplicates, internal-search results, tracking parameters, and intentionally excluded URLs.
Apply the same rule to each locale. Include a French, Arabic, or English page only when that local destination actually exists and is useful. A forced redirect or incomplete translation does not become an appropriate search result because it appears in XML. This review belongs in a responsible SEO Morocco launch or redesign process.
An XML sitemap needs a <urlset> root and namespace, a <url> element for each page, and an absolute URL in <loc>. It must be UTF-8; XML-reserved characters, including & in query URLs, need entity escaping. <lastmod> is optional and should represent a meaningful page update, not routine sitemap generation. Google says it ignores changefreq and priority Google Search Central — Build and submit a sitemap.
Make automation follow publishing reality
Many CMS platforms generate sitemaps. Check what happens when a page is published, redirected, moved to draft, or deleted. Automation helps when it reliably follows these lifecycle changes; otherwise it can spread mistakes across thousands of URLs.
A manual file can suit a small, stable site. Frequent publishing or a large catalogue calls for generation from publishing data. Avoid updating every <lastmod> without a meaningful content change.
One sitemap is limited to 50,000 URLs or 50 MB uncompressed. Larger sites should split files and use a sitemap index. Grouping by page type, product set, or language can help investigate processing in Search Console, but is not a ranking technique. See the Sitemaps protocol for file and index requirements.
Use robots directives sparingly
Audit the live file before editing
Fetch robots.txt for every relevant origin. www.example.ma and shop.example.ma can each have a separate file. Record existing rules, affected paths, their reason, and owner. An old staging directive or broad rule can have unexpected impact after a migration.
A minimal file might read:
```txt User-agent: * Disallow: /search/
Sitemap: https://www.example.ma/sitemap.xml ```
Use that example only when /search/ has no crawl value. Sitemap: declares a sitemap location and can appear more than once. It does not override Disallow or replace checking processing in Search Console. RFC 9309 defines user-agent groups and allow/disallow paths; other records may be handled separately by crawlers RFC 9309.
Validate impact, not only syntax
Disallow: / blocks the entire site for the matching group. Google supports some pattern characters, but crawler behaviour can vary. Keep rules short, purposeful, and tested against representative URLs.
After publication, check robots.txt, the sitemap, priority pages, and changed folders. Record the date, rule, owner, intent, and rollback route. Coordinate with landing pages so a campaign page is not inadvertently inaccessible to a crawler.
Investigate sitemap–robots conflicts in order
A Search Console “submitted but blocked by robots.txt” message shows a mismatch, not a universal cause of non-indexing. Scheme, host, trailing slash, parameters, and redirected destination can change which rule applies.
- Define the desired state: indexable, non-indexable, private, redirected, or removed.
- Check whether that exact URL or its final redirect destination is in the sitemap.
- Compare it with robots rules, canonical, page/meta or HTTP robots directive, response status, and internal links.
- Correct the source of conflict: remove a non-search candidate from the sitemap, or allow crawling when the crawler must read the page and its directive.
- Recheck in the relevant tools without assuming a fixed processing or indexing date.
A non-indexed page may instead involve usefulness, internal links, server errors, canonical selection, duplicates, or rendering. Sitemap and robots files are diagnostic controls, not an indexing button.
Limits and review points
A clean sitemap cannot compensate for an unhelpful page or no usable path to important content. A clean robots file cannot control every crawler or secure public material. XML validity does not show that an URL deserves to represent the site in search.
Review both files around a migration, CMS change, redesign, language launch, catalogue update, or indexability change. Where the same work changes content, conversion, and measurement, digital strategy and tracking & analytics can clarify ownership and validation without performance promises.
FAQ — XML sitemaps and robots directives
Does an XML sitemap guarantee indexing?
No. It is a discovery signal. Check accessibility, internal links, canonicalisation, indexability, and page usefulness as well.
Should robots.txt remove a public HTML page from Google?
Not by default. A blocked URL can still be discovered elsewhere. Use an accessible noindex, removal, or access control according to the actual goal.
Where should the sitemap be declared?
Add an absolute Sitemap: URL to robots.txt, submit it through Search Console, or use both. These are signalling methods, not a guarantee that every URL will be used.
Should every site URL be included?
No. Include canonical, indexable pages meant for search. Leave out redirects, duplicates, tests, unhelpful parameter URLs, and intentionally excluded pages.