A site with several hundred pages without a sitemap leads to the same result: visitors going in circles and a search engine ignoring half of the content. The sitemap solves this problem by listing useful URLs in a format readable by indexing bots and, in its HTML version, by internet users. However, this file must be structured methodically, as a catch-all sitemap produces the opposite effect of what is intended.
XML Sitemap or HTML Sitemap: Two Files, Two Distinct Uses
The two are often confused. The XML sitemap is a technical file intended for search engines. It indicates the URLs to explore, their last modified date, and sometimes a frequency of updates. Google, Bing, and other crawlers read it to find pages they might not have discovered through internal linking.
The HTML sitemap, on the other hand, is a visible page of the site. It serves as a navigation plan for visitors, especially those who cannot locate content through the main menu. The W3C recommends offering multiple navigation methods (menu, search, sitemap) to cover different user profiles.
In practice, a site needs both versions. The XML feeds indexing, while the HTML enhances the browsing experience. When you can explore the sitemap of pharmaplanet.net, you can see how a sitemap organizes URLs into readable categories, facilitating both human reading and automated crawling.
Filtering URLs in the Sitemap: What You Include Changes Everything
The classic trap is to generate an automatic sitemap via your CMS and never touch it again. WordPress, Shopify, or Wix create default files that list all pages, including tag pages, date archives, internal search results, or paginated duplicates. The result: the search engine wastes its crawl budget on pages of no value.
A well-structured sitemap contains only the pages you actually want indexed. Specifically, you should exclude:
- Legal notice pages, privacy policy, and terms and conditions (unless they target a specific keyword)
- Internal search results pages, which create nearly infinite URLs without original content
- URLs with sorting or filtering parameters, common on e-commerce sites
- Noindex pages, as requesting the crawling of a page you refuse to index sends a contradictory signal
This filtering is not a technical luxury. On a site with a few dozen pages, the impact remains limited. On a product catalog or an editorial site with thousands of articles, the selection of URLs in the sitemap directly conditions the speed of indexing for recent content.
Hierarchical Structure of the Sitemap and Internal Linking
A flat sitemap, meaning a list of URLs without any grouping logic, fulfills its minimal technical role. Bots crawl the list and visit the pages. But this misses a more powerful navigation lever.
Organizing URLs by Thematic Silo
The best practice is to break the sitemap into coherent subsets. An e-commerce site can separate its sitemaps by product category. An online media outlet can create a sitemap by editorial section. The sitemaps.org protocol allows for an index file (sitemap index) that points to several child sitemaps, each limited to a specific scope.
This organization reflects the structure of the site and strengthens internal linking. A sitemap that reproduces the actual structure of the site helps engines understand the hierarchy of content. A category page groups product pages, and an editorial folder encompasses articles: the sitemap translates this logic into ordered URLs.
The Last Modified Date as a Freshness Signal
The lastmod tag in the XML sitemap indicates to bots when a page was last modified. Many CMSs automatically populate this date, but some update it with every save, even without any real content change. When all pages display today’s date, the signal loses all value.
It is beneficial to update this tag only when there is a substantial change in content. Engines use this information to prioritize recrawling recently modified pages.
Accessibility of the Sitemap on Mobile and via Keyboard
One angle that most guides overlook: the HTML sitemap is useless if it is inaccessible on small screens or via keyboard navigation. The W3C recommends designing the presentation for different window formats and ensuring that navigation elements remain understandable after text enlargement.
For an HTML sitemap, this means that the list of URLs must remain readable without horizontal scrolling on mobile. Links should have a sufficient clickable area, and the structure by headings (h2, h3) should allow assistive technologies to jump from one section to another.
Since June 2025, the European accessibility directive applies to certain categories of digital services, including e-commerce. Accessibility of the navigation path, including the sitemap, is becoming a regulatory obligation for the sites concerned. Automated checks are not enough: W3C recommendations suggest complementing them with real-world testing with users.
Declaring and Maintaining Your Sitemap Over Time
Creating a clean sitemap is pointless if engines do not know where to find it. Two declaration methods coexist:
- Add the line Sitemap followed by the full URL of the file in the robots.txt at the root of the site
- Submit the sitemap directly in Google Search Console or Bing Webmaster Tools, which also allows tracking crawl errors
- Combine both approaches to cover all engines, including those that do not offer a dedicated console
Maintenance is as important as the initial creation. When a page is deleted, it should be removed from the sitemap. When a section is added, the file should be updated or a new child sitemap created. An outdated sitemap sends bots to 404 errors, which degrades the quality of crawling across the entire site.
The sitemap file is not a static document generated once and for all. It is a management tool that reflects the actual state of the site at a given moment. It should be maintained like one maintains a catalog or a menu structure: regularly, by removing what no longer exists and adding what has just been published.
