Troubleshooting Indexing Issues: When Your Website Is Crawled but Not Indexed
If you’ve recently launched a website and notice that search engines are crawling your pages but not indexing them, it can be a perplexing experience. This situation often indicates underlying issues that need to be addressed to improve your site’s visibility in search engine results.
Common Scenario: Crawlability but No Indexing
One frequent scenario involves search engine bots successfully crawling your website but being unable to index your pages. An example of such a message might resemble: “Page fetch: robots.txt unreachable.” This suggests that while bots can visit your site, they’re encountering obstacles preventing them from properly reading your instructions — particularly the robots.txt file, which guides search engines on which pages to crawl or avoid.
Understanding the Background
In your case, you purchased your domain through GoDaddy and set up redirects to Format.com, a platform designed for building photography websites. Redirects and hosting configurations can sometimes interfere with how search engines access your content, especially if certain files like robots.txt are inaccessible or misconfigured.
Why Does This Happen?
- Robots.txt Accessibility Issues: If the robots.txt file is missing, misplaced, or inaccessible due to server configuration issues, search engines may have difficulty understanding your site’s crawling permissions.
- Redirects and Hosting Setup: Redirects from your custom domain to third-party platforms can sometimes create challenges for search engines, especially if the redirect setup isn’t SEO-friendly.
- Server Connectivity and Reachability: Technical issues such as server downtime or misconfigured DNS settings can also hinder search engine bots from accessing crucial files.
Practical Steps to Resolve the Issue
- Verify Robots.txt Accessibility
- Use online tools like Google Search Console’s URL Inspection Tool or third-party robots.txt testers to verify whether your robots.txt file is accessible and correctly configured.
-
Ensure that your robots.txt file is located at the root domain (e.g., yourdomain.com/robots.txt) and is publicly reachable.
-
Check Your Redirect Configuration
- Confirm that the redirects from your domain to Format.com are set up properly and do not block or obstruct search engine bots.
-
Consider using canonical tags to clarify the preferred version of your pages.
-
Consult Hosting and Platform Support
- Reach out to GoDaddy support to ensure that your DNS and hosting settings are correct.
- Contact Format.com support to verify that their platform allows proper crawling and indexing of redirected