Stop guessing whether Google sees your pages. This guide covers every method to check index status — from the Site: operator to Search Console, API scripts, and bulk diagnostics. You will learn the exact workflow we use when a client reports missing traffic.
Every SEO has been there: you publish a well-researched article, build three internal links, submit the sitemap, and wait. Two weeks later, zero impressions. You run a site:yourdomain.com/url check — nothing. The page is not indexed. This scenario is the norm, not the exception. Google's crawl budget is finite, and your content competes against millions of other URLs. Checking index status is the first diagnostic step, not a report you run once a month. In practice, when you audit a site with 50,000 pages, you will find that 15-30% of them are not indexed for reasons that range from blocks in robots.txt to weak internal linking depth.
A common situation we see: agencies who rely only on Search Console's 'Index Coverage' report miss pages that are indexed but not serving — soft 404s, canonicalized to wrong URLs, or trapped in a crawl loop. That is why this guide teaches you to check URL index status using multiple methods, each with its own blind spot. The Web Vitals documentation is a good technical reference for understanding how page performance can influence indexation decisions, but the core bottleneck remains: you cannot fix what you do not measure.
| Method | How It Works | Best For | Hidden Risk / Limitation |
|---|---|---|---|
| Site: operator Type site:domain.com/url in Google search | Returns snippet if indexed. No snippet = not indexed or unknown. | Quick ad-hoc checks on any device. No login needed. | May return stale cached data. Does not work for subpages if domain has noindex tags on parent. Do not rely on it for bulk. |
| Search Console URL Inspection Paste URL in Google Search Console | Shows index status, last crawl, canonical, coverage issue, and request indexing button. | Authoritative single-URL check. Also shows why a URL is not indexed (e.g., blocked, soft 404, alternate page). | Requires property owner access. Rate limit: ~600 checks per day. Does not show index status for URLs under different properties. |
| Index Coverage report (GSC) Filter by 'Excluded' or 'Error' | Lists all URLs with issues like 'Crawled - currently not indexed', 'Discovered - currently not indexed', 'Soft 404', 'Blocked by robots.txt'. | Bulk analysis of index gaps. See patterns like thin content or redirect loops. | Data can be 2-7 days old. Excluded URLs group similar errors. You must export and de-duplicate. Missing pages may not appear at all if never crawled. |
| Python Google Indexing API Script using indexing.googleapis.com | Submit up to 200 URLs per day for priority indexing. Can also query status for job posting or broadcast events. | Large-scale index monitoring for job ads or live events. Programmatic checks. | API scope limited to job posting and broadcast Event types. Standard web pages not supported. OAuth setup required. See our Python Google Indexing API setup guide. |
| Index status checker tools Third-party SaaS like Sitebulb, Screaming Frog, or custom scrapers | Automate site: operator or SERP API calls. Some integrate with GSC. | Agencies managing 10+ sites. Bulk index audits with reporting. | Many tools get blocked by Google for excessive site: queries. SERP API costs add up. False positives when Google shows 'missing' due to location or personalization. |
Type <code>site:example.com/url</code> in a private browser window. If snippet appears, the URL is likely indexed. If not, go to step 2.
Paste the exact URL. Read the coverage status. If it says 'URL is not on Google', note the reason (blocked, not found, etc.).
Inspect robots.txt and meta robots. A blocked URL will show 'Blocked by robots.txt' or 'Blocked by noindex' in GSC.
Go to GSC Crawl Errors report. Filter by the URL. Common issues: 5xx server errors, redirect chains, DNS timeouts. See our <a href="https://googlecrawlw.vercel.app/google-crawl-errors">Google crawl errors diagnostic</a>.
If the URL is valid and not blocked, click 'Request Indexing' in GSC. For bulk fixes, use the sitemap resubmit or the API.
Google's own data shows that a significant portion of crawled pages end up not indexed. The two most common GSC statuses are 'Crawled - currently not indexed' and 'Discovered - currently not indexed'. The first means Googlebot visited the page but decided not to include it in the index. The second means Google knows the URL exists (via sitemap or link) but has not yet crawled it. These are fundamentally different problems. A page that is crawled but not indexed often suffers from low content quality, thin value, or duplicate detection. A discovered but not crawled page is usually a priority issue — you need more internal links or a faster crawl rate. We have a dedicated guide on pages not indexed diagnostic that walks through 12 specific causes and fixes.
Edge cases you will encounter: JavaScript-rendered content that Google fails to execute, pages behind login walls that leak a 'noindex' from the server, and orphan pages with zero internal links. The most frustrating failure is a page that shows as indexed in the Site: operator but returns a 404 status code in a server log — that is a soft 404 that Google will eventually drop from the index. Always cross-check with server logs, not just GSC.
We audited a client with 12,400 product pages. Using the Index Coverage report in GSC, we exported all URLs with status 'Excluded' and found 3,820 URLs (31%) under 'Crawled - currently not indexed'. The breakdown: 2,100 were product pages with auto-generated descriptions under 200 words, 1,200 were filter pages with no unique value, and 520 were canonicalized to the wrong variant.
We set up a Python script using the requests library to check each URL's HTTP status and robots meta tag. Of the 3,820 excluded URLs, 740 returned a 200 but had a noindex meta tag due to a staging environment leak. Another 310 returned a 302 redirect loop. The cost: 2 hours of scripting, 3 hours of analysis. The result: we removed noindex from 740 pages, added unique content to 1,200 filter pages, and fixed canonicals. After 4 weeks, 2,100 of those 3,820 pages became indexed.
| GSC Status / Symptom | Likely Root Cause | Operational Fix | Risk of Ignoring |
|---|---|---|---|
| URL is not on Google No history in GSC | Page never crawled. No internal links. Sitemap missing or invalid. | Add to sitemap. Build 2-3 internal links from indexed pages. Use URL Inspection to request indexing. | Page remains invisible indefinitely. Crawl budget wasted on other URLs. |
| Crawled - currently not indexed Page was visited but excluded | Thin content, low word count, duplicate content, or low page quality signal. | Improve content depth (add 500+ words unique value). Consolidate with canonical. Remove or noindex if not useful. | Page may be recrawled and re-evaluated in 2-4 weeks, but if no change, it falls into a crawl hole. |
| Discovered - currently not indexed URL known but not crawled | Low crawl priority. Deep site architecture. No external links. Slow server response. | Increase internal link density. Submit via sitemap. Reduce server response time below 200ms. Check core web vitals. | Page may stay in this state for months. New content from competitors gets indexed first. |
| Blocked by robots.txt Disallowed rule blocks Googlebot | Accidental wildcard in robots.txt. Blocked directory. Test environment leak. | Edit robots.txt to allow the URL. Test with GSC robots.txt tester. Recrawl after fix. | All URLs under the blocked path remain invisible. Fix is quick but often overlooked. |
| Soft 404 Returns 200 but content is empty or error page | CMS returns 200 status for 404 pages. Empty search results pages. Login redirect with 200. | Return proper 404 or 410 status code. Remove empty pages from sitemap. Add canonical to existing page. | Google may treat entire site as low quality if soft 404s are common. Wastes crawl budget. |
Confirm the URL is in your sitemap and the sitemap is submitted in GSC.
Open a private browsing window to avoid personalized search results.
Run a server header check to confirm the URL returns 200 and has no meta robots noindex.
Verify that the page has at least one internal link from an indexed page on your site.
Check robots.txt for any disallow rule that might block the URL or its parent directory.
Ensure the page is not a canonical copy of another URL (check rel=canonical).
If using JavaScript rendering, test the page in Google's Rich Results Test to see the rendered content.
Use the Site: operator in a private browser: <code>site:yourdomain.com/url</code>. If a snippet appears, the URL is indexed. This method is free and requires no login. However, it may show stale data. For authoritative results, use Google Search Console's URL Inspection tool, which is also free and gives you the exact index status and any errors.
Export the Index Coverage report from Google Search Console. Filter by status 'Submitted and indexed', 'Excluded', or 'Error'. De-duplicate the list. For deeper analysis, use a crawler like Screaming Frog SEO Spider that integrates with GSC API or run a custom script to check each URL via the Site: operator (be careful with rate limits).
Submission is a request, not a guarantee. Common reasons: the page has a noindex meta tag, is blocked by robots.txt, returns a 4xx or 5xx status, or Google deemed the content low quality. Check Search Console URL Inspection for the specific reason. If it says 'Crawled - currently not indexed', improve the content and internal linking.
The Google Indexing API allows programmatic submission and status checks for URLs, but only for JobPosting and BroadcastEvent types (schema.org). It does not work for standard web pages. To use it, set up a Google Cloud project, enable the API, create service account credentials, and use the Python client library. See our <a href="https://pythongoogleindexingu.vercel.app/python-google-indexing-api-setup">Python Google Indexing API setup guide</a> for step-by-step code.
Use a multi-site dashboard like Sitebulb or custom scripts that loop through GSC properties. Each property has a daily limit of ~600 URL Inspection API calls. For bulk checks, rely on the Index Coverage report exported per site. Set up automated weekly reports to track changes in 'Excluded' status over time.
This status means Google visited the page but chose not to index it. Evaluate page quality: is the content thin, duplicate, or auto-generated? Add 500+ words of unique, useful content. Improve internal linking. Ensure the page has a clear purpose. If the page is not valuable, remove it or add noindex. Monitor the page's status after 2-4 weeks.
Wait 24-72 hours. Then check the Index Coverage report in GSC. Filter by 'Submitted and indexed' for your sitemap. If the URL does not appear there, run a URL Inspection check. If it says 'URL is not on Google', the sitemap may have an error, or the URL is blocked. Validate your sitemap with a tool like XML Sitemap Validator.
Common errors: 5xx server errors (500, 502, 503), 404 not found, redirect chains exceeding 10 hops, DNS resolution failures, and robots.txt blocks. Google Search Console's Crawl Errors report lists these. For a full diagnostic, see our <a href="https://googlecrawlw.vercel.app/google-crawl-errors">Google crawl errors diagnostic</a> page.
Checking index status is the first step, not the last. The real work begins when you find pages that are not indexed and you understand why. Use the methods above to build a repeatable workflow. For every URL you check, document the status, the cause, and the fix. Over time, you will see patterns: thin content on product pages, broken internal linking on blog archives, or crawl budget wasted on paginated filters. Address the root cause, not the symptom. The tools and guides linked throughout this article — from Web Vitals to the pages not indexed diagnostic — will help you go deeper. But the most important asset is your own disciplined process: check, diagnose, fix, verify.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.