How to Avoid Stumbling Blocks for Search Engine Spiders and Get Indexed

I have often been asked the same question by various website owners and other SEO enthusiasts and I received that same old question once again in an email from one of my blog readers yesterday. To summarize, his problem is that his website has been there for 3 months now but all its pages are not indexed. While I replied back to his email, I thought it would make a good blog post to put the tips on how to get your page indexed in Google and other search engines.

It is the job of search engine spiders to crawl and index websites we just need to provide them with the minimum requirement that they need to access a website and that minimum requirement is LINKS !

Search engine spiders move from one website to another website by following links, so if you are creating a new website and want it indexed by search engines, all you will need is to establish some inbound links from already indexed pages to your website and you will be well on your way to the Google index.

Now the question is, if it is that easy even then why websites are often not indexed or often partially indexed ( just the home page or may be a few pages and most pages remains un – indexed ) ?

We need to understand that Google bot or any search engine spider for that matter is nothing but an executable program and they are not human beings. Webmasters often create stumbling blocks for search engine spiders that prevents the bots from indexing a website properly. It is important that we keep away from them. So what are the stumbling blocks for search engine spiders ?

First let’s talk about issues that would simply STOP the search engine spiders.

• Pages that require log in. A search engine spider cannot sign up and submit the log in information – they are not human 🙂
• Pages accessible via select form and submit button ( again the same logic applies)
• Pages that require a drop down menu to be accessed.
• Web pages that are accessible only via a search box
• Documents that are blocked by robots Meta tag or a robots.txt file
• Bait and Switch pages. Pages that redirect to some other page before displaying the content. This is basically cloaking technique and search engines can ban a website for using this technique.

While the above factors are absolute no-no s if you plan to get your website indexed in search engines, there are other websites that still face indexing problem though they do not have any of the above factors. Why?

There are some factors which might not stop the search engine bots completely but make it extremely difficult for search engine spiders to crawl. If you have any of these factors on your website that might prevent all your pages from getting indexed.

• Dynamic URL s with more than two parameters – Spiders find it very difficult to crawl this type of pages and they are pretty reluctant on these pages because these pages often result in errors when accessed by non-human visitors.
• Pages that are buried below the third level on your website. To put in simple words, try to make all your pages accessible from the home page within a max limit of 3 clicks. Pages below the third level can also be indexed only if there are good numbers of external back links to those pages.
• Pages requiring a session id or cookie. For human visitors the browser can retain the session id or cookies; search engine spiders would often fail to retain these cookies or session id.
• Pages that contains frames often makes the page inaccessible to search engine spiders.
• If your pages are linked from pages that contain more than 100 links to other pages, the chances are very low the spiders would crawl all the links and index your pages.

If you have read the above and keep these simple points in mind you should no have any problem to get the search engine spiders to index your webpage. And if it still doesn’t work give me a shout and I will take a look 🙂