Search engines employ automated processes or robots, casually known as ’spiders’ or ‘crawlers,’ to find various sites. They’re an important part of the whole internet infrastructure, but why is that so? What do they do exactly?

Robots actually have the same basic functionality that earlier browsers had. Just like these early browsers, search engine robots do not have the ability to do certain things. Robots cannot get past password protected areas. They do not understand frames, Flash movies, nor Images or JavaScript. Even if you use a robot, you have to click the buttons on your website. They can cease to function while using JavaScript navigation or when indexing a dynamically generated URL. A search engine robot retrieves data and finds information and links on the web.

The ’submit url’ function places the url into a list of urls the robots are going to explore. Even without submitting your url directly, robots will try to find your site by following links. That’s why building visibility through a web of links is important.

By collecting and following links, robots manage tn transport themselves all over the internet. Think of it as an internet equivalent of the roads we use in our lives. Robots travel on the roads and read the signposts so they know what leads to where.

When the robots return, the information they gathered is assimilated into the search engine’s database. Through a complex algorithm, this data is interpreted and web sites are ranked according to how relevant they are to various topics that would be searched for. Some of the bots are quite easy to notice – Google’s is the appropriately-named Googlebot, where Inktomi utilizes a more ambiguous bot named Slurp. Others may be difficult to identify at all.

A robot ‘reads’ your site by collecting data on any visible text, on tags you may have in the coding of your page, and on any links available. These are the things that determine what the search engines ‘think’ your content is about, so these are the things you really need to pay attention to when building a site that you want to have high visibility in search results.

If you?re interested in seeing which pages the spiders have visited on your website, you can check your server logs or the results from your log statistics. From this information you?ll know which spiders have visited, where they went, when they came, and which pages they crawl most often. Some are easy to identify, such as Google?s ?Googlebot,? while others are harder: ?Slurp? from Inktomi, for example. In addition to identifying which spiders visit, you can also find if any spiders are draining your bandwidth so that you can block them from your site. The internet has plenty of information on identifying these bad bots. There are also certain things can prevent good spiders from crawling your site, such as the site being down or huge amounts of traffic. This can prevent your site from being re-indexed, though most spiders will eventually come by again to try re-accessing the page.

Justin Harrison is an internationally recognised Internet Marketing Consultant expert who provides world class Search Engine Optimization to website owners. For more information visit: http://www.seorankings.co.za

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

If you enjoyed this post, make sure you subscribe to my RSS feed!