SEO Articles

The Role of the Robots.txt File to Improve Site Ranking!


Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com


MORE RESOURCES:

Internet Marketing News

SEO Consult Maintains Its Ranking as The UK's Number 1 Search ...
MSNBC - 3 hours ago
TopSEOs, the independent authority on search vendors, has ranked SEO Consult, the UK's leading Search Engine Optimisation (SEO) company, as the number one ...
There's No Shortcut to Good SEO...Or Is There? Search Engine Watch
SEO's View of Google's Greatest Achievements of 2008 Search Engine Roundtable
SEO Specialist Offers Free Services to Small and Midsize Companies PR Web (press release)
Practical Ecommerce (subscription) - CRN
all 38 news articles


Bruce Clay, Inc. Releases New Free SEO Tools
SYS-CON Media, NJ - 35 minutes ago
Both tools compile select data from Google, Microsoft and Yahoo! and put only the most useful SEO information within easy reach. ...


Increase Traffic Using Article marketing, Article Submission and ...
Meadow Free Press, ID - 9 hours ago
If you are a business person and have a website, you may have heard the term SEO or search engine optimization. Article marketing is a very effective way to ...
New Free One-Stop Website Providing Expert Resources Launched ... 1888 Press Release (press release)
all 2 news articles


This Time, SEO Gets Personal
Mediapost.com, NY - 7 hours ago
Tom Pick reminds us that CEOs and other executives need to appreciate the importance of their personal brand, but he doesn't mention that employees need to ...


How to use Google Analytics to Track SEO Rankings
Marketing Pilgrim, NC - 3 hours ago
With Google and other search engines continuing to roll out personalization updates, it is becoming increasingly difficult for SEOs to get an idea of their ...


SEO Googler How-Tos
Mediapost.com, NY - Jan 5, 2009
Get Google to read your keywords first. Things Google ignores. Don't link to sites in "bad neighborhoods." And, how to get your site listed in the search ...


Jeremy's SEO Blog: You need Google Chrome
NewsFactor Network, CA - 22 hours ago
Ok, maybe need is strong word, but you should definitely check out Chrome, Google’s new web browser. I’ve downloaded it myself and found it to be pretty ...


Chart of the Week: SEM/SEO Most Important Marketing Tactic
Practical Ecommerce (subscription), Grand Junction - Jan 5, 2009
Search engine marketing and optimization, which embodies both SEO and pay-per-click efforts, were clearly the most important marketing tools in the survey ...


Google Warns About Social Media "Schemes" in SEO Guide
Search Engine Roundtable - Jan 5, 2009
Remember a few months back when Google published their internal SEO guide? Well, Michael Gray asked if Social Media links is an easy way to SEO death row. ...
Newbie Thread of Week: Alphabetized META Tags Search Engine Roundtable
all 2 news articles


SEO Bots Rank Sites Better, Oh My!
Mediapost.com, NY - Jan 5, 2009
The post suggests we could begin to see the "first serious SEO Surfbot Nets" rollout in a few months. While the term "bots" typically connotes "malicious ...

SEO - Google News

Home | Sitemap | Forums | Contact Us | Advertise
© 2006 SEO Articles. A Real Web Talk Presentation.
Network Sites : Index Rated Directory | SEO Care Directory | Humour Inc. | Free Mobile Ringtones | A1hosts | SEO Blog
the four required words