SEO Articles

Search Bots, Crawlers, and Spiders


If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/


MORE RESOURCES:

Search Engine Watch

Bruce Clay, Inc. Releases New Free SEO Tools
MSNBC - 1 hour ago
Both tools compile select data from Google, Microsoft and Yahoo! and put only the most useful SEO information within easy reach. ...
Ecommerce Know-How: Information Architecture to Improve SEO and ... Practical Ecommerce (subscription)
Two New Free SEO Tools Released by Bruce Clay Search Engine Watch
The Role of Contextual Links In SEO Is Immense Promotion World (press release)
BigNews.biz (press release) - MSNBC
all 61 news articles


Search Engine Land

Considering In-house SEO? How to ease into in-house SEO Smoothly
Search Engine Land, CT - 10 hours ago
In-house SEO can be more cost efficient, but it can also be very costly because in-house SEO has many unique challenges. If you are planning to bring SEO ...
How PPC and SEO Can Benefit Companies In A Struggling Economy Promotion World (press release)
all 2 news articles


BizAtomic Launches Version 5.1 of its SEO Optimized E3 Ecommerce ...
Emediawire (press release), WA - 6 hours ago
St Augustine, FL (PRWEB) January 7, 2009 -- BizAtomic.com, a leading provide of Ecommerce Websites and SEO services, today announced the launch of version ...


Indianapolis SEO, IndianapolisSeo.com, an SEO and Web Design Firm ...
PR Web (press release), WA - 11 hours ago
Indianapolis SEO, www.IndianapolisSeo.com, a Web design and search engine optimization firm is launching the new website for PJ's College of Cosmetology ...


IndiaBook Indian SEO Company Launches Search Marketing Outsourcing ...
PowerHomeBiz.com (press release) - 4 hours ago
January 7, 2009 ( PowerHomeBiz ) - India -- India SEO a Professional Internet marketing Firm located in Chandigarh, northern part of India, since 1999 has ...


Search Engine Journal

Search for SEO eBooks and PDF Tutorials
Search Engine Journal - 3 hours ago
There are several really cool groups there found for [SEO] search each containing really good books. 2. Google offers a nifty advanced operator allowing to ...


Increase Traffic Using Article marketing, Article Submission and ...
Meadow Free Press, ID - Jan 6, 2009
If you are a business person and have a website, you may have heard the term SEO or search engine optimization. Article marketing is a very effective way to ...
New Free One-Stop Website Providing Expert Resources Launched ... 1888 Press Release (press release)
all 2 news articles


How to use Google Analytics to Track SEO Rankings
Marketing Pilgrim, NC - 21 hours ago
With Google and other search engines continuing to roll out personalization updates, it is becoming increasingly difficult for SEOs to get an idea of their ...


This Time, SEO Gets Personal
Mediapost.com, NY - Jan 6, 2009
Tom Pick reminds us that CEOs and other executives need to appreciate the importance of their personal brand, but he doesn't mention that employees need to ...


Chart of the Week: SEM/SEO Most Important Marketing Tactic
Practical Ecommerce (subscription), Grand Junction - Jan 5, 2009
Search engine marketing and optimization, which embodies both SEO and pay-per-click efforts, were clearly the most important marketing tools in the survey ...

SEO - Google News

Home | Sitemap | Forums | Contact Us | Advertise
© 2006 SEO Articles. A Real Web Talk Presentation.
Network Sites : Index Rated Directory | SEO Care Directory | Humour Inc. | Free Mobile Ringtones | A1hosts | SEO Blog
the four required words