HOME | EBOOKS | COMMUNITY | TELECOMMUTING JOBS | ADVERTISE
Work at Home Moms

Welcome to MyMommyBiz



Creating Your Robots.txt File

You should not create your robots.txt file in your web creator program, but create it in either NotePad or TextPad. It should be saved as robots.txt (NOT with an .html or .htm extension). It will then be uploaded to the root directory, so it will appear as http://www.example.com/robots.txt

Allowing all bots
Your basic robots.txt file (what you likely want to use if you haven't been using one until now) will allow all bots to index all pages of your site. The * is generic to refer to all bots.

User-agent: *
Disallow:

Disallowing all bots
Your basic robots.txt file would not allow any bot to access any page on your site they can find.

User-agent: *
Disallow: /


Disallowing specific bots
If you would like to not allow certain bots to index your pages, you can refer to bots by name and disallow them. For example, many webmasters prefer not to allow Google's Image Search bot to index any of the images on their site.

User-agent: Googlebot-Image
Disallow: /

Disallowing Specific Pages & Directories
You may want to not allow bots to index certain pages or directories on your site. Do keep in mind that if you have any private directories that are listed in your robots.txt that you should password protect them. Otherwise you have just alerted rogue bots (that ignore contents of your robots.txt file or specifically target disallowed pages and directories) as well as any competitors to your private directories. In these cases, you should disallow robots using meta tags instead.

cgi-bin is a common directory to disallow, since it normally contains scripts, many that don't need indexing and which can go through a lot of bandwidth as a result.

User-agent: *
Disallow: /cgi-bin/
Disallow: /privatedirectory/
Disallow: /directory/largepage.htm

Once it is completed, save it as robots.txt and upload it to your server.

Alternatively, you can use our free robots.txt generator to have us create your robots.txt generator for you.

MyMommyBiz Sponsors



MAIN SECTIONS

Business Idea Central
Home Party Plan Central
Loans & Financing
Telecommuting Jobs
Mystery Shopping
Paid Surveys List
MyMommyBiz eBooks
eBusiness
Search Engine Optimization (SEO)
Selling on eBay
Marketing & Publicity
Advice & Resources
Goals & Motivation
Web Design Help
Web Hosting
Craft Business Central
Education
Spyware & Viruses Help
Business Software
Business Directory
Classifieds & BizOps
Avoiding Scams

IN DEPTH CHANNELS

PARTY PLAN CENTRAL
Company List
Choosing a Company
Leading Your Downline
Fundraisers
Marketing
Recruiting
Hostess Tips
Business Issues

TELECOMMUTING
Articles
Job List
Message Board

MYSTERY SHOPPING
200+ Company List

EBOOKS
Dog Bakery
Candy Wrappers
Dog Day Care
Personalized Letters
Personal Chef

LOANS & FINANCING
Business Loans
Credit Cards
Venture Capital
Angel Investors

BUSINESS IDEA CENTRAL

EBUSINESS

MOMS IN BUSINESS
Join FREE Directory

CRAFT BUSINESS

BUSINESS DIRECTORY
View Directory
Pink Ribbon Listings


WEB DESIGN HELP

MARKETING & PUBLICITY


ADVICE & RESOURCES

.


About Us | Contact Us | Submission Guidelines |Privacy policy
Copyright 2001-2012