Building sitemaps and robots.txt the easy way?

[bell88]

A recent check of my website suggested that a sitemap and a robots.txt would help my rankings... if this is true is there an easy way of doing this?

I read the info on the Google sitemap tool and got confused so it may be better for me to outsource this to Elance or rentacoder or is there an easier way of doing it myself?

If not any ideas on what it would cost to get someone to do it for me.

Thanks in advance.

[Tim Buchalka]

Howdy,

Yes I would suggest this would be a good idea to do if you have not already done it.

1. Sitemaps.

If your running a blog it can be a fairly easy process of installing a plugin like Google XML Sitemaps Plugin by Arne Brachhold.

Yes it is an option to get someone to create an xml sitemap for you on rentacoder or other outsourcing sites. See this outsourcing post for details about outsourcing.

If you are not running a blog you so have options still if you don't want to outsource. There are lots of free services to create sitemaps for you.

One popular one is

Create your Google Sitemap Online - XML Sitemaps Generator

The free version will index around 500 pages and I believe they charge a fee for an unlimited standalone version.

Once you have done that you get a file and then upload that file to your server. It's then recommended that you "tell" Google about it by logging into your Google webmaster tools account.

Below is a video showing you how to do this...



2. Robots.txt file

There is a similar process here, you can find a plugin for wordpress or "edit your own" file.

See this plugin as one example of a robots.txt file for wordpress.

Here is a robots.txt file generator/analyzer

Robots.txt Tutorial

You do the same thing here, create it and upload it and then "tell Google" about it via the webmaster tools.

I hope this helps. Let me know if you have any other questions about it.

Cheers


Tim

A recent check of my website suggested that a sitemap and a robots.txt would help my rankings... if this is true is there an easy way of doing this?

I read the info on the Google sitemap tool and got confused so it may be better for me to outsource this to Elance or rentacoder or is there an easier way of doing it myself?

If not any ideas on what it would cost to get someone to do it for me.

Thanks in advance.

[bell88]

Thanks for that.

The main site is an OSCommerce shopping cart.

Thanks for the link, that seems to be exactly what I need at the moment - and its free!

I have read that you can dynamically generate sitemaps - please explain for me... does this mean that you need to update your sitemaps as you change the content of the site so that those pages do not get "ignored"?

Also, I have several small sites in on the same hosting account that are html pages that are unrelated content and are from redirects of other domain names. Should these be included on the sitemap or can I select what directories are mapped and submit them separately?

Thanks for all your help so far. Graham.

[Tim Buchalka]

Thanks for that.

The main site is an OSCommerce shopping cart.

Thanks for the link, that seems to be exactly what I need at the moment - and its free!

I have read that you can dynamically generate sitemaps - please explain for me... does this mean that you need to update your sitemaps as you change the content of the site so that those pages do not get "ignored"?

Also, I have several small sites in on the same hosting account that are html pages that are unrelated content and are from redirects of other domain names. Should these be included on the sitemap or can I select what directories are mapped and submit them separately?

Thanks for all your help so far. Graham.

Hi Graham,

Dynamically created sitemaps get updated automatically as new content is produced without you needing to do anything manual (e.g. run a program, create a sitemamp and upload it, etc).

Typical platforms that have dynamic sitemaps are blogs and forums such as this one. You could imagine the hassle of creating a sitemap for a forum like this with lots of new content being added each day.

There is good value in keeping a sitemap updated because it "points" google to your new content and can assist in that content being found sooner and being indexed, etc.

For the record, I did a search and found this, which may help you.

Dynamic SiteMap - osCommerce Community Support Forums

If you have a relatively static oscommerce site (not updating to new products, etc much) then you maybe do not the dynamic version.

Generally speaking I would not have unrelated content in the sitemap as you have indicated unless you have a good reason to do so.

Most of this type of software will enable you to choose the directories, etc to be indexed. The worse case is you could open the sitemap file that is created with notepad and remove these extra files/directories if for some reason they do not enable you to do choose specific folders, etc.

If you did decide to use the dynamic sitemap, then most likely it would only include the content from your oscommerce site so that might be another way around it.

Regards


Tim

[bell88]

Thanks Tim,
That was clearly explained and I had in done in less than 15 minutes - after spending hours of reading stuff.
AND it was really useful because I can get full use of Google webmaster tools now. Our bead wholesale website (Beads4u.com.au Wholesale plastic beads in Australia) is now flying. Cheers

[Tim Buchalka]

Thanks Tim,
That was clearly explained and I had in done in less than 15 minutes - after spending hours of reading stuff.
AND it was really useful because I can get full use of Google webmaster tools now. Our bead wholesale website (Beads4u.com.au Wholesale plastic beads in Australia) is now flying. Cheers

Hi Graham,

Glad we were able to help you out. Hopefully this thread will help others with a similar question.

Google webmaster tools is a very useful and important tool that I think everyone should be using.

Good luck with your rankings!

Cheers


Tim