Google XML Site Maps – Why You Must Sign Up For It.

Since Google is the most popular search engine at this present time, it’s imperative that Google can properly search your website. The only way to know whether or not Google is crawling your site completely is by submitting a Google XML site map through Google Webmaster Tools.

Top Three Reasons Why You Should Use Google Webmaster Tools:

1. You need to know if your website has been completely crawled by Google.
2. You need to know if Google has encountered any problems while crawling your site.
3. You need to know how many duplicate title tags and meta description tags your site has.

The first thing to do is to go to the Google Webmaster Tools start page and sign up:

http://www.google.com/webmasters/start/

Once you have signed up for Google Webmaster Tools (GWT) and your site has been verified, the next step is to create an XML site map file and upload it via GWT. Creating a site map can be slightly difficult. Here are some methods to create a complete, XML compliant site map.

1. www.xml-sitemaps.com

This handy website will crawl through your site and generate a sitemap.xml file for you. Once you download your sitemap.xml file, simply put it in the root directory of your website and notify Google that your site map is ready in GWT.

The only problem with this site map generating service is that if may not find all your webpages (URLs). This could be the result of webpages that are not linked properly, or instances where xml-sitemaps.com times out while generating your .xml file.

2. Using Your Own Sitemap Generating Script.

We highly recommend having an automated script created that will generate the site map file for you. WordPress has a site map plugin and there are also free online scripts that will crawl your site and spit out a xml site map in your root directory.

The only hard part about having an automated script installed on your site is that you have to verify that the script is actually picking up every URL on your site. Someone in your organization will have to do a manual check to make sure that the automated site map script works. Hopefully you don’t have thousands of pages already :).

The best way to do this is to get a report of all the files contained in your website. We usually just do a complete download of every file and directory from our webserver, then use a program like jedit to copy and paste the file names into excel. Next, we sort the excel spreadsheet to only list .html and .php files (or whatever extensions have content on them). This is a good way to make an inventory of all your website pages.

Finally, we cross reference all the actual webpages on our spreadsheet with what the automated script is listing in terms of webpages. If they both show the same results then we know our script works.

The lesson here is that you can’t always take a script or a services word for it. It can be a costly mistake. Always do sanity check to see if your automated help is providing the results you want.

3. Content Management Systems and Dynamically Created Content

This is where the verification process get tricky. You can never trust that a plugin or script is actually accounting for all your URLs correctly. The only way to be sure if your plugin or script is accounting for all your URLs is to:

1) Test it out on a small set of URLs.

Or

2) Set up an independent test site to see if your plugin or script is working correctly.

This can be a mind racking process depending on how big your site is and what management system you use to contain it. Please comment on what methods you found that work best for you.

Thanks!