I was pretty excited about getting sitemaps working, so much so that I recently wrote about it. Turns out I had a couple bugs in my implementation. When I was on diamondcomparer.com, I would do something like show all the categories / products that diamondcomparer didn’t actually offer. Also turns out that pretty well each shopping site had more than 50,000 urls, which means I had to break things up a bit. Plus, I wasn’t zipping things, and I wasn’t real confident I was doing everything right. So, I decided to use google’s open source code for generating sitemaps, which I figured handled everything I was looking for.
In the past I had used the google code for crawling directories, but now I needed to pull from a database to my list of urls. Well, turns out the google code can handle that as well. You just dump the urls to a file, make a config file explaining a few things and then away you go. Was really not too bad. Stayed up till three am last night getting this to work
While I am here, have you seen chrome’s xml viewer? Yeah, me neither, it just dumps to the screen.
I have been tracking google (and others) crawling my stuff, and it looks like the product pages haven’t been getting crawled. I am hoping this helps that out. Guess we shall see. I am now generating these files and pinging the search engines nightly. Really would like to get traffic based on product pages being indexed well.
Enjoy!
Earl
pretty dang cool, I think.