Awhile ago, I wrote about LogHelper. Well, sort of. Turns out I just mentioned that I hadn’t really mentioned it before then went on my way. Nothing too remarkable I guess, I would just like to parse folks’ rather arbitrary logs and allow them to do some pretty set analytics. The theory is that folks would send me their logs, I would ingest them and output some pretty graphs. As a proof of concept, I tried to create and eat my own dog food with HolaServers and LogHelper. I scp’d logs from HolaServers to another box, parsed the logs there with good old pig (which I have written about), and display graphs using the Google Visualization API. The cool graphs look like this
A little side note. Sometime ago, I made it so you could email a photo to your HolaServers site. I would check once a minute for messages and act accordingly. Well, the code I was using logged quite a bit and I didn’t realize it was logging anything. Long story short, I filled up /tmp and my logging stuff started emailing me every few minutes. So, I commented out a cron or two and quit doing my LogHelper stuff. A week or two later I discovered that the mail checker was the real culprit but couldn’t figure out how to get the logging to work again. Even if I had the following few lines, I would have saved several nights / hours.
- the magic conf file is here conf/loghelper.com/mylogadm.conf, which needs to be copied to /etc/mylogadm.conf
- on client boxes, this is the magic cron
- 5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/sbin/logadm -p now /opt/coolstack/apache2/logs/mylog -v -f /etc/mylogadm.conf
- on the server box, this is the magic cron
- 2,7,12,17,22,27,32,37,42,47,52,57 * * * * perl -I/export/home/earl/svn/lib /export/home/earl/svn/crons/loghelper/go
If you want to see the stats in action, just sign up for an HolaServers site, get some traffic, and then wait a few minutes to see things show up.
Enjoy!
Earl