spack++

Maybe I should wait to write three of them, but if I worked for the onion, I think I would submit the following for consideration

• Study finds global warming caused by computers studying global warming.
• Two NFL athletes move into training camp facility to fight for “first one to practice, last one to leave” title.

What do you think?  Should I send them my resume?  Maybe I’ll wait till I get a third one.

Enjoy!

Earl

So, I have a couple famous friends and one of them asked if I could write a little app that could help him send emails to folks on his totally legit email list.  Well, since it is a totally legit email list, folks that really do want to hear from him, I felt pretty comfortable writing the app on google appengine.  About a day after he asked for a little help, google released an offline taskqueue that could be used nicely for sending emails.  Rather serendipitous, I thought.

Lesson in non-engineer customers: sometimes they want you to do things that you think you can’t do, but with a little effort you can.  My friend said he would like to just maintain spreadsheets in google docs for who to send to, then have my program look at the spreadsheets and send accordingly.  I was like, “yeah, I don’t think we can do that.”  We started to discuss how to maintain the email list just with my app, how to keep things in sync, how to query for users in a certain country or state, etc.  Messy.  Well, turns out using the gdata api, I can authenticate a user, talk to google docs, allow the user to pick a spreadsheet / worksheet, then pull stuff like email addresses.  Awesome!

Took a few nights, but I have something up and running.  My friend says he will give me some nice powered by links when he sends and I am hoping for a blog post here or there.

If you happen to have a totally legit email list (very serious about that part) and would like to trade some sending help for some publicity / marketing, please drop a line to cahille AT yahoo DOT com.

Enjoy!

Earl

For ages I have been meaning to add some sort of search to mycomparer.  Well, we’re live!  I spent may four hours total on it over two nights, and implemented the following features

1. walk through each word in the query and search categories and += matching categories
2. walk through each word in the query and check against upcs
3. walk through each word in the query and check against affiliate ids, like searching by asin
4. good old full text search via mysql

I did the first three the first night, and started the full text search.  Here’s what I had to do.

• Create a table
  • CREATE TABLE sh_product_my_text (
    product_id INT NOT NULL,
    FOREIGN KEY (product_id) REFERENCES sh_product(id) ON DELETE CASCADE,
    my_text TEXT NOT NULL,
    FULLTEXT(my_text),
    timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
    ) ENGINE=’MYISAM’;
• then in my shopping db population process, I populated the table with some product stuff
• I change a query like “computers netbook -wireless” to “+computers +netbook -wireless”
• the above ends up in ? for
  • SELECT product_id FROM sh_product_my_text WHERE MATCH(my_text) AGAINST(? IN BOOLEAN MODE) limit 20;
• also added–key_buffer_size=1024M to my mysql config.  pretty terrible before this change, pretty good after

Course, I implemented it as a service and tie into the service via my Template::Plugin::WebService with code that looks like

[% USE web_service = WebService %]
[% search_ref = web_service.webservice_call(‘/api/shop/search’, form) %]

Can’t tell you how cool I think that is.  If I decide to serve straight from flex or something, then it is pretty well no code change.

And that’s about it.  Give it a go.

Enjoy!

Earl

Awhile ago I wrote some (I think) cool stuff for pig that allowed for parsing apache logs.  Unfortunately I wrote my stuff on an old branch.  Didn’t really know it was an old branch and that everything I wrote would need to get ported, but there you go.  Recently, someone ported my stuff (which was awesome!), and folks at cloudera are blogging about it.

Years ago, I wrote this (I thought) cool stuff, Data::Fallback, which would allow you to pull data from various sources.  I don’t think anyone in the world ever used it.  Like ever.  In fact I discovered memcached and I quit using it.  Kind of cool that folks might actually use some stuff I wrote.

Earl

One of my “many” hosting customers was mentioning that when he logs into his site, he sometimes sees an error.  Well, it turns out I would sometimes see that error and he mentioning it inspired me to look into it.  It comes down to speed.  I have my admin stuff hosting on google app engine and it talks to my backend via web service.  Turns out that google doesn’t want to host slow serving pages, meaning pages that take more than like five seconds to load.  And it turns out that my web service would sometimes take more than five seconds.  There were a couple issues.

1. Memcached helps me not hit the database.  I used to have servers at 10.1.1.1 and 10.1.1.2.  A little while ago I quit running the 10.1.1.2 server, but was still checking it in the code.  Think I would hit some timeout which wasn’t too long, but it slowed me down enough to annoy google.
2. Memcached is all about what the memkey is.  You look up values based on a memkey.  Well, I call it memkey anyway.  For me, if the memkey doesn’t return something, I hit the database and then add the memkey.  Well, in my code for getting a user’s configuration, I had hard coded $memkey = time, which means that each time the code ran, I would fail to get the conf.  I guess that someday in the past I wanted to generate the conf each time, and then just happened to commit.  Oops.
3. Added an index or two to mysql, but don’t think that helped too much.

I am afraid that folks would try and login, get an error and give up.  For sure they wouldn’t likely tell their friend to come sign up for a site.

Enjoy!

Earl

Let’s supposing that you have log files of some sort pouring in and you want to put aggregate data representing the logs into an rdmbs.  To begin, let’s start with a blank slate, i.e., just dumping the data in.  And let’s have a simple table, that in mysql is created via

CREATE TABLE `history` (
`id` int(11) NOT NULL auto_increment,
`hits` int(11) NOT NULL,
PRIMARY KEY  (`id`)
);

I did a pass each for both MyISAM and Innodb with a million inserts.

A multi-value insert for three values looks like this

INSERT INTO today (hits) VALUES (?), (?), (?)

Then I execute with the three values.

The fact that inserts with 1000 values start to approach the load data infile numbers is a little compelling.  But let’s suppose that we want to do every insert from a bulk load but we want to have a table (like history above) that has aggregate data, += style.  Is it possible?  Sure.

Here is one approach for mysql:

1. Create a temp table, which I will call today
2. Bulk load the data into today
3. Run the query INSERT INTO history (SELECT * FROM today) ON DUPLICATE KEY UPDATE history.hits = history.hits + today.hits;
4. Drop today

I would like to apply this strategy and contribute some pig code that allows for bulk insert.  This would (I think) allow for some pretty large scale aggregating all from with a “simple” pig script.  Would also like to start using chukwa, but it looks a little tough.  I think the architecture would then look something like

web servers -> chukwa -> pig -> mysql

Think then I would be pretty well at yahoo! or facebook scale.

Guess we’ll see how it all goes 🙂

Enjoy!

Earl

I was pretty excited about getting sitemaps working, so much so that I recently wrote about it. Turns out I had a couple bugs in my implementation.  When I was on diamondcomparer.com, I would do something like show all the categories / products that diamondcomparer didn’t actually offer.  Also turns out that pretty well each shopping site had more than 50,000 urls, which means I had to break things up a bit.  Plus, I wasn’t zipping things, and I wasn’t real confident I was doing everything right.  So, I decided to use google’s open source code for generating sitemaps, which I figured handled everything I was looking for.

In the past I had used the google code for crawling directories, but now I needed to pull from a database to my list of urls.  Well, turns out the google code can handle that as well.  You just dump the urls to a file, make a config file explaining a few things and then away you go.  Was really not too bad.  Stayed up till three am last night getting this to work

While I am here, have you seen chrome’s xml viewer?  Yeah, me neither, it just dumps to the screen.

I have been tracking google (and others) crawling my stuff, and it looks like the product pages haven’t been getting crawled.  I am hoping this helps that out.  Guess we shall see.  I am now generating these files and pinging the search engines nightly.  Really would like to get traffic based on product pages being indexed well.

Enjoy!

Earl