samp on OpenSolaris

January 2nd, 2009

I spent quite a bit of time getting mod_perl, apache and mysql to play nice on solaris (the nevada builds) some few months ago.  I eventually found CoolStack and after some amount of effort got up and running.  A nice private wiki page documented each command needed to get up and running on a solaris zone.  Recently I got a new X4150 from Sun and wanted to try out their project indiana stuff, which I guess is really OpenSolaris proper.  Oh, and I want my samp to run from inside a zone, did I mention that?

My goal for this post is to go from a fresh OpenSolaris 2008.11 zone and to get this one-liner to work

  • perl -e “use DBI;my \$dbh=DBI->connect(‘DBI:mysql:database=mysql’, ‘root’, ”);print ‘ping=’ . \$dbh->ping . qq{\n}”

Here is an overview

  1. Install OpenSolaris 2008.11.
  2. Create a zone, which I called tester3, since my first couple attempts didn’t go so hot
  3. Install some packages
  4. Start some services
  5. Install some perl packages
  6. Give it a go!

After 1 and 2, let’s see where we are.  Just for kicks, let’s give our one-liner a shot.  I get

-bash: perl: command not found

Turns out that the zones create in OpenSolaris are pretty dang bare, as per this article.  In fact, many programs, like less and telnet, that I expected to have there, weren’t.  Using this page, and this command

  • pkg search -lr telnet

helped quite a bit.  Looks like my zone starts with 54 packages.  Sun’s Web Stack project provides an amp-dev package, so let’s install that and a couple other things for 3.

  • pkg install sunstudioexpress SUNWperl584usr amp-dev

As per the article above, the packages get pulled down and all together those three packages look to be 585.31 megs, so it takes a minute or ten to get downloaded via my cable modem connection.

Once installed I try again and get

  • Can’t locate DBI.pm in @INC (@INC contains: /usr/perl5/5.8.4/lib/i86pc-solaris-64int /usr/perl5/5.8.4/lib /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int /usr/perl5/site_perl/5.8.4 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int /usr/perl5/vendor_perl/5.8.4 /usr/perl5/vendor_perl .) at -e line 1.  BEGIN failed–compilation aborted at -e line 1.

DBD::mysql seems to install better with mysql up and running, so as per the Administering MySQL Server section of this page, we

  • svccfg import /var/svc/manifest/application/database/mysql.xml
  • svcadm enable application/database/mysql:version_50

The DBD::mysql tests want to run mysql_config, so we need to get that in the PATH

  • export PATH=$PATH:/usr/mysql/5.0/bin

Now we’re ready to install DBI and DBD::mysql.  The first time or two through I tried installing the SUNWpmdbi package, but it didn’t play well (perhaps because it is like a year old), so instead I do

  • perl -MCPAN -e shell
  • install Test::More DBI DBD::mysql

Perhaps a little too hard, but now, so, nice!

  • root@tester3:~# perl -e “use DBI;my \$dbh=DBI->connect(‘DBI:mysql:database=mysql’, ‘root’, ”);print ‘ping=’ . \$dbh->ping . qq{\n}”
  • ping=1

Time to try out mod_perl.

If any Sun or web stack folks are reading, including packages for DBD::mysql and a more modern DBI would be great 🙂  And maybe adding the mysql bin to the path?

Enjoy!

Earl

thoughts?

October 29th, 2008

"So, I've been thinking of trying a little lipstick."

“So, I’ve been thinking of trying a little lipstick.”

I smell bacon!

October 24th, 2008

For years I have been interested in parsing logs.  Did a fair amount of it in a past life.  I think pig changes pretty well everything.  Back in the day, I wrote some parsing in perl, tried to figure out how to group things, colate, += these logs into a database, etc.  Well, for me, pig makes the parsing pretty slick.

My pig scripts generally look like

REGISTER coolJarOne.jar

REGISTER secondJar.jar

Those jars can contain user defined functions (UDFs), for doing custom log parsing.

Then, let’s slurp in a log

raw = LOAD ‘myLog.txt’ USING org.apache.pig.piggybank.storage.apachelog.CombinedLogLoader() AS (remoteAddr, remoteLogname, user, dayTime, method, uri, proto, status, bytes, referer, userAgent);

So that one line will parse one of apache’s combinedLogs.  And yeah, I wrote it.  It isn’t commited in yet, but I hope to submit a patch tonight-ish.

Then, this little bit of pig latin pulls out all the refers from my log

refererRaw1 = FOREACH raw GENERATE MyDateExtractor(dayTime), host, remoteAddr, referer;
refererRaw2 = FILTER refererRaw1 BY com.loghelper.MyLength($3) > 1;
refererRaw3 = GROUP refererRaw2 BY ($0, $1, $2, $3);
refererRaw4 = FOREACH refererRaw3 GENERATE FLATTEN($0), COUNT($1);
STORE refererRaw4 INTO ‘log-referers.txt’ USING PigStorage();

so log-referers.txt is a tab delimited file, which is rather easy to parse and += into mysql.  The beauty is that you can supposedly do it all in parallel.  Haven’t done that yet, but hopefully soon.

A few exciting things have been happening for me.

  1. My pig patches (which I call bacon) have been committed in.  Check it out.
  2. I tied in my pig stuff to parse logs from good old holaservers.
  3. I tied into the Google Visualization API and am now generating some pretty killer graphs.
  4. Tonight I took a friend’s combined log, ran it through my system and folded it on in to the above graphs.

Course, all the pig and hadoop (the parallel processing stuff) work will hopefully be the basis of good old loghelper.

where is my van?

October 3rd, 2008

(originally posted September 16th, 2008)

So, after my awesome marketing, I actually got some few signups.  And sadly, though perhaps not too surprisingly, most all the signups are spammers.  Looking like rather dumb linker sites, and some auto-redirect sites.  I nearly linked to them, but there you go.  Think I will block with rather extreme prejudice.  By email, by ip, each site for the offending user, etc.

I used to fight against such things at freeservers.  Sounds like I need to write a script or two that can detect problems and block offending sites.  Not really what I want to be working on, but I sure don’t want to pay for folks to peddle their spammy sites.

Finding offensive javascript is one thing, but I am thinking that for offensive images maybe I will use amazon’s mechanical turk for such things.  In order to get my feet wet, I have the following HIT (Human Intelligence Tasks) currently live

In English, please type in your favorite clean joke. Jokes that aren’t clean will not be accepted. You are welcome to provide an explanation for why you like your joke so much.
For a nickle a joke, it has been fun to watch.  Think my favorite weird response has been
There was one litle man whit his van and wherever he goes he say i am a litlle man this is my van and he robs a Bank.After 20 states finally he come’s in Bosnia and say i am a litlle man where is my van. I am from Bosnia and Hercegovina and this joke is one of many about bosnian people I had to change the joke a litlle bit to write in english but it didn’t lose his charm. I like this joke because most people Bosnian people survived everything and they are clever and most of them thives.
And the joke I will most likely keep repeating?
Q: What is brown and sticky?
A: a stick
Enjoy!
Earl

head in the clouds

October 3rd, 2008

(originally posted September 16th, 2008)

Recently I went to an amazon web service thing up in salt lake.  Turns out the cloud (they say) is all the rage.  Folks doing pretty well everything in the cloud.  I recently moved the start of holaservers to google app engine.  Here is a before pic

Pretty well every non-S3 thing was a single point of failure.

And now

So, for a bit of work, I have a highly available front page.  And for some reason, it is free until I start getting like five million hits a month.  Yeah, not really bumping up to that.

At the aws thing, I learned that I could do all my holaservers stuff in their cloud.  Didn’t really think I could do ftp or kind of hard web serving via my own custom apache, but yeah, can totally do both.  Sounds like EC2 can pretty well just give me a virtual opensolaris box.  I am kind of tempted to start moving stuff on over.  Besides a pretty big aws learning curve, and quite a bit of work, I would have to start putting out $72 a month, which won’t really be happening till I make some money or become a startup finalist.  Think I find out on October 3rd.

Earl

a little marketing music

October 3rd, 2008

(originally posted September 8th, 2008)

Well, am finally going to give it a little bit of a go.  I just posted HolaServers to the Google App Engine Gallery, and you can hopefully see me here.  Wonder if I will get my first real signup based from it.  I have analytics going (actually just added a couple more pages), so I should hopefully be able to measure the deluge pretty well.  Also have the whole every hit goes into mysql going, so we shall see.  I am afraid of spammers or inappropriate contenters, but there you go.

I also entered the Amazon Startup Challenge for HolaServers and good old LogHelper, which it turns out, I haven’t blogged about.  Well, won’t that be an awesome entry.  I figure I have a decent shot at making the finals, which would mean at least $5000 in amazon aws credits.  That would turn out the be real money, since I will be paying out of pocket for S3 as I am getting started.  Also, I would quite like to have my mysql hosted on say EC2, which it looks like is getting more doable daily.  Not sure how they ensure that your mysql daemon stays up.  To do such a thing directly with amazon is like $72 / month (ten cents / hour * 24 hours / day * 30 days / month).  Not too bad once I am getting upgrades, but to start, I think it rather prohibitive.  My co-lo is only like $100 per month and that’s for 2U of servers.  Winning the contest and getting 50k in cash and 50k in credits would also be cool.

So lately I have been trying to get the styles to line up a bit better and though I am not all done, I would feel pretty hopeless without good old firebug.  With firebug you can mouse through the dom and the browser highlights what you’re over.  Also, coming from a guy who used to make a living reading some not so good perl, css is a bit hard to read / follow.  Guess I would get better at such things, but I am not there yet.

Are the next generations of designers going to be great at design, html / css, javascript, templates and the like?  Can someone point me in the direction of some such person?  I would really like to be able to set someone up with eclipse, python and getting holaservers working on their laptop, have the mythical designer edit the templates and commit them when they’re done.  Is that so wrong?

Enjoy,

Earl

a little paranoid about android

September 4th, 2008

For years and years I have been vaguely to quite interested in locations.  Kind of travelling around stuff.  Like shortest routes and the like.  And the more I think about it, the more I am interested in being able to track where I go daily.  Not sure why I would ever care, but here are types of questions I may sometimes care about

  1. How many times have I eaten at yamato in the past few months?
  2. Do I get to work earlier during the summer?
  3. How long does it take me to walk to / from my car?
Impossible to answer?  Perhaps not.  For awhile I thought about buying a garmin device or the like, carrying it around with me all the time, uploading bread crumbs to a server somewhere, tieing into some maps api, etc.  Sounds terrible, and heck, I can barely remember my phone sometimes.  Also, some clunky gps thing would likely look a bit dorky in my pocket, and folks would be asking stuff like, “why do you carry that around all the time again?”
Turns out there are a couple phones (systems) with geo apis.  One day it hit me that I could just buy an iPhone, program it up and be on my way.  A few things slowed me down.  Two listed on the iPhone OS wikipedia page.
  1. Apple has not announced any plans to enable Java to run on the iPhone.
  2. The SDK requires an Intel Mac running Mac OS X Leopard.
  3. My buddy Matt was spending a lot of time with perhaps not a lot of progress.
  4. The whole register and then get blocked anyway wasn’t real compelling.
There are a few reasons I started to consider android, also partly from wikipedia.
  1. Also, at least for now, software installed by users must be written in Java
  2. Includes a device emulator, tools for debugging, memory and performance profiling . . .
  3. . . . a plugin for the Eclipse IDE.
  4. By buddy Drew might actually write some Java code with me.
  5. Heck, it’s Google, they do things right, right?
Turns out I am now a fulltime Java developer (as per my resume), so the Java thing is pretty dang compelling.  I also working in Eclipse pretty well all day, so that was exciting.  And I figured the emulator would be cool.  And yeah, Google!  Who loves Google?  Heck, who doesn’t.  I am using Chrome right now!  How cool is that?  How cool is android?  Well, over the past few nights, not very.
I read that there was a way to spoof location data in the emulator.  Also read that there was a sqlite api.  I used to play with sqlite quite a bit and have years of sql experience, so rock on.
Getting a hello world! app going was pretty easy.  Then I tried the geo stuff.  Figured there would be a simple api, a few google-written examples to follow and I would be on my way.  Well, I read through some of the docs and mostly got lost.  Just too much.  I found a blog, dated Christmas 2007 that talked walked through some geo api stuff.  I pasted the code into eclipse and . . . it didn’t compile.  Looks like an abstract method got added and hadn’t been implemented.  Not so hardened.  There was accompanying code at google code, and I tried the latest release with similar results.  I checked the blog and found a more recent post saying he had tried for a few hours to get things working,  but to no avail.  I checked out his most recent code from subversion, learned a little about kml and was on my way.
So, one api, some few changes.
Then I thought I would try to insert my location points into sqlite.  Had a similar experience with the docs.  Figured that there would be a nice jdbc interface and it would look much like what I do at work for hours most days.  No such luck.  I found another blog, this one posted December 1, 2007 that walked through a simple sqlite implementation.  Cut and pasted the code.  Didn’t compile.  I think this line right here
db = ctx.openDatabase(DATABASE_NAME, null);
I think there was no matching openDatabase method.  Amazing to me.
Two apis, neither works with code from just a few months ago.
What about google’s sample code?  I looked, I think pretty hard, and couldn’t see a single geo or sqlite example.  Maybe the gravity stuff will someday interest me, but not so much today.
So, yeah, not real impressed.  A little while ago, I spent a day or two at work looking at google’s recently open sourced protocol buffers.  Got it working without too much pain.  Apis worked as advertised.  There were simple examples that I could change to match my investigations.  Maybe those guys could help out android stuff.  I am pretty blown away by my prima facie android experience.  They gotta know that folks gotta get some things working within a night or two, or they might leave and not return.
Is google just getting too full of themselves?  Cell phone os?  We can do that, we’re google!  I had to pass like twenty interviews to get this job, I rule!  I eat (for now) free food every day!  If some dorky senior java engineer from utah wants to do some android, let him figure it out on his own time.  I work at google!
I watched a couple google io videos in the background while I was working today.  The presenter in this one starts his talk by saying something like “this talk is about a C++ library and is going to be pretty technical, so if you’re here for some javascript, you might want to evaluate being here.”  Just kind of rubbed me the wrong way.
In this one, the presenter had (I think) code that had to be changed to compile, and his laptop eventually had to be rebooted.  He also (I think) kind of talks down to someone asking about CSS.
Anyway, enough of my rant.  Mostly I would like to get some android stuff working, and I guess android is still beta, but I wouldn’t be too big of a fan to get stuff working only to have apis change.
I had written that chrome seemed pretty cool, but then my line spacing broke in this entry 🙂  And I need to add that I have played a fair amount with google app engine, and the docs there were quite good.  Oh, and I bought a url for my android stuff, http://crumbtracker.com, which I registered via google and it all went well.
Also want to mention that I noticed tonight a few (ok, quite few) comments on this blog from folks I didn’t know, that might not even be spammers.  Welcome!
enjoy!
Earl

say hola to google app engine!

August 17th, 2008

Well, I bit the bullet and moved the front page and administration portions of holaservers.com over to google app engine.  It was a fair amount of work, and you might be asking why.  Well, here are a few reasons

  1. At good old united online, quite a bit of money and effort was expended trying to keep the front page highly available and serving fast.  Well, would be nice to get such services nearly for free.
  2. A good excuse to learn some more python.
  3. Kind of because I could.  I had much of the required information available via web services, so it wasn’t TOO hard to move things over to google.
  4. If a user is logged into Gmail and on holaservers, I can send email via their Gmail account.  I am hoping this is huge.  It means that I am not sending any unsolicited email from spack servers.  Folks are really just sending to their contacts via their Gmail accounts.
  5. Maybe I could go speak at a conference about it.  I have done (I think) a pretty good proof of concept for moving portions of legacy systems to google app engine.

So why not move everything?

  1. At the heart of holaservers is an ftp server.  Kind of the vision for getting stuff to your site.  Since google AFAIK only listens on port 80 for their stuff, ftp is out of the question.  I guess I could get a flash / java / whatever file uploader working that speaks my webservice, and there you go.
  2. DNS.  bob.holaservers.com would have to be set up as a different application to resolve to google.  You only get ten, so there you go.  Kind of the same story with bob.com.

As part of the move, I got my stuff working on selenium again.  SO cool!  It has caught many a bug.  Probably the hardest thing about moving over was breaking cache.  I wrote a little web service proxy to handle my old javascript requests and the app engine really tried to hang on to cache.  I ended up sending everything through a single method that included a .epoch variable to break on through.  I also needed to add something to proxy through the remote ip address securely since I depend on such things quite a bit.

Python huh?  Yeah, python.  Got to admit that it has been a little rough going.  I got paid to learn perl while a math grad student at BYU, and continued to learn it on the job for seven years at united, et. al.  While at united I got paid to do some java training and the church has been kind enough to pay me over the last year to learn more java.  So for perl and java, I could spend quite a bit of time learning this and that.  Python I have really just tried to get things done and don’t have any sort of big picture.  “Python” has also meant django, which has a learning curve all its own.  And since I only work on the stuff nearly each night, I have to keep relearning how to get the size of an array, etc.  It took me a good twenty minutes to get a couple email canonization regular expressions working tonight.  Perl would have been wire speed.

Enjoy!

Earl

ready for takeoff!

July 25th, 2008

Here’s the elevator pitch for my cool new site:

Folks

  1. post a thesis
  2. post arguments for and against said thesis
  3. rate arguments
  4. comment on theses
  5. comment on arguments

What do you think?  Dumb enough to work?  I think it could.  I call it 5arguments.com.  Right now, I have an alpha version working.  So what?  Well, first off, good point.  Second, I wrote it in Python using the new Google App Engine, which is, so far, actually pretty cool.  I figure it is dumb enough that it should actually take off.  Think I will add a few google ad words just to help out with the mortgage and the like.

Hoping to having something up on it pretty quick.

Enjoy!

Earl

here comes the sun . . . server

July 16th, 2008

Awhile ago, a palindromic friend of mine told me about a deal from Sun that is explained here.  Basically, if I agreed to release my product on solaris within nine months, I could get a free server.  It says that the product can’t currently be available on solaris, and since holaservers isn’t really launched or available for real on solaris, I figured I was fine.  I signed up for the partner program and was (I think) automatically approved.  Once approved for the partner program (not the server) I saw requirements about fulltime and revenue, neither of which I am too flush with.  I kind of gave up.

A few days later I got a call from a Sun employee welcoming me to the partner program.  I explained my hopes for the server, and disappointment in the requirements and he said to apply anyway.  It took a couple months, but I did.  I had to update a url, and send an email (maybe two) about my business plan and the like.  I figured the requirements would quickly come back to haunt me, but instead I got an email containing

Dear Sun Partner,

Congratulations! Your request for the Port Now! Hardware Offering has been approved. Please allow approximately 4-12 weeks for delivery of your system.

It was awesome.  I sent in some paperwork the next day and am anxiously waiting.

Of course, if you look at the specs for the x4150, you may notice a few variable things, like

  • One or two Intel® Xeon® Processors . . .
  • 16 DIMM slots . . .
  • Up to eight SAS disk drives . . .

Not related, but I did just see “Dual redundant, hot -swappable power supply” which is pretty awesome.  I am just really curious to know what I am going to actually get.  A two proc box with a total of eight cores, a ton of ram and eight SAS drives would just rock!  An empty shell or a box with no ram, no drives and four cores would also rock, just not quite as much.

To help sooth my curiosity, I emailed Sun support and also called them.  In response to my call, today I got (and let me say, I feel I fully explained my question) an email which contained

Hello Earl,

Here is the URL for the server specifications

http://www.sun.com/servers/x64/x4150/specs.xml

If you have any questions please let me know.

Regards