joeldg - navigating the surface

I run my own server but I need to use dreamhost servers in the course of my day and in dealings with people and I would like to relay some of the pain: Most professional hosting is good but for anyone out there who are having issues with their hosting there you should consider leaving dreamhost because clearly they are great for little girls ‘first blog’ who use free webmail somewhere instead of trying to rely on the rubber bouncy balls of dreamhosts servers. If you need uptime, don’t apply, if you need stability don’t bother, need speed? no can do, they have the crappiest bandwidth I have ever experienced. Need a database, if it works, they will manage to garble it. Need email for business? no way, and if you need it for business it should not be time critical and be able to sit for days. At this point in seeing friends and people I have/and do work with dealing with the suck I am at a loss how this company can stay in business with barely being able to host plain html .. let alone anything else.

Before you go off, I worked on the team that kept datacenters running for two weeks through one of the worst natural disasters to ever happen and we did it without downtime in the middle of hurricane and through looting etc. Dreamhost on the other hand had a little heatwave and went flopping over like a rotten fish.

So, maybe that was not their fault? Well, for one of the sites for our work, this morning the site was down with mysql errors, no email from or to them for a while because dreamhost mail works like Iraqi power. And then when mail got through it came out that there was an sql query that they could not figure out how to stop so they did us the wonderful favor of RENAMING the mysql tables to stop the query from running, no email, nothing. And to top it all off, they then say “learn about mysql EXPLAIN” (because it is not their fault in any way) on a site that has been running for a long while with no issues and no complex queries. I should not even need to “explain” to someone how unacceptable this is.

Constant downtime, neverending mail troubles, geeksquad rejects as system admins and legions of people sending around coupon codes trying to save money… Do yourself a favor, protect yourself by using anyone else.

I want to do a little rant about web2.0 and then on some new things, like Google getting beat to the punch at one of the greatest new things to come out of this 2.0 gooball (it isn’t a bubble, just the graphics look that way)

Lets be clear about something, “Web2.0 is just web1.0 with prettier javascript” it is not a new web, it is not really anything new, it is fueled by ONE library and that libarary is named prototype and it is the glue that has allowed this thing we call “web2.0″ to come to where we are currently.

Prototype allows us to *not* have to hack around in nasty javascripts that are smeared throughout our html, it allows us to have access to the DOM in unprecedented ways and from the prototype wellspring has come all the libraries like script.aculo.us and moo.fx and dojo and.. and.. From these libraries have come plugins for frameworks like ruby on rails, django and cake which have taken prototype and made it easily accessible to your application. What the combined effect of these frameworks doing this is, is that now you really don’t need to know javascript inside and out to make use of the full power of it.

So, right now all you need is to look around you and see what people do online and with computers and even in real life and just like the mousetrap analogy you figure out a way to make it “better”, “easier” ot even just “fun”. You can get a cheap account at a webhost (however, due to many recent experiences and from listening to people you do NOT want to use dreamhost, I am without email for days at a time with one site I work with) then because “cheap webhosts” have a tendency to limit your bandwidth and your drivespace you can’t really run a very busy site on one. Now comes into play some new stuff which has recently emerged.

Amazon s3 (simple storage service) which I will make a bold prediction about.. “Google will be coming out with a storage system similar to s3 within a year of right now” and I would be willing to bet money on it. Yahoo will also more than likely try it as well.

Anyway, S3 is pretty amazing, it is quick easy and cheap cheap cheap. You host your larger files, user content, images, basically anything and it is almost a drop-in replacement for hosting your own files. It is metered so scales “with you”. If you are currently having to run one of those “platinum” plans from a webhost, you can drop down to their “basic” plan and host your images and files with S3 and save yourself a few hundred dollars per year.

The main reason I mention S3 is that after looking over the service and some of the implementations and playing around with my account I am impressed that it works so well and basically allows anyone with a few bucks to start a site that can be basically be scale-paid. Enough to the point that I can make another prediction, scaled webhosting which is based only on your gigabyte disk and line usage is most likely already out there, but will become the new defacto system and guys like cpanel, plesk and the large hosting providers like serverbeach will be quick to step up and get on board with this.

I am sure the apparently very smart people at Amazon saw this and so they said. Well, lets sell servers using the same model. And so was born Amazon EC2 (elastic compute cloud) need a new box? just fire up an image you have already uploaded (or use one provided) and there it is for ten cents per instance hour, or $2.50 per day per server or $75 per month before the standard costs for the S3 gigabytes/bw (totals come to about half of what I pay for my dedicated box)

So now we have metered unlimited space, any file can be turned into a torrent by simply adding “?torrent” after the file name (to save you some $), metered bandwidth and even metered instance time on a machine, it is all “pay as you go” and is built and designed for the smaller shops or even the medium sized businesses which could use a little work. To give you an idea of the savings, I worked for a mid-town company here in New York that paid thousands of dollars per month in bandwidth overages which would have amounted to perhaps $50 combined on S3.

To top all this off, Amazon now has a “People task” quick job board you can do called Mechanical Turk where you can hire people to do the little things for you, or take on little jobs in your spare time.

Today Google launched Google apps for your domain which really does nothing for me because I am not interested in branding my own version of gmail, everyone already uses the real gmail. They are usually pretty good about getting stuff out, but this is the first time I have really seen them being beat in something that is normally their domain.
When did Amazon get “cool” and Google doing the boring stuff?

So, for your little web2.0 startup the above links should be plenty enough to turn it out on a dime and also use and get your hands on the good stuff.

Some references:

http://www.holovaty.com/blog/archive/2006/04/07/0927

http://blog.pairwise.net/2006/08/28/a-simple-amazon-php-s3-class/

http://overstimulate.com/articles/2006/08/24/amazon-does-it-again.html

http://blog.pairwise.net/2006/08/22/amazon-s3-the-holy-grail-of-bandwidth-problems/

Well, I decided to finally switch to a more powerful cms for this site. Have some needs currently and coming up that require a little more granularity than eggblog could provide, though I do miss it’s simplicity and quickness.
I am pretty impressed so far with wordpress however and think I will be sticking with it for a while.
You may notice the dates are all changed on the posts so they show as being posted today, this is because I just quickly dumped all the older ones in.

Not my normal topic, but this is a new show on television called Heroes. Basically the idea is that ordinary people discover they have extraordiary abilities and what they do with those..

The full one hour first episode can be downloaded here and a four minute “super trailer” can be downloaded here the first episode is promising, though we are just being introduced to the main characters.

I think my favorite character was Hiro (yes, the name) who is a Japanese salary worker bored with his ordinary life and who has read too many comics.

The show has some leanings to LOST style of storyline where everyone is connected and there are a lot of hidden little gems in the show that I caught, and I am sure as the series progresses it will be more pronounced.

Not too many shows aside from anime that I go out of my way to check out, this one was worth it..

Tomorrow I am going to post about my findings with YM4R (googlemaps and geocoding) in rails and also do a walk through perhaps on some of the AJAX rails calls that I find handy.. (I am learning here as well, sometimes it helps for me to try and explain to others)

My no-signup, super-quick mouse movement and click-tracker has been released on the new umbrella site http://dreamvendors.com so stop by and check it out. It takes a minute to set up and get started tracking how people visit your site.

I set it up so that there is no signup necessary, each individual page that it is called from will be stored and logged.
I will post more about it later and perhaps set up the pages more, they are quick templates for that pages I set up.

36,000 mouse and click points in I discovered my initial experiment in mouse position tracking had a slight flaw in it, as I didn`t expect the script to bail so early.. So much of the tracking I would have gotten was lost, but never fear, there has been a fix thanks to the PHP function ignore_user_abort(TRUE) in the update script which under situations of a click would bail rapidly while the browser moved to a new page or loaded different content.

FIRST THING: TO VIEW THE STATS, CLICK THE LITTLE ICON IN THE LOWER LEFT. (BLUE IS CLICKS, GREEN/ish IS MOUSE)

I got the mouse track logging back on track… The script was bailing out and I fear I may have some some stats. Otherwise is working well and seems to be attracting some attention from people looking into ways to see what people are doing with their mouse on your page.

I am considering using a technique to identify the location of certain elements of a page and try to attach the div`s to them when displaying stats.. This will cure some of the ills of text-size changes, and from the vastness of screen resolutions.. Though, there is something to learn about resolutions as well and it IS kind of easy to see.. SO… I may just make multiple display modes in the overlay.

I am also looking into the mozilla canvas for the items on the overlay display which could programatically handle them very well.

I have made some changes to statsbox size on the overlay as well as colors to view what is what..

I am mostly swamped by work and life though, so it is difficult to spend too much time.. Perhaps I can steal some time at some point this weekend to get this packaged into a hosted app.

I was looking at a good mouse/click tracking solution that does visualization of where people are clicking on a site and what people mouse-over the most.. I looked at crazyegg.com and tested out their beta but didn`t like the fact their javascript overwrote every link on my page. I also didn`t like that they limited my click tracking and plan to charge for it..

(EDIT: this probably won`t look very nice or even work in IE, but IE is five years old people.. so `maybe` you should upgrade? If you are using IE your browser is probably older than your computer, think about that….)

for those that cannot see what it is supposed to look like:

Now.. it is not that I plain don`t trust so many people as I would rather control what is going on with my own stats, being a programmer, I like to keep tabs on these things. So, I did my own which is now running on this page. You can click on the little thermometer in the lower left (if you are in firefox, otherwise IE may put it in the uppper left) and the page overlay will pop up which shows you in green (progressive by mouseover) where people move their mouse around the site then, in blue where they are clicking.. This is a running `live` total and obviously limits the queries and the entries to the most relevant possible data and limits the display to 500 mouse areas and 500 click areas (progressive color by most for each spot) anything more than a thousand extra divs on a page and your browser is going to start choking, so let s keep it simple from the start and work up.

HOW IT WORKS:

1. Register mouse moves
2. Mark all mouse moves for the first fifteen seconds on the page via a timer (in case they don`t click)
3. Register all clicks with prior mouse moves on the page, this is reset with each click
4. A mouse move is regarded as a ten pixel change in the X or Y value of the mouse. All clicks are marked regardless
5. Stats are displayed sorted by clicks DESC and then divs are created based on a normalization of the hits

TODO:

• Fix the issue with browser size… great for fixed upper-left sites, bad for centered sites..
• Maybe brighten up the colors some, they are a bit washed now, though, I am interested to see it after a week, so will wait..
• cleanup the js… there is some dangles going on..
• cache the results script, it will get out of hand with more than 100 sites otherwise
• actually release it

Obviously this has a lot of cleanup and might turn into a hosted app for people who have PHP and such but don`t really feel like managing their own stats and just want to plop a php script and a some js in a dir and have immediate results.. We will see..
But for now, I am collecting stats so I can better tweak things..

Not bad for two afternoons work

I tried an experiment to see just how good the panoramic software was and trotted down to Grand Central mid-day and took a series of photo`s from the stairs.

I took a series of six photos from the stairs in grand central, I was curious if having people walking around in a photo would really mangle up the algorithms.. Guess what, it did. However, manually going in a fine-tuning the control points and adding some at key points and whalla, no problem getting the images to line up and spit out a nice panorama.

If you look at the image in the lower right, there is an interesting bit with two guys who seem joined and are walking in different directions, and there is one guy with his head cut off and another guy taking his place, but otherwise I am still amazed by the stichwork these programs do on photos.

I am starting a series of panoramic shots of New York city, and I am going to start with this short article on creating panoramic images in linux..

I have a digital camera and have been too lazy and have better things to spend my money on than memory sticks, so.. I can take six photo`s with it, so I am always trying to figure out what to photograph so that I am not wasting shots, would it not be better to just have one GOOD photo than a series of so-so ones.. I started looking into panoramic images.

Now, I run gentoo linux, so clearly the imaging tools are available I just needed to research it.

There is a half-baked tutorial/infopage at the gentoo wiki on the tools to use. Though, most of those tools you really don`t need, the things you do NEED are Autopano tools, Hugin and finally enblend which will do softblending over your images so that there are no harsh lines from color shifts due to light differences and glare.

With those tools, it is pretty easy to follow this tutorial on how to use the tools. Though, I prefer to use enblend internally from Hugin, so don`t follow that tutorial to the letter.

I had taken some photos from a rooftop here in Manhattan, and sat down to work on this. The following image took me about ten minutes, using tools I have never touched, to make the following.

Six images, quick fast and turned into a panorama..
There is one smudgy spot, which is just because I messed up the photo, but otherwise, going from images to panoramic that fast is awesome.

I started a photo gallery here on the right and will try to add panoramics to them here and there as I take my camera out. I doubt my callphone camera takes decent enough photo`s for doing this, but I will try that also.

I get lot of people asking me about AJAX and what library to use, so I will write up here my personal preferences on what to use and why.

First and foremost THE library to use is Prototype, I use a light version of prototype that comes with moo.fx lib.

Prototype is the `core` not the end means for doing ajax calls. For easy easy ajax work you need to look at moo.fx, the prototype lite and moo.ajax can be downloaded easily from here. The reason for using moo.fx is that it is a super small library which gets the job done.

AJAX calls in moo.ajax are so easy it is ridiculous and with prototype $(`ID`).property style work you can just plop whatever into whatever easy as pie.

For “full-featured” libraries for doing effects, I would suggest Rico which has nice smooth animations and scrolling and it`s drag and drop lib is really nice. Though, the reality is you will seldom ever really need this functionality so use sparingly.

The other popular lib is script.aculo.us which I personally like the list sorting portion of, but the library is big and chunky and in general I feel it is just over the top and moo.fx is generally a better choice.

Using small and tight javascript libraries is kind to your users, makes your page feel snappy (vrs that chunky web2.0 feel on sites that include 30 js libs because they just HAVE to have drag and drop on things) and it is also easier on you because you don`t need to read a book to use them. I was up and working with moo.ajax and moo.fx in no time, and any questions I had could be easily solved by either looking over the source (which is 3kb) or poking around in the documentation they have on their site.