joeldg - navigating the surface

Templating webpages is a tricky business with a lot of issues and options, and the vast majority of people do it totally and completely wrong. A template, in terms of documents, is any document fragment with blank spaces designed for you to ‘fill’ in. It is something that establishes a pattern. In terms of web pages, a template is the design and interface aspect of a site, it should be entirely separate from the logical portions of the site.

Let me restate that “it needs to be entirely separate from the logical portions of the site”. The primary reason for this is maintainability. The reason we use templates in the first place is to organize our development and have two distinct layers, the logic and the design. Remixing logic into design (as smarty and others do) is counter to the primary reason we chose to use templates in the first place. With a proper template system if there is a database or looping issue we know where to look right away and we don’t have to sift through the 20 templates that make up a page in order to find which one has the bug.
So, I will introduce the templatling system I wrote that forces this issue. (it will appear below) The first thing we need to do in creating a perfect template system is balance out ease of use, speed, and determine how it will function and where it will store it’s template files. It also will need ‘complete’ separation of logic and design.

A few things:

If I request index.php in the root directory of a site, the template engine will look in TPL_BASEDIR + “/index.php/” (yes, a directory of the name of the file) for the individual parts of that template. We will also have a global directory for headers, footers and the engine will look backwards up to three directories to try and find a template.
Template files will have a .tpl extension and will NOT be eval’d so that no code can be run in them and there will be no temptation for the “quick fix” of adding code to them. There will be one exception to allowing eval’d code and that will be code that is in html styled <php> tags for outputting dates or other small single variables. (this was an addition to help in transition from systems like smarty).

We want to be able to set some scheme defaults for our template, things we don’t want to bother with in the logic portion, mostly these will be color codes for css files and pages. (this helps to keep design out of the programmers hands)
We need to be able to have multiple sites using the same template engine.

We need to be able to sling sql statements and query result sets to the engine and have it use those if possible. This is a very DRY approach, similar to how rails does some things.

It needs to be able to process css and javascript files as well as html fragments.

We want a set of functions that also are design related for doing things inside the templates if we need, things similar to the “cycle()” ruby function which are automatically called, but are set and forget. This is useful when dealing with row colors so the programmer does not need to worry about colors or stye classes.

We want the template system to be fast so we build in a mechanism so that the pre-processed templates are not loaded more than once.

What we end up with is a fast and proven system, see:

The template system( txt )

To use this you would include the file and set up a template name replace.tpl which has the contents of:

“Here is the {MYVARIABLE} for sale”

in PHP you do :

$replacements[’template’] = ‘replace.tpl’;

$replacements[’myvariable’] = “house”;

template($replacements);

and you will get “Here is the house for sale”

If you want, you can also use your templates from javascript as well. I have some big plans and loads of ideas for ways to use JSON and prototype.

A very similar system was in use at my old work (this is an improvement and revamped version I wrote for another job a while back), we managed 20+ programmers of various skill levels with this template engine and it worked perfectly. Sites managing tens of millions of hits per day use this system and it works well.

We also never had to poke around in HTML to look for a programming bug.

Try it, you might like it. You can even easily turn it into a class, if you are running PHP5 turning it into a class will not have any slowing effects as it did in prior versions..
p.s. Later I will post some old code for doing in-memory content caching using shared memory (schmop) it is easily fitted into the above template system to speed it up even more.

I run my own server but I need to use dreamhost servers in the course of my day and in dealings with people and I would like to relay some of the pain: Most professional hosting is good but for anyone out there who are having issues with their hosting there you should consider leaving dreamhost because clearly they are great for little girls ‘first blog’ who use free webmail somewhere instead of trying to rely on the rubber bouncy balls of dreamhosts servers. If you need uptime, don’t apply, if you need stability don’t bother, need speed? no can do, they have the crappiest bandwidth I have ever experienced. Need a database, if it works, they will manage to garble it. Need email for business? no way, and if you need it for business it should not be time critical and be able to sit for days. At this point in seeing friends and people I have/and do work with dealing with the suck I am at a loss how this company can stay in business with barely being able to host plain html .. let alone anything else.

Before you go off, I worked on the team that kept datacenters running for two weeks through one of the worst natural disasters to ever happen and we did it without downtime in the middle of hurricane and through looting etc. Dreamhost on the other hand had a little heatwave and went flopping over like a rotten fish.

So, maybe that was not their fault? Well, for one of the sites for our work, this morning the site was down with mysql errors, no email from or to them for a while because dreamhost mail works like Iraqi power. And then when mail got through it came out that there was an sql query that they could not figure out how to stop so they did us the wonderful favor of RENAMING the mysql tables to stop the query from running, no email, nothing. And to top it all off, they then say “learn about mysql EXPLAIN” (because it is not their fault in any way) on a site that has been running for a long while with no issues and no complex queries. I should not even need to “explain” to someone how unacceptable this is.

Constant downtime, neverending mail troubles, geeksquad rejects as system admins and legions of people sending around coupon codes trying to save money… Do yourself a favor, protect yourself by using anyone else.

I want to do a little rant about web2.0 and then on some new things, like Google getting beat to the punch at one of the greatest new things to come out of this 2.0 gooball (it isn’t a bubble, just the graphics look that way)

Lets be clear about something, “Web2.0 is just web1.0 with prettier javascript” it is not a new web, it is not really anything new, it is fueled by ONE library and that libarary is named prototype and it is the glue that has allowed this thing we call “web2.0″ to come to where we are currently.

Prototype allows us to *not* have to hack around in nasty javascripts that are smeared throughout our html, it allows us to have access to the DOM in unprecedented ways and from the prototype wellspring has come all the libraries like script.aculo.us and moo.fx and dojo and.. and.. From these libraries have come plugins for frameworks like ruby on rails, django and cake which have taken prototype and made it easily accessible to your application. What the combined effect of these frameworks doing this is, is that now you really don’t need to know javascript inside and out to make use of the full power of it.

So, right now all you need is to look around you and see what people do online and with computers and even in real life and just like the mousetrap analogy you figure out a way to make it “better”, “easier” ot even just “fun”. You can get a cheap account at a webhost (however, due to many recent experiences and from listening to people you do NOT want to use dreamhost, I am without email for days at a time with one site I work with) then because “cheap webhosts” have a tendency to limit your bandwidth and your drivespace you can’t really run a very busy site on one. Now comes into play some new stuff which has recently emerged.

Amazon s3 (simple storage service) which I will make a bold prediction about.. “Google will be coming out with a storage system similar to s3 within a year of right now” and I would be willing to bet money on it. Yahoo will also more than likely try it as well.

Anyway, S3 is pretty amazing, it is quick easy and cheap cheap cheap. You host your larger files, user content, images, basically anything and it is almost a drop-in replacement for hosting your own files. It is metered so scales “with you”. If you are currently having to run one of those “platinum” plans from a webhost, you can drop down to their “basic” plan and host your images and files with S3 and save yourself a few hundred dollars per year.

The main reason I mention S3 is that after looking over the service and some of the implementations and playing around with my account I am impressed that it works so well and basically allows anyone with a few bucks to start a site that can be basically be scale-paid. Enough to the point that I can make another prediction, scaled webhosting which is based only on your gigabyte disk and line usage is most likely already out there, but will become the new defacto system and guys like cpanel, plesk and the large hosting providers like serverbeach will be quick to step up and get on board with this.

I am sure the apparently very smart people at Amazon saw this and so they said. Well, lets sell servers using the same model. And so was born Amazon EC2 (elastic compute cloud) need a new box? just fire up an image you have already uploaded (or use one provided) and there it is for ten cents per instance hour, or $2.50 per day per server or $75 per month before the standard costs for the S3 gigabytes/bw (totals come to about half of what I pay for my dedicated box)

So now we have metered unlimited space, any file can be turned into a torrent by simply adding “?torrent” after the file name (to save you some $), metered bandwidth and even metered instance time on a machine, it is all “pay as you go” and is built and designed for the smaller shops or even the medium sized businesses which could use a little work. To give you an idea of the savings, I worked for a mid-town company here in New York that paid thousands of dollars per month in bandwidth overages which would have amounted to perhaps $50 combined on S3.

To top all this off, Amazon now has a “People task” quick job board you can do called Mechanical Turk where you can hire people to do the little things for you, or take on little jobs in your spare time.

Today Google launched Google apps for your domain which really does nothing for me because I am not interested in branding my own version of gmail, everyone already uses the real gmail. They are usually pretty good about getting stuff out, but this is the first time I have really seen them being beat in something that is normally their domain.
When did Amazon get “cool” and Google doing the boring stuff?

So, for your little web2.0 startup the above links should be plenty enough to turn it out on a dime and also use and get your hands on the good stuff.

Some references:

http://www.holovaty.com/blog/archive/2006/04/07/0927

http://blog.pairwise.net/2006/08/28/a-simple-amazon-php-s3-class/

http://overstimulate.com/articles/2006/08/24/amazon-does-it-again.html

http://blog.pairwise.net/2006/08/22/amazon-s3-the-holy-grail-of-bandwidth-problems/

amazing green leaf

Originally uploaded by joeldg.

Eve and I took a trip yesterday to Wave hill park in new york and I uploaded a bunch of photos from it. This is an image of a leaf that I saw and had to take a picture of because the light coming through it looked so interesting.
This is also partial a test of posting to my site from flickr, so checking it out.

There is a whole photo set of the day, obviously not all of them, but I ran out of upload space for this month at flickr…

Part of the reason that I switched to wordpress is that I wanted a platform that had native mo-blogging support. I considered hacking around in other simple packages but in the end wordpress won me over. Eve and I ordered some new phones which should be delivered soon (monday). I am pretty excited, the phone is the nokia 6682 and eve found a pretty good deal (clicking picture will open up the amazon page)

The phones is one of the nokia series 60 GSM smartphones that is normally about $500-$600. It is a USB, Bluetooth enabled phone that can do video calling, play mp3’s, take up to an hour of video, connect to laptop easy and has a 1.3 mega-pixel camera with flash, it comes with the lifeblog software which you can use to keept track of your comings and goings. The total for the order was $0.02 which I didn’t know they could actually charge such a low amount to a credit card. There are some stats and reviews of the phone online obviously. Anyway, told some people I would post the link to the one cent smartphone deal at amazon, so there ya are.
and here is a larger img

note: this is more of a note to myself than anything
To do latitude/longitude radius from a point in mysql, first you need to have a table with latitude/longitude pairs for items to look up. You can find various databases online with geocoded data.

Now lets assume we are on a spot near Union Square in New York.
its coded location is: Latitude: 40.7383040 and Longitude: -73.99319

We assume the following distances in relation to our earth’s radius (R)
6378137 meters, 6378.137 km, 3963.191 miles, 3441.596 nautical miles
We will use these in our computation for distance from point if we want to use miles, kilo’s or meters from our starting point, if you really wanted to get crazy then 6378137 meters = 20925646.3 feet so you could literally search for something within several hundred feet of yourself.

We use these units in the following SQL to determine how we want to determine our distance from point of origin, so using R = 3963.191 to give us our distance back in miles. For this example we want to see what is 1.5 miles from this point in our database.

note:PI = 3.141592653589793, mysql’s pi() function returns 3.141593 so if you need finer grain granularity then use the above constant, latitude and longitude are the names of fields in my database in the query below


select asciiname,latitude,longitude, acos(SIN( PI()* 40.7383040 /180 )*SIN( PI()*latitude/180 )
)+(cos(PI()* 40.7383040 /180)*COS( PI()*latitude/180) *COS(PI()*longitude/180-PI()* -73.99319 /180)
)* 3963.191 AS distance
FROM allcountries
WHERE 1=1
AND 3963.191 * ACOS( (SIN(PI()* 40.7383040 /180)*SIN(PI() * latitude/180)) +
(COS(PI()* 40.7383040 /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* -73.99319 /180))
) < = 1.5
ORDER BY 3963.191 * ACOS(
(SIN(PI()* 40.7383040 /180)*SIN(PI()*latitude/180)) +
(COS(PI()* 40.7383040 /180)*cos(PI()*latitude/180)*COS(PI() * longitude/180-PI()* -73.99319 /180))
)

This can be used over the results of a sub-query as well, so if you have a huge dataset you can search for a squared area by just adding and subtracting from both longitude and latitude taking that result set and running this query over it. The query give back results from closest to furthest.

In tests over 6.2M records on my machine here, this took 21.3 seconds to complete without subquery first and 0.05 seconds after subquery over the 6.2M records. latitude and longitude being indexed.

Well, I decided to finally switch to a more powerful cms for this site. Have some needs currently and coming up that require a little more granularity than eggblog could provide, though I do miss it’s simplicity and quickness.
I am pretty impressed so far with wordpress however and think I will be sticking with it for a while.
You may notice the dates are all changed on the posts so they show as being posted today, this is because I just quickly dumped all the older ones in.

Just a quicky:

Every developer I have sent the link to gotapi to has come back and told me that it is now their home page or one of the tabs that opens when they start firefox. It has become and indispensable tool for me and those I work with. On one fast page you have the docs and fast lookups for every web development language (with the notable exception of python). My gotapi I keep with the following “HTML, CSS, Javascript, Prototype, MySQL, PHP, Rails Ruby and Ruby stdlib.

If you are a developer, go there, all the JAVA api`s are listed, C and C++, XML, actionscript etc.. It is the one stop shop for documentation.

To the guys who wrote it .. “Thank you”

Secondly, Protolize the ultimate web2.0/ajax/rails/cool list of links that is always just what you need.

Some other notables:
programming cheatsheets sort of like the “key cards” they supply in video games, but for programming languages.. Handy to print out and staple to some vertical surface at eye-level

Learning ruby links a collection of links that other people have found that I thought were worthwhile.

CSS event selectors sort of like behavior, but a little easier to use.

nice protype-based image cropper script that I can think of many uses for. One being to crop an image so that a section can be used as an avatar/icon for a website.

Open web design and Open source web design both great resources for that fast template you might need to mock something up or just get a framework for a site setup.

Just a few links I wanted to organize.

Not my normal topic, but this is a new show on television called Heroes. Basically the idea is that ordinary people discover they have extraordiary abilities and what they do with those..

The full one hour first episode can be downloaded here and a four minute “super trailer” can be downloaded here the first episode is promising, though we are just being introduced to the main characters.

I think my favorite character was Hiro (yes, the name) who is a Japanese salary worker bored with his ordinary life and who has read too many comics.

The show has some leanings to LOST style of storyline where everyone is connected and there are a lot of hidden little gems in the show that I caught, and I am sure as the series progresses it will be more pronounced.

Not too many shows aside from anime that I go out of my way to check out, this one was worth it..

Tomorrow I am going to post about my findings with YM4R (googlemaps and geocoding) in rails and also do a walk through perhaps on some of the AJAX rails calls that I find handy.. (I am learning here as well, sometimes it helps for me to try and explain to others)

there is no cpaddon module for rails, so you gotta get your hands dirty. This is the fast way I got it to run, I will makes notes where needed. This is current as of August 2006 with the latest “stable” cpanel WHM.

As the root user, install ruby, gems, rails, fcgi, mod_fastcgi and add a configuration line to http.conf and restart as follows ( condensed from: here):

$ cd /usr/local/src
$ wget ftp.ruby-lang.org/pub/ruby/ruby-1.8.4.tar.gz
$ tar -xvzf ruby-1.8.4.tar.gz
$ cd ruby-1.8.4
$ ./configure && make && make install

$ cd /usr/local/src
$ wget rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
$ tar -xvzf rubygems-0.8.11.tgz
$ cd rubygems-0.8.11
$ ruby setup.rb

$ gem install rails

$ cd /usr/local/src
$ wget fastcgi.com/dist/fcgi-2.4.0.tar.gz
$ tar -xvzf fcgi-2.4.0.tar.gz
$ cd fcgi-2.4.0
$ ./configure && make && make install

$ cd /usr/local/src
$ wget fastcgi.com/dist/mod_fastcgi-2.4.2.tar.gz
$ tar -xvzf mod_fastcgi-2.4.2.tar.gz
$ cd mod_fastcgi-2.4.2
$ /usr/local/apache/bin/apxs -o mod_fastcgi.so -c *.c
$ /usr/local/apache/bin/apxs -i -a -n fastcgi mod_fastcgi.so

$ gem install fcgi

$ mkdir -p /tmp/fcgi_ipc
$ chown nobody.nobody /tmp/fcgi_ipc -R
$ chmod 755 /tmp/fcgi_ipc -R

Then in /etc/httpd/conf/httpd.conf add

LoadModule fastcgi_module libexec/mod_fastcgi.so
<IfModule mod_fastcgi.c> 

FastCgiIpcDir /tmp/fcgi_ipc/
AddHandler fastcgi-script .fcgi

< /IfModule>

$ su people
$ cd ~
$ rails test
$ cd public_html
$ ln -s ../test/public/ rails
$ cd ../test/
$ chmod -R 777 tmp/
$ cd public
$ chmod 755 dispatch.fcgi
$ vim .htaccess