Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(/home2/people/www/blog/tmp) is not within the allowed path(s): (/home/people/:/usr/lib/php:/usr/local/lib/php:/tmp) in /home/people/public_html/blog/wp-content/plugins/wassup/lib/wassup.class.php on line 268

Warning: Cannot modify header information - headers already sent by (output started at /home/people/public_html/blog/wp-content/plugins/wassup/lib/wassup.class.php:268) in /home/people/public_html/blog/wp-content/plugins/wassup/wassup.php on line 1265
joeldg - navigating the surface » general

general


If you are a rails, or a java mobile ninja and live near New York city you may want to send in a resume to us. See: our blog post showing the two positions we are hiring for. As the lead engineer, I can tell you a bit about the projects you would be working on and the tech we need.

Ruby on rails, we need someone with experience in ROR, someone who can display a competence in ROR and is familiar with “railzy” ways of doing things. We will ask for sites and possibly code samples.

MySQL: We ‘use’ mysql, we don’t just have a database, we are using mysql 5 specific functions and it would be helpful if you are familiar with views, stored procs and triggers.

Linux: All of our products run on linux, it is good if you know linux and server basics.

The Java mobile (J2ME) guy we hire will be on the cutting edge of mobile applications working on a product that will be featured by mobile handset makers and carriers. It is pretty high profile job, and you would be working with some of the brightest minds in the business.

Clearly, we are looking for a proven track record and experience in java mobile, if you think you fit the bill, then by all means apply via the link at the top of this post.

funny naruto

A while back I started watching the Anime Naruto, and loved the storyline of the original part of it. It went on perfectly until about episode 140, when the television part actually ‘passed’ where the Manga (drawn comic) was at. They didn’t really know what to do, so they created all these filler episodes until #219. I couldn’t stand the pain of watching the horrible filler episodes so I

finally gave up at about episode #180.

All the while the manga continued to develop the story and is much further along now. So, they basically pretend that the last 80 or so episodes didn’t happen and that he was “training” and

we re-enter the story two and half years later, and Naruto has grown up some.

And grown up he has. He is taller, finally got a new wardrobe and we can only assume he has picked up some fancy new abilities, this episode is perhaps the most anticipated Anime release in history as people are all kinds of crazy over it and there are multiple websites that are devoted

naruto showing us his growth

to all things Naruto.

It is a crazy episode, and they have a lot of catching up for people on what is going on.

When I downloaded the torrent, there were about 20,000 seeders and so it literally was the fastest 250MB I have ever downloaded.

I also noted that Naruto’s headband has changed colors.

Each of the kids (including Konahamaru and his buddies) have grown up, and each has their own little explanation for the audience showing their size in comparison to how tall they were. The Sasuke storyline finally picks back up and we are teased with it in the very beginning of the episode with a glimpse at where this episode and the following ones are leading.

Naruto's new ability ;)Anyway, if you are not familiar with Naruto, watch the eps on one of the sites (some have streaming episodes even) up to about episode ~130 to ~140 and then skip ahead to episode 220 (Shippuuden 001-002)
Pick it up at http://yhbt.mine.nu/t/ The wait is finally over.

You can also pick up Bleach there as well, if you are not familiar with Bleach, it is kids who work as Reapers, vrs the Ninja school of Naruto.

I have done a ton of research into animating sprites for a web page via javascript and am curious if I am on the right track.
Basically, I am looking at the following options:
1) SVG animated sprites, calling functions within the svg file itself from js on the page.. like walk_up(), walk_left() etc.. I have not tested this yet, but I want to generate a prototype of it to see if it would work. I know this is possible (see: this killer svg example for how JS works within an svg file in firefox.
2) DHTML with CSS cropping .. switching fast between styles to ‘animate’ the sprite.
3) Animated gifs. (this is more of a last resort)
4) Other.

What I am trying to determine is what would the best way be to animate a little walking guy on top of another image. Also, there may be multiple little walking guys.
I am going to avoid flash because I actually want to do this without flash, and to keep the file size (thus bandwidth) down.

Any ideas?
I miss anything?

powered by performancing firefox

I have had an interest in simulating weather systems for a while. And have given a lot of thought to what makes a weather system ‘random’ and how to achieve this in a programmatic way for usage in a “game-world” to have a true weather system. I have done some additions (much reprogramming actually) to a fla file that was created on another site. I woke up this morning after half-sleeping thinking about this and hacked this out, using a base from the original fla file, I played around and worked on randomization and such and wrapped this up in about an hour.

The idea being that you have two “attractors” which represent low pressure systems, and two repelling spots, which represent the high pressure systems. The little lines represent clouds or weather of some kind.
You can drag the high and low pressure systems around to see how they interact, and they also move randomly by themselves.

In a full blown system (not a flash mock up) these high and low pressure systems would have destination points randomly selected and migrate slowly towards them.

If you sit and watch this for a while you will see that just by moving the points around there are some very interesting interactions with them and you can effectively create “jet-streams” where the clouds prefer to travel, and that a move of one high or low pressure spots will change the entire way the weather system moves.

One of the things is that in games, or in any simulation, weather is never really a factor and is not accurately represented (sometimes it is randomly introduced). I think this is a huge part of a game world that is being and has always been overlooked.

For a non-mockup of this, I would/plan to, have a random number of high and low pressure systems (with at least three) and each would have a pre-defined lifetime and fluctuating strength of attract/repulse. The number of clouds/weather would also not be pre-set and would change (as per seasons) and could be modified based on speed and possibly combine in strength based on if they are pushed together, vrs these which do not combine. I would also implement curling about the systems also to better represent actual cloud systems, in this mockup the clouds do not have lifetimes either, they are permanent, each should have a randomly determined lifespan and intensity. However, this would enter into some computational complexity that would take a lot of processing power for each iteration.

I have been working a lot these last few months, and when I wasn’t working I have been trying not to overload on development and have stopped myself from working on any new projects on the side. I have finally been able to get my head above water and start to take a look at some of the new things coming out and I feel compelled to discuss Adobe Flex.

I have never been a big fan of Flash. As a linux user, I have been irritated by Flash9 not being ported for a while and did not consider flash to be something I would ever work in or support. Then I heard some rumbling about this thing called Flex and heard rumors about another thing in development that works with it called Apollo. Apollo is scheduled to be released early this year (meaning within the next two months) and as I understand it, you build an app in Flex using HTML, JS and some flash actionscript sprinkled in, you export it through Apollo to a desktop app for Mac, Windows (and hopefully linux) and suddenly you can take your webapp and make a dockapp for windows, or a background desktop app and it all connects to your website for updates and data. This is how I understand it, and how it is presented. (see the screenshots from the demo).

Flex is picking up some momentum, there is a Flexblog with a ton of examples and nifty demo’s and a ton of blogs linked off the Adobe flex page. There is also a prerelease signup for Flex3 which has more support and such for Flash9.

One of the things which got me interested in Flex was the skinning ability (i.e see this) and that it is a simple matter to have it connect to PHP/Ruby or Python as a gateway to MySQL which is clearly where it becomes interesting. Adobe offers a “try” version of it for free on the website so you can play around with it and quickly make a nice frontend to a webapp.

Flex is apparently also compatible with Mobile devices, through Flash lite (flashlite? heh..) I have not tried this out yet as I am still playing around learning the Flex interface. If it works out perhaps this is something I could recommend to work for building apps on top of Socialight.

Anyway, from my initial impressions, Flex is something I think is worth checking out (at the very least) and the charting is well worth it for doing general all purpose charts of data (I get sick of looking at plaintext lists of numbers). Will be something I work with a lot more. Obviously, if I choose to work with Flex a lot you will see some apps appear here from time to time.

I have decided to combine these two reviews.

First: Children of Men.

I normally write about things I cam currently working, projects or little things I create in ‘code-space’ but recently one of the things that I have ended up talking to people about is the movie Children of men. Every person who sees this movie *needs* to talk to someone about it because it has such a profound impact.
The movie is dsytopian, and bleak. It is depression and somehow uplifting, it is a strange combination and a powerful story lies underneath the absolutely perfect film making.

Many scenes in the movie are filmed as ‘long shots’ which basically means they have the camera rolling for the whole time and not scenes cut and spliced together, what you see is what you get. For this alone, it is amazing and a shining example of a film that, like it characters, does not take the easy road.

There are so many things to say about this movie I would not know where to start.

Primarily, the things I noticed while watching this gripping and incredibly dense movie were:

1) The primary character (Theo) is a complete pacifist and never picks up a gun throughout the entire movie, when he finally had to hit another character (Sid) he does so in such a way as you know he has never done anything like that before. Don’t expect Theo to turn into rambo, he isn’t and his character stays true to that.
2) We only see and hear what Theo can see and hear, any conversation off screen or through a window are hidden from the audience. This is a subtle but effective tool in making us connect with the character.
3) The aforementioned ‘long-shots’ are not overdone, actually, nothing is over done you almost wish they had done more of those incredible shots.
4) The audience is placed into the world that Theo lives in, it is as if you suddenly find yourself in that world and there is no real attempt to force feed you the details of it, you discover yourself as you go just what is happening and why, but it is not forced on you. It feels like you just walked into something and you connect more with the characters because of this (not in spite).

5) There were real horrors in this movie and part of that horror is that the violence was treated the same as anything else, it just was. There was no special treatment on violence, it was very (as Eve would say) ‘matter of fact violence’ and this just served to amplify it’s effect on the viewer.

6) This movie is long, but it only feels like 30 minutes.

7) No person in the theatre made a sound during the entire movie, everyone was riveted to the edge of their seats and forgot they had popcorn and drinks. Hell, they forgot they were in a theatre.
I am so glad I saw this movie in the theatre, I could not imagnie watching it on a little television. It really does demand the large screen and the full sound.

8) As the movie progresses, the camera get closer to Theo, by the end of the movie we are running practically alongside Theo. This again is subtle, but involves you deeply with Theo and his perceptions.

If you have not seen it, I would recommend this movie to almost anyone (except friends who have children as this particular movie seems to strike a nerve with parental types and they hate it before it starts)

Tonight I am seeing Pan’s Labyrinth and have heard it is equally amazing.

I am SO glad we are out of the glut of crappy films that we were in from 2003-2006

Editing this post now that I have seen Pan’s Labyrinth.

Pan’s is first and foremost a dark dualistic movie, and secondarily can be taken on it’s fantasy elements.

The story is dense and replete with symbolism, both internally and in the world today. The movie starts with a girl named Ofelia and her pregnant mother (who is not doing well in that endeavor) as they are brought from their home in ‘the city’ (which one we are never told) to a mountainous encampment of her stepfather, a commander of troops who is going through the motions of being a leader as he feels it should be done in living up to his brutal fathers memory, his job that he does not enjoy but does with ruthless precision is to root out a last enclave of rebels in the mountains near his encampment. His encampment is an old stone mill house, with a large main house and barns and quarters for his men.
Ofelia is an imaginative child whose eyes we see the world through, her world of imagination is tied into and interacts with the world as she sees it. As events happen in the ‘real world’ of the story, her world of imagination keeps up with it and offers us insight into some of it. For instance, as the general is cutting rations and has a stockpile of medicines needed by the desperate locals, and we see the local peasants waiting in line for meager bread, Ofelia’s world has her in a dangerous place, with a key to get a knife from a locked wall, where she is forced to walk past a vast feast of food that she must not eat. As is classic in this type of story, she cannot resist and samples some of the food and barely makes it out losing two of the fairies that accompany her. This is on par with Mercedes, the housekeeper whose brother and the rebels in the woods are hunted by the general. She has to walk past the reserves of medicine and has secretly kept a spare key to the store room which is given to the rebels. Etc..
Mercedes is portrayed as Ofelia as grown up, the Ofelia that has stopped believing in fairies and fauns, but one who is familiar with Ofelia’s imagination, and takes the place of protector of Ofelia (even to an intense example in one very brutal scene). Mercedes story is in-line with Ofelia’s story as are all the characters.

There were many subtle nods to all the famous stories, stories we see Ofelia carry around in a stack of books as she is doing the tasks that the faun gives her. The faun is introduced early in the film as Ofelia’s fairy leads her to him at the center of the old stone Labyrinth, near the encampment and which Mercedes is strangely familiar with, telling her stories of her own mother saying to not trust fauns.
I could not mention much more without giving away the story entirely, but suffice to say that in this story, the imaginative elements are deeply tied to the real world events, some that even Ofelia could not know of. And the story circles and turns a delightfully dark and twisted path to it’s end where all the stories come together and we are presented with an ending that is both tragic and wonderful as Ofelia has done her duty to protect her brother from harm while, the grown Ofelia/Mercedes has also protected her brother. We are confronted with the sadness of growing out of imagination and the wonder of it at the same time, the ending is compelling on many levels and will leave you much to think about.
Some things to watch and parallels in the movie.

1) The eyeless man, appears to be a parallel with the government the general follows, and the dignitaries who have recently come to the house blindly spouting things they themselves do not believe, eating at a table of plenty allowing none other to touch. (notice the pile of shoes that is in only one shot)
2) The faun is kind of a father figure, though he is a guide.
3) Mercedes looks like a grown up Ofelia, while Ofelia does not look like her mother. As the story progresses, Mercedes becomes more the mother and the mother becomes more the wife to the general.
4) The frog that poisons the forest tree and that holds the golden key, turned inside out by three stones I ‘think’ is a foreshadowing to the general and three stab wounds he gets. It could also be a more over-arching subtle message that would need a further viewing to see.

5) The death of the two fairies, I am not 100% sure about, but it could be an allusion to the two rabbits hunters, or it could be the two spies the general discovers.
6) The stick bug fairies are a definite relation to the rebels in the forest, leading the way, but not always knowing the way (as in Ofelia decided a different lock to unlock, as the rebels should not have unlocked the lock they did).

And on it goes.

Excellent film, excellent effects that are not overdone, excellent acting and over all beautiful movie with contrasts and subtle references galore. You will be very glad to see this movie and like me will want to see it again.

Rails memcached is not very easy to introduce to a large rails installation. Memcached also chews up a lot of memory on the box and overall cached model does not work the way I needed it to. Basically, I have just a “few” queries that I needed to cache because pagination sucks just that bad in rails.
So, I built my own cache, similar to how I build them in PHP except I am not using disk cache, I am using MySQL itself to cache it’s own results

First, we need a table to hold all this info (note the ‘blob’ field)

CREATE TABLE `cacheditems` (
`id` int(11) NOT NULL auto_increment,
`cachekey` varchar(255) default NULL,
`created` datetime default NULL,
`expires` datetime default NULL,
`content` longblob,
`cachehit` int(11) NOT NULL,
PRIMARY KEY  (`id`),
KEY `cacheditems_cachekey_index` (`cachekey`),
KEY `cacheditems_created_index` (`created`),
KEY `cacheditems_expires_index` (`expires`)
)

Then we create a model called “cacheditem” which has the following functions

require 'digest/sha1'
class Cacheditem < ActiveRecord::Base

def self.checkfor(sql)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info "%%% checking for key #{key}"
#logger.info "%%% checking by sql #{sql[0]}"
Cacheditem.find( :first, :conditions => [ “cachekey = ? AND expires > NOW()”, key] )
end

def self.getcached(sql)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info “%%% getting by key #{key}”
#logger.info “%%% getting by sql #{sql[0]}”
getc = Cacheditem.find( :first, :conditions => [ “cachekey = ?”, key] )
hitcount = getc.cachehit + 1
Cacheditem.update(getc.id, {:cachehit => hitcount})
Cacheditem.delete_all “expires < NOW()"  # cleaner
return Marshal.load( getc.content )
end

def self.storeresult(sql, result)
key = Digest::MD5.hexdigest(Marshal.dump(sql))
logger.info "%%% storing by key #{key}"
content = Marshal.dump(result)
logger.level = (4) # this stops display in logs of the marshal data
ci = new()
ci.cachekey    = key
ci.created       = Time.now()
ci.expires        = 30.minutes.from_now() # change as needed
ci.content        = content
ci.cachehit       = 0
ci.save
return  result
end

end

Then, in application.rb I added the following function

def find_by_sql_cache(sql)
iscached = Cacheditem.checkfor(sql)
if iscached
Cacheditem.getcached(sql)
else
result = connection.select_all(sanitize_sql(sql), "#{name} Load").collect! { |record| instantiate(record) }
Cacheditem.storeresult(sql, result)
end
end

just throw “_cache” after any “find_by_sql” statement you have a need to cache and there you are.

This works very fast, very well, and doesn’t hog your memory. It cleans up after itself in the database, and perhaps it does that too much.. It would be easy to add in a standard garbage collection function which runs on a random but I felt this gave me much better stats of the actual thirty-minute cache…
If you use zabbix for monitoring your network, you can have fun graphs of cache statistics by adding the following to your zabbix_agentd.conf

UserParameter=mysql.totalcache,mysql --batch --skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e "SELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;" | cut -f1
UserParameter=mysql.cachehits,mysql --batch --skip-column-names -D {YOUR_DATABASE} -u{YOUR_USERNAME} -p{YOUR_PASSWORD} -e "SELECT count( * ) AS total, SUM( cachehit ) AS amount from cacheditems;" | cut -f2

It looks roughly like:

chart

Where cached queries are simply the number of queries that are cached in blue and in red(ish) are the cache hits or queries that we saved from actually running.

It is working pretty well for me.. and decreased the load on our servers by a large amount.

Socialight, the project that I have been working on for a while now with some great people is now fully live.

Socialight is a website which is also accessible via WAP and SMS/MMS as well as a java application (which will be released shortly). It allows you to view virtual geolocated “sticky notes” which you can leave tagged on a spot on the map so that others can come along and find them, comment on them, etc.. They can be organized into channels, which are collections of similar notes and can be private or ‘contact’ only. The site is a social network, but taken out into the real world, where you are connected through your phone in the physical place you are at. For instance, here is my socialight badge, it will show the latest sticky note I have created:
badge

This is my latest note on the site, if you were my contact, and happen to be in the area of union square, the site would have sent you a message with this note content the moment I posted it and also if you later came along, you could browse on your phone and find this when you are in that location.

It is a step into the future, it is bleeding edge, this is the future of social networks and mobile communications. Normal social networks are very abstract and exist only on the web and are similar to bulletin boards of the old days. We have taken this concept and moved it into the real world, where you are physically located.

Some people say, “it is like myspace for my phone” I like to think of it more as something totally new, a new concept, a new way to communicate.

Just a note to say I am still alive, I have been buried in work for weeks and weeks.

We will be releasing a new version of the site on monday, and have dome some pretty amazing things. Working in rails has been ‘interesting’ to say the least, I still would not consider myself an expert by any means but rails is not too shabby.

Originally I was planning to write an entire article about the fallacies of trying to use rails for everything, I still might, but I can see its uses and its merits in speed work.

Monday will be a big day and I will be posting here about the new release of Socialight in the meantime, stop by the site and sign up for the notification of the release and see me on the cool video (I am the guy in the BRIGHT red shirt) …

Templating webpages is a tricky business with a lot of issues and options, and the vast majority of people do it totally and completely wrong. A template, in terms of documents, is any document fragment with blank spaces designed for you to ‘fill’ in. It is something that establishes a pattern. In terms of web pages, a template is the design and interface aspect of a site, it should be entirely separate from the logical portions of the site.

Let me restate that “it needs to be entirely separate from the logical portions of the site”. The primary reason for this is maintainability. The reason we use templates in the first place is to organize our development and have two distinct layers, the logic and the design. Remixing logic into design (as smarty and others do) is counter to the primary reason we chose to use templates in the first place. With a proper template system if there is a database or looping issue we know where to look right away and we don’t have to sift through the 20 templates that make up a page in order to find which one has the bug.
So, I will introduce the templatling system I wrote that forces this issue. (it will appear below) The first thing we need to do in creating a perfect template system is balance out ease of use, speed, and determine how it will function and where it will store it’s template files. It also will need ‘complete’ separation of logic and design.

A few things:

If I request index.php in the root directory of a site, the template engine will look in TPL_BASEDIR + “/index.php/” (yes, a directory of the name of the file) for the individual parts of that template. We will also have a global directory for headers, footers and the engine will look backwards up to three directories to try and find a template.
Template files will have a .tpl extension and will NOT be eval’d so that no code can be run in them and there will be no temptation for the “quick fix” of adding code to them. There will be one exception to allowing eval’d code and that will be code that is in html styled <php> tags for outputting dates or other small single variables. (this was an addition to help in transition from systems like smarty).

We want to be able to set some scheme defaults for our template, things we don’t want to bother with in the logic portion, mostly these will be color codes for css files and pages. (this helps to keep design out of the programmers hands)
We need to be able to have multiple sites using the same template engine.

We need to be able to sling sql statements and query result sets to the engine and have it use those if possible. This is a very DRY approach, similar to how rails does some things.

It needs to be able to process css and javascript files as well as html fragments.

We want a set of functions that also are design related for doing things inside the templates if we need, things similar to the “cycle()” ruby function which are automatically called, but are set and forget. This is useful when dealing with row colors so the programmer does not need to worry about colors or stye classes.

We want the template system to be fast so we build in a mechanism so that the pre-processed templates are not loaded more than once.

What we end up with is a fast and proven system, see:

The template system( txt )

To use this you would include the file and set up a template name replace.tpl which has the contents of:

“Here is the {MYVARIABLE} for sale”

in PHP you do :

$replacements[’template’] = ‘replace.tpl’;

$replacements[’myvariable’] = “house”;

template($replacements);

and you will get “Here is the house for sale”

If you want, you can also use your templates from javascript as well. I have some big plans and loads of ideas for ways to use JSON and prototype.

A very similar system was in use at my old work (this is an improvement and revamped version I wrote for another job a while back), we managed 20+ programmers of various skill levels with this template engine and it worked perfectly. Sites managing tens of millions of hits per day use this system and it works well.

We also never had to poke around in HTML to look for a programming bug.

Try it, you might like it. You can even easily turn it into a class, if you are running PHP5 turning it into a class will not have any slowing effects as it did in prior versions..
p.s. Later I will post some old code for doing in-memory content caching using shared memory (schmop) it is easily fitted into the above template system to speed it up even more.

« Previous PageNext Page »


Warning: is_writable() [function.is-writable]: open_basedir restriction in effect. File(error_log) is not within the allowed path(s): (/home/people/:/usr/lib/php:/usr/local/lib/php:/tmp) in /home/people/public_html/blog/wp-includes/wp-db.php on line 199