joeldg - navigating the surface

I am starting a series of panoramic shots of New York city, and I am going to start with this short article on creating panoramic images in linux..

I have a digital camera and have been too lazy and have better things to spend my money on than memory sticks, so.. I can take six photo`s with it, so I am always trying to figure out what to photograph so that I am not wasting shots, would it not be better to just have one GOOD photo than a series of so-so ones.. I started looking into panoramic images.

Now, I run gentoo linux, so clearly the imaging tools are available I just needed to research it.

There is a half-baked tutorial/infopage at the gentoo wiki on the tools to use. Though, most of those tools you really don`t need, the things you do NEED are Autopano tools, Hugin and finally enblend which will do softblending over your images so that there are no harsh lines from color shifts due to light differences and glare.

With those tools, it is pretty easy to follow this tutorial on how to use the tools. Though, I prefer to use enblend internally from Hugin, so don`t follow that tutorial to the letter.

I had taken some photos from a rooftop here in Manhattan, and sat down to work on this. The following image took me about ten minutes, using tools I have never touched, to make the following.

Six images, quick fast and turned into a panorama..
There is one smudgy spot, which is just because I messed up the photo, but otherwise, going from images to panoramic that fast is awesome.

I started a photo gallery here on the right and will try to add panoramics to them here and there as I take my camera out. I doubt my callphone camera takes decent enough photo`s for doing this, but I will try that also.

I get lot of people asking me about AJAX and what library to use, so I will write up here my personal preferences on what to use and why.

First and foremost THE library to use is Prototype, I use a light version of prototype that comes with moo.fx lib.

Prototype is the `core` not the end means for doing ajax calls. For easy easy ajax work you need to look at moo.fx, the prototype lite and moo.ajax can be downloaded easily from here. The reason for using moo.fx is that it is a super small library which gets the job done.

AJAX calls in moo.ajax are so easy it is ridiculous and with prototype $(`ID`).property style work you can just plop whatever into whatever easy as pie.

For “full-featured” libraries for doing effects, I would suggest Rico which has nice smooth animations and scrolling and it`s drag and drop lib is really nice. Though, the reality is you will seldom ever really need this functionality so use sparingly.

The other popular lib is script.aculo.us which I personally like the list sorting portion of, but the library is big and chunky and in general I feel it is just over the top and moo.fx is generally a better choice.

Using small and tight javascript libraries is kind to your users, makes your page feel snappy (vrs that chunky web2.0 feel on sites that include 30 js libs because they just HAVE to have drag and drop on things) and it is also easier on you because you don`t need to read a book to use them. I was up and working with moo.ajax and moo.fx in no time, and any questions I had could be easily solved by either looking over the source (which is 3kb) or poking around in the documentation they have on their site.

note this entry is way out of date and the code is mostly gone

just an interesting tidbit

I like interesting ideas and interesting effects with js and transparency in particular.
Try the following link..
click here

You will notice that a interesting corner box appears in the upper right.. I am experimenting with this and having it being called via ajax, so that you can be sitting there on a page and as an event comes in that needs your attention, you can pop up this cornerbox which will not bother the user but they will be aware of what needs to be done and are offered a link to check it out or to dispell the effect.

I tried to get this to work with IE, I did.. but, the png fixing script cannot seem to fix the transparency in a hidden cell..

Funny how all the “fixes” are for IE now..

Short paper on large scale data indexing for retrieval.

When dealing with large amount of data, at times it is necessary to index outside of your current database, particularly when space is an issue.

The new MySQL archive storage engine is perfect this scenario, however you still need to index the data as the archive engine is compressed and does not support indexes (and rightfully so).

There are several camps of thought in this arena:

• hashed indexes
• Bucket systems (a data-structure)
• bloom filters (hash filtering)

with a hashed index, you need to deal with very specific queries and the hash tables will take up large amounts of space.

with a bucketed system the data is highly organized, but space will still be a major factor as will be retreival and again you will need to store an additional database that lists available data. Each piece of an index is broken down. You set up directories of the data like 0-9_a-z_A-Z (62 directories) and in each create an additional set of 0-9_a-z_A-Z, thus when retrieving the data for record number “DJ192″ you would find that item in /hashdirs/D/J/DJ192 (very simply put this is a backet system, there are obviously much more advanced versions of bucketing, even within databases)

Bloom filters were designed (quite a while back) for a different usage, but have proven to be fast, small and can be tweaked for reliability in this area.

A bloom filter is a “ONE-WAY” hash, meaning that you cannot derive the type of data stored in a filter, you can only check for the existance of a piece of data. Traditional bloom filters are used in spellchecking applications, RFID identification for stores and anywhere that you may have enourmous numbers of distinct chunks of data.

De Gan filters is my improvement on bloom filters which has been able to store enourmous amounts of data in a hashed thread-table. It is not perfect and needs to have some tweaks made on it, but for one-way hashing tables in a scripting language, it is well formed and fast (even without bitshifting implemented)

Well, that is great but how is this useful?

First, you will never search your indexes on items that are not in the database, second you can greatly speed up your searches by using this filter and third when you create the record in the filter, you can also store a combine hash (md5 over the sha1/md5 combo) and use that to reference your data. In the case of a text article, this is useful for word pairs/pairing hashing so that you can easily search them and know that your data retreival will not be wasting any milliseconds checking for data that is non-existent or actually combing through the data.

For small(ish) sets mysql fulltext is fine, I have had no problems or cpu issues on a database with over 5,000,000 records running fulltext searches over multiple fields. However, in the case where you go to 50M, 100M or 500M (depending on your hardware) you are going to see some drastic reductions in performance of searches, especially if the database is being heavily used.

With the filters I developed, it is entirely possible to fit 100M records in a one-way hash in about 20-25megs of space, in a mysql database, this would equate to the records and the indexes at most likely 200meg to 500m depending obviously on what is in the table, though for even an indexed lookup it will be much larger than 25meg for a hash. Additionally, you will need to store alongside the table a further index of the hashes, the best way to do this is bucketing the databases and tables (database A-Z etc and tables A-Z) which will be large, but will be much faster than a straight single table.

does this work?

Ask google, the founders original intent and design was published for all to view and study. It is like a roadmap on how to build a monster ever-scalable search engine

edit also found the google spasehash project that google released.

I remember working on old computers and thinking that by 2000 we would all have amazing computers that could do so much and by 2005 we would have have computers built into our eyeglasses, houses, and everything around us.

I remember working on old computers and thinking that by 2000 we would all have amazing computers that could do so much and by 2005 we would have have computers built into our eyeglasses, houses, and everything around us. It is true that cars have done a lot recently, but desktop computers still suck. We still use mice which were invented in 1963 to move around, 43 years, we still rely on keyboards (which are fine) and we use basic windowing systems to navigate data within our computer. What happened to those systems in Johnny Mnemonic, minority report and Paycheck with all the interfaces there? Are we doomed to be stuck in 2d computer mentality forever? It was said that virtual reality would be viable once computers were faster, well, they ARE faster.

A few years ago, I wrote a windows application that could take apart a file, split it into defined chunk sizes, y-encode it, and upload it to “free” web hosting sites with various names and even concealed the filedata inside of HTML.

A few years ago, I wrote a windows application that could take apart a file, split it into defined chunk sizes, y-encode it, and upload it to “free” web hosting sites with various names and even concealed the filedata inside of HTML. It would upload the data into multiple places so that the file would survive a freehost “cleaning”. It would then create a file that could be “encoded” with a password which you could pass around to your friends or post anywhere and if people used the same program and loaded the “map” file they would download the various bits, uncloak them and then rebuild the file on your local box.. The idea was a killer, but it had a few issues, one the free hosts would catch on fast, freehosts would also be harmed by this idea as people would be uploading multiple copies of ISO`s all over the net.. BUT, it did solve the port blocking problem with some other apps and “was” technically valid html.. I created and tested this app, even wrote a “mass account generator” to go along with it, which was never finished before I decided to abandon the project after speaking with a company lawyer.

Mashups are pretty cool, of course they are all over and a lot are seemingly thrown together.

Mashups are pretty cool, of course they are all over and a lot are seemingly thrown together.

There are some sites that catalouge all the hacks they find. There are some good resources on all things google. Some New York city real estate maps that look like Sim City other speciality maps of New York (non google). weather is covered, history is just starting to be covered. People are using them with geocoding to do guestbooks, traffic stats, and visitor indexes (too many links there) and Yahoo just jumped in the fray, with Microsoft having previously done their version (which of course had no api or anything useful that people could do with terraserver)

The bane of my existence..
I hate having to code to get something to work in I.E.

The bane of my existence..
I hate having to code to get something to work in I.E.

You can get a site working perfect in Firefox, which follows all current standards perfect, with doctypes and all, and then open it up in IE and you always have to make some IE specific *hack* to get it to work.

Why do people continue to use the steaming pile when there are better alternatives which won`t infect your computer, spam you, and slow down your computer out there. Firefox people! and it is actually updated so you can see the web the way it is supposed to look.

Every developer I know right now is facing the same issue.. It is the IE frustration, the IE headache..

The Joel on Software guy Joel Spolsky wrote a piece about why he won`t use the term web2.0 which I found interesting for a few reasons..

The Joel on Software guy Joel Spolsky wrote a piece about why he won`t use the term web2.0 which I found interesting for a few reasons..

The term web2.0 is actually not the exact definition of using major revision numbers, because technically we would be on maybe web 1.84565 which I get..

However, the “2.0″ in the web2.0 is really in the way we “think” about development for the web. It has been a while since we have had to drastically alter the way we thought about development of web-based apps. When starting from scratch there are entire new systems in effect for the actual process.

Computer stew was probably the best `real` commentary on the bubble while it was happening..

Computer stew(no downloads from this link) was a daily comedy show that got me through working long long days back in the bubble days..John Hargrave the man behind the `stew` is funny and still seems to be going.

You can find the complete collection here.

Computer stew was probably the best `real` commentary on the bubble while it was happening.. There is nothing close to it now (or since), but then who is going to actually `pay` fulltime people anymore to create a comedy webcast? (zdnet really should have kept them around)