Stefan Koopmanschap's RSS

Location-aware web applications are rising, and I'm personally very interested in this space. Besides the obvious cool-factor, I think there are a lot of uses for location-aware information. This post is a short overview of geo-related standards and api's.

The heart of it all lies in 2 numbers, latitude and longitude.

How to get these numbers?

However you publish the GPS coordinates, the information will at one point in time come from a user. However, asking a user to enter these numbers is not the best end-user experience.

Use a map

A very common way is simply ask a user to pinpoint their location on a map. The most common two providers for these are:

• Google maps.
• Yahoo maps.

Google maps is the most popular (no numbers!), but if you are looking for more flexibility, yahoo maps might be the best choice. Yahoo maps has an Actionscript API which enables you to add a lot of customizations. Justin Everett-Church has some cool examples of what's possible. Yahoo also provides a REST api, allowing you to just fetch the tiles and do all the stitching yourself.

Address to GPS translation.

This is called Geocoding. Yahoo has a very easy to use REST API to do this translation. Google also provides an API, which is also able to spit out kml as well.

Browser plugins!

W3c currently has a draft for a geo location api, which is also part of the HTML5 movement.

The user interface is nice too. You are asked by the site if you are ok with supplying your coordinates, which then get sent back to the client.

The best implementation I've seen from this is the Geode extension. I can highly recommend to install the add-on and give the demo's a shot. This extension will only work if your computer has WiFi. It makes use of Sky Hook, which has mapped locations of wifi routers across the world. It mainly seems to have done the urban areas in north america and europe.

I actually moved a month or two ago from 43.651904 -79.428498 to 43.645466 -79.448729. For the first month our router was still mapped to the old address, but it updated recently to reflect my new address. Creepy, but cool!

Gears also has a javascript API, which might be a bit more common. My personal experience has been that I've only been able to get information to up to 4 decimals, which placed me in a different part of town.

Firefox 3.1 will also come with the api built in HOWEVER, it serves as an empty shell for extension implementors. You'll still need to install an extension to actually provide the coordinates, which could work through WiFi mapping, Cell-tower triangulation (the method the iPhone uses) or manual input. An extension that does this through manual input of an address is 'Geolocation'.

From devices

There are a couple devices on the market that track geo information. I wrote a small overview earlier today, and the most impressive seems to be the Amod AGL3080. Its conceptually very simple and when you plug it into your machine, it will show up as an external harddrive with a CSV.

Next to that more camera-phones appear on the market that have support for embedding GPS information within the EXIF data of JPEG's. Most notably recent blackberries and the iPhone. Its reasonably easy to extract this information if it is available. PHP has an extension to read it out, but you'll need to do some conversion from the stored coordinates, as they are specified as the (now less common) degrees, minutes, seconds format.

Publishing

Clearly the easiest way to publish location information is through maps, but the concept of adding this meta-data to the documents you produce (be that html, api's, rss ..) can be much more interesting. This will allow people to re-use the data and present it somewhere else.

Geo RSS

Adding geo information to RSS is an easy one. RSS and Atom is already very widely used. Originally used for feeding blogposts, it has gotten much wider usage such as aggregating pictures, video, etc.

georss.org has some simple examples on how to do this. Flickr is already pretty big on GEO information, it confuses me why they haven't yet integrated the standard.

HTML meta tags

This will allow you add GPS information to a specific site or page There's two ways to define this:

<meta name="geo.position" content="62.300626;-84.023437" />
<meta name="ICBM" content="62.300626, -84.023437" />

There's no harm in including both. It's not very clear to me which one is used more, but the former seems to have more popularity in recent implementations.

Geo microformat

There is also a microformat spec for geo information. Personally, I'm not a big fan of microformats, but would like to be proven wrong in its usefulness.

Microblogging

A standard way to include GPS information for microblogging (and with microblogging, 99% reads twitter.), using a very easy format. This allows cool applications such as Twittervision and Twinkle. The latter had an interesting review on the ever offensive Something Awful.

KML

KML is also on its way to become a standard for the geo-web. KML covers much complexer drawing and is already supported by a number of apps, most prominently Google Earth and Nasa Worldwind. People who have google earth installed can be directed straight to a location on the map by serving a .kml file.

Conclusion

Well, I think the geo-aware web has a big and bright future. Having appropriate standards in place to consume and publish this information is very important. Even though a minimal amount of users will immediately benefit from for example geo information in your html head section. Allowing other developers or applications to access this information today will help pave the future for a fully geo-enabled web. I say all this realizing this blog does not provide this information, but hey.. nobody is perfect :). I can proudly say however I have some geo-aware applications in the works.

I'm very interested in your ideas of examples of interesting use of GPS information! If I missed an important or relevant API or you have a cool example, make sure to leave a comment.

/me goes off to find a restaurant near me and wonders where my friends would hang out on a cold saturday night in december.

Like half of the web development community, I recently became really interested in location-enabled stuff ;)

I really want to get a GPS tracker (for christmas!, hint hint!), there seem to be a couple out there, so I'm wondering if there's any people who have such a device and what their experiences are. Ideal features:

• Small! I don't need to display, I just want to drop it in my backpack and forget about it.
• Has USB, all I need is something that spits out cvs data (or similar format) on the drive which I can extract.
• Somewhat good battery life (>24h).
• Most importantly, charges through USB. Batteries is a no-go (for any device for that matter, I'm looking at you wii controller.)

So we have some options:

Professional mini tracker key

Pro:

• Title of the product is completely written in uppercase, has to mean something right?
• Logs data every second.

Con:

• Uses AA batteries.
• No specs on battery life.
• No specs on how to extract the data, but has USB.
• On the expensive side ($278.99).

TRACKING KEY LAS-1505

Pro:

• "Ideal For Parents Who Want To See If Their Teenagers Are Speeding", seriously.. the page says that literally.

Con:

• Runs on AA batteries.
• Price ($206).

Amod AGL3080

Pro:

• Looks like a more decent product.
• Price: $69
• Uses standard format for data (works as csv on usb drive), accessible with my scripting powers.
• Works on a set interval or 'Push to log'.
• Works for 15 hours (good enough).

Con:

• AA batteries.

Zoombak ZMBK200

Pro:

• Totally rad name!
• A/C wall charger! (the only one so far).

Con:

• Battery life for 150 location requests.
• Subscription fee! (this is where I'm not even going to look for more cons).

Trackstick 2

Pro:

• Reasonable price: $114.
• Exports to CSV.

Con:

• 1MB flash drive.
• AA batteries.

Conclusion so far:

The Amod AGL3080 seems by a stretch the most impressive device, not only does it seem to have a good battery life, and plenty of storage. Just judging from the page on amazon, it quite frankly seems the most professional, despite it having the lowest price on the list.

Dear lazyweb, am I making a mistake? Are there better devices out there?

For some of our long-running processes we use PHP. It makes total sense from our perspective, because we can re-use all our existing business logic from our main PHP web application.

To make things more efficient, I recently started some work on using forks and have a couple of worker processes around.

This application is essentially the core of our transcoder. The parent process would retrieve new jobs from the queue and fire up a number of workers to actually transcode the file. The main problem is that the parent opens up a MySQL connection and fires off some queries. After the child process is done, it actually closes the MySQL connection regardless of if it was actually used or not.

This means I'll have to close all mysql connections before forking, and re-connecting right after. No big deal, but still at least a bit annoying.

<?php

$db = new MySQLi('hostname','user','password');

if (pcntl_fork()) {

    $status = 0;
    // parent
    pcntl_wait($status);

   $result = $db->query('select version()');
   if ($db->error) echo $db->error;

} else {

   // the child process does nothing and exits gracefully 

}

?>

Output:

MySQL server has gone away

In order to play MP4 files while they are still downloading (either by for example Quicktime of Flash), in most cases you'll need to move the so called 'MOOV' atom to the beginning of the file.

The MOOV atom is in essence a block of data, containing an index of the rest of the file. Most authoring tools will place this data on the end of the file, because the size of the block will have to change dynamically as changes are made to the video. If the atom was right at the beginning, the entire file would have to be rewritten for a lot of smaller changes.

So, there are a couple of tools that do this for you, but none of them seem to work really well. FFMpeg contains a tool called 'qt-faststart', but I had to kill it after I discovered it had been running for 12 hours on a relatively small mp4 file.

Some suggestions on the FFMpeg mailing list led me to two other tools, MP4Box and MPEG4IP. MP4Box dies with a big:

MP4Box /tmp/508842_22.mp4  -inter 500
Saving /tmp/508842_22.mp4: 0.500 secs Interleaving
*** glibc detected *** double free or corruption (!prev): 0x08071108 ***
Aborted

On top of that their forum seems mostly dead. MPEG4IP simply has the following message on their download page:

NOTE: currently, the tarball is corrupt, and I do not have a way to remake it at this point. Hopefully that will change in a few weeks.

Dear lazyweb, has anyone else came across similar issues? How would one go about solving this? :(

Update: Reading through it again, it seems as if the mpeg4ip project has simply died in 2007.

Update 2: My problem with MP4Box was solved! I needed to explicitly supply the -out parameter.

Daniel B. Ravicher from PUBPAT did a great talk about the patent system and more importantly, how it doesn't work. Worth watching if you are in the tech industry and if you're interested in the subject.

The PUBPAT's mission statement: "PUBPAT Represents the Public's Interests Against Undeserved Patents and Unsound Patent Policy". Their website could be a bit more glamorous and less dry, but it is an important cause that affects all of us.

It just came to my attention that Zend Framework 1.7 was released, along with AMF support.

This is good news for people looking for a solid AMF implementation for PHP, because Zend with Adobe backing will likely have a lot more eyes on it, and there's a good chance this will result in a high quality implementation.

This implementation has similar design goals as SabreAMF, which begs the question how relevant SabreAMF still is. I'm not really sure myself. I sort of feel SabreAMF served its purpose well. It (mainly AMF3) has been a reference implementation for many other projects such as PyAMF, Red5, some ruby implementation which I know forgot the name of and unless I'm mistaken also AMFPHP and this very Zend_AMF.

Note that these guys never actually gave any credit ;), so I might very well be lying here.. I mainly just overheard this in the various mailing lists and from different people.

So yea, my personal goal has been to be the first open source AMF3 implementation, and build a very clean implementation people can easily drop into existing business logic. I feel I've achieved this, and its been a fun couple of years working on this, not to mention that it has helped putting my name out there, just a little bit.

I'll keep actively maintaining for the people that placed their bets on SabreAMF, recently Asbjørn Sloth Tønnesen has been helping a lot with this as well, but I'll probably start directing people to the Zend implementation for future setups.

Let me know what you guys think. Is there still value in keeping SabreAMF growing, or should it fully go to maintenance mode? Also, thanks a lot to all the people out there that submitted bug reports, patches, blogged about it or simply used it in production! This was my first open source project (hopefully not the last), and it has been a lot of fun :).

Last night the CNMA awards were held here in Toronto, and I turned out to go home with an award!

Although I still feel I didn't really deserve such a prize, I feel honored and proud of the team.

Thanks guys! Its been an amazing ride so far, and also thanks to the CNMA organization and judges.

Just released a new alpha for SabreDAV.

Its been almost 5 months since the last one, the wait was mostly related to the fact that on of my main goals was to pass all litmus tests. Today I committed the last fix for this. There is one outstanding 'FAIL' in litmus, but this is due to a bug in litmus, not SabreDAV.

Changes:

• Passing all litmus tests.
• Added an example for a basic file-server.
• Custom property support in the API.
• Shared lock support.
• Depth support to locks.
• Locking on unmapped url's as per RFC 4918.
• Advertising DAV class 1, 2 and 3 support.
• Ever growing documentation.

Download 0.4.

Also, I'd love to hear your feature request. We're closing in on being feature complete for standard WebDAV and the system is pretty stable in a live environment (has been tested for quite some time..). I'm not completely fixed yet on how the API works, so I'd love to hear some suggestions on what you'd like to see, or what you don't like at all.

So my good friend siep bought Sony headphones a year or two ago. After a while, they broke right at the left earpiece. Kinda sucked.. He didn't have any receipts, so there wasn't a lot that could be done.

They are pretty high quality headphones. Well, at least in terms of audio quality. So he tried to repair them with some screws and some metal:

But they broke again, this time on the other side.

He actually had given the set to me, because I just gave him a new pair for christmas, the MDR-V500. They also broke:

And then, I got another pair of Sony headphones for my birthday in April. This time the MDR-7505. Hoping they fixed some of the issues with a new model, I had full confidence. On top of that I was extra careful this time and made sure they were around my neck when I transported them. It was no use:

We went to the store when the second pair broke, and also now with the third pair. They won't do anything however without the receipts while this is such an obvious faulty product. On top of that I have two other friends with the same stories about the headphones. $400 worth of headphones down the drain.

I called Sony today and they tell me I should go back to the store to try to get a copy of the receipt. I'm going to try to get that, but not really happy with that solution either. It might give me 1 set back at most, while I really want my money back for all 3 of them.

A list apart was probably one of the first feeds I added to my RSS reader, and up until this day I enjoy reading the high quality articles.

Just last Tuesday they published an article around progressive enhancement with css, which covered a rather funny example from Andy Clark's site, which is completely black and white in IE6, along with modified low-fi images.

I couldn't resist doing this myself as well, so if you hit this site using IE6 now this is how it will look like:

.

The code:

<!--[if lte IE 6]>
  <style type="text/css"> body { filter: gray(); } </style>    
<![endif]-->

This has been covered on other sites even when FP10 was still beta, but sadly I've just ran into this one right now.

Flash Player 10 no longer allows uploads to be invoked from within javascript, and requires an actual mouseclick or buttonpress event from within flash. For the people using tools like swfupload or the (old version of-) YUI uploader, this is bad news, as these tools will simply break.

The fix was made as a security measure against phishing-type attacks, and this has been the behaviour in browsers for as long as I can remember, so if its any consolidation.. it's a good thing really.

The Flash Blog mentions:

Similarly, another possibility is to overlay a transparent SWF button over the HTML content so that again, the user clicking happens in Flash and not in HTML.

Which is actually kind of funny, in a time where 'Clickjacking' is big news, and I would also strongly discourage such attempts to make sure you have a solution that still works in 2009. I'd like to see an example of this, just to test if noscript's clickjacking protection works.

The only real options out there are to simply switch back to good old html uploaders, or do your uploader interface entirely in flash. We'll probably be implementing the former in the short term to work on the latter down the road. The updated version of the YUI uploader also takes the flash approach.

SolarPHP's framework now comes with SQL adapter for MySQL replicated setups since the 1.0.0alpha2 release. The adapter automatically picks master or slave databases depending on the type of query. If its a SELECT, it will go to a slave, if its a UPDATE, DELETE, etc. it will automatically go to the master.

Let me just put it out there, I think that's a really bad idea. While this will work for certain situations, it is advertised as "The Solar_Sql_Adapter_MysqlReplicated adapter handles all the connections and switching-around for you, and you don't need change a single line of application code.".

Quite frankly, I started out with a similar approach. It worked fine for our staging environment, which didn't experience a lot of load, when it went to the live machine things went bad.

Suddenly I realized there were more than couple places in the application that had to do SELECT based on information that was just INSERT'ed. After the INSERT was done, the SELECT was performed on the slave, while the data hadn't arrived yet from the master. This caused some pretty freaky errors. Hoping to fix the problem, I decided to always execute any query on the master after an data-altering statement was done. Turned out that the master got quite a bit more load than the slaves after that change.

Things got worse when there was a replication lag of say, a second. Users would modify some data by submitting a form, and when they ended up on a resulting page the data was gone!

Some of the things we looked at was using a whitelist or blacklist for queries that are always ok/not ok to go to the slave, but considering we then ended up with decoupling a logical block of code, we went for the pragmatic approach. Always pick the master or slave depending on what is required for a specific piece of functionality. Yes, it might require a bit more thinking as you're working on your database code, but the behaviour is predictable and reliable.

Just posted a new SabreAMF version to googlecode, containing mostly bugfixes and a few enhancements. (download). Updating is highly recommended, this release was supposed to get out a bit earlier, but I wanted to test it in a live environment for a bit first.

Changes:

• We're now throwing an SabreAMF_InvalidAMFException instead of a generic Exception in the event on a corrupt/incorrect AMF request. (contribution by: Asbjørn Sloth Tønnesen.)
• Fixed a bug in the standard Recordset object (identified by 'datafirm').
• Fixed string reference problem occuring with multiple AMF3 bodies, identified by sylvinus and fixed with help from the PyAMF team. Thanks guys!
• Fixed a bug in AMF3 integer encoding. For some small ranges you would end up with the wrong integers in PHP. The integer decoder is also a lot faster now (Fixed by Kevin Martin).
• SabreAMF_Server now allows alternative inputs than php://input (patch by Asbjørn).

Thanks guys!

MySQL allows comments to be added with the standard /* */ syntax. If your application is getting more complex, it can become more difficult to figure out where certain queries originate from.

By simply adding a comment in front of your query, it will be very easy to spot the origin of some of your queries.

/* recent blogposts */ SELECT id, title, time FROM blogposts ORDER BY time DESC limit 10

The comment will show up as the first thing in SHOW [FULL] PROCESSLIST, MySQL administrator and your log/slow-log files.

In trying to get more out of our webservers using a Lighttpd and PHP-FastCGI setup, I've come across some major issues that make it difficult to use. I hope this post will warn people of some of the bugs they might encounter and workaround that might need to be implemented until some of these are fixed.

First off, the parent PHP-CGI process spawns n number of children, depending on your PHP_FCGI_CHILDREN. However, if your webserver (lighttpd) is stopped, or restarted, the parent process does not kill its children and they all get orphaned.

The only way to get around this easily is by making sure that as soon as you need to stop or restart your webserver, you do a 'killall php-cgi' while the server is still down. There's a PHP bugreport open, which seems to indicate the issue also happens in Apache. Vote for it!

The second, more severe issue is that when you hit maximum capacity for your PHP backend, lighty will start serving HTTP 500 errors for all PHP requests, and does not seem to stop doing this until the webserver is restarted altogether. Although not completely sure, these bugs seem to be the relevant ones:

• PHP bugs #41229 and #43610.
• Lighttpd bugs #784, and #897.

So yea, based on this information it turns out that there's a clear need for some smart process killing/webserver restarting scripts if you'd like to switch to lighttpd in a high load environment. I got pretty scared trying this after finding these bugs. Makes me think no one really tried it out under heavy loads, which leads me to some questions.. Hopefully some readers of this feed have some experience here.

1. Are you using lighttpd or an other alternative 'light' webserver using PHP under high load environments? Have you experienced similar issues?
2. What are good ways around PHP's FCGI buggy behaviour (the buggy part is that PHP's parent process should return FCGI_OVERLOADED instead of timing out.) Should FCGI be avoided altogether at this point?
3. What is a good way to come up with settings for 'max-procs' and 'PHP_FCGI_CHILDREN'. Reading other people's comment on this on the web people are all across the board, ranging from 1 for max-procs and 200 for PHP_FCGI_CHILDREN, to the exact opposite. Supposedly APC is isolated to 1 group of processes, so getting at least bigger groups of processes is important.
4. And most importantly, whats your moms favourite color?