So what are they and why are they so cool, and what/who the hell is “Mongo”?

Let’s deal with the what first. “Traditional” database systems are Relational, they are characterised by strictly defined tables, strong relations between tables and their ACID compliance. It simple terms that means you say:

“I want a table of users and each one will have a first name, a last name and an email address, and each of those will be a ‘text’ entry of a maximum of 255 characters”.

If at a later date you want to add more info, then you have to go back and alter your table definition, give the old rows default values and fix any coding bugs that relied on the previous structure (not that you’d ever develop such a dependant application). Now the approach in a document oriented database is that you say:

“I want a table of users”

Then you can give each one whatever data you want. you may have one user with a name and address, another with a name and phone number only, a third might have a phone number and address but no first name. All these records can co-exist! That’s pretty different and I really didn’t know whether it was a good idea. So I played with MongoDB.

MongoDB is really cool. I installed it my laptop simply and fired up the shell to play. The syntax is very javascript-like, becasue it is javascript! So we can do this.

Notice two amazing things here.

1. We never created the table! we simply call db.myFirstMongoDB… which creates the “collection” (as they are called in the document oriented world). We certainly never defined a schema!
2. We passed an object with nested data. Not just simple fields! We can use this in a much more useful way in the next example

So why might you want to do this. Consider a website with articles. Articles are written by a person, they are published on a date and the have content, tags, comments, view counts, trackbacks, and a proprietary rating system. If this where a relational database, we are looking at the following tables:

Joining all that info together is not only complex, but can be a fair amount of load on the system. As more info wants to be added by the website owners changes become more complicated and difficult. However with Mongo I can hold the article in one collection like this:

[

So that might look like a more complex entity. It is, but you can search for values in sub-documents, you can filter results on existence of certain keys, and you can pull all this info one result. Your previous 7 tables and 8 joins, just became 1 collection and no joins. guess which is quicker?

Try to pull articles with a certain tag: “balls”? In SQL you’d be doing this:

And that would just get you the list of Id’s and you’d still need to get the info for each article. In Mongo with the collection we described above, you’d do:

How much easier is that, and we’ve now got all the info, not just the “id”s! I was just amazed at how easy that is!

Of course this simplicity comes with a price, you lose the relational elements making tight relationships harder to work with. Also you some ease in aggregating things like “tags”, but MapReduce support makes this very efficient anyway.

In my next blog I’ll talk about Lithium, Mongo and how fast you can build a full blown web app with these tools.

Universal Plug and Play is used for 2 mains things as far as I can tell.

1. Dynamically opening holes in firewalls/NAT configurations for inbound connections to services. (bad!
2. Media Discover/playback/control on a local network. (good!)

The first is bad in my opinion. I don’t want the ability for some software behind my firewall to allow connections into my network. It might make some software function a little more smoothly, but if it’s that important that connections can be made inbound I would have set up a port-forward myself.

So I am only concerned with the second scenario.

The situation was that the Xbox360 can Stream Music/Pictures and Video from a Windows based PC. Great, except if you haven’t got a windows based PC, or you have all your media on a fileserver running Ubuntu Server and don’t want to have to have another PC on just to stream to your Xbox360. So I did some digging and it turns out that the Xbox360 uses a “standardised” UPnP discovery protocol. I say “standardised” because it’s not quite compliant, but close enough.

I look at the options available and try uShare first. It looks good, but the Xbox won’t see it and it can’t read the ID3 info from my music - rendering the whole thing useless.

Next TwonkyMediaServer. I read somewhere that the linux demo doesn’t expire. I hope not, because it worked beautifully pretty much out of the box. The only gripe is that there’s no nice init scripts and I have to run a shell script every time I want to start/stop/status/restart it. But it read my media and the Xbox360 recognised it and I haven’t looked back.

So I thought, Wow, this UPnP thing is pretty cool. but it must be more than this?. So I booted up a Windows7 PC. I had had trouble with this machine because Windows Media Player didn’t like the fact that all my music was on a Read-Only SMB share. That and I hate the way the Windows won’t let you connect to different shares on the host with different credentials! How rubbish!

Still WMP12 didn’t like my read-only filesystem. However, with Twonky running on the network, all the music just “appeared” like magic in the “Other Libraries” section. nice.

Then I thought of another thing that had been annoying me. Why can’t I stream media from my fileserver to my Android phone over Wifi?

Well, guess what? There’s an App for that!

Andromote can connect to UPnP media servers and play back the content. Also, and this is quite cool, I can specifiy the media server as Twonky, specify the Media Renderer (i.e. what plays it) as my Windows7 PC, and use the Andromote App as the Control Point - allow full media browsing, playback, skipping, seeking and volume control of the PC from my phone. Pretty neat, eh?

The whole time, all the servers and renderers are auto-discovered, no configuration necessary. That’s pretty cool.

So in summary UPnP media is in 3 distinct parts and the best bit is that each part can be on a separate physical machine, or 2 on 1, or 1 each on 3 separate physical devices. Very flexible. The parts:

Media Server — Serves the media - duh. i.e. media is physically stored (or accessed from) here.

Media Renderer — Plays back the media - simples.

Media Control Point — Controls the Media. Reads media info and tells the renderer what to play.

So the most common configuration is the server on one machine and the renderer and control point in another. That’s how the Xbox360 works.

So that was my experience into UPNP. I realise now that it was slightly unecessary as my old Xbox, running XBMC, would have (according to their wiki) worked as the UPNP media server out of the box and is conencted to all my shares, and isn’t a hassle to turn on as it sits right next to my Xbox360. It’ll be my fallback if Twonky does expire…

As it turns out, I was beaten to the post by Duncan MacKensie (http://duncanmackenzie.net/Blog/put-up-a-rest-api-for-xbox-gamertag-data) who hosts a webservice to retrieve gamer data from Microsoft. I could find no details about how this service works, where the data comes from or anything! Either he has a relationship with Microsoft, or he scrapes xbox.com but either way, the data seems pretty consistent and reliable. Actually it turns out this information was right there on his website… http://www.duncanmackenzie.net/Blog/if-you-are-wondering-where-i-get-my-xbox-live-info So he gets it as part of his membership to the Xbox Community Developer Program.

However, the webservice is great, and returns XML which is fine, but I thought it would be more useful to me to have a PHP API for this data. So I wrote one which retrieves data from Duncans webservice.

Then I thought wouldn’t it be great to be able to use this dynamically in a webpage, so I wrote a service frontend which will return JSON formatted data. Then I thought wouldn’t it be useful to let other people use this as well, so I modified it and it can now cope with JSONP requests with a “_callback” parameter.

OK, so what does this all mean.

The PHP

The class is called gamertag and the usage is very simple:

which outputs something like this:

Array
(
    [Gamertag] => thechriswalker
    [AccountStatus] => Silver
    [State] => Valid
    [ProfileUrl] => http://live.xbox.com/member/thechriswalker
    [TileUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatarpic-l.png
    [AvatarFullUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatar-body.png
    [Country] => United Kingdom
    [Location] => Bradninch
    [Bio] =>
    [Reputation] => 58.72229
    [ReputationImageUrl] => http://live.xbox.com/xweb/lib/images/gc_repstars_external_12.gif
    [Zone] => Recreation
    [GamerScore] => 230
    [PresenceInfo] => Array
        (
            [Valid] => true
            [Info] => Last seen 12/29/09   playing Modern Warfare® 2
            [Info2] =>
            [LastSeen] => Tue, 29 Dec 2009 21:35:22 +0000
            [Online] => false
            [StatusText] => Offline
            [Title] => Modern Warfare® 2
        )

    [RecentGames] => Array
        (
            [0] => Array
                (
                    [Name] => Modern Warfare® 2
                    [TotalAchievements] => 50
                    [TotalGamerScore] => 1000
                    [Image32Url] => http://tiles.xbox.com/tiles/Z+/tF/12dsb2JgbA9ECgQJGgYfVl5UL2ljb24vMC84MDAwIAABAAAAAPhq63g=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/CE/Vx/0Gdsb2JhbC9ECgQJGgYfVl5UL2ljb24vMC84MDAwAAAAAAAAAP9eRRc=.jpg
                    [LastPlayed] => Tue, 29 Dec 2009 21:32:52 +0000
                    [Achievements] => 9
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%60m%2fl%3b%7cw&compareTo=thechriswalker
                )

            [1] => Array
                (
                    [Name] => PGR 4
                    [TotalAchievements] => 60
                    [TotalGamerScore] => 1250
                    [Image32Url] => http://tiles.xbox.com/tiles/Y1/qn/0Gdsb2JgbA9ECgR8GgMfWSlaL2ljb24vMC84MDAwIAABAAAAAP+IWnw=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/DP/ST/12dsb2JhbC9ECgR8GgMfWSlaL2ljb24vMC84MDAwAAAAAAAAAPi89BM=.jpg
                    [LastPlayed] => Mon, 28 Dec 2009 16:59:25 +0000
                    [Achievements] => 5
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%15%18*iAq%0b&compareTo=thechriswalker
                )

        )

)

So now we can easily get at the data. The source code for the class (which is not fully tested, but the basics work!) is at http://thechriswalker.net/xbox360/gamertag.source.php (NB it requires either PHP5 (for json_encode) or the PEAR Services_JSON class if you want to use the “getJSON()” method).

The JSON

PHP is well and good but what if I want to use a JSON/JSONP (JSONP is for cross-domain information requesting and is very useful for public information services, see http://en.wikipedia.org/wiki/JSON#JSONP) request, well, that can be done at http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG for the straight JSON or http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG&_callback=YOUR_CALLBACK_FUNCTION_NAME for JSONP.

The first returns just JSON with a content type “application/json” and the second returns a javascript function call to your callback function with the JSON object as the only parameter and a content type of “text/javascript”.

These enabled me to build a simple Google Gadget to display a Gamercard:

So I thought, OpenSolaris. It’s the original implementation of the filesystem it must be great, so I export my zpools and boot from the LiveCD (another great reason to try it).

However it wasn’t going to be that easy… When I try to import my zpools in Solaris I get errors stating the vdev’s are corrupt! Something like:

status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.

Scared I boot back to linux to find they are all fine.

Back to Solaris and they still look corrupt. So I search and find this on the ZFS-FUSE Google Group.

Sun claims best practice is to use an entire disk when assigning disks to vdevs, rather than just partitions. So that’s what I did in linux. However the Linux implementation differs form the Solaris one and whole disks added in Solaris get a single partition first and that is added to the vdev.

Great! Now I have these zpools I can’t use anywhere else. To test I used a pool I set up as a mirror and detached a vdev (one disk). I then repartitioned the disk to have one big partition and also wiped the ZFS pool info from it (easier said than done! Ghost pools seem to be a problem!). Then I re-inserted the new partition into the the zpool and waited for resilver. Then I did the same with the other drive, detach from pool, partition, attach partition and resilver.

Bingo. It works in Solaris! Now I just have to find a spare 750Gb disk so I can do the same for my JBOD style zpool…

This was also my first major experience with Linux as a whole, so I was trying to keep things simple. So I did a default type install, with LVM on the 2×200GB disks to give one logical 400GB partition.

All was fine and dandy until I wanted to upgrade a couple of years later and not knowing much about LVM, wiped all my data. Yes, ALL my data. I had some important stuff backed up, but a lot of non-critical but frustrating to lose data (many GBs of painstakingly ripped and encoded CDs and Audiobooks for example).

I was gutted, and vowed not to use LVM again (note my solution to my mistake and lack of knowledge - never use the software again! Well thought out!). So I rebuilt the server using CentOS 4.x ( I can’t remember…) and added another 250GB disk. This time I kept each disk seperate with it’s own filesystem.

This worked fine for ages until the disks were nearly full and I spent all my time moving data from one disk to another to gain a few GBs extra on another disk. Nightmare.

So I decided it was time to start again and did some research into the available filesystems people use for fileservers, how software RAID can be implemented and so on. In this time I came across ZFS and I couldn’t believe what it could do. It seemed to good to be true. I now know this to be false and it can’t do everything, but it can do an awful lot!

So how can I use ZFS? Well, if you use any Solaris variant, then you probably already are. Otherwise chances are slim that you’ve encountered it. I didn’t really want to go down the OpenSolaris route, but instead stick to Linux where I’m more comfortable. This poses issues, as the licencing behind ZFS from Sun doesn’t fit with GPL, so no kernel support for ZFS. Instead someone (I’d love to reference but I forget his name, google will surely help - http://google.com/search?q=ZFS+FUSE) has ported ZFS to FUSE which is great except for the massive memory usage and the poor performance.

Those actually sound like quite big issues! Well, the memory thing is not so bad as it seems. ZFS is not memory instense (well, actually it constantly uses about 256Mb so it’s not light) but uses available memory cache to store data in case it needs it again. A good idea, as the memory is otherwise just doing nothing! The Performance is pretty poor though. It’s enough for my network and to stream music, pictures and video over my network. But single file transfer never hits much more than 8Mb/s read or write.

Now I’m interested in trying some of the other OS options, as ZFS is easy to “export” from one OS and “import” into another. What are the options?

• Option 1: OpenSolaris – Seems like a good idea. The native platform, has a good AMP stack so should be fine for my needs. Have heard some stability issues, but not so much with ZFS, but the OS itself.
• Option 2: Nextenta – Solaris Kernel with GNU userland. Supposed to be a good happy medium, sounds too much like a compromise to me…
• Option 3: BTRFS – different route totally but a new filesystem designed not to mirror ZFS, but to take the best of everything it can. Some aspects very like ZFS. Very promising project, but too young.

Kinda only leaves OpenSolaris… I’ll let you know how I get on.

And I got really excited and really angry.

I got angry, because most people weren’t excited for the right reasons, or were dismissive / “anti” because they didn’t really see what Google has done.

But because I am so excited about Wave (not just Google Wave), I thought I’d post this to help you understand what Google Wave actually is and what it isn’t.

What Google Wave isn’t

Google Wave isn’t Wave.

Wait a second! Why have you dropped the “Google”, what’s “Wave” all about?

Firstly let’s take an analogy, with something you are no doubt familiar with. Consider the statement:

Google Mail is not email.

Imagine that email didn’t exist and Google showcased Google Mail. People might think that email was Google Mail, but we all know that isn’t the case.

The same for Wave. Google have designed a messaging system which can be run by anyone. They have designed the protocols it will use which anyone can implement. This messaging system is not Google Wave, but Wave itself.

What Google Wave is

Google have designed a UI for Wave, which is Google Wave.

Don’t get me wrong, I like their interface. It is written in HTML5 using the Google Web Toolkit and it is super slick.

However, the Wave itself is far more exciting.

What Wave is

Wave is email redesigned for the 21st century. Email was born before the internet and has been flogged and flogged into how we use it now. It is inefficient, single one-way message based, bad with files, incosistent with rich styling, non-collaborative and basically just way past it’s “sell by date”.

With Wave we no longer deal in single messages, but conversations; where content evolves over time and can be “played back” to provide the user with context. Everything email and IM can do and more. They have designed the system to be extensible and so can communicate with other systems. They also designed it to work over a federated networks model, so many people running their own Wave servers can communicate in the same rich way as if they were communicating with their people on their own server.

Amazing really, and provided we can bridge the backwards compatibility gap^* (which we can, remember Wave is extensible!) then I see Wave replacing email altogether.

When Google Wave goes live, Google will also be releasing the server for anyone to run, learn, enjoy and stop using email with. I am looking forward it immensely.

[update 2009-08-06] Further developments have shown that yes “gateways” have been designed in, so Wave can replace email and any other messaging/collaboration protocols/tools if someone creates a “gateway” for it.

Doesn’t sound like much, but if you are like me, you’ll not only want to finish all the levels but also go back and experiment with loads of other different ways to finish in the vain hope of finding the perfect solution. A solution that excels by being so simple, or so elegant. So unusual, or so complex. I could play for hours. But I won’t go on for hours.

I will leave you with some of the best contraptions I have managed to come up with, and the first one you should really vote as amazing, because (and I know this sounds a bit arrogant) is ingenious!

• unusual technique! See if you can guess how it works before you start it!
• Minimalistic elegance, perfect timing.
• I saw a solution with more than one spinny thing, but this seemed a cooler solution (notice the pole to slow down the last ball slightly…)
• A cool way to do “The Ball”

That’s all for the moment, but I’m sure I’ll post more contraptions in the future…

Tranforms have been implemented on a few browser using proprietary syntax, and the same is true of WebKit. As we shall see further down the article though, the WebKit transitions make them so much more powerful.

But what are they? They modify an element, without displacing it’s original setting within the document, which means we can move it about and the spacing doesn’t change.

Remember these will only work in WebKit based browsers, for example Safari for Mac, Chrome for Windows or Midori for Linux

An example: how about rotation.

element:hover {
  -webkit-transform:rotate(180deg);
}

Not very exciting, but combine this with a transition! (Try putting you mouse in the middle of the box for best effect)

element:hover {
  -webkit-transform:rotate(180deg);
}
element {
  -webkit-transition:-webkit-transform 1s linear
}

Much more useful! But there’s more, we can scale, we can translate (change its position on the screen) and also we can change the “center” for the transform. Usually the rotation happens from the center like above, but we could do it from closer to the left:

element:hover {
  -webkit-transform:rotate(180deg);
  -webkit-transform-origin:30% 50%;
}
element {
  -webkit-transition:-webkit-transform 1s linear;
  -webkit-transform-origin:30% 50%;
}

Or we could scale and move and rotate! (you might need to track it with you mouse to keep the animation! this is a demo after all!)

element:hover {
  -webkit-transform:rotate(180deg) scale(2) translate(-100%,0);
  -webkit-transform-origin:30% 50%;
}
element {
  -webkit-transition:-webkit-transform 1s linear;
  -webkit-transform-origin:30% 50%;
}

<div id='hey-look-at-me'> <div id='untarget'> <pre>
CSS:
#hey-look-at-me:target pre {
  outline:2px solid red;
  color:#f0f;
}
#hey-look-at-me pre {
  -webkit-transition-property:outline, color;
  -webkit-transition-duration:500ms;
}
</pre> </div> </div>

target me! untarget me!

NB: Notice how I nested the PRE to stop unwanted excessive screen movement.

So, what can we do with all this? Well, the culmination of the post, the Mac OSX style Dock menu, using no Javascript. Yes, that’s right a bulging docked menu, with no javascript.

Just so you remember, there no javascript in the demo.

Check out the Javascript free OSX Dock Menu Demo.

But what is it? Simply put, it’s a way to create a transition between to CSS states (hope that cleared everything up for you!)

Better, an example: You have two CSS rules for an element, say a link, and for it’s :hover state. like this:

Now usually, the browser would switch instantly between these two states, but with transitions we can slow it down into a smooth animation.

We can specify which propertys to transition and which to allow to change instantly. Here we can transition in the background and color properties.

Here’s two links, the they are identical except that the second has the transisions applied and if you’re using a WebKit based browser then it will fade nicely between states.

I will flick between states. Do I fade or do I flick?

Next time I’ll go into a bit more detail and show how using CSS Transforms as well we can make an OSX Dock style menu using CSS alone!

Why bother though? Well, they are used mostly in Japan, where mobile phones are generally all Internet-enabled, so on an advert you might have a QR code for a URL where more info can be found. So you whip out your phone, point the in-built camera at the code, and bingo, it takes you to the website. Also you could have a film opening encoded in QR as a calendar event, so when you scan it it puts a reminder in your calendar. Useful no?

On a personal level though, when QR codes are not big in England, why should I care. Well, they are useful for transferring information I want to a phone. For example, I have a bookmarklet on my browser that converts the current page I’m looking at into a QR code (courtesy of the Google Chart API) and open the image in a new tab. Very useful if I find a page and think, “I’ll want to look at that later” I can make the QR scan it with my phone, and the page goes into it’s browser history for my later enjoyment.

For those of you interested the bookmarklet code is (should be all on one line, but for the page width’s sake…):

Try it! (will generate the QR code for this page, but a lot bigger than the one by the title)

You may have noticed that there are QR codes for the permalinks to these posts aswell, so you can easily bookmark them with a capable phone. Another useful function that my phone can do is to display the QR code for a Contact in my address book, so I can easily share that with another QR enabled phone without having to “pair” them with bluetooth, or use Infrared (if the phone still has it!) or text it or whatever. Much more convenient. I just need to find someone else with a QR enabled phone…