theChrisWalker.net

I have been following a PHP Rapid Application Development Framework called Lithium with much interest (for other reasons which I fully intend to blog about later) and they are the ones I owe for turning me on to document oriented database systems.

So what are they and why are they so cool, and what/who the hell is “Mongo”?

Let’s deal with the what first. “Traditional” database systems are Relational, they are characterised by strictly defined tables, strong relations between tables and their ACID compliance. It simple terms that means you say:

“I want a table of users and each one will have a first name, a last name and an email address, and each of those will be a ‘text’ entry of a maximum of 255 characters”.

If at a later date you want to add more info, then you have to go back and alter your table definition, give the old rows default values and fix any coding bugs that relied on the previous structure (not that you’d ever develop such a dependant application). Now the approach in a document oriented database is that you say:

“I want a table of users”

Then you can give each one whatever data you want. you may have one user with a name and address, another with a name and phone number only, a third might have a phone number and address but no first name. All these records can co-exist! That’s pretty different and I really didn’t know whether it was a good idea. So I played with MongoDB.

MongoDB is really cool. I installed it my laptop simply and fired up the shell to play. The syntax is very javascript-like, becasue it is javascript! So we can do this.

Notice two amazing things here.

1. We never created the table! we simply call db.myFirstMongoDB… which creates the “collection” (as they are called in the document oriented world). We certainly never defined a schema!
2. We passed an object with nested data. Not just simple fields! We can use this in a much more useful way in the next example

So why might you want to do this. Consider a website with articles. Articles are written by a person, they are published on a date and the have content, tags, comments, view counts, trackbacks, and a proprietary rating system. If this where a relational database, we are looking at the following tables:

Joining all that info together is not only complex, but can be a fair amount of load on the system. As more info wants to be added by the website owners changes become more complicated and difficult. However with Mongo I can hold the article in one collection like this:

[

So that might look like a more complex entity. It is, but you can search for values in sub-documents, you can filter results on existence of certain keys, and you can pull all this info one result. Your previous 7 tables and 8 joins, just became 1 collection and no joins. guess which is quicker?

Try to pull articles with a certain tag: “balls”? In SQL you’d be doing this:

And that would just get you the list of Id’s and you’d still need to get the info for each article. In Mongo with the collection we described above, you’d do:

How much easier is that, and we’ve now got all the info, not just the “id”s! I was just amazed at how easy that is!

Of course this simplicity comes with a price, you lose the relational elements making tight relationships harder to work with. Also you some ease in aggregating things like “tags”, but MapReduce support makes this very efficient anyway.

In my next blog I’ll talk about Lithium, Mongo and how fast you can build a full blown web app with these tools.

You may notice an Xbox360 related theme in my current posts, as after getting one, most of my technology thought process is involved in something to do with it. Hence the Xbox Live Gamercard API I wrote about before. Now we get on to UPnP.

Universal Plug and Play is used for 2 mains things as far as I can tell.

1. Dynamically opening holes in firewalls/NAT configurations for inbound connections to services. (bad!
2. Media Discover/playback/control on a local network. (good!)

The first is bad in my opinion. I don’t want the ability for some software behind my firewall to allow connections into my network. It might make some software function a little more smoothly, but if it’s that important that connections can be made inbound I would have set up a port-forward myself.

So I am only concerned with the second scenario.

The situation was that the Xbox360 can Stream Music/Pictures and Video from a Windows based PC. Great, except if you haven’t got a windows based PC, or you have all your media on a fileserver running Ubuntu Server and don’t want to have to have another PC on just to stream to your Xbox360. So I did some digging and it turns out that the Xbox360 uses a “standardised” UPnP discovery protocol. I say “standardised” because it’s not quite compliant, but close enough.

I look at the options available and try uShare first. It looks good, but the Xbox won’t see it and it can’t read the ID3 info from my music - rendering the whole thing useless.

Next TwonkyMediaServer. I read somewhere that the linux demo doesn’t expire. I hope not, because it worked beautifully pretty much out of the box. The only gripe is that there’s no nice init scripts and I have to run a shell script every time I want to start/stop/status/restart it. But it read my media and the Xbox360 recognised it and I haven’t looked back.

So I thought, Wow, this UPnP thing is pretty cool. but it must be more than this?. So I booted up a Windows7 PC. I had had trouble with this machine because Windows Media Player didn’t like the fact that all my music was on a Read-Only SMB share. That and I hate the way the Windows won’t let you connect to different shares on the host with different credentials! How rubbish!

Still WMP12 didn’t like my read-only filesystem. However, with Twonky running on the network, all the music just “appeared” like magic in the “Other Libraries” section. nice.

Then I thought of another thing that had been annoying me. Why can’t I stream media from my fileserver to my Android phone over Wifi?

Well, guess what? There’s an App for that!

Andromote can connect to UPnP media servers and play back the content. Also, and this is quite cool, I can specifiy the media server as Twonky, specify the Media Renderer (i.e. what plays it) as my Windows7 PC, and use the Andromote App as the Control Point - allow full media browsing, playback, skipping, seeking and volume control of the PC from my phone. Pretty neat, eh?

The whole time, all the servers and renderers are auto-discovered, no configuration necessary. That’s pretty cool.

So in summary UPnP media is in 3 distinct parts and the best bit is that each part can be on a separate physical machine, or 2 on 1, or 1 each on 3 separate physical devices. Very flexible. The parts:

Media Server — Serves the media - duh. i.e. media is physically stored (or accessed from) here.

Media Renderer — Plays back the media - simples.

Media Control Point — Controls the Media. Reads media info and tells the renderer what to play.

So the most common configuration is the server on one machine and the renderer and control point in another. That’s how the Xbox360 works.

So that was my experience into UPNP. I realise now that it was slightly unecessary as my old Xbox, running XBMC, would have (according to their wiki) worked as the UPNP media server out of the box and is conencted to all my shares, and isn’t a hassle to turn on as it sits right next to my Xbox360. It’ll be my fallback if Twonky does expire…

So, got decided to join most of my friends and I got an Xbox 360. Me being me though, I got interested in the way that all the information about your “Gamertag” is stored an accessible on the xbox.com website. Wouldn’t it be fun to do something with this data!

As it turns out, I was beaten to the post by Duncan MacKensie (http://duncanmackenzie.net/Blog/put-up-a-rest-api-for-xbox-gamertag-data) who hosts a webservice to retrieve gamer data from Microsoft. I could find no details about how this service works, where the data comes from or anything! Either he has a relationship with Microsoft, or he scrapes xbox.com but either way, the data seems pretty consistent and reliable. Actually it turns out this information was right there on his website… http://www.duncanmackenzie.net/Blog/if-you-are-wondering-where-i-get-my-xbox-live-info So he gets it as part of his membership to the Xbox Community Developer Program.

However, the webservice is great, and returns XML which is fine, but I thought it would be more useful to me to have a PHP API for this data. So I wrote one which retrieves data from Duncans webservice.

Then I thought wouldn’t it be great to be able to use this dynamically in a webpage, so I wrote a service frontend which will return JSON formatted data. Then I thought wouldn’t it be useful to let other people use this as well, so I modified it and it can now cope with JSONP requests with a “_callback” parameter.

OK, so what does this all mean.

The PHP

The class is called gamertag and the usage is very simple:

which outputs something like this:

Array
(
    [Gamertag] => thechriswalker
    [AccountStatus] => Silver
    [State] => Valid
    [ProfileUrl] => http://live.xbox.com/member/thechriswalker
    [TileUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatarpic-l.png
    [AvatarFullUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatar-body.png
    [Country] => United Kingdom
    [Location] => Bradninch
    [Bio] =>
    [Reputation] => 58.72229
    [ReputationImageUrl] => http://live.xbox.com/xweb/lib/images/gc_repstars_external_12.gif
    [Zone] => Recreation
    [GamerScore] => 230
    [PresenceInfo] => Array
        (
            [Valid] => true
            [Info] => Last seen 12/29/09   playing Modern Warfare® 2
            [Info2] =>
            [LastSeen] => Tue, 29 Dec 2009 21:35:22 +0000
            [Online] => false
            [StatusText] => Offline
            [Title] => Modern Warfare® 2
        )

    [RecentGames] => Array
        (
            [0] => Array
                (
                    [Name] => Modern Warfare® 2
                    [TotalAchievements] => 50
                    [TotalGamerScore] => 1000
                    [Image32Url] => http://tiles.xbox.com/tiles/Z+/tF/12dsb2JgbA9ECgQJGgYfVl5UL2ljb24vMC84MDAwIAABAAAAAPhq63g=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/CE/Vx/0Gdsb2JhbC9ECgQJGgYfVl5UL2ljb24vMC84MDAwAAAAAAAAAP9eRRc=.jpg
                    [LastPlayed] => Tue, 29 Dec 2009 21:32:52 +0000
                    [Achievements] => 9
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%60m%2fl%3b%7cw&compareTo=thechriswalker
                )

            [1] => Array
                (
                    [Name] => PGR 4
                    [TotalAchievements] => 60
                    [TotalGamerScore] => 1250
                    [Image32Url] => http://tiles.xbox.com/tiles/Y1/qn/0Gdsb2JgbA9ECgR8GgMfWSlaL2ljb24vMC84MDAwIAABAAAAAP+IWnw=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/DP/ST/12dsb2JhbC9ECgR8GgMfWSlaL2ljb24vMC84MDAwAAAAAAAAAPi89BM=.jpg
                    [LastPlayed] => Mon, 28 Dec 2009 16:59:25 +0000
                    [Achievements] => 5
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%15%18*iAq%0b&compareTo=thechriswalker
                )

        )

)

So now we can easily get at the data. The source code for the class (which is not fully tested, but the basics work!) is at http://thechriswalker.net/xbox360/gamertag.source.php (NB it requires either PHP5 (for json_encode) or the PEAR Services_JSON class if you want to use the “getJSON()” method).

The JSON

PHP is well and good but what if I want to use a JSON/JSONP (JSONP is for cross-domain information requesting and is very useful for public information services, see http://en.wikipedia.org/wiki/JSON#JSONP) request, well, that can be done at http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG for the straight JSON or http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG&_callback=YOUR_CALLBACK_FUNCTION_NAME for JSONP.

The first returns just JSON with a content type “application/json” and the second returns a javascript function call to your callback function with the JSON object as the only parameter and a content type of “text/javascript”.

These enabled me to build a simple Google Gadget to display a Gamercard:

As my last post said, I’m using ZFS and love the way it works, but am having trouble with the performance of ZFS-FUSE. It’s not unusable, I just expected more.

So I thought, OpenSolaris. It’s the original implementation of the filesystem it must be great, so I export my zpools and boot from the LiveCD (another great reason to try it).

However it wasn’t going to be that easy… When I try to import my zpools in Solaris I get errors stating the vdev’s are corrupt! Something like:

status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.

Scared I boot back to linux to find they are all fine.

Back to Solaris and they still look corrupt. So I search and find this on the ZFS-FUSE Google Group.

Sun claims best practice is to use an entire disk when assigning disks to vdevs, rather than just partitions. So that’s what I did in linux. However the Linux implementation differs form the Solaris one and whole disks added in Solaris get a single partition first and that is added to the vdev.

Great! Now I have these zpools I can’t use anywhere else. To test I used a pool I set up as a mirror and detached a vdev (one disk). I then repartitioned the disk to have one big partition and also wiped the ZFS pool info from it (easier said than done! Ghost pools seem to be a problem!). Then I re-inserted the new partition into the the zpool and waited for resilver. Then I did the same with the other drive, detach from pool, partition, attach partition and resilver.

Bingo. It works in Solaris! Now I just have to find a spare 750Gb disk so I can do the same for my JBOD style zpool…

I have a fileserver at home. I have done for many years now. The first incarnation was based on Fedora Core 3 (which will tell you *exactly* how long ago that was…) and was very simple. I had an old Xbox HDD (8GB Seagate) and 2 shiny new 200GB Western Digiatl drives for storage.

This was also my first major experience with Linux as a whole, so I was trying to keep things simple. So I did a default type install, with LVM on the 2×200GB disks to give one logical 400GB partition.

All was fine and dandy until I wanted to upgrade a couple of years later and not knowing much about LVM, wiped all my data. Yes, ALL my data. I had some important stuff backed up, but a lot of non-critical but frustrating to lose data (many GBs of painstakingly ripped and encoded CDs and Audiobooks for example).

I was gutted, and vowed not to use LVM again (note my solution to my mistake and lack of knowledge - never use the software again! Well thought out!). So I rebuilt the server using CentOS 4.x ( I can’t remember…) and added another 250GB disk. This time I kept each disk seperate with it’s own filesystem.

This worked fine for ages until the disks were nearly full and I spent all my time moving data from one disk to another to gain a few GBs extra on another disk. Nightmare.

So I decided it was time to start again and did some research into the available filesystems people use for fileservers, how software RAID can be implemented and so on. In this time I came across ZFS and I couldn’t believe what it could do. It seemed to good to be true. I now know this to be false and it can’t do everything, but it can do an awful lot!

So how can I use ZFS? Well, if you use any Solaris variant, then you probably already are. Otherwise chances are slim that you’ve encountered it. I didn’t really want to go down the OpenSolaris route, but instead stick to Linux where I’m more comfortable. This poses issues, as the licencing behind ZFS from Sun doesn’t fit with GPL, so no kernel support for ZFS. Instead someone (I’d love to reference but I forget his name, google will surely help - http://google.com/search?q=ZFS+FUSE) has ported ZFS to FUSE which is great except for the massive memory usage and the poor performance.

Those actually sound like quite big issues! Well, the memory thing is not so bad as it seems. ZFS is not memory instense (well, actually it constantly uses about 256Mb so it’s not light) but uses available memory cache to store data in case it needs it again. A good idea, as the memory is otherwise just doing nothing! The Performance is pretty poor though. It’s enough for my network and to stream music, pictures and video over my network. But single file transfer never hits much more than 8Mb/s read or write.

Now I’m interested in trying some of the other OS options, as ZFS is easy to “export” from one OS and “import” into another. What are the options?

• Option 1: OpenSolaris – Seems like a good idea. The native platform, has a good AMP stack so should be fine for my needs. Have heard some stability issues, but not so much with ZFS, but the OS itself.
• Option 2: Nextenta – Solaris Kernel with GNU userland. Supposed to be a good happy medium, sounds too much like a compromise to me…
• Option 3: BTRFS – different route totally but a new filesystem designed not to mirror ZFS, but to take the best of everything it can. Some aspects very like ZFS. Very promising project, but too young.

Kinda only leaves OpenSolaris… I’ll let you know how I get on.