What are WebSockets?

WebSockets are a connection that the browser makes to a server. Unlike a normal HTTP request, this is not a simple “send request”, “receive response” connection. This is a persistent and full-duplex connection. They can run over port 80 (or 443 for a WebSocket over TLS/SSL) and so have little difficulty with firewalls. There are some concerns over their interaction with proxies, particularly transparent ones. Once the browser sets up the WebSocket, you can send UTF8 text over it, and receive text data from the server. All this sending and receiving over data requires no more headers and only a 2 byte overhead per message.

Why are they going to be so useful?

It’s really the last sentence of that last paragraph that is important. The problem is trying to get data from the server when it’s available. Web apps using AJAX (or similar techniques), have to ask for data and then get a response, i.e. polling for updates or using Comet-like long polling techniques – where the request is made and the connection opened but the response is delayed until the server decides it has something useful to say. The Comet technique allows much better real-time updates from the server, but there’s still the overhead of the HTTP request headers and then the response headers/content.

WebSockets aim to alleviate both issues. Once the socket is open, data can flow from the server to the client whenever the server decides it wants to. Also the client can send information to the server without have to send any extra headers, any info it receives from the server is also header free and therefore the process is very efficient. HTTP headers tend to bulk up with Cookies and remain the same size irrespective of the payload sent.

Do you have an example I can actually use?

I do, but it’s not exactly ready for distribution yet. I built a very basic chat client that uses WebSockets as a demo for myself. The hardest bit was writing a socket server in PHP that could handle multiple clients and manage the broadcasting of data to them. I will post all the code in part 2 — watch this space! (in the mean time a search for “websocket” on Google Code will find you some examples (it’s what I based my code on…)

And you don’t have to have root access!

OK, so how does it work. There’s two parts. One on the PC (or Mac…) and one on the phone. The phone bit is easy thanks to the man behind Proxoid a proxy App for Android. It’s free and available in the Marketplace. Also it’s a tiny download and has almost zero configuration! The PC bit requires the Android SDK, which is a free download from Google. Here are the instructions:

Phone Setup

1. Get “Proxoid” from the marketplace (market://search?q=proxoid from your phone or use the QR code)
2. Enable USB Debugging from Settings > Applications > Development > USB Debugging
3. Run Proxoid and connect your phone to your PC by USB.
4. (optional) Choose the port you want to use (default: 8080)

PC Setup

These steps are pretty generic, as they will be different for different operating systems. I’ll give full instructions for Linux Mint 8 (the distor I am using) which should work for most Ubuntu/Debian type systems.

1. Download Android SDK from http://developer.android.com/sdk/index.html
2. Extract it somewhere.
3. Start the ADB server (more code below)
4. Start the port forwarding.
5. Change your network proxy

So on Mint that goes along the lines of:

That will bring up the network proxy setup. I have a “location” set for the proxy which is a SOCKS Proxy at “localhost” on port 8080 (or whichever yort u choose).

Open you’re browser and bingo Internet over 3G – try http://whatismyip.com to check you’re not on your usual IP address!

As the SOCKS proxy pretty much only works for HTTP and web based traffic you might want to use FoxyProxy for Firefox which allows you to change proxy settings quickly and easily. Definitely a good idea for windows users, not sure how easy it is on a Mac. Also I wrapped the ADB commands into a single script that I can run to start the tethering.

There you go. Tethered 3G internet access for Android phones!

So far not free and non-jailbroken way to do this on an iPhone (please someone prove me wrong!)…

UPDATE: 15th May 2010 I have since found that Proxoid doesn’t always work for me. However the same effect can be achieved (and more securely) using ConnectBot (or any other SSH client supporting Dynamic Port Forwarding) to setup a SOCKS proxy SSH tunnel.

I had been using a VPS to host this website (amongst other things) and it was getting old. It ran on CentOS 4 and used Plesk as a control panel. PHP was down at version 4.2.2 or something and MySQL still at version 4.x. I didn’t understand server administration as well as I do now and the whole thing was a bit of a mess. So I decided a fresh start, with a VPS on more powerful hardware and built cleanly.

So I went for a basic VPS from 34SP.com with no control panel and using Ubuntu 9.04 as the OS. They set up the server quickly and so I went about configuring it how I wanted. There were a couple of snags along the road and so I’ll tell you about them here too.

The Basics, disabling root.

First things first, the VPS came with the root account enabled and that as the only user on the system. Now I like to have my root account disabled and use sudo, so I created an admin user, granted sudo rights and disabled the root account. The command used are below.

Create user and set password:

Now in the sudoer’s file you’ll see a line for root which looks something like:

duplicate this line but with you’re username:

save and exit. Now log out of root and back in as admin, then disable the root account:

and we’re done!. Now on this system my home directory wasn’t created (probably because I didn’t give the right flags in the useradd command. So we need to make it.

LAMP Stack

34SP had installed Apache but not PHP or MySQL, both of which I wanted. In fact I wanted the following:

• Apache with mod_rewrite
• PHP5.3+ (5.3 is important to me) with APC, Xdebug and MongoDB
• MongoDB server
• MySQL5 server

I also needed to migrate my blog from it’s current location, to the new server. The Blog uses Wordpress 2.8.something and the latest version at time of writing is 2.9.2 (time to upgrade!). Wordpress doesn’t mention PHP5.3 support so this could be fun.

So, first hurdle, the default Ubuntu Repositories do not have packages for PHP5.3. This leaves 2 options, compile from source or find a repositiory which does have the packages. Knowing which I’d prefer and having already heard of dotdeb.org I added the dotdeb.org PHP5.3 repository to my sources.list and then I could install PHP5.3.2 and xdebug and apc right from apt!

After editing /etc/apt/sources.list:

But I also wanted MongoDB which is only available as a PECL module. So I need the php5-dev package. unfortunately, this gave me an error about libtool not being the correct version. and the only way to resolve it (that I could find) was to use libtool from the previous Ubuntu release.

Then it worked like a dream and I could finally use pecl

That was the PHP module, but what about the actual MongoDB server itself? Well, there’s a repository for that as well. Add it to your sources.list, update then you can apt-get install mongodb. no configuration necessary. The phpinfo() output shows the mongo extension installed correctly and the mongo shell functions to show the mongodb server is working fine.

OK, now time to get my blog up and running, so I create a new virtual host in /etc/apache2/sites-available and use the apache tool to enable it sudo a2ensite thechriswalker.net. I have all my virtual hosts in /var/www/vhost//{httpdocs,logs}, I change to the httpdocs directory and grab wordpress wget http://wordpress.org/latest.tar.gz, unzip and browse to it to see if it worked!

Well, of course it didn’t – I hadn’t done a couple of things. First I needed to restart Apache to enable the Virtual Host I had created. Then I realised that the DNS hadn’t propagated so I needed to set a static hosts entry on my local computer. This done I tried again – success!

So I dumped the database from the old VPS, copied (scp’d) my custom theme, plugins and some other non-wordpress content across to the new one. On the new one, I imported the wordpress database and tried to access it again. It took me through a 2 step “we need to update your database” (it was 2.8 not 2.9 remember) screen which worked great! That was easier than expected and I haven’t seen anything behave oddly under PHP5.3 – result.

More

Now I just need to setup the email side of things, I want to be able to send and recieve mail securely through this server, but thats a job for another day… Also I moved the email for this domain over to Google Apps, as I like GMail so much, so I may not even bother with the email part of this!

So what are they and why are they so cool, and what/who the hell is “Mongo”?

Let’s deal with the what first. “Traditional” database systems are Relational, they are characterised by strictly defined tables, strong relations between tables and their ACID compliance. It simple terms that means you say:

“I want a table of users and each one will have a first name, a last name and an email address, and each of those will be a ‘text’ entry of a maximum of 255 characters”.

If at a later date you want to add more info, then you have to go back and alter your table definition, give the old rows default values and fix any coding bugs that relied on the previous structure (not that you’d ever develop such a dependant application). Now the approach in a document oriented database is that you say:

“I want a table of users”

Then you can give each one whatever data you want. you may have one user with a name and address, another with a name and phone number only, a third might have a phone number and address but no first name. All these records can co-exist! That’s pretty different and I really didn’t know whether it was a good idea. So I played with MongoDB.

MongoDB is really cool. I installed it my laptop simply and fired up the shell to play. The syntax is very javascript-like, becasue it is javascript! So we can do this.

Notice two amazing things here.

1. We never created the table! we simply call db.myFirstMongoDB... which creates the “collection” (as they are called in the document oriented world). We certainly never defined a schema!
2. We passed an object with nested data. Not just simple fields! We can use this in a much more useful way in the next example

So why might you want to do this. Consider a website with articles. Articles are written by a person, they are published on a date and the have content, tags, comments, view counts, trackbacks, and a proprietary rating system. If this where a relational database, we are looking at the following tables:

Joining all that info together is not only complex, but can be a fair amount of load on the system. As more info wants to be added by the website owners changes become more complicated and difficult. However with Mongo I can hold the article in one collection like this:

[

So that might look like a more complex entity. It is, but you can search for values in sub-documents, you can filter results on existence of certain keys, and you can pull all this info one result. Your previous 7 tables and 8 joins, just became 1 collection and no joins. guess which is quicker?

Try to pull articles with a certain tag: “balls”? In SQL you’d be doing this:

And that would just get you the list of Id’s and you’d still need to get the info for each article. In Mongo with the collection we described above, you’d do:

How much easier is that, and we’ve now got all the info, not just the “id”s! I was just amazed at how easy that is!

Of course this simplicity comes with a price, you lose the relational elements making tight relationships harder to work with. Also you some ease in aggregating things like “tags”, but MapReduce support makes this very efficient anyway.

In my next blog I’ll talk about Lithium, Mongo and how fast you can build a full blown web app with these tools.

Universal Plug and Play is used for 2 mains things as far as I can tell.

1. Dynamically opening holes in firewalls/NAT configurations for inbound connections to services. (bad!
2. Media Discover/playback/control on a local network. (good!)

The first is bad in my opinion. I don’t want the ability for some software behind my firewall to allow connections into my network. It might make some software function a little more smoothly, but if it’s that important that connections can be made inbound I would have set up a port-forward myself.

So I am only concerned with the second scenario.

The situation was that the Xbox360 can Stream Music/Pictures and Video from a Windows based PC. Great, except if you haven’t got a windows based PC, or you have all your media on a fileserver running Ubuntu Server and don’t want to have to have another PC on just to stream to your Xbox360. So I did some digging and it turns out that the Xbox360 uses a “standardised” UPnP discovery protocol. I say “standardised” because it’s not quite compliant, but close enough.

I look at the options available and try uShare first. It looks good, but the Xbox won’t see it and it can’t read the ID3 info from my music – rendering the whole thing useless.

Next TwonkyMediaServer. I read somewhere that the linux demo doesn’t expire. I hope not, because it worked beautifully pretty much out of the box. The only gripe is that there’s no nice init scripts and I have to run a shell script every time I want to start/stop/status/restart it. But it read my media and the Xbox360 recognised it and I haven’t looked back.

So I thought, Wow, this UPnP thing is pretty cool. but it must be more than this?. So I booted up a Windows7 PC. I had had trouble with this machine because Windows Media Player didn’t like the fact that all my music was on a Read-Only SMB share. That and I hate the way the Windows won’t let you connect to different shares on the host with different credentials! How rubbish!

Still WMP12 didn’t like my read-only filesystem. However, with Twonky running on the network, all the music just “appeared” like magic in the “Other Libraries” section. nice.

Then I thought of another thing that had been annoying me. Why can’t I stream media from my fileserver to my Android phone over Wifi?

Well, guess what? There’s an App for that!

Andromote can connect to UPnP media servers and play back the content. Also, and this is quite cool, I can specifiy the media server as Twonky, specify the Media Renderer (i.e. what plays it) as my Windows7 PC, and use the Andromote App as the Control Point – allow full media browsing, playback, skipping, seeking and volume control of the PC from my phone. Pretty neat, eh?

The whole time, all the servers and renderers are auto-discovered, no configuration necessary. That’s pretty cool.

So in summary UPnP media is in 3 distinct parts and the best bit is that each part can be on a separate physical machine, or 2 on 1, or 1 each on 3 separate physical devices. Very flexible. The parts:

Media Server — Serves the media – duh. i.e. media is physically stored (or accessed from) here.

Media Renderer — Plays back the media – simples.

Media Control Point — Controls the Media. Reads media info and tells the renderer what to play.

So the most common configuration is the server on one machine and the renderer and control point in another. That’s how the Xbox360 works.

So that was my experience into UPNP. I realise now that it was slightly unecessary as my old Xbox, running XBMC, would have (according to their wiki) worked as the UPNP media server out of the box and is conencted to all my shares, and isn’t a hassle to turn on as it sits right next to my Xbox360. It’ll be my fallback if Twonky does expire…

As it turns out, I was beaten to the post by Duncan MacKensie (http://duncanmackenzie.net/Blog/put-up-a-rest-api-for-xbox-gamertag-data) who hosts a webservice to retrieve gamer data from Microsoft. I could find no details about how this service works, where the data comes from or anything! Either he has a relationship with Microsoft, or he scrapes xbox.com but either way, the data seems pretty consistent and reliable. Actually it turns out this information was right there on his website… http://www.duncanmackenzie.net/Blog/if-you-are-wondering-where-i-get-my-xbox-live-info So he gets it as part of his membership to the Xbox Community Developer Program.

However, the webservice is great, and returns XML which is fine, but I thought it would be more useful to me to have a PHP API for this data. So I wrote one which retrieves data from Duncans webservice.

Then I thought wouldn’t it be great to be able to use this dynamically in a webpage, so I wrote a service frontend which will return JSON formatted data. Then I thought wouldn’t it be useful to let other people use this as well, so I modified it and it can now cope with JSONP requests with a “_callback” parameter.

OK, so what does this all mean.

The PHP

The class is called gamertag and the usage is very simple:

which outputs something like this:

Array
(
    [Gamertag] => thechriswalker
    [AccountStatus] => Silver
    [State] => Valid
    [ProfileUrl] => http://live.xbox.com/member/thechriswalker
    [TileUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatarpic-l.png
    [AvatarFullUrl] => http://avatar.xboxlive.com/avatar/thechriswalker/avatar-body.png
    [Country] => United Kingdom
    [Location] => Bradninch
    [Bio] =>
    [Reputation] => 58.72229
    [ReputationImageUrl] => http://live.xbox.com/xweb/lib/images/gc_repstars_external_12.gif
    [Zone] => Recreation
    [GamerScore] => 230
    [PresenceInfo] => Array
        (
            [Valid] => true
            [Info] => Last seen 12/29/09   playing Modern Warfare® 2
            [Info2] =>
            [LastSeen] => Tue, 29 Dec 2009 21:35:22 +0000
            [Online] => false
            [StatusText] => Offline
            [Title] => Modern Warfare® 2
        )

    [RecentGames] => Array
        (
            [0] => Array
                (
                    [Name] => Modern Warfare® 2
                    [TotalAchievements] => 50
                    [TotalGamerScore] => 1000
                    [Image32Url] => http://tiles.xbox.com/tiles/Z+/tF/12dsb2JgbA9ECgQJGgYfVl5UL2ljb24vMC84MDAwIAABAAAAAPhq63g=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/CE/Vx/0Gdsb2JhbC9ECgQJGgYfVl5UL2ljb24vMC84MDAwAAAAAAAAAP9eRRc=.jpg
                    [LastPlayed] => Tue, 29 Dec 2009 21:32:52 +0000
                    [Achievements] => 9
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%60m%2fl%3b%7cw&compareTo=thechriswalker
                )

            [1] => Array
                (
                    [Name] => PGR 4
                    [TotalAchievements] => 60
                    [TotalGamerScore] => 1250
                    [Image32Url] => http://tiles.xbox.com/tiles/Y1/qn/0Gdsb2JgbA9ECgR8GgMfWSlaL2ljb24vMC84MDAwIAABAAAAAP+IWnw=.jpg
                    [Image64Url] => http://tiles.xbox.com/tiles/DP/ST/12dsb2JhbC9ECgR8GgMfWSlaL2ljb24vMC84MDAwAAAAAAAAAPi89BM=.jpg
                    [LastPlayed] => Mon, 28 Dec 2009 16:59:25 +0000
                    [Achievements] => 5
                    [GamerScore] => 115
                    [DetailsURL] => http://live.xbox.com/en-US/profile/Achievements/ViewAchievementDetails.aspx?tid=%09%5d%3a%15%18*iAq%0b&compareTo=thechriswalker
                )

        )

)

So now we can easily get at the data. The source code for the class (which is not fully tested, but the basics work!) is at http://thechriswalker.net/xbox360/gamertag.source.php (NB it requires either PHP5 (for json_encode) or the PEAR Services_JSON class if you want to use the “getJSON()” method).

The JSON

PHP is well and good but what if I want to use a JSON/JSONP (JSONP is for cross-domain information requesting and is very useful for public information services, see http://en.wikipedia.org/wiki/JSON#JSONP) request, well, that can be done at http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG for the straight JSON or http://thechriswalker.net/xbox360/?gamertag=YOUR_GAMERTAG&_callback=YOUR_CALLBACK_FUNCTION_NAME for JSONP.

The first returns just JSON with a content type “application/json” and the second returns a javascript function call to your callback function with the JSON object as the only parameter and a content type of “text/javascript”.

These enabled me to build a simple Google Gadget to display a Gamercard:

So I thought, OpenSolaris. It’s the original implementation of the filesystem it must be great, so I export my zpools and boot from the LiveCD (another great reason to try it).

However it wasn’t going to be that easy… When I try to import my zpools in Solaris I get errors stating the vdev’s are corrupt! Something like:

status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.

Scared I boot back to linux.

All the disks and zpools are fine. So, back to Solaris and they still look corrupt. So I search and find this on the ZFS-FUSE Google Group.

Sun claims best practice is to use an entire disk when assigning disks to vdevs, rather than just partitions. So that’s what I did in linux. However the Linux implementation differs form the Solaris one and whole disks added in Solaris get a single partition first and that is added to the vdev.

Great! Now I have these zpools I can’t use anywhere else. To test I used a pool I set up as a mirror and detached a vdev (one disk). I then repartitioned the disk to have one big partition and also wiped the ZFS pool info from it (easier said than done! Ghost pools seem to be a problem!). Then I re-inserted the new partition into the the zpool and waited for resilver. Then I did the same with the other drive, detach from pool, partition, attach partition and resilver.

Bingo. It works in Solaris! Now I just have to find a spare 750Gb disk so I can do the same for my JBOD style zpool…

This was also my first major experience with Linux as a whole, so I was trying to keep things simple. So I did a default type install, with LVM on the 2×200GB disks to give one logical 400GB partition.

All was fine and dandy until I wanted to upgrade a couple of years later and not knowing much about LVM, wiped all my data. Yes, ALL my data. I had some important stuff backed up, but a lot of non-critical but frustrating to lose data (many GBs of painstakingly ripped and encoded CDs and Audiobooks for example).

I was gutted, and vowed not to use LVM again (note my solution to my mistake and lack of knowledge – never use the software again! Well thought out!). So I rebuilt the server using CentOS 4.x ( I can’t remember…) and added another 250GB disk. This time I kept each disk seperate with it’s own filesystem.

This worked fine for ages until the disks were nearly full and I spent all my time moving data from one disk to another to gain a few GBs extra on another disk. Nightmare.

So I decided it was time to start again and did some research into the available filesystems people use for fileservers, how software RAID can be implemented and so on. In this time I came across ZFS and I couldn’t believe what it could do. It seemed to good to be true. I now know this to be false and it can’t do everything, but it can do an awful lot!

So how can I use ZFS? Well, if you use any Solaris variant, then you probably already are. Otherwise chances are slim that you’ve encountered it. I didn’t really want to go down the OpenSolaris route, but instead stick to Linux where I’m more comfortable. This poses issues, as the licencing behind ZFS from Sun doesn’t fit with GPL, so no kernel support for ZFS. Instead someone (I’d love to reference but I forget his name, google will surely help – http://google.com/search?q=ZFS+FUSE) has ported ZFS to FUSE which is great except for the massive memory usage and the poor performance.

Those actually sound like quite big issues! Well, the memory thing is not so bad as it seems. ZFS is not memory instense (well, actually it constantly uses about 256Mb so it’s not light) but uses available memory cache to store data in case it needs it again. A good idea, as the memory is otherwise just doing nothing! The Performance is pretty poor though. It’s enough for my network and to stream music, pictures and video over my network. But single file transfer never hits much more than 8Mb/s read or write.

Now I’m interested in trying some of the other OS options, as ZFS is easy to “export” from one OS and “import” into another. What are the options?

• Option 1: OpenSolaris – Seems like a good idea. The native platform, has a good AMP stack so should be fine for my needs. Have heard some stability issues, but not so much with ZFS, but the OS itself.
• Option 2: Nextenta – Solaris Kernel with GNU userland. Supposed to be a good happy medium, sounds too much like a compromise to me…
• Option 3: BTRFS – different route totally but a new filesystem designed not to mirror ZFS, but to take the best of everything it can. Some aspects very like ZFS. Very promising project, but too young.

Kinda only leaves OpenSolaris… I’ll let you know how I get on.

And I got really excited and really angry.

I got angry, because most people weren’t excited for the right reasons, or were dismissive / “anti” because they didn’t really see what Google has done.

But because I am so excited about Wave (not just Google Wave), I thought I’d post this to help you understand what Google Wave actually is and what it isn’t.

What Google Wave isn’t

Google Wave isn’t Wave.

Wait a second! Why have you dropped the “Google”, what’s “Wave” all about?

Firstly let’s take an analogy, with something you are no doubt familiar with. Consider the statement:

Google Mail is not email.

Imagine that email didn’t exist and Google showcased Google Mail. People might think that email was Google Mail, but we all know that isn’t the case.

The same for Wave. Google have designed a messaging system which can be run by anyone. They have designed the protocols it will use which anyone can implement. This messaging system is not Google Wave, but Wave itself.

What Google Wave is

Google have designed a UI for Wave, which is Google Wave.

Don’t get me wrong, I like their interface. It is written in HTML5 using the Google Web Toolkit and it is super slick.

However, the Wave itself is far more exciting.

What Wave is

Wave is email redesigned for the 21st century. Email was born before the internet and has been flogged and flogged into how we use it now. It is inefficient, single one-way message based, bad with files, incosistent with rich styling, non-collaborative and basically just way past it’s “sell by date”.

With Wave we no longer deal in single messages, but conversations; where content evolves over time and can be “played back” to provide the user with context. Everything email and IM can do and more. They have designed the system to be extensible and so can communicate with other systems. They also designed it to work over a federated networks model, so many people running their own Wave servers can communicate in the same rich way as if they were communicating with their people on their own server.

Amazing really, and provided we can bridge the backwards compatibility gap^* (which we can, remember Wave is extensible!) then I see Wave replacing email altogether.

When Google Wave goes live, Google will also be releasing the server for anyone to run, learn, enjoy and stop using email with. I am looking forward it immensely.

[update 2009-08-06] Further developments have shown that yes “gateways” have been designed in, so Wave can replace email and any other messaging/collaboration protocols/tools if someone creates a “gateway” for it.

Doesn’t sound like much, but if you are like me, you’ll not only want to finish all the levels but also go back and experiment with loads of other different ways to finish in the vain hope of finding the perfect solution. A solution that excels by being so simple, or so elegant. So unusual, or so complex. I could play for hours. But I won’t go on for hours.

I will leave you with some of the best contraptions I have managed to come up with, and the first one you should really vote as amazing, because (and I know this sounds a bit arrogant) is ingenious!

• unusual technique! See if you can guess how it works before you start it!
• Minimalistic elegance, perfect timing.
• I saw a solution with more than one spinny thing, but this seemed a cooler solution (notice the pole to slow down the last ball slightly…)
• A cool way to do “The Ball”

That’s all for the moment, but I’m sure I’ll post more contraptions in the future…