Rich Wareham

Getting Started with Electronics

2015-03-03T00:00:00+00:00

Introduction

Towards the end of 2014 I decided that I wanted to learn digital electronics. As someone who would self-identify as a software person I only had a relatively wooley understanding of what happened at the “sand layer” of my computer. I’ve set myself the project of designing and making a 1980s-era computer called “Búri”. I’ve recorded a few YouTube videos on it:

This post covers the “bootstrapping” process I went through to get myself to a place where I could start learning electronics. Its the set of things I wish I knew when I started.

The essentials

Unlike software, electronics is fundamentally concerned with individual things. And the thing about things is that the incremental cost of making and delivering an individual thing is far greater than software. (Making a copy of a piece of software is very nearly free.) Hence you’ll need to spend some money. I spent quite a bit of money but if I were to bootstrap again, this is what I’d buy:

£5.48 - Arduino UNO-alike.
£3.73 - Solderless breadboard, wires and power supply.
£16.74 - Electronics component starter kit.

Prices inclusive of postage and correct at time of writing. Total cost: £25.95 or, again as of writing, around $45 (US). Note that I’m not endorsing these individual eBay sellers; YMMV when ordering from China. I was also lucky in that my Dad, on hearing of my new hobby, found a large box of chips, resistors, capacitors, etc. in the garage. Most of my projects have now been driven more by what I have for free in those boxes rather than what I’ve bought.

You can never have enough solderless breadboards. I bought a A4-sized whiteboard from Poundland and stuck breadboards to it. This is a lot cheaper than buying one of the metal-backed “advanced” breadboard modules. You can see it in action in the videos above and, in an earlier state, below:

(The video itself is the first test of using a MAX7219 chip to drive a set of seven segment displays from the Big Box. This ended up forming the core of the debug board for my homebrew 6502 computer.)

Less essential essentials

Aside from discrete components like resistors, capacitors and LEDs, if you want to play with digital electronics like I did you’ll need some 7400 series logic. I inherited a lot from my Dad in the Big Box O’ Joy. That being said, the chips you’ll definitely want a handful of are:

74HC00 - NAND gates.
74HC02 - NOR gates.
74HC04 - NOT gates.
74HC74 - D-type flip-flops.
74HC165 - 8-bit “output” shift register.
74HC595 - 8-bit “input” shift register.

The latter two chips are useful if you’re using an Arduino to drive lots of outputs or getting lots of inputs. You’ll be surprised at how small “lots” can be.

There’s a good Computerphile video on the use of flip-flops in computer memory if you don’t know your “D-type”s from your “JK-type”s.

Knowledge

Once you’ve got your equipment, you need to start learning. I started with a little basic electronics knowledge from school and a couple of courses at University. I was rather dismissive of electronics ad University viewing it as a subject deeply steeped in “magic”. As I’ve gotten older, I’ve realised that electronics people viewed software as similarly being the domain of wizards and sorcery. Such is any skill when viewed from outside.

That being said I was aware, perhaps distantly, of the following:

Voltage and current are different like “pressure” and “flow” are different. Current is the flow of electrons and voltage is how much those electrons want to flow.
Capacitors let alternating current pass but block direct current.
Diodes let current flow in one direction only.
Transistors act like an electrically controlled switch although I was more than a little fuzzy on the difference between NPN, PNP, bipolar junction, MOSFET, etc, etc.
Resistors obey Ohm’s law. (Otherwise known as the Very Important Rule.)
Kirchhoff’s circuit laws. (Otherwise known as “current and voltage don’t appear or disappear by magic”.)

Of these, the first is the most important. If you’re not sure of the difference between voltage and current you’re going to have a bad time and you should watch some YouTube videos to get a good handle on the difference.

Aside: more correctly, I was aware that current is the flow of the places where electrons want to be. Conventional positive-to-negative current flow predates the discovery that electrons are negatively charged!

I had, therefore, something of a schoolboy electronics knowledge with some pretty large gaps but YouTube provided some important gap-filling mortar. I can’t stress how useful being able to sit down with a cup of tea and watch the odd ten minutes of electronics tutorial videos was.

A first project

The first circuit that almost anyone does these days is the Arduino “blinking LED” circuit. This consists of an LED, a resistor and an Arduino. There’s not much to that but I like to try and learn a little bit of theory from everything I do. The first question which popped into my head was “why do we need the resistor?” I’ll let the interested reader Google for better explanations than I could write here but suffice it to say that I found it pleasing to be able to calculate what value that resistor should have. Even the simplest of circuits can be a learning opportunity.

From deep in my old bedroom in my parents’ house came the book Adventures with Microelectronics. I’m sure many similar books exist which were printed after the colours brown and yellow went out of fashion. This book has a few simple circuits which can be built on a breadboard and which cover the basics of wiring things up, designing oscillators, etc. If you could find an introductory e-book on something Arduino related, that’d probably be just as good. Working through the book was the work of a single day but was fun to “get my feet wet”.

A stretch project

I found it important to set a “stretch goal”: a project which seemed imposing and difficult to begin with but could be broken down into stages which would spark off simple projects along the way. As outlined in the introuction, I chose to implement a 1980s-style microcomputer. The thought was that if I can get to the point where I can compile C, I had “closed the gap” in my knowledge between chip and code.

YouTube: let a thousand lectures bloom

YouTube is an amazing resource for learning while in the bath. Queue up some videos, draw some suds and lie and relax. If you;re learning electronics, searching for “arduino” will consume many happy hours. If you want some specific recommendations:

Computerphile.
Julian Ilett. His “postbag” segment lets you in to all the fun little electronics modules you can get from eBay.
EEVblog “Fundamentals Friday” YouTube playlist.
The Ben Heck Show. This is more for inspiration than eductaion but is still fun.

Conclusions

This post contains some of what I wished I knew when starting out in electronics. Specifically, it gives an initial, affordable, shopping list. I hope you find it useful!

Fonts from old ROMs with Python

2015-02-03T00:00:00+00:00

This post is also available as an IPython notebook which may be downloaded or viewed online.

Introduction

For a personal project recently I needed a simple monospaced 8x8 pixel font. I could, of course, make my own but I thought it’d be nice for nostalgia reasons to try and extract the Acorn 8x8 font used in the old BBC Microcomputer. Using Python we can directly extract the font from a zipfile containing a ROM image which we download from the web. We can then use the imgur API to upload the resulting font directly to the web.

Implementation

You can find a copy of the BBC Model B memory map online. The relevant section for our needs is:

PAGE 192 (&C0) to 194 (&C2) : OS ROM

C000-C2FF Character font lookup table

This tells us that the character font table is in the first 0x300 (or 768) bytes of the OS ROM. The OS ROMs are available from multiple sources but there is a page on a BBC Micro games site which has a link.

OS_ROM_URL = 'http://www.bbcmicrogames.com/roms/os12.zip'

This is a zip file and so we can use the Python zipfile module to examine the file after downloading it into a bytes object via the requests library:

import requests         # downloading the zipfile
from io import BytesIO  # treating a bytes object as file
import zipfile          # parsing the zipfile

# Fetch the ROM and check that the GET succeeded
os_rom_zip_req = requests.get(OS_ROM_URL)
assert os_rom_zip_req.status_code == 200

# Parse the body as a zip file
os_rom_zip = zipfile.ZipFile(BytesIO(os_rom_zip_req.content))

There should only be one file in the archive which is the one we want:

print('Archive contents: ' + ','.join(os_rom_zip.namelist()))
assert len(os_rom_zip.filelist) == 1

Archive contents: OS12.ROM

So, we want to get the first 0x300 bytes:

font_table = os_rom_zip.read(os_rom_zip.filelist[0])[:0x300]

We can use the bitarray module along with numpy to extract the character font as a 1-bit array. The characters are stored as one byte per row and so our final font array should be 8 columns wide and 8 × number of characters high.

import numpy as np
from bitarray import bitarray

font_table_array = bitarray(endian='big')
font_table_array.frombytes(font_table)
font = np.array(font_table_array.tolist()).reshape((-1, 8))

n_chars = font.shape[0] // 8
print('Number of characters: {0}'.format(n_chars))

Number of characters: 96

Let’s look at the first few characters just to check:

%matplotlib inline
from matplotlib.pyplot import *

imshow(font[:8*3, :], interpolation='none', cmap='gray')

<matplotlib.image.AxesImage at 0x7f344fc0db38>

It would be nice to see the entire font as one image. Let’s first split it into one array per character:

char_arrays = np.split(font, n_chars)

# check that we've split correctly
imshow(char_arrays[20], interpolation='none', cmap='gray')

<matplotlib.image.AxesImage at 0x7f344fbebd68>

Now we do some advanced juggling to reshape the character array:

font_image = np.vstack(tuple(
    np.hstack(row).reshape((8, -1)) for row in np.array(char_arrays).reshape((-1, 16, 8, 8))
))

print('Generated font image of shape: ' + 'x'.join(str(x) for x in font_image.shape))

imshow(font_image, cmap='gray', interpolation='none')

Generated font image of shape: 48x128

<matplotlib.image.AxesImage at 0x7f344fb65208>

And we’re a winner. The only thing left to do is to save it as an image for some further processing. We’ll use the imgurpython library to upload our result directly to imgur. I’ve created a file called imgur-credentials.json containing the client id and secret I obtained by registering an application at https://api.imgur.com/. The file looks something like this:

{
    "clientId": "some magic hex string",
    "clientSecret": "another magic hex string"
}

The first thing to do is to create an imgur client to upload the file:

import json
from imgurpython import ImgurClient

imgur_creds = json.load(open('imgur-credentials.json'))
imgur_client = ImgurClient(imgur_creds['clientId'], imgur_creds['clientSecret'])

Using the pillow library we can save the image to a temporary file and the upload it. Note the use of the with statement to make sure that the temporary file is deleted.

import tempfile
from PIL import Image

with tempfile.NamedTemporaryFile(suffix='.png') as tf:
    Image.fromarray(np.where(font_image, 255, 0).astype(np.uint8)).save(tf.name)
    upload_result = imgur_client.upload_from_path(tf.name)
print('Image uploaded to: {0}'.format(upload_result['link']))

Image uploaded to: http://i.imgur.com/7mZVBee.png

IPython has some built-in support for displaying images from URLs. Let’s use that to look at the result:

from IPython.display import Image
Image(upload_result['link'])

Conclusion

This little post has shown how we can extract fonts directly from old microcomputer ROM images with Python. As a bonus, we upload the resulting font back to the web via the imgur API.

Optical Character Recognition in Python: Transcribing the Turing code

2014-11-09T00:00:00+00:00

This post is also available as an IPython notebook which may be downloaded or viewed online.

This post talks about how to do a simple Optical Character Recognition task in Python. But first, why did I want to do this in the first place?

The University of Manchester is, as I’m writing this, running a competition as part of a tie in with the forthcoming film The Imitiation Game. The film centres around Alan Turing and his time as a cryptographer at Bletchley Park near Oxford.

Appropriately enough, the competition involves decrypting (or deciphering) a set of clues which it is claimed will reveal the location where Alan Turing hid some bars of silver.

The first and third problems are straightforward enough and I’ll not spoil the competition by posting solutions here.

Let’s take a look at the second problem:

# Import PyLab to generate plots and set the default figure size
# to something a little nicer.
%pylab inline
rcParams['figure.figsize'] = (16,9)

# Use PIL and requests to fetch the second clue:
import requests
from PIL import Image

r = requests.get('http://www.maths.manchester.ac.uk/cryptography_competition_the_imitation_game/code2_full_res.png')
assert r.status_code == 200

from io import BytesIO
code2_im = Image.open(BytesIO(r.content))

Populating the interactive namespace from numpy and matplotlib

Let’s take a look at the clue:

imshow(code2_im)

<matplotlib.image.AxesImage at 0x7f0cb0374410>

As you can see, none of the characters here are the usual alphabetic ones. There’s an eight and a three but the rest are mathematical symbols. Before I can start doing any decrypting, I need to transcribe these symbols. I could do this manually but a) I don’t know which letters to use for which symbols and b) this is highly error prone. Instead, I wondered if I could write a Python program to extract a list of symbols for me.

Extracting images for each symbol

The first thing to do is snip an image containing each symbol in the input. Counting them, I see the grid is 24 symbols across and 15 down. Let’s try to form an array of centre pixel locations for each symbol. Using IPython, I can fidle around with the values below until the dots line up roughly with the symbols:

# Fiddle with these numbers a bit...
xs, ys = np.meshgrid(np.linspace(93, 1146, 24), np.linspace(135, 984, 15))

# Until these dots lie at the centres
imshow(code2_im, interpolation='none')
scatter(xs.flat, ys.flat, alpha=0.5)

<matplotlib.collections.PathCollection at 0x7f0cb02add90>

OK, those points look good. Let’s snip out 16 pixels either side of these centre points into one “region” per symbol:

# Get the image as a grayscale image
code2_grey = np.asarray(code2_im.convert('F'))

# region box "half width"
bhw = 16
regions = []
for x, y in zip(xs.flat, ys.flat):
    x, y = int(x), int(y)
    regions.append(code2_grey[y-bhw:y+bhw,x-bhw:x+bhw])
    
# Since the final line only has 9 symbols on it, ignore the final
# 15 regions (set them to blank)
regions = regions[:-15] + [np.zeros_like(regions[0]),] * 15

print("Number of regions: " + str(len(regions)))

Number of regions: 360

We can also write a little convenience function to view all of these snipped out regions:

def tile(mats, rows=None):
    """Tile arrays from *mats* into a larger array optionally
    specifying the number of rows.
    
    """
    if rows is None:
        rows = int(np.sqrt(len(mats)))
    cols = int(np.ceil(len(mats) / float(rows)))
    w, h = mats[0].shape[:2]
    out = np.zeros((rows*h, cols*w))
    for idx, m in enumerate(mats):
        r = idx // cols
        c = idx % cols
        out[r*h:(r+1)*h, c*w:(c+1)*w] = m
    return out

snipped_regions = tile(regions, rows=15)
imshow(snipped_regions, cmap='gray', clim=(0,255))

<matplotlib.image.AxesImage at 0x7f0cb0222390>

Excellent, they look nice. The next step will be to match each region to a corresponding “symbol id”.

Computing feature vectors

We can’t directly compare the pixel values in each region. It is likely that my rough centre-points are not exact and there is a non-uniform background in the image. (I.e. the “wartime paper” effect.) Instead we want to manipulate the images to exctract a vector of features which are tolerant to small shifts or small changes in intensity. Fortunately I have just a thing in my little bag of tricks a Python implementation of the Dual-Tree Complex Wavelet Transform.

It’s not required to know the specifics of the transform but see what happens if we transform the first region and one translated by one pixel:

# Create a "default" DTCWT
from dtcwt import Transform2d
dtcwt_2d = Transform2d()

# Transform the first region and that region shifted a little
t1_im = regions[0]
t1 = dtcwt_2d.forward(t1_im)
t2_im = np.roll(regions[0], -1, axis=0)
t2 = dtcwt_2d.forward(t2_im)

# Show images side-by-side
subplot(1,2,1)
imshow(t1_im, cmap='gray')
subplot(1,2,2)
imshow(t2_im, cmap='gray')

<matplotlib.image.AxesImage at 0x7f0cb01af050>

Again, the technicalities of the transform are not of interest to us here but the DTCWT converts an image into a “lowpass” (or “blurred”) low-resolution image and a set of “highpass” (or “edge-like”) complex-valued images.

Each highpass image is an array of NxMx6 complex numbers. Let’s take a look at a slice from the first symbol:

# Look at first slice of second (level "3")) highpass image
subplot(1,2,1)
imshow(np.abs(t1.highpasses[2][...,0]), interpolation='none')
subplot(1,2,2)
imshow(np.abs(t2.highpasses[2][...,0]), interpolation='none')

<matplotlib.image.AxesImage at 0x7f0cb00f6950>

Although the images have been shifted a little, the DTCWT coefficients are very similar. We’ll use this to perform matching. Firstly, transform each region into a “feature vector” consisting of the DTCWT highpass coefficients from level 3 onwards:

def compute_region_fv(region):
    t_region = dtcwt_2d.forward(region)
    return np.concatenate(list(
        np.abs(y).ravel()
        for y in t_region.highpasses[2:]
    ))

feat_vecs = np.array(list(compute_region_fv(r) for r in regions))
print("Feature vectors has shape: " + str(feat_vecs.shape))

Feature vectors has shape: (360, 96)

Matching symbols

We need a set of symbols to match with and so I manually went through the image and found the indices of the first example of each kind of symbol:

# Indices of "exemplar" regions
exemplars = [
    0,1,2,3,4,5,7,11,12,13,14,15,16,17,18,
    20,26,30,35,37,49,50,51,57,60,95,97,102,143,322,
    feat_vecs.shape[0]-1, # Final "blank" region
]
print("Symbol count: " + str(len(exemplars)-1))

Symbol count: 30

Let’s look at them:

imshow(
    tile(list(regions[e_idx] for e_idx in exemplars)),
    cmap='gray', clim=(0,255), interpolation='none'
)

<matplotlib.image.AxesImage at 0x7f0cb004bfd0>

We’re going to match regions by computing the total difference between the DTCWT coefficients and those of the examplar regions. Then we’ll assign that region a symbol id based on which exemplar it is closest to.

# Compute total differences to each exemplar for each region
diffs = np.zeros((feat_vecs.shape[0], len(exemplars)))
for e_idx, r_idx in enumerate(exemplars):
    # Get exemplar's feature vectore
    e_fv = feat_vecs[r_idx:r_idx+1,:]
    
    # Difference from each feature vector to the exemplar
    delta = feat_vecs - np.repeat(e_fv, feat_vecs.shape[0], axis=0)
    
    # Compute total absolute difference
    diffs[:, e_idx] = np.sum(np.abs(delta), axis=1)
    
# Use Numpy's argmin function to return the index of the smallest difference
match_idx = np.argmin(diffs, axis=1)

Has it worked? Let’s find out. We’ll create a “reconstructed” image by copying the exemplar symbol for each map into the appropriate position in the output image:

synthetic_snipped_regions = tile(list(regions[exemplars[m_idx]] for m_idx in match_idx), rows=15)
imshow(synthetic_snipped_regions, cmap='gray', clim=(0,255))

<matplotlib.image.AxesImage at 0x7f0caba37e90>

That looks pretty good. What about if we compute the difference between this and the original:

imshow(synthetic_snipped_regions - snipped_regions, cmap='gray', clim=(-255,255))

<matplotlib.image.AxesImage at 0x7f0caba1d710>

Looks good. If we had got any matches wrong we would have a very strong white or black line where the matched symbol differed from the actual. The final step is to turn the match indexes into a transcription. We’ll write out the code again using the symbol index in place of the symbol:

transcription = ''
for s_idx, s in enumerate(match_idx):
    transcription += '{0:0>2} '.format(s)
    if s_idx % 24 == 23:
        transcription += '\n'
print(transcription)

01 02 03 04 05 02 06 05 01 06 07 08 09 10 11 12 13 14 04 15 04 04 05 
10 16 04 13 07 17 04 13 14 03 18 13 19 07 00 01 14 13 12 17 14 09 10 
20 21 22 12 25 09 20 02 23 13 19 24 15 04 24 09 10 18 03 12 16 09 20 
20 08 16 05 12 13 23 07 09 06 08 25 16 09 13 19 02 05 18 10 05 13 25 
26 00 13 12 22 27 13 12 24 09 10 14 06 12 10 27 27 13 12 24 15 27 13 
02 01 15 11 13 19 02 05 07 20 02 02 09 20 12 07 09 03 08 09 19 08 28 
16 03 18 13 19 08 07 13 18 12 10 03 17 24 09 10 04 06 21 06 02 10 04 
26 00 06 16 15 11 24 12 03 04 16 25 07 26 02 13 12 15 06 16 10 11 13 
05 11 02 19 20 24 13 27 25 04 05 07 09 10 16 25 02 21 20 12 27 13 02 
13 14 16 15 08 15 16 09 25 05 05 10 04 22 03 16 24 13 27 16 24 06 04 
10 01 12 13 06 05 23 06 12 07 13 27 00 01 12 10 26 19 02 06 12 11 01 
02 10 12 13 19 08 20 24 13 14 13 12 17 04 13 08 20 07 13 16 22 02 27 
15 04 03 02 11 13 05 22 10 04 11 12 01 23 24 20 05 18 01 06 17 12 15 
26 16 00 03 12 25 04 10 00 29 00 03 11 09 15 04 22 19 16 15 04 26 00 
25 04 25 24 25 06 02 16 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 

Or, maybe, it’s easier if we use alphabetic characters. (Note that we’ll also end up using some punctuation since there are more than 26 symbols.)

transcription = ''
for s_idx, s in enumerate(match_idx):
    transcription += '{0} '.format(chr(65+s))
    if s_idx % 24 == 23:
        transcription += '\n'
print(transcription)

A B C D E F C G F B G H I J K L M N O E P E E F 
N K Q E N H R E N O D S N T H A B O N M R O J K 
E U V W M Z J U C X N T Y P E Y J K S D M Q J U 
C U I Q F M N X H J G I Z Q J N T C F S K F N Z 
E [ A N M W \ N M Y J K O G M K \ \ N M Y P \ N 
E C B P L N T C F H U C C J U M H J D I J T I ] 
P Q D S N T I H N S M K D R Y J K E G V G C K E 
Z [ A G Q P L Y M D E Q Z H [ C N M P G Q K L N 
E F L C T U Y N \ Z E F H J K Q Z C V U M \ N C 
C N O Q P I P Q J Z F F K E W D Q Y N \ Q Y G E 
C K B M N G F X G M H N \ A B M K [ T C G M L B 
L C K M N T I U Y N O N M R E N I U H N Q W C \ 
\ P E D C L N F W K E L M B X Y U F S B G R M P 
K [ Q A D M Z E K A ^ A D L J P E W T Q P E [ A 
B Z E Z Y Z G C Q _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Conclusions

We’ve successfully used Python to avoid the tedium of having to transcribe a sheet of symbols by hand. Techniques such as this are used by actual OCR programs to be tolerant to small changes in character shape. Obviously I won’t be writing any more about the further decoding of this clue until after the closing date for the competition :).

Making a Little Planet Panorma using Python and Scikit Image

2014-09-29T00:00:00+00:00

This post is also available as an IPython notebook which may be downloaded or viewed online.

Recently I shared a post on G+ which had a Google “photosphere” transformed into the “little planet” projection. This post (and associated IPython notebook) will walk you through the way I created the planet image using scikit-image. This was my first steps with scikit-image since previously I had been using OpenCV. Unfortunately OpenCV is a bit of a pain to install and so I was looking for a pip-installable library with the functionality I required.

Loading the initial panorama

As usual, we need a little bit of boiler-plate to locad matplotlib and numpy. I’m also going to import the Pillow library for loading images (although I think scikit-image could do that).

# IPython magic to allow inline plots
%matplotlib inline

import os # for file and pathname handling functions
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image

# Set the default matplotlib figure size to be a bit bigger than default
plt.rcParams['figure.figsize'] = (16,9)

On my phone, the photosphere is stored as a single JPEG file. Let’s use Pillow to load the image and display it via matplotlib:

pano = np.asarray(Image.open(os.path.expanduser('~/Downloads/PANO_20140927_124513.jpg')))
plt.imshow(pano)

<matplotlib.image.AxesImage at 0x7f91a1106090>

Beautiful. The next step is to work out how to warp this image.

Image warping via scikit-image

Looking at the appropriate section in the scikit-learn documentation, we see that there is a handy function called warp which will do exactly what we want. The function takes, amongst other inputs, the image to warp, the output image shape and a function which specifies the warp itself.

This function takes a N⨉2 array of (x,y) co-ordinates in the output image and turns them into a corresponding array of co-ordinates in the input image. Let’s just have a go at that using a very simple function which scales and shifts the co-ordinates:

from skimage.transform import warp

def scale_by_5_and_offset(coords):
    out = coords * 5
    out[:,0] += 1000
    out[:,1] += 300
    return out

plt.figure(figsize=(10,10)) # A square figure for square output
plt.imshow(warp(pano, scale_by_5_and_offset, output_shape=(256,256)))

<matplotlib.image.AxesImage at 0x7f917c93c610>

The “little planet” projection

The panorama we have is in the “equirectangular” projection which means that the y-co-ordinate represents the up-down angle and the x-co-ordinate represents the left-right angle. The little planet projection takes a different approach: the up-down angle is now specified by the distance from the centre of the image and the left-right angle is given by the angle from the horizontal in the image.

Given a particular output image shape, therefore, we can convert a co-ordinate in that image to a radius and angle, or 𝑟 and 𝜃 pair:

# What shape will the output be?
output_shape = (1080,1080) # rows x columns

def output_coord_to_r_theta(coords):
    """Convert co-ordinates in the output image to r, theta co-ordinates.
    The r co-ordinate is scaled to range from from 0 to 1. The theta
    co-ordinate is scaled to range from 0 to 1.
    
    A Nx2 array is returned with r being the first column and theta being
    the second.
    """
    # Calculate x- and y-co-ordinate offsets from the centre:
    x_offset = coords[:,0] - (output_shape[1]/2)
    y_offset = coords[:,1] - (output_shape[0]/2)
    
    # Calculate r and theta in pixels and radians:
    r = np.sqrt(x_offset ** 2 + y_offset ** 2)
    theta = np.arctan2(y_offset, x_offset)
    
    # The maximum value r can take is the diagonal corner:
    max_x_offset, max_y_offset = output_shape[1]/2, output_shape[0]/2
    max_r = np.sqrt(max_x_offset ** 2 + max_y_offset ** 2)
    
    # Scale r to lie between 0 and 1
    r = r / max_r
    
    # arctan2 returns an angle in radians between -pi and +pi. Re-scale
    # it to lie between 0 and 1
    theta = (theta + np.pi) / (2*np.pi)
    
    # Stack r and theta together into one array. Note that r and theta are initially
    # 1-d or "1xN" arrays and so we vertically stack them and then transpose
    # to get the desired output.
    return np.vstack((r, theta)).T

We’re now very nearly in a position to generate our first little planet picture. In our original panorama 𝑟=0 corresponds to the bottom of the picture (i.e. maximum y-co-ordinate) and 𝑟=1 corresponds to the top (i.e. a y-co-ordinate of zero). Similarly 𝜃=0 corresponds to an x-co-ordinate of 0 and 𝜃=1 corresponds to the maximum x-co-ordinate. We can write a function to convert these co- ordinates:

# This is the shape of our input image
input_shape = pano.shape

def r_theta_to_input_coords(r_theta):
    """Convert a Nx2 array of r, theta co-ordinates into the corresponding
    co-ordinates in the input image.
    
    Return a Nx2 array of input image co-ordinates.
    
    """
    # Extract r and theta from input
    r, theta = r_theta[:,0], r_theta[:,1]
    
    # Theta wraps at the side of the image. That is to say that theta=1.1
    # is equivalent to theta=0.1 => just extract the fractional part of
    # theta
    theta = theta - np.floor(theta)
    
    # Calculate the maximum x- and y-co-ordinates
    max_x, max_y = input_shape[1]-1, input_shape[0]-1
    
    # Calculate x co-ordinates from theta
    xs = theta * max_x
    
    # Calculate y co-ordinates from r noting that r=0 means maximum y
    # and r=1 means minimum y
    ys = (1-r) * max_y
    
    # Return the x- and y-co-ordinates stacked into a single Nx2 array
    return np.hstack((xs, ys))

Let’s test our mapping functions using the warp function:

def little_planet_1(coords):
    """Chain our two mapping functions together."""
    r_theta = output_coord_to_r_theta(coords)
    input_coords = r_theta_to_input_coords(r_theta)
    return input_coords

plt.figure(figsize=(10,10))
plt.imshow(warp(pano, little_planet_1, output_shape=output_shape))

<matplotlib.image.AxesImage at 0x7f917c8ab2d0>

That’s not a bad first attempt but it would be nicer if we had a bit more horizon and a little less ground. That is to say we’d like to map 𝑟 a bit so that it increases quite rapidly to start with but flattens off a little. Usefully the square root function behaves a little like that:

rs = np.linspace(0, 1, 100)
plt.plot(rs, np.sqrt(rs))

[<matplotlib.lines.Line2D at 0x7f917c88e7d0>]

So let’s modify our little planet projection to take the square root of 𝑟:

def little_planet_2(coords):
    """Chain our two mapping functions together with modified r."""
    r_theta = output_coord_to_r_theta(coords)
    # Take square root of r
    r_theta[:,0] = np.sqrt(r_theta[:,0])
    input_coords = r_theta_to_input_coords(r_theta)
    return input_coords

plt.figure(figsize=(10,10))
plt.imshow(warp(pano, little_planet_2, output_shape=output_shape))

<matplotlib.image.AxesImage at 0x7f917c7eeb90>

That’s better. The castle and trees look less warped. It would be nice to have the castle come out at a slightly different angle though. We can do that by shifting theta:

def little_planet_3(coords):
    """Chain our two mapping functions together with modified r
    and shifted theta.
    
    """
    r_theta = output_coord_to_r_theta(coords)
    
    # Take square root of r
    r_theta[:,0] = np.sqrt(r_theta[:,0])
    
    # Shift theta
    r_theta[:,1] += 0.1
    
    input_coords = r_theta_to_input_coords(r_theta)
    return input_coords

plt.figure(figsize=(10,10))
plt.imshow(warp(pano, little_planet_3, output_shape=output_shape))

<matplotlib.image.AxesImage at 0x7f917c7d4790>

That’s nicer. There’s possibly a little too much sky so, finally, let’s just zooom in a bit by scaling $r$ down a bit:

def little_planet_4(coords):
    """Chain our two mapping functions together with modified and
    scaled r and shifted theta.
    
    """
    r_theta = output_coord_to_r_theta(coords)
    
    # Scale r down a little to zoom in
    r_theta[:,0] *= 0.75
    
    # Take square root of r
    r_theta[:,0] = np.sqrt(r_theta[:,0])
    
    # Shift theta
    r_theta[:,1] += 0.1
    
    input_coords = r_theta_to_input_coords(r_theta)
    return input_coords

plt.figure(figsize=(10,10))
plt.imshow(warp(pano, little_planet_4, output_shape=output_shape))

<matplotlib.image.AxesImage at 0x7f917c738290>

Saving the result

I’m happy with this image now. Let’s use Pillow to save the result:

# Compute final warped image
pano_warp = warp(pano, little_planet_4, output_shape=output_shape)

# The image is a NxMx3 array of floating point values from 0 to 1. Convert this to
# bytes from 0 to 255 for saving the image:
pano_warp = (255 * pano_warp).astype(np.uint8)

# Use Pillow to save the image
Image.fromarray(pano_warp).save(os.path.expanduser('~/Pictures/little-planet.jpg'))

IPython can display images directly fiven a filename. Let’s just check it saved correctly:

from IPython.display import Image as display_Image
display_Image(os.path.expanduser('~/Pictures/little-planet.jpg'))

Summary

In this post we gave a quick example of how to interactively play with an image and warp it from an equirectangular projection to the popular “little planet” projection.

Verifying identity on social media with keybase.io and command-line tools

2014-09-24T00:00:00+00:00

I recently joined keybase.io which is an interesting experiment in verifiable assertions of identity on social media. The concept is simple: if I tweet a message which can be verified as being signed by my private key then you can trust that, at that point, my twitter account was controlled by someone who also had access to my private key. The same assertions can be made about GitHub accounts by signing a gist which makes a similar statement.

Assuming that one trusts a user to be savvy enough to revoke these assertions if they believe their private key is compromised, this provides a fairly strong assertion that various social media accounts are controlled by the same entity. Of course one could do this already oneself using GPG but keybase.io provides a relatively nice workflow and a pretty website. Do not underestimate the advantages which this can bring.

Why is this useful?

I must admit that I do not have a long list of reasons why it is useful to tie social media account identities together and associate them with a public key apart from it being a sort of social media “verified tick” which is available to anyone.

That being said, there are a few use cases I can think of. If one is a software developer which makes signed releases but is otherwise anonymous, e.g. TrueCrypt, then it would be useful to assert that a twitter account is controlled by the same person. To flip this in the other direction, if you know that a Twitter account is controlled by someone with access to a private key then you can have some confidence that anything signed by that private key came from the same person. Finally, it’s useful to be able to say that @john_q_rockstar on Twitter is the same as the person running https://johnqrockstar.com/ without relying on there being a Twitter “verified” tick.

So, how does it work?

The central keybase.io concept is that one should be able to verify for oneself that a social media account or DNS domain was at some point controlled by someone in possession of the private half of a public key which you trust. Let’s work through an example of verifying that my public key is associated with richwareham.com. (Note that DNSSEC does not solve this exact problem although it certainly works in a related space.)

It would be nice to be able to verify richwareham.com without trusting keybase.io infrastructure to actually do the verification. In that way, keybase.io can be seen as a nice tool for creating the verifications without being a critical, or trusted, component in the chain.

Importing the public key

The first thing that one needs to do is to import my public key. Obviously I already have mine imported but one can always import it again:

$ curl -s https://keybase.io/richwareham/key.asc | gpg --import
gpg: key 4E3D7E0D: "Rich Wareham (Personal) <rjw57@cantab.net>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Let’s just check the fingerprint of that key:

$ gpg --fingerprint rjw57@cantab.net
pub   4096R/4E3D7E0D 2011-10-12
      Key fingerprint = 4E27 6278 B082 CE98 E5A9  618E 7BA6 A2C7 4E3D 7E0D
uid                  Rich Wareham (Personal) <rjw57@cantab.net>
sub   4096R/4D4C8881 2011-10-12

This fingerprint can be verified against both my keybase.io profile page and the MIT public key server. It should also appear in your web of trust if you have any trust in the keys which signed mine.

Getting the assertion from DNS

We now need to see what is asserted in DNS. We can use the dig tool to get any TXT records from richwareham.com:

$ dig txt richwareham.com | grep keybase
richwareham.com.        7687    IN      TXT     "keybase-site-verification=AnLTOBDGpT9Efr0DJn4X4C7VcnwZF_ZRT6Vx8_pk_J8"

The important record here is the one containing keybase-site-verification. This also contains the following apparently magic string:

AnLTOBDGpT9Efr0DJn4X4C7VcnwZF_ZRT6Vx8_pk_J8

What the assertion actually is

On the associated proof page for my DNS assertion you can see a walk through of how that magic string was generated. It indicates that everything started from a JSON-formatted assertion message which was signed by my public key. Let’s use GPG to verify that by pasting in the GPG signed statement from the website:

$ gpg <<EOL
> -----BEGIN PGP MESSAGE-----
> Version: GnuPG v1
> 
> owGbwMvMwMRYvWzRcT/bOl7G0we+JDGEKO5kqVZKyk+pVLKqVspOBVNpmXnpqUUF
> RZl5JUpWSiapRuZmRuYWSQYWRsmplhappomWZoYWqeZJiWaJRsnmJqnGKeapBilK
> OkoZ+cUgHUBjkhKLU/Uy84FiQE58ZgpQFIv6UoiEiXmqRaqFgaGhhWGyibmxqbmx
> iXmyWaKZmZm5oYmBAUhhcWpRXmJuKlB1UWZyRnliUWpGYq5SrY4SUKIsMzkV5OyU
> /NzEzDxUJXrJ+blA/QVF+SX5yfk5QMmUvGKQvpLKApBp5alJ8VAj4pMy81KAPgcq
> L0stKs7MBxplCFSZXJIJstjQxNDQBOhxSyMdpdSKgsyi1HiQZYam5mYWBkAAsiS1
> DGikAdAzZqmJhsaJyZbmqWlmZmmmiYaWyQZJaeYmKSlJySYmaSlJxkZmJolGaQZJ
> RkmJiUkGBqYGlompBubGFoapSiBPFeblK1kZA52ZmA40sjgzPS+xpLQoVamWq5NJ
> hoWBkYmBjZUJFHkMXJwCsChVOsz/z7JYubSvKVrRcRKfrdRDrYenTvLc9vr2fcnq
> VPuHGeuV2apPxnP94vM3PZcRKCjfH7rxo4PY89Iz0xZ5HLPdpfPgtimHy5u+jZ+i
> JbjWn3WyUVz1o+73Kle7oN/FJgXuERN+ap7mddU1M5Y5+3j2ujkznPW63Jbedb5V
> 4129ke9m563VRyMvPeVYfUhfo3zzzxBxUTmFtfsP/ou5qfBv+R9TC83jry+39WvL
> mqrf/r9j93Zln+6QS1kc3qveahSzyy7426Jx/cDpW/POPqhnXvVRt9OtwbvM686c
> 3La7lXOOci7L0XpgnB+9PlE9ztk9SfpK3JZfrtMLV32fkKn80XRS3T7fg0Lvgk88
> zjLwy11SeGT+z9mtmo8ebJubUnDnw4IghdT5fr9DZ8w/p3Sc+2Tcct7NTkInxaUZ
> HfQy3wVPvVpvWFWlHPn/1fSiYz+ynpnfUZ+wq/nhy05rU/np3IcMze8cO/V60ctl
> By7xlTqYmqVYndve7bxLZV1TKP86s/1aCxYwf31//d3a/a9VG+7t7is8KpmTIuC3
> I7egrTU7M7ajOnfO7xNbLlo48H0Ta70yb5q33IV5/NtrK2Q0OHv23/fRl/IL3KXy
> uWHBuxZHWavSiXZPTKzY9D3fy+yZ2mWl3STNs3fJvpv/dqqF5kytS1t7esYu9oqX
> Z39fWa/svfjzQpcLwR5652bfvGBixnb+YG5J7B+hWWX1lwE=
> =bND4
> -----END PGP MESSAGE-----
> EOL
{"body":{"key":{"fingerprint":"4e276278b082ce98e5a9618e7ba6a2c74e3d7e0d","host":"keybase.io","key_id":"7ba6a2c74e3d7e0d","uid":"747e8e801181c47357347c6a66671400","username":"richwareham"},"service":{"domain":"richwareham.com","protocol":"dns"},"type":"web_service_binding","version":1},"ctime":1411496192,"expire_in":157680000,"prev":"0e8e6ea13ac97ef66f5a19c0bf74ddbc44fdb3264a2f0b2baab00509ae07381e","seqno":3,"tag":"signature"}
gpg: Signature made Tue 23 Sep 2014 19:16:36 BST using RSA key ID 4E3D7E0D
gpg: Good signature from "Rich Wareham (Personal) <rjw57@cantab.net>"

Indeed this is a message signed by my key. The message includes a fingerprint of my public key, my keybase.io username, the DNS domain I was asserting is mine and when I made the assertion. OK so I trust that I actually asserted once that I control richwareham.com. But I could just as easily assert I control google.com. We need to use the TXT record on richwareham.com to actually check this.

Getting the expected hash

The next stage is to take the actual signed message and compute a SHA-256 hash of its binary representation. Fortunately that’s pretty easy. Stripped of the header and footer, a GPG message is just a Base64-encoded binary blob. We can use the base64 utility to decode it, the sha256sum utility to compute the SHA-256 hash of the contents and then use the ever-wonderful xxd utility to convert the hex-string produced by sha256sum into a binary file containing the 256-bits (or 32-bytes) of hash:

$ base64 -d << EOL | sha256sum - | xxd -r -p expected.bin
> owGbwMvMwMRYvWzRcT/bOl7G0we+JDGEKO5kqVZKyk+pVLKqVspOBVNpmXnpqUUF
> RZl5JUpWSiapRuZmRuYWSQYWRsmplhappomWZoYWqeZJiWaJRsnmJqnGKeapBilK
> OkoZ+cUgHUBjkhKLU/Uy84FiQE58ZgpQFIv6UoiEiXmqRaqFgaGhhWGyibmxqbmx
> iXmyWaKZmZm5oYmBAUhhcWpRXmJuKlB1UWZyRnliUWpGYq5SrY4SUKIsMzkV5OyU
> /NzEzDxUJXrJ+blA/QVF+SX5yfk5QMmUvGKQvpLKApBp5alJ8VAj4pMy81KAPgcq
> L0stKs7MBxplCFSZXJIJstjQxNDQBOhxSyMdpdSKgsyi1HiQZYam5mYWBkAAsiS1
> DGikAdAzZqmJhsaJyZbmqWlmZmmmiYaWyQZJaeYmKSlJySYmaSlJxkZmJolGaQZJ
> RkmJiUkGBqYGlompBubGFoapSiBPFeblK1kZA52ZmA40sjgzPS+xpLQoVamWq5NJ
> hoWBkYmBjZUJFHkMXJwCsChVOsz/z7JYubSvKVrRcRKfrdRDrYenTvLc9vr2fcnq
> VPuHGeuV2apPxnP94vM3PZcRKCjfH7rxo4PY89Iz0xZ5HLPdpfPgtimHy5u+jZ+i
> JbjWn3WyUVz1o+73Kle7oN/FJgXuERN+ap7mddU1M5Y5+3j2ujkznPW63Jbedb5V
> 4129ke9m563VRyMvPeVYfUhfo3zzzxBxUTmFtfsP/ou5qfBv+R9TC83jry+39WvL
> mqrf/r9j93Zln+6QS1kc3qveahSzyy7426Jx/cDpW/POPqhnXvVRt9OtwbvM686c
> 3La7lXOOci7L0XpgnB+9PlE9ztk9SfpK3JZfrtMLV32fkKn80XRS3T7fg0Lvgk88
> zjLwy11SeGT+z9mtmo8ebJubUnDnw4IghdT5fr9DZ8w/p3Sc+2Tcct7NTkInxaUZ
> HfQy3wVPvVpvWFWlHPn/1fSiYz+ynpnfUZ+wq/nhy05rU/np3IcMze8cO/V60ctl
> By7xlTqYmqVYndve7bxLZV1TKP86s/1aCxYwf31//d3a/a9VG+7t7is8KpmTIuC3
> I7egrTU7M7ajOnfO7xNbLlo48H0Ta70yb5q33IV5/NtrK2Q0OHv23/fRl/IL3KXy
> uWHBuxZHWavSiXZPTKzY9D3fy+yZ2mWl3STNs3fJvpv/dqqF5kytS1t7esYu9oqX
> Z39fWa/svfjzQpcLwR5652bfvGBixnb+YG5J7B+hWWX1lwE=
> EOL
$ hexdump -C expected.bin
00000000  02 72 d3 38 10 c6 a5 3f  44 7e bd 03 26 7e 17 e0  |.r.8...?D~..&~..|
00000010  2e d5 72 7c 19 17 f6 51  4f a5 71 f3 fa 64 fc 9f  |..r|...QO.q..d..|
00000020

Now we have a file, expected.bin, which contains a 256 bit (32 byte) hash of our signed assertion. To prove that I control richwareham.com, I added a TXT record which contained an encoded form of that hash.

Getting the actual hash from DNS

It’s asserted that the magic string we found in the DNS TXT record should be the web-safe Base64 encoding of that very hash. The assumption being that only someone in control of the domain would be able to insert a TXT record which is cryptographically equivalent to their assertion of control.

Web-safe Base64 is just normal Base64 with + and / replaced with - and _. We can use the Unix tr utility to swap those characters and decode the Base64.

$ echo 'AnLTOBDGpT9Efr0DJn4X4C7VcnwZF_ZRT6Vx8_pk_J8' | tr -- -_ +/ | base64 -d >actual.bin
base64: invalid input
$ hexdump -C actual.bin 
00000000  02 72 d3 38 10 c6 a5 3f  44 7e bd 03 26 7e 17 e0  |.r.8...?D~..&~..|
00000010  2e d5 72 7c 19 17 f6 51  4f a5 71 f3 fa 64 fc 9f  |..r|...QO.q..d..|
00000020

The invalid input warning here can be ignored since it relates to a missing = character on the end of the hash used to indicate the correct padding. If you are concerned, you can verify that appending an = to the hash removes the warning.

Comparing the hashes

We could manually compare the hex-dumps from the expected and actual hashes but there exists a utility whose sole purpose is to compare files so let’s use it:

$ diff -s actual.bin expected.bin
Files actual.bin and expected.bin are identical

The hashes match and so we may conclude that I have once had control of richwareham.com.

Summary

I’m not sure if keybase.io will take off as a concept but it’s a nice idea. The technology to cryptographically assert control over social media accounts has existed for some time but keybase.io wraps it up into a nice bundle. They also have some rather spiffy little web-tools which allow one to verify messages from people you follow (or “track”) on keybase.io and, if one trusts their Javascript crypto, you can even host an encrypted private key with them.

I already have a GPG key and I’d rather not upload it, even encrypted, to a random website. Recognising this, the entire keybase verification workflow can be done client-side on your machine using GPG. Although I signed the assertions of control hosted on keybase.io, I did so on my machine using GPG without ever sending the private key to them. This is the key advantage of public key cryptosystems and so it’s nice to see keybase.io supporting this.

I’m still waiting to see what the killer use-case for it will turn out to be.

Windows Development from a Linux Programmer’s Perspective

2014-09-17T00:00:00+00:00

I recently had to do a little bit of Windows development after having managed to avoid it for around seven years. This post documents my experiences from the point of view of an experienced software developer from a non-Windows background.

The precise reasoning behind why I needed to do some Windows development isn’t that important for this post but, for the curious, I’ve included a brief summary at the end of the post. The short-version: I wanted to write a server which would advertise itself over Zeroconf and stream depth frames from a Kinect for Windows 2 sensor over the network using ØMQ.

Don’t fight the inevitable

It’s a mistake to try and fight the common culture on a particular platform. My usual tool for writing research software is Python. It is an excellent language and ecosystem for writing scientific research software on Linux-like platforms but there is quite the impedance mismatch on Windows. I decided that it would be best to try and embrace the Windows development culture and write a Kinect 2 streaming server “the Windows way”. This also meant I would avoid the future problem of getting Python to talk to the Kinect for Windows SDK.

The “Windows way” in this case is undoubtedly .NET and Visual Studio. What follows are some of my experiences trying to determine what the Windows way is and following it. I should note that the last time I did any Windows programming was a long time ago in a different job and in a completely different field. It was also in C++. I’ve very little experience with .NET development. This was the project to learn. The code is available. Idiom-fixing pull requests are very welcome.

Visual Studio Express Edition

It’s madness to charge people for the privilege of adding value to your software platform so it’s nice to see that Microsoft make the express edition of Visual Studio available at zero-cost. It is, however, extremely confusing that you need to know that you need the “for Windows Desktop” version in order to actually develop programs; the similarly named “for Windows” edition being good only for writing software for both users of Windows Phone OS.

(Aside: perhaps Apple might want to consider quite how many times people need to pay a $99 bounty to write code for their OS, get their development tools, get something onto one of their phones, etc. Development on Apple used to be a dream in 2004. Ten years later it is a nightmare.)

Visual Studio itself is… OK. I don’t really like tools which hide things from you, particularly if you’re a software developer. Visual Studio skirts the boundary of “too much magic” but in general I didn’t find myself worrying where on disk a particular magic bit of configuration is stored.

I’m not really an IDE guy but I will admit that for an almost complete novice .NET developer, the code-completion is useful. The quality of the MSDN documentation, however, is pretty poor. One line summaries of methods, in particular, are inexcusable. Also the MSDN website is really… slow… to… browse. I’d rather have usage examples and discussion of APIs next to or within the reference. Having them scattered all over MSDN makes quickly learning about something pretty hard.

The MSBuild tool is a welcome change since the last time I touched Windows development. Being able to run tests, build code and even package up the results from the command line or automated scripts is an absolute essential.

C# and .NET

I’d say that C# is a very well designed language in that it is utterly unsurprising. Designing something new to be unsurprising is very difficult and it is a complement to C# to describe it as such. If you can get by in Object Pascal and Java then you’ll make yourself understood in C#. I suspect I was programming with a thick C++ accent but I don’t think I was being too offensive to the C# idiom.

I like the Go-style divorce between namespaces/modules and file names on disk; source code files are best organised for the convenience of the developer of a library whereas namespaces are best organised for the convenience of a consumer. Having this distinction recognised is welcome.

External dependencies

This has traditionally where Windows has been the most painful. Thankfully I only had three external dependencies.

The NetMQ .NET port of ØMQ is available through the NuGet package manager. NuGet is a solution which should’ve existed ages ago. It’s sad that it required a third-party Open Source project to plug such a gaping hole.
Apple’s Bonjour API is exposed via COM objects. This necessitates the installation of a MSI package to get all of Bonjour up and working but an advantage of COM is that one doesn’t need to hard-code or discover DLL paths in your build system.
The Kinect SDK itself is a bit of a pain. At the moment I don’t actually have access to a Windows 8 box and so I had to do some tricks to pull the DLLs out from the installer package directly since it flat-out refuses to install on any other version. (I with there was an option in MSI packages to only install a subset of the contents.) This is only necessary on my current development machine while I wait for a Windows 8 machine to become available. The Kinect .NET API lives in the global assembly cache (GAC) and so, again, I don’t need to worry about hard-coding paths to find DLLs.

The handling of dependencies was actually the part of this whole project where I was the most pleasantly surprised. NuGet has solved a lot of the problems which I used to have with Windows development.

Source control

Microsoft finally woke up to the fact that git is the source control system which the cool kids are using. And no one more obviously yearns to be one of the cool kids than Microsoft does at the moment. Visual Studio’s git integration is basic but functional. I didn’t try to see how easy it was to clone/push/pull/merge/etc from the GUI because I only used the GUI for simple “commit all the changes” type activity. For everything else, I just used git for windows and Git bash. There’s not much to say about git bash: it’s git and bash. Thankfully vim is included as well. I ended up using vim to edit things like READMEs. Visual Studio’s editor is overkill for editing text files and I just couldn’t get on with things like Notepad++. I’m also not fashionable enough to use Sublime Text. I suspect editors are just too personal a thing to change at the drop of a hat.

While we’re talking about git bash, the terminal (“console” in Micro-speak) is appalling. It’s 2014 and you should be able to a) resize the window and b) copy and paste without resorting to hidden menus. Even Windows PowerShell seems to have been hit with the same ugly stick.

Testing

The unit test framework in Visual Studio is perfectly acceptable. It’s nice that the test runner uses reflection to determine automatically which tests to run. (I’m used to such a thing with testing frameworks on Python.) It’s also nice that each test is run in its own domain meaning that silly mistakes like dangling threads are easy to pick up on.

The final code is less tested than I’d like. Only writing the server-side portion means that you’re missing one of the ends for end-to-end testing. That’s not Windows’ fault though.

Documentation tools

I think the poor focus of Microsoft on documentation is particularly apparent here. There is the most perfuctuary support for API documentation. Essentially you can write arbitrary XML in a comment abover your method, class, etc definition and have it concatenated together into an XML output file. There’s no HTML documentation tool built into Visual Studio and the solutions which exist are clunky at best.

I suspect that the people at Microsoft think that code-completion is a substitute for documentation.

Continuous integration

I love Travis CI. Having a free service which will observe a GitHub repository, check out any new commits and then try building and testing your software in a fresh VM is wonderful. Having had to set up a Jenkins server in a previous job I cannot tell you how nice it is to have a system like Travis. It’s brain-dead and simple but, like Unix, that is a feature in itself. It’s hard to do simple well and Travis does it excellently.

Travis CI only supports Linux test machines. (Or, at least, the free service only support Linux.) That’s not going to fly for Windows. Luckily someone clearly saw a gap in the market and set up AppVeyor. It’s not quite as nice as Travis but it’s also a lot younger. It’s also not quite as simple to use but it is opinionated. That means that you’ll have a nice time if you don’t fight the “Windows way”. A project which is laid out in the usual Visual Studio way will Just Build (TM).

All of my external dependencies are either NuGet-able or are packaged as re-distributable .msi instalers. After some Googling for the correct PowerShell incantation, getting Bonjour and the Kinect SDKs auto-installing on the test boxes was easy enough and the NuGet packages are downloaded automatically by MSBuild.

I chose to have the non-NuGet dependencies in a separate repository. Having big binary blobs in your source repo is a Bad Idea. It’s also a needless waste of bandwidth for someone who clones your repo when they already have the appropriate SDKs installed.

Packaging

This is a little frustrating. For some reason, Microsoft saw fit to not ship the support for packaging projects with the express edition of Visual Studio. There is only support for publishing a “ClickOnce” package. It’s OK as a mechanism but I’m reserving judgement until I actually try to deploy this software onto a separate machine.

GUIs

The way that Windows Forms does GUI designing is a little dirty. I prefer a more explicit separation between UI layout and logic. (WPF seems to be better in that regard but, from what Googling I performed, it looks like WPF is a bit of a dead-end for developing GUIs.) As it was, there would be the tiniest of tiny bits of GUI development for the server. I wanted most of the guts to live in a library (or “assembly” in Micro-speak) and have the GUI basically be a start/stop button and a log window. See below:

I’m a firm believer in separate modules and composable software and so the GUI shell is just that: a shell. The core logic is very small. There’s also a console version of the server in the repository which is actually how I run it most of the time. Since there is so little of it, having the GUI code be a bit ugly isn’t a problem.

Summary

If you are willing to do things the “Windows way” then writing software on Windows is relatively painless. The packaging and dependency management story is a lot better nowadays but it still is lacking far behind Linux. Debian has had apt-get for over 15 years now and NuGet is not a lot better than apt was back then.

It’s still too hard to set up a project in a way which encourages others to build it; you need to jump through some hoops to lay it out in a way where your development machine isn’t special and NuGet is only a partial solution to dependency hell. It is better than it was though and shows that the Microsoft ecosystem is becoming a little more friendly to the “Open Source” model of small projects shared widely.

Overall, I think this little experiment is a success and I’m well placed to add actual real Kinect support to my code once the Windows 8 machine arrives. With the CI system and GitHub, maybe I could even get a student to do it?

Addendum: the problem

Below I’ll describe the actual problem I was facing and why I decided to learn an entirely new platform, new language and new IDE to solve it.

Firstly, a little context: this year I will have some Masters’ degree students working on a robotic project which hopes to use a Microsoft Kinect for Windows version 2 sensor. (This is the one that comes with the XBox One console.) The project is a research project and, to make life easier for the students, we will be using the ROS to wire together the various algorithms we will be using and to control the robot arm.

When using ROS, the path of least resistance is to use Ubuntu Linux. While there are some very capable drivers for the original Kinect sensor, there are unfortunately not currently any drivers for the Kinect version 2 sensor which I consider stable enough to let students loose on. There is a reverse engineering effort which may bear fruit but, as of writing, it is not mature enough for our uses.

Being a fan of the simplest (a.k.a. “laziest”) solution, I decided that it would be best to have a dedicated Windows 8 computer with the Kinect plugged into it. (This is proving a challenge, however, since any spare machines at work tend to be running Linux. The number of Windows 8 machines is quite small.) The computer would then stream the depth buffer into the rest of our Linux-based pipeline. The precise nature of the research project means that sub-100 millisecond latency isn’t really required.

Similarly it isn’t a requirement that the depth stream be completely uncompressed. A quick back-of-envelope calculation reveals that a 1080p60 16-bit depth buffer is around 400 megabytes per second. This is far more than the network at work can deal with. Some brief experimentation showed that one can non-linearly quantise a depth buffer to 8-bit precision using something like the μ-law algorithm and then compress it with JPEG in real-time to get the required bandwidth down to a more manageable 1 or 2 megabytes per second.

All of this experimentation was done in Python and the code is available if you want to take a look. The experimental code also includes a simple client/server pair. The server advertises itself over Zeroconf and uses ØMQ to stream depth frames over the network. The idea being that we don’t want to spend time configuring IPs, subnet masks, etc, etc. Ideally we can just plug the Windows 8 machine into the ROS machine with a network cable and “magic” will happen. Every required configuration option is another potential point of failure.

So far, so good. All of the development at this point had been done in Linux with Python and a little bit of code which mocked up the depth stream I’d be getting from the Kinect. This allowed some rapid development and the ability to actually test the client and server bits by having the test suite start both on the same machine and provide a “mock” Kinect device.

I was vaguely aware that the received opinion is that Python development on Windows is a little hard but when I finally got around to finding a Windows box to try installing the software on I found out how bad it was. Firstly, the “official” Python is almost unusable. For anything other than the bleeding-edge Python 3.4 release, installing simple tools like pip is a pain. Getting all the pieces in line to actually pip install a C-backed module like pyzmq is even worse. Continuum Analytics’ Anaconda installer makes life a lot easier but it still requires quite an extensive amount of fiddling to get a machine set up for development.

This goes against my natural inclination against “special snowflake” machines. One of the advantages of having pip-like tools in the first place is that one can use generic VMs in the cloud to build-test your code on each commit. Then you have an explicit, repeatable and automated recipe for turning your GitHub repository into installed software. Having to set up a special development machine will come back to haunt you the first time you need to get someone else to build (or install) your software.

Counting Sneezes from the Web

2014-09-05T00:00:00+00:00

This post is also available as an IPython notebook which may be downloaded or viewed online.

The sneeze count website catalogues one person’s sneezes since 2007. A work colleague asked how difficult it would be to extract a list of timestamps from the site. Could we then plot sneezes over time? This post shows how easy this is to do using Python.

Fetching the data

Conveniently there is an RSS feed available for the site which can be fetched via Python’s built in urlopen function:

from urllib.request import urlopen
print('...')
print('\n'.join(urlopen('http://sneezecount.joyfeed.com/feed').read().decode('utf8').splitlines()[20:30]))
print('...')

...
	<item>
		<title>Four thousand and nineteen</title>
		<link>http://sneezecount.joyfeed.com/four-thousand-and-nineteen/</link>
		<comments>http://sneezecount.joyfeed.com/four-thousand-and-nineteen/#comments</comments>
		<pubDate>Fri, 22 Aug 2014 12:41:25 +0000</pubDate>
		<dc:creator><![CDATA[Peter]]></dc:creator>
				<category><![CDATA[Sneezes]]></category>

		<guid isPermaLink="false">http://sneezecount.joyfeed.com/?p=8608</guid>
		<description><![CDATA[Seesaw, Twycross Zoo Moderate to strong &#8220;I&#8217;m not heavy enough. And you&#8217;re too heavy.&#8221;]]></description>
...

There is also a paged query parameter whch lets one get the next page of results:

from urllib.request import urlopen
print('...')
print('\n'.join(urlopen('http://sneezecount.joyfeed.com/feed?paged=2').read().decode('utf8').splitlines()[20:30]))
print('...')

...
	<item>
		<title>Four thousand and nine</title>
		<link>http://sneezecount.joyfeed.com/four-thousand-and-nine/</link>
		<comments>http://sneezecount.joyfeed.com/four-thousand-and-nine/#comments</comments>
		<pubDate>Sat, 09 Aug 2014 17:09:39 +0000</pubDate>
		<dc:creator><![CDATA[Peter]]></dc:creator>
				<category><![CDATA[Sneezes]]></category>

		<guid isPermaLink="false">http://sneezecount.joyfeed.com/?p=8587</guid>
		<description><![CDATA[Dining room, Malt Barn, Brize Norton Moderate Accepting the offer of an elderflower Prosecco]]></description>
...

Python also has a built in XML parsing module, the ElementTree module. Usage is, ahem, elementary:

import xml.etree.ElementTree as ET

def parse_xml_url(url):
    tree = ET.parse(urlopen(url))
    return tree.getroot()

root = parse_xml_url('http://sneezecount.joyfeed.com/feed?paged=2')
print(root)

<Element 'rss' at 0x7f958d5ccf70>

An RSS feed encodes each post with an item tag and each publication date within the item as a pubDate. A quick and dirty hack to extract all dates from an RSS feed is simply to list all the pubDate elements. According to WordPress’s documentation, the pubDate tages are RFC822 dates. Python has a module to deal with that too! The email.utils module has a parsedate function. The value can be passed directly to time.mktime() to get a timestamp.

from email.utils import parsedate
import time

def extract_dates(root):
    return list(time.mktime(parsedate(elem.text)) for elem in root.iter('pubDate'))

print(', '.join(str(t) for t in extract_dates(root)))

1407600579.0, 1407520308.0, 1407389858.0, 1407381140.0, 1407344647.0, 1407255800.0, 1407141500.0, 1407069426.0, 1407009164.0, 1406970003.0

To get dates for a given feed, therefore, we need simply page through the feed starting at page 1 until we get a result with no items.

from urllib.parse import urljoin
from urllib.request import HTTPError

def dates_from_feed_url(url):
    paged = 0
    all_dates = []
    while True:
        paged += 1
        page_url = urljoin(url, '?paged={0}'.format(paged))
        try:
            dates = extract_dates(parse_xml_url(page_url))
        except HTTPError:
            # Interpret a HTTP error (e.g. 404) as us reaching the end of the list
            break
        all_dates.extend(dates)
        if len(dates) == 0:
            break
    return all_dates

# Test with a feed URL for a single month:
ts = dates_from_feed_url('http://sneezecount.joyfeed.com/2014/08/feed')
print(', '.join(str(t) for t in ts))

1408707685.0, 1408628545.0, 1408563641.0, 1408537639.0, 1408439866.0, 1408275986.0, 1408256591.0, 1408167669.0, 1408131465.0, 1407820127.0, 1407600579.0, 1407520308.0, 1407389858.0, 1407381140.0, 1407344647.0, 1407255800.0, 1407141500.0, 1407069426.0, 1407009164.0, 1406970003.0, 1406870597.0

We can get all dates for all time by using the full feed URL:

timestamps = dates_from_feed_url('http://sneezecount.joyfeed.com/feed')

If we look at the first page of the website, as of writing there are four thousand and nineteen sneezes. Let’s just ckeck that we’ve got them all:

print('Fetched {0} sneeze timestamps'.format(len(timestamps)))

Fetched 4019 sneeze timestamps

Plotting

Let’s use matplotlib to plot the timestamps. We first use some IPython magic to load the pylab environment:

%pylab inline
rcParams['figure.figsize'] = (14,9) # Set the default figure size

Populating the interactive namespace from numpy and matplotlib

plot(timestamps, np.arange(len(timestamps)))

title('Sneezes over time')
xlabel('Timestamp')
ylabel('Sneezes')

grid('on')

Possibly more interesting is a histogram of intervals between sneezes:

# Use three hour bins out to 5 days
hist(np.diff(np.asarray(timestamps) / (60*60)), bins=np.arange(0, 5*24, 3))
title('Histogram of times between sneezes')
xlabel('Hours')
ylabel('Count')
grid('on')

Summary

In this post we showed how the Python standard library has all the tools we require to scrape a website and plot some interesting figures using data from it.

HMAC: How I securely deploy to this site from Travis CI

2014-09-02T00:00:00+00:00

Overview

This website has dynamic content like my link shortener and static content like this blog post. I have Travis CI set up to build my website’s static content whenever a new commit is made to the github repository. My website has a very simple method of updating itself: HTTP POST-ing a zip file to a magic URL will unzip that file over the static file directory. This post is how I try to secure that mechanism while preserving the simplicity of a “deploy via cURL” model.

The problem

This website’s source is hosted on github although the site itself is hosted on OpenShift. If you look at the site’s source code you’ll see that it’s just a pretty simple Python web application using the flask web framework for Python.

Another thing you might notice is that none of the content is in that repository. That’s because having a full dynamic website for my simple little corner of the web is overkill. Instead I use jekyll to generate a static website from a pile of Markdown files and HTML templates. All of this magic is hosted in a separate repository.

Why separate the repos like this? Well there is some non-trivial downtime associated with pushing a new version of an OpenShift webapp. Previously I had a lot of pre-build and build hook magic which would install jekyll, install bower, build the static site, etc which was inappropriate for what is, essentially, a very simple Python webapp.

By separating out the static content from the dynamic content I can configure the webapp to simply serve any URL is doesn’t know about from the static directory. All very nice, clean and modular but how do I update the static files on my website when I make a change?

I tend to subscribe to the “Unix laziness” philosophy which holds that spending some time automating a solution will make things a lot easier in the long run. The repo holding the static site content is monitored by a corresponding Travis-CI job. Travis CI is very good at installing the various Node.js, Ruby and JavaScript dependencies to build the site and so we should make use of its strengths.

So my problem is: I have a webapp serving static content from a directory on the filesystem and I have Travis with a freshly built pile of HTML, CSS, etc. How do I get the new content over to the webapp in a secure way? “Secure” here means that anyone with read access to the webapp repository, static content repository and Travis CI configuration file shouldn’t be able to scribble all over my website(!)

Simple is better

I wanted a very simple way to deploy a new bundle of static content. Initially I thought github releases would be the way forward. My plan was to make Travis create a new release associated with the static content repo on each successfully built commit to master. Unfortunately github is geared up to make releases only when there’s an associated tag. Having to tag each and every commit seemed ugly and, as usual when things seem ugly, it’s worth taking a step back and thinking about the problem. A build of the static content isn’t really a “release” in the github sense; once it’s deployed I don’t need to keep it around forever more.

After a bit more thought, I decided to make the deployment process a kind of “fat webhook”. When Travis built the static content, it would zip it up and then HTTP POST that file to some URL. (In this case, https://www.richwareham.com/static-content.) The Python webapp would take the payload and unzip it into the static files area. Handily Python has a zipfile module which can handle zipped up data directly.

So far, so simple. We only need to write two functions. The first is a utility function to take a destination directory, ensure it exists and to unzip the contents of a file object to the directory:

def update_static(destdir, fileobj):
    logging.info('Extracting new static site...')
    z = zipfile.ZipFile(fileobj, 'r')
    if not os.path.exists(destdir):
        os.makedirs(destdir)
    z.extractall(destdir)

The second is a flask handler for the URL:

@app.route('/static-content', methods=['POST'])
def update_static_content():
    update_static(STATIC_SITE_DIR, request.files['archive'].stream)
    return 'OK'

All Travis now needed to do was to zip up the new static site content into a file called, for example, site.zip and POST it to my website using something like cURL:

$ curl -i -F "archive=@site.zip" https://www.richwareham.com/static-content

This works beautifully but it is massively insecure: anyone can modify my site’s content. To overwrite or create a file on my site an attacker need only POST a zipfile to a URL which is plainly listed in the Tavis configuration file.

Shhh… It’s a (shared) secret

HMAC or “hash-based message authentication code” is a standard technique to compute a token given a message and a secret value. Anyone in possession of the same secret value can then verify that the message is authentic. (Or, at least, was generated by someone who also knew the secret.)

In principle it’s quite simple: the deploy script on Travis combines the zip file and secret together and takes a hash. It then sends the zip file and computed hash to the webapp. The webapp computes its own version of the hash from the given zip file and its own copy of the shared secret. If the hashes match then I know the person who sent me the zip file knows the shared secret.

HMAC itself is a little more subtle than this but fortunately there is a implementation in the Python standard library. We’re free to choose our own hash algorithm and so, as a good default, I chose SHA256.

The complete Python code to compute a HMAC from an open file object is very small:

import hashlib
import hmac
# ...
digest = hmac.new(shared_secret, file_obj.read(), hashlib.sha256)
token = digest.hexdigest() # hex-digit string containing HMAC

On the Travis side, this is wrapped up into a small calc_hmac.py script. On the webapp side we need only create a check function and modify our request handler to use it:

def check_hmac(fileobj, secret, provided_digest):
    # Compute expected digest
    digest = hmac.new(secret, fileobj.read(), hashlib.sha256)

    # Does this digest match? Use compare_digest() because it it constant time.
    # (Timing attacks are subtle. Crypto is hard :(.)
    return hmac.compare_digest(provided_digest, digest.hexdigest()):

@app.route('/static-content', methods=['POST'])
def update_static_content():
    # Get a file object pointing to the archive sent with the request
    archive = request.files['archive'].stream

    # Check the HMAC which was provided matches the shared secret
    archive.seek(0)
    hmac_ok = check_hmac(archive, SHARED_SECRET, request.values['hmac'])
    if not hmac_ok:
        return abort(403)

    # Unzip the static files
    archive.seek(0)
    update_static(STATIC_SITE_DIR, archive)
    return jsonify({ 'status': 200, 'message': 'OK' })

We’re nearly there. All that’s needed now is a way to get the shared secret to Travis and the OpenShift webapp in a secure manner.

Both OpenShift and Travis have the ability to securely store “secret” information. Travis allows you to store encrypted environment variables and OpenShift recently added secure environment variable support.

It’s straightforward to arrange for both Travis and OpenShift to have the same shared secret exposed via the STATIC_SITE_SECRET variable using a little bit of bash:

$ SECRET=`dd if=/dev/urandom count=1024 | sha1sum`
$ cd $OPENSHIFT_APP   # cd to checkout of OpenShift app repo
$ rhc set-env STATIC_SITE_SECRET=$SECRET
$ cd $STATIC_SITE     # cd to checkout of static site repo
$ travis encrypt --add STATIC_SITE_SECRET=$SECRET

Note that this can be automated via a cron job and so the shared secret need not stay constant. The Python webapp can now just use os.environ['STATIC_SITE_SECRET'] for the shared secret.

Summary

Above, I outlined the method I use for pushing new static site content to my website without requiring a time-consuming re-deployment. The astute among you will notice two potential problems which will need to be considered at a later date:

The static site contents are just unzipped atop the old ones and so it isn’t possible to delete files.
Because the zip file is just unzipped over the static files directory, the update isn’t atomic. A better solution would be to unzip files to a temporary directory and use some symlink magic to update them atomically.

For my tiny website neither of these problems are major at the moment although I may re-visit the solution at a later date.

Also, crypto is hard. Even though this HMAC based system is about a simple as it can get, there may still be ways for a sufficiently motivated attacker to push content to my website. I’m defending against the trivial “LOL his website accepts any zip thrown at it”-style attacker, not someone willing to put real effort into defacing my website :). That being said, if you do find some obvious way to subvert this method, I’d be interested to hear it. I’m always willing to learn more about crypto because if anyone claims they know it all, they’re lying or mad.

Can we predict England’s traffic?

2014-09-01T00:00:00+00:00

In September 2014, we presented a poster on some of the ancillary research I am doing as part of my latest project supported by the Centre for Sustainable Road Freight. A PDF of the poster is available to download for those that are interested. The source for the poster is hosted on the University git service.

Overview

The poster focusses on a single part of our wider research project on gathering traffic flow data. Part A of the project aims to infer traffic flow directly from video streams but knowing the current state of the world is only one part of the problem; we are also investigating what techniques can be used to predict the near future.

Data wrangling

In a previous post I outlined some of the basic steps required to get hold of the UK Highways Agency’s real time data feed and wrangling it in Python. I recommend reading that post if you’re interested in the details of how the data is fetched and parsed.

What data is available?

The UK Highways Agency publish an XML document four times an hour which gives the latest measurements for stretches of roads (or links) in England. A link has a direction; each carriageway of a road will appear as a separate link but a link may have multiple lanes. This information includes the current traffic flow (vehicles/hour), mean speed (km/hour) and occupancy (%). An occupancy of 100% means that the road is fully occupied, nose-to-tail, on each lane.

How do you get it?

The XML document itself is available via a public URL which may simply be fetched. We have developed an automated system based in the Cloud which fetches this document every 15 minutes, parses the XML and strips the information we currently do not use. The XML stream itself is 120MB an hour and so this “data reduction” step is important. We have a full archive of the XML stream for around six months which is nearly 150GB in size even when compressed.

How do you store it?

Our system in the cloud does two things. Firstly it provides an archive of the day’s traffic flows, occupancies and speeds which is downloaded and integrated into a local SQLite database on our site. This database is the source for the time-series data for individual links. The cloud based system also translate the current state of England’s traffic into a JSON-formatted document which can easily be consumed by our web-based visualisation tool (XML, despite its web-heritage, is rather inconvenient as a file format for today’s web.) On a more technical note, our cloud service also supports CORS (Cross-Origin Resource Sharing) which is again a technical requirement for making use of the data in our visualisation platform.

Analysis

Our central assumption is that all links exhibit similar patterns over the course of a day. This matches our intuition that we expect morning, evening, daytime and night time behaviours to be different for any given link but all links will have some degree of, for example, “morning rush hour”.

Traditionally this data reduction step would be performed by taking a large number of “training” samples of daily traffic behaviour and using some projection-based technique such as Principal Component Analysis to extract a small set of “basis vectors” or “components” which can be scaled and summed together to approximate any of the training samples with minimal error.

These techniques are well known and often very useful as initial data reduction step. The problem is that the basis vectors produced are not guaranteed to be positive. Usually this is not a problem but in traffic analysis one of the few things we can say with certainty is that speeds, occupancies and traffic flows are positive.

Taking a naïve PCA of 40,000 traffic flow daily samples results in these components:

The only directly interpretable component is the first cone which corresponds to the normalised mean of all traffic flows. The rest are not very illuminating. In the case of traffic analysis one has a lot of prior knowledge available which does not obviously map onto these components.

Instead of PCA, we can make use of a relatively novel technique: non-negative matrix factorisation. In this technique one forces all of the components to be positive. The downside is that the factorisation is no-longer unique. To combat this, one usually factorises while minimising some “sparseness measure”. In our case we factorise while trying to keep the components as “peak-like” as possible. Fortunately the NMF implementation we are using, scikit-learn, directly supports this. The non-negative fundamental components look like this:

It is now far easier to directly interpret these components in terms of “morning rush hour”, “night time flow”, etc.

By resolving our training flows onto these components we end up with 5 components which describe the flow on one link over an entire day rather than one number for each quarter hour sample. This is around a 5% data reduction.

We may now reason using these components to derive “typical” component values for each flow for each weekday. Typical results from this projection are shown in the poster.

Prediction

Data reduction is all well and good but it is all for naught if the reduced data doesn’t represent the actual data. To this end we investigated how well our “typical week” will predict a new week which wasn’t present in our training data. To add some challenge, the new week included a Bank Holiday Monday which wasn’t present in the training set.

Reconstructing our “typical week” for each link resulted in surprisingly good performance. Remember that this is the simplest form of prediction; we assume that each week is like the typical week. Even with such a simple model our prediction has a median relative error of only 7%. Plotting a scatter chart of predicted flow versus actual flow showed a tight adherence to the perfect 1:1 correspondence line. The bank holiday was clearly visible in the predicted output.

Summary

We investigated whether non-negative matrix factorisation of traffic flows in England could provide large scale data reduction with low error while providing basis components which retained interpretable. We showed that even reducing data to 0.25% of the original we could predict future flows with a median relative error of 7%.

Given these results we intend to proceed with traffic prediction working directly on the reduced traffic data which we believe to be a more tractable problem than dealing with the full dataset directly.

References

Non-negative Matrix Factorization (Wikipedia references)
Principal Component Analysis
Scikit-learn: a Python package for machine learning

Meet Emma, our latest robot

2014-08-01T00:00:00+00:00

Our lab has a new member. For next year’s Masters projects, we are partnering with a local tech consultancy to develop a robotic project. We’ll be using the CrustCrawler AX-18A Smart Robotic Arm in the project. Ultimately we’re going to have four: one per student. Before that, though, we need to make sure that we know what bits to get. She took quite a while to put together but now Emma lives!

In case you’re wondering, one possible project aim is to peel a banana with a scalpel. The arm is, somewhat playfully, named after Mrs Emma Peel from The Avengers.

Rich Wareham

Getting Started with Electronics

Introduction

The essentials

Less essential essentials

Knowledge

A first project

A stretch project

YouTube: let a thousand lectures bloom

Conclusions

Fonts from old ROMs with Python

Introduction

Implementation

Conclusion

Optical Character Recognition in Python: Transcribing the Turing code

Extracting images for each symbol

Computing feature vectors

Matching symbols

Conclusions

Making a Little Planet Panorma using Python and Scikit Image

Loading the initial panorama

Image warping via scikit-image

The “little planet” projection

Saving the result

Summary

Verifying identity on social media with keybase.io and command-line tools

Why is this useful?

So, how does it work?

Importing the public key

Getting the assertion from DNS

What the assertion actually is

Getting the expected hash

Getting the actual hash from DNS

Comparing the hashes

Summary

Windows Development from a Linux Programmer’s Perspective

Don’t fight the inevitable

Visual Studio Express Edition

C# and .NET

External dependencies

Source control

Testing

Documentation tools

Continuous integration

Packaging

GUIs

Summary

Addendum: the problem

Counting Sneezes from the Web

Fetching the data

Plotting

Summary

HMAC: How I securely deploy to this site from Travis CI

Overview

The problem

Simple is better

Shhh… It’s a (shared) secret

Sharing the secret

Summary

Can we predict England’s traffic?

Overview

Data wrangling

What data is available?

How do you get it?

How do you store it?

Analysis

Prediction

Summary

References

Meet Emma, our latest robot