Martijn Koster's Pages

Installing Distributed Solr 4 with Fabric

2013-05-22T00:00:00+00:00

I wrote an article “Installing Distributed Solr 4 with Fabric” about deploying SolrCloud with Fabric. Code is on github.com/LucidWorks/solr-fabric.

My VM strategy and server worked great for developing/testing this!

A Mini-ITX server

2013-05-12T00:00:00+00:00

I put together a small server today, to serve as DVR server for a CCTV system.

Requirements

This server connects to an office LAN and a separate LAN for the IP cameras, so I needed two network interfaces. It lives in a small, minimally ventilated room, so I wanted a small form factor, but with sufficient airflow and enough space to work with comfortably. The room is used as an office, so I wanted minimal noise. The software is used remotely; there is no permanently connected screen/keyboard, so I wanted remote management. I wanted virtualisation support (VT-x and VT-d), but have no need for too much CPU power. This machine is not mission-critical, so there is no need for RAID.

Parts

To satisfy the minimal size/noise requirements I chose a Mini-ITX case. There are not many Mini-ITX motherboards that offer dual ethernet, virtualisation, and remote management. I’ve been intrigued by the DQ77KB’s unusual features for some time, and after some further research, and some advice from mini-itx.com (thanks Ewan!), I settled on the following specification:

Silverstone SUGO SG05B Mini-ITX Chassis
Intel DQ77KB Socket 1155 Thin-ITX Motherboard
Intel i5-3470S 3rd Gen Core 2.9GHz CPU
External 160W AC Adapter 19V
Akasa 7106HP Low Profile Heatsink
Kingston 8GB DDR3 1333 SODIMM
Intel 525 Series 60GB mSATA Solid State Hard Drive, from overclockers.co.uk
3.5in SATA HD, re-purposed

Observations

Of note:

For remote management, the CPU needs graphics and Intel vPro Technology support. The i5-3470S was the cheapest CPU with those features that my supplier stocked.
The DQ77KB is a Thin-ITX board that needs a 19V PSU. I picked an external supply to keep the heat out of the case, keep the noise low, and allow for easy future replacement.
I selected the SG05B for its good ventilation (large slow chassis fan), sturdiness (steel body) and internal space.
The SG05B comes with a PSU that I do not need. The recently announced a SG05-Lite comes without a PSU, but that is not yet widely available in the UK. I simply removed the PSU, and covered the case hole with some hexagonal mesh scavenged from a junk ATX PSU and cut to size.
The DQ77KB has mSATA support, which I like for keeping the build clean a simple. The mSATA board fits into a slot at one end, and is supported by screwposts on the other. The screws were very tight, and unscrewing actually unscrewed the posts; I had to hold them with pliers.
The low-profile heatsink was not strictly required, but I like how quiet this model is, and how its design encloses the fan which helps keep it away from cables.
I’m not delighted by the Thin-ITX power specification: having to source a specific 19V supply is a hassle, and the power socket has no lock and could easily be accidentally knocked out. For this particular build, the low-profile of Thin-ITX didn’t matter; I would have been quite happy with a standard Mini-ITX board.
The total bill of materials is higher than a typical Mini-ITX build; if you can drop some of the requirements, you can save money by choosing a smaller case, a more mainstream board frmo a more cost-conscious manufacturer, and a cheaper CPU.

Some photos:

Remote Management

Setting up remote management was more involved than say my SuperMicro server with IPMI. There is good information in these two sources:

How-To Geek “How to Remotely Control Your PC (Even When it Crashes)”
Michael Kuron: “Using Intel AMT’s VNC server”

I used a separate IP address for AMT, with the same ethernet interface also using a second address for normal LAN access.

The biggest head-ache was the display resolution: Ubuntu (grub and the framebuffer) happily chose the largest resolution supported by the Intel graphics card, which ended up making my Chicken-of-the-VNC window for AMT too large to use on my Macbook Pro. After various fruitless experiments with GRUB_GFXMODE/GRUB_GFXPAYLOAD_LINUX in the /etc/default/grub, I eventually fixed that by specifying:

GRUB_CMDLINE_LINUX="video=1024x768-24"

See the ArchLinux page about KMS for details.

Software Install

I upgraded the BIOS to the latest KBQ7710H.86A.0051 (from KBQ7710H.86A.0038), using the F7 method and a USB stick.

Ubuntu 12.04 LTS installed without any problems.

It takes 25 seconds from Power-on to login, and from reboot to login, including 2 seconds of grub delay.

kvm-ok reports “KVM acceleration can be used”, VirtualBox runs fine.

Conclusion

It’s small, quiet, powerful; good stuff.

Cloning VMs with KVM

2013-03-24T00:00:00+00:00

Now that I have my shiny new server, it needs virtual machines.

I have two usecases:

long-running VMs to run services (such as OpenLDAP, Postfix, Jenkins), to do development, and be the target of ongoing deploys. These should be isolated, dependable, and have fast and consistent performance.
short-lived VMs to run single-node or multi-node tests. These should be quick to bring up in a consistent known state, and scriptable.

I use KVM, because it is open source, part of the kernel, widely supported and reliable. I’ve run it on a colo for years, and various ISP use it for production cloud platforms (ByteMark, Dutch Cloud, Digital Ocean). I manage it with virt-manager and script it with libvirt.

Not everything is rosy:

Documentation is scattered.
Virt-manager’s UI is basic.
Libvirt’s snapshot support is incomplete:
- “snapshots of inactive domains not implemented yet”
- “revert to external disk snapshot not supported yet”
the virt-clone tool canot write to a fresh LVM volume (“Clone onto existing storage volume is not supported”).
on Ubuntu AppArmor gets in the way when you manage multiple snapshot image files (see this Launchpad bug).
libvirt’s use of XML for domain configuration is a bit annoying to script.
QCow2’s internal snapshots take longer than I would like; about 10 seconds instead of 1 second for external snapshots.
and whenever you do any virtualisation, networking details are always fiddly.

So here is the workflow that I settled on:

Create a base image with a standard OS install
For the long-running VMs, I use separate LVM volumes to serve as raw virtual disk, clone a base image, and configure the guest. That takes under a minute.
For the short-lived VMs, I convert the disk of a long-running VM to qcow2 to serve as a backing image, and then create qcow2 images for each domain that use this backing store, configure that guest, and then create another backing store based on that, which is what the domain is configured to use. Rolling back is then simply a matter of re-creating the last qcow image. That takes a few seconds.

I’ll illustrate these in more detail.

Base Image Creation

I like to install the base from a GUI, using the standard installer, and complete it manually, so that it matches what users will see.

I run vn4server, and connect from my workstation. I install virt-manager, define a storage pool (my /dev/vg_vms LVM volume group), then create an 8G image base on ubuntu-12.10-server-amd64.iso, and name that domain ubuntu-base-vm.

For disk partitioning I use “Guided – use entire disk” rather than my usual “Guided – use entire disk and set up LVM”. There are three reasons for this:

I don’t really need LVM on these fairly small virtual disks
the installer names the volume group after the host, which will look strange on the clones
the volume group ends up in an extended partition, where virt-resize doesn’t resize logical volumes

In the Software selection I pick “OpenSSH server”.

Finally I login to the console, and add my ssh public key to .ssh/authorized_keys so that I can login to clones remotely later, and do an aptitude update; aptitude upgrade for good measure.

At some point I might swith to virt-install for this step.

Network Preparation

First, I’m making a list of VM names, IP addresses and MAC addresses. I’ll later use this list to configure the network interface of the KVM domain configuration, and the networking configuration files in the guest. For convenience I match vm names with ip addresses, such that vm111 is on 192.168.0.111. I wrote a little script that generates the address and the MAC (per this tip). For long-term VMs you can just edit the resulting file and change the name.

#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# generate vm network range

import virtinst.util
for num in range(110, 130+1):
    print("vm{0}\t192.168.0.{0}\t{1}".format(num, virtinst.util.randomMAC()))

which I can use like this:

python generate-ips.py  > ips.txt

and produces:

$ head -n 2 ips.txt 
vm110   192.168.0.110   00:16:3e:39:a5:53
vm111   192.168.0.111   00:16:3e:1e:0b:51

The instructions in this document do not rely on DNS configuration, but it is nice to give your VMs DNS names. On my LAN I use OpenWRT, and I can generate the configuration for its /etc/config/dhcp:

#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# generate openwrt /etc/config/dhcp config

import sys
for line in sys.stdin:
    (ip, name, mac) = line.split()
    print("config domain\n\toption name '{0}'\n\toption ip '{1}'\n\n".format(name, ip))
    print("config host\n\toption mac '{0}'\n\toption name '{1}'\n\toption ip '{2}'\n\n".format(mac, name, ip))

run like:

python generate-openwrt.py < ips.txt

and copy/paste into my router.

Long-term VMs: Clone to raw LVM guests

Here are the steps to do a tichk provisioning of a VM. I’ll use bash variable $VM for the VM name.

VM=vm111

First create the volume, of the same size as the original. For example, for a VM named vm111, I create a volume named vms-vm111 in the volume group vg_vms:

size=`sudo lvs -o lv_size --unit=b --noheadings /dev/vg_vms/ubuntu-base-vm | sed 's/^ *//'`
echo size=$size
sudo lvcreate --size=$size --name=vms-$VM vg_vms

Alternatively you can specify a large size, e.g.:

sudo lvcreate --size=20G --name=vms-$VM vg_vms

Next, I copy the base image:

sudo virt-resize --expand sda1 \
    /dev/vg_vms/ubuntu-base-vm /dev/vg_vms/vms-$VM

The ‘sda1’ parameter indicates the partition inside the guest that should be exanded; in this case the root partition.

Now we need to create a KVM domain to use that disk. I can copy the XML definition of the base image, and update the device path, the MAC address, and generate a new UUID. To make that easier, I use this python script:

#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# modify-domain.py -- modify a KVM domain
#
# Copyright (C) 2013 Martijn Koster
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:  The above copyright notice and
# this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import re, sys, uuid
from lxml import etree
from optparse import OptionParser

parser = OptionParser()
parser.add_option("--name")
parser.add_option("--new-uuid", action="store_true")
parser.add_option("--device-path")
parser.add_option("--mac-address")
(options, args) = parser.parse_args()

tree = etree.parse(sys.stdin)

if options.name:
    name_el = tree.xpath("/domain/name")[0]
    name_el.text = options.name

if options.new_uuid:
    uuid_el = tree.xpath("/domain/uuid")[0]
    uuid_el.text = str(uuid.uuid1())

if options.device_path is not None:
    if options.device_path[0] is not '/':
        sys.exit("device_path is not an absolute path")
    source_el = tree.xpath("/domain/devices/disk[@device='disk']/source")[0]
    source_el.set('dev', options.device_path)
    if re.match('.*\.qcow2$', options.device_path):
        driver = 'qcow2'
    else:
        driver = 'raw'
    driver_el = tree.xpath("/domain/devices/disk[@device='disk']/driver")[0]
    driver_el.set('type', driver)

if options.mac_address is not None:
    if not re.match("([0-9a-f][0-9a-f]:){5}[0-9a-f][0-9a-f]", options.mac_address):
        sys.exit("{0} is not a valid MAC address".format(options.mac_address))
    mac_el = tree.xpath("/domain/devices/interface[@type='bridge']/mac")[0]
    mac_el.set('address', options.mac_address)

print(etree.tostring(tree, pretty_print=True))

so that I can do:

mkdir -p tmp
virsh dumpxml ubuntu-base-vm > tmp/ubuntu-base-vm.xml
mac=`egrep "^$VM"'\s' ips.txt | awk '{print $3}'`; echo $mac
python ./modify-domain.py \
    --name $VM \
    --new-uuid \
    --device-path=/dev/vg_vms/vms-$VM \
    --mac-address $mac \
    < tmp/ubuntu-base-vm.xml > tmp/$VM.xml
virsh define tmp/$VM.xml
virsh dumpxml $VM

Finally, we need to configure the guest’s networking details. The virt-sysprep tool can help with that, but doesn’t regenerate openssh keys, or /etc/hosts. So I wrap it with some scripting. I use some some templates:

For the networking:

mkdir -p templates
cat > templates/network-interfaces <<NET
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address IP_ADDRESS_GOES_HERE
    network 192.169.0.0
    netmask 255.255.255.0
    broadcast 192.168.0.255
    gateway 192.168.0.1
    dns-nameservers 192.168.0.1
    dns-search vlab1.stalworthy.net
NET

The hosts file:

cat > templates/hosts <<HOSTS
127.0.0.1   localhost
IP_ADDRESS_GOES_HERE   VM_NAME_GOES_HERE

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
HOSTS

And a script to run in the host:

cat > templates/configure.sh <<SCRIPT
#!/bin/bash
# Run in the host, with the cwd being the root of the guest

set -x
cp tmp/network_interfaces.VM_NAME_GOES_HERE etc/network/interfaces
cp tmp/hosts.VM_NAME_GOES_HERE etc/hosts

# re-generate the keys. Letting virt-sysprep remove the keys
# is insufficient, and they don't get automatically regenerated
# on boot by Ubuntu. A dpkg-reconfigure fails for some reason,
# and doing a boot-time script is overkill, so just do it now explicitly.
rm etc/ssh/ssh_host_rsa_key etc/ssh/ssh_host_rsa_key.pub
rm etc/ssh/ssh_host_dsa_key etc/ssh/ssh_host_dsa_key.pub
rm etc/ssh/ssh_host_ecdsa_key etc/ssh/ssh_host_ecdsa_key.pub
ssh-keygen -h -N '' -t rsa -f etc/ssh/ssh_host_rsa_key
ssh-keygen -h -N '' -t dsa -f etc/ssh/ssh_host_dsa_key
ssh-keygen -h -N '' -t ecdsa -f etc/ssh/ssh_host_ecdsa_key
SCRIPT

Now we can use those templates to generate host-specific versions, and prepare the image:

ip=`egrep "^$VM\s" ips.txt | awk '{print $2}'`; echo $ip
sed -e "s/IP_ADDRESS_GOES_HERE/$ip/g" -e "s/VM_NAME_GOES_HERE/$VM/g" < templates/hosts > tmp/hosts.$VM
sed -e "s/IP_ADDRESS_GOES_HERE/$ip/g" -e "s/VM_NAME_GOES_HERE/$VM/g" < templates/network-interfaces > tmp/network-interfaces.$VM
sed -e "s/IP_ADDRESS_GOES_HERE/$ip/g" -e "s/VM_NAME_GOES_HERE/$VM/g" < templates/configure.sh > tmp/configure.sh.$VM
chmod a+x tmp/configure.sh.$VM
sudo virt-sysprep -d $VM \
  --verbose \
  --enable udev-persistent-net,bash-history,hostname,logfiles,utmp,script \
  --hostname $VM \
  --script `pwd`/tmp/configure.sh.$VM

Now the guest is ready and can be started:

virsh start $VM

Now that you’ve gone through this once, and the templates are in place, you can use this script to make this a one-liner:

$ time ./clone.sh vm112
...
Domain vm112 started

real    0m55.401s

Which is nice.

Short-term VMs: Thin-provisioning with Qcow2

Ubuntu’s libvirt installation has AppArmor configuration which limits what guests can write to. Here we’ll be using different files, and you’ll get permission errors. The easiest way around this is to use this workaround and reboot:

cat >> /etc/apparmor.d/abstractions/libvirt-qemu <<OEM
/var/lib/libvirt/images/** r,
EOM

For the short-lived VMs, we start by creating a clone of the base, in qcow2 format:

sudo qemu-img convert -O qcow2 /dev/vg_vms/ubuntu-base-vm /var/lib/libvirt/images/ubuntu-base-vm-readonly.qcow2
sudo chmod u-w /var/lib/libvirt/images/ubuntu-base-vm-readonly.qcow2

This can be used as backing file by multiple VMs, and must not be modified. From that base image, create a thin clone image for this specific VM:

VM=vm114
sudo qemu-img create -f qcow2 -b /var/lib/libvirt/images/ubuntu-base-vm-readonly.qcow2 /var/lib/libvirt/images/$VM.qcow2

then define a KVM domain for it, and configure it:

mkdir -p tmp
virsh dumpxml ubuntu-base-vm > tmp/ubuntu-base-vm.xml
mac=`egrep "^$VM"'\s' ips.txt | awk '{print $3}'`
python ./modify-domain.py \
    --name $VM \
    --new-uuid \
    --device-path=/var/lib/libvirt/images/$VM.qcow2 \
    --mac-address $mac \
    < tmp/ubuntu-base-vm.xml > tmp/$VM.xml
virsh define tmp/$VM.xml
virsh start $VM

at this point you can start the VM and check it works. Next we make a snapshot to serve as a clean starting point for future runs:

virsh destroy $VM
sudo qemu-img create -f qcow2 -b /var/lib/libvirt/images/$VM.qcow2 /var/lib/libvirt/images/$VM-start.qcow2
virsh dumpxml $VM > tmp/$VM.xml
python ./modify-domain.py \
    --device-path=/var/lib/libvirt/images/$VM-start.qcow2 \
    < tmp/$VM.xml > tmp/$VM-start.xml
virsh define tmp/$VM-start.xml

virsh start $VM

Now we can use it. When we’re done and want to reset the VM, we can do:

virsh destroy $VM
rm /var/lib/libvirt/images/$VM-start.qcow2
qemu-img create -f qcow2 -b /var/lib/libvirt/images/$VM.qcow2 /var/lib/libvirt/images/$VM-start.qcow2
virsh start $VM

which takes less than 2 seconds:

$ time sudo ./reset-vm.sh vm114
Domain vm114 destroyed

Formatting '/var/lib/libvirt/images/vm114-start.qcow2', fmt=qcow2 size=8388608000 backing_file='/var/lib/libvirt/images/vm114.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 
Domain vm114 started

real    0m1.311s

Closing

We’ve seen you can automate thick-provisioning VMs through cloning in under a minute, and rollback thin-provisioned VMs in seconds. Which is nice.

I’m really looking forward to future versions of libvirt/virt-manager adding support for these things through their API/UI.

For now, I’ll see how well this setup works in practice, and perhaps experiment with some alternatives.

SuperMicro server for virtualisation

2013-03-16T00:00:00+00:00

I’ve acquired a new lab machine, for virtualisation and deployment development. The goal is quickly spinning up VMs based on snapshots, do a deploy and end-to-end test of cluster software, and spin it all down again. I’m doing that regularly on remote machines, but wanted that capability locally, for ease of use, speed, and lack of resource competition. In addition I want to have some VMs running distro repository cache, run Jenkins, do development etc. My existing server hardware is old and not very scalable, so it was time to invest in better capability.

The requirements were:

support virtualisation (VT-X/VT-d)
scale to lots of VMs, so:
- CPU: many cores. No need to be super fast. Keep electricity consumption low and heat down. Dual CPU to get the core count up while keeping CPU costs in check.
- RAM: a decent amount to start, with future expansion potential. ECC. 64G to start with.
fast IO for installs, but no huge requirements for large datasets. Dual SSD (one for the system, one for VMs), with one fixed disk for storing distributions. No need for RAID.
pedestal case (no rackmount), and reasonably quiet, because it gets deployed in a test lab / office.
remote management
get a system that will last and be useful for many years, but not spend silly money.

which gets you into professional server territory. I’ve been self-building workstations for years, but this time I resisted temptation and decided to save myself the time and risk. I opted for the following specification from Sentral, where I had sourced a 1U Tyan server some years back:

Case: SuperMicro 4U SM Tower c/w 665W Super Quiet (25dB) PSU (CSE 733TQ-665B)
Motherboard: SuperMicro X9DRi-F. Dual Socket R (LGA 2011). IPMI 2.0 and KVM, 16 DIMM sockets.
CPU(s): Dual Intel E5 2630L (6/12*2Ghz/15Mb cache) 60W. Note that this is the low-power variant.
Memory: 64GB DDR3 1600 ECC REG (4*16GB)
HDD: 1* 3TB SATA Seagate Barracuda (ST3000DM001-1CH166)
SSD: 2* 240GB Intel 520 Series SSD (SSDSC2CW240A3)
CDRW: LITE-ON DVD Burner (iHAS124-04)

Cost a bit over £2500 ex VAT delivered, which I think is good value, if quite an investment. The build and delivery took two weeks; a bit longer than normal because of my non-standard CPU choice, and testing that much RAM. Sentral was pleasant to deal with: no on-line configuring ordering of credit card payment, but a few quick emails back and forth, a BACS transfer, plus regular pro-active updates worked as well. Plus, it’s actually nice to have the personal interaction (hi Dave).

Initial impressions:

Nice looking case. The side door is a bit fiddly to close, seems to work best lying on its side.
Quality of components is good. Sentral did a nice job of cable management.
It’s much quieter than a 1U rackmount, but louder than my self-builds where I optimise for quiet. In particular the front case fan is noisy; the vents in front of the fan and the shroud behind probably don’t help. I may decide to move the machine, or look into replacing the front fan, but for now it’s not too bad.
I connected the management interface and first normal interface to the LAN, found the new MAC address on my router, connected to IPMI, and configured it from there.
It feels fast.
Ubuntu 12.10, KVM and VirtualBox run fine.
It’s sipping around 120 Watt at idle.

Obligatory screendump, 2 CPUs x 6 cores x 2 HyperThreads = 24 processors in htop:

It will take some time to do some experimentation and complete the full setup of the lab environment. I’m looking forward to it already.

Boto with IAM roles

2013-03-01T00:00:00+00:00

In a previous blog post I explored IAM roles, with a patched s3cmd. I have since discovered that boto supports IAM roles too, and that makes things even easier. It even comes with a fetch_file command to use without writing any Python code.

So overall, what I want to happen during EC2 instance creation (in a VPC) is:

set the hostname to the EC2 instance ID, and configure /etc/hosts
add minimal packages
register this instance in the DNS
use IAM roles to securely download bootstrap code and data from S3, and execute that

The userdata I use for this looks like this:

Content-Type: multipart/mixed; boundary="===============0933669979118751095=="
MIME-Version: 1.0

--===============0933669979118751095==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="my-cloudconfig.txt"

#cloud-config

preserve_hostname: true
manage_etc_hosts: false

bootcmd:
- cloud-init-per instance my_set_hostname sh -xc "echo $INSTANCE_ID > /etc/hostname; hostname -F /etc/hostname"
- cloud-init-per instance my_etc_hosts sh -xc "sed -i -e '/^127.0.1.1/d' /etc/hosts; echo 127.0.1.1 $INSTANCE_ID.MYDOMAIN $INSTANCE_ID >> /etc/hosts"

cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - [scripts-user, always]
 - keys-to-console
 - phone-home
 - final-message

apt_sources:
- source: deb http://archive.ubuntu.com/ubuntu precise multiverse
- source: deb-src http://archive.ubuntu.com/ubuntu precise multiverse
- source: deb http://archive.ubuntu.com/ubuntu precise-updates multiverse
- source: deb-src http://archive.ubuntu.com/ubuntu precise-updates multiverse
- source: deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse
- source: deb-src http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

packages:
- language-pack-en
- python-pip

--===============0933669979118751095==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="my-userdata-userscript.txt"

#!/bin/sh

cd /root

pip install --upgrade boto

python <<PY
import boto
from boto.s3.key import Key

route53 = boto.connect_route53()
changes = ResourceRecordSets(route53, "MY_HOSTED_ZONE_ID")
change = changes.add_change("CREATE", instance_id + ".MY.DOMAIN","A")
change.add_value(local_ipv4)
changes.commit()

c = boto.connect_s3()
b = c.get_bucket('MYBUCKET')
k = Key(b)
k.key = '/MYBOOTSTRAP.sh'
k.get_contents_to_filename('MYBOOTSTRAP.sh')
PY

chmod u+x ./MYBOOTSTRAP.sh
./MYBOOTSTRAP.sh
--===============0933669979118751095==--
EOM

Fun stuff.

s3cmd with IAM roles

2012-12-25T00:00:00+00:00

In AWS, delivering credentials to an instance has traditionally been problematic. You can pass them in via instance user data at instance creation time, but that’s somewhat inconvenient, and instance data cannot be changed after the instance has started. You can pass them in by pre-loading on an AMI, but that’s even more hassle and static. Or you can have the instance fetch them from somewhere else at boot time, for example through CloudInit, but that then only moves the problem because you then need to secure that fetch. And of course you can copy credentials to the instance after it has booted, but then you have to wait until the ssh sever has regenerated its keys, and you must have security groups and VPC routes to allow access, and it precludes you running an instance manually from the AWS Console.

I’ve ignored that problem for a while, simply relying on Chef to distribute credentials. But for a current project I’m trying to do without custom AMIs and a Chef server: I want to run an instance, and have it do some setup and secure configuration at creation time. For example, it could setup an OpenVPN server in a VPC, or could create a shared secret for Chef Encrypted Data Bags prior to registering as a Chef client.

In June 2012, AWS announced IAM roles for EC2 instances – Simplified Secure Access to AWS service APIs from EC2 which addresses this very use case: it allows you to pass IAM credentials to instances.

So my plan was:

make credentials available to the instance, using IAM roles
create a script to download further scripts and credentials from a private S3 bucket
fetch and execute this script at boot time, using CloudInit

Creating the IAM Role

For the first step, the user guide has a section Granting Applications that Run on Amazon EC2 Instances Access to AWS Resources which explains the approach, and illustrates it with a video that uses the AWS Console. I prefer doing IAM configuration from the command-line, so that I can document and script it more easily, and track changes in a git repo. What is not obvious is that when you use the command line, you need to explicitly create a instance profile as a separate step.

iam-rolecreate -r myrole -p / -s ec2.amazonaws.com -v
iam-roleuploadpolicy -r myrole -p mypolicy -f ./mypolicy.txt
iam-instanceprofilecreate -p / -r mypolicy -s myrole

See the AWS Identity and Access Management CLI Reference for details.

I first ran an instance from the AWS Console by hand, and then determined how to do this from my ruby scripts that use Fog:

server = @compute.servers.create(
        ...
        :iam_instance_profile_name => 'myrole'
    )

With the instance running, I verified that the metadata service made the credentials available:

wget -O - -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
wget -O - -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole'

and verified that they worked by downloading the latest EC2 API Tools and Java, and running ec2-describe-instances, which was one of the permissions granted in the policy file.

Temporary credentials in s3cmd

I tried using the AccessKeyId and SecretAccessKey from the metadata in by s3cmd .s3cfg file, with a simple s3cmd ls s3://mybucket/ command, but got:

ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

The reason is that for IAM temporary credentials to work, you need to provide the token from the metadata, in a separate HTTP header when talking to the AWS S3 API (see this FAQ). Some googling found someone using security token for s3cmd put (with this change) but just using --add-header=x-amz-security-token didn’t work; it looks like s3cmd only uses that for put/sync/cp/mv.

There is a popular open feature request to Support IAM roles / instance profiles; it’d be great if you could just tell s3cmd to disengage its config file and “use the metadata service”. But, I found this post and patch which at least adds minimal support. For convenience I pulled it into my fork.

So then it’s just a matter of copying the credentials from the metadata to the s3 config file. Bear in mind that they expire; you’ll need to re-generate them, and the simplest is to do that for every use. Here is an example with bash and jq:

wget -O mycreds -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole'
SECRET_KEY=`jq -r '.SecretAccessKey' <mycreds`
ACCESS_KEY=`jq -r '.AccessKeyId' <mycreds`
TOKEN=`jq -r '.Token' <mycreds`
cat >s3cfg <<EOM
[default]
access_key = $ACCESS_KEY
secret_key = $SECRET_KEY
security_token = $TOKEN
EOM
s3cmd --config s3cfg ls s3://mybucket/

I’d like to see this (or better) support in s3cmd, but the last release was nearly a year ago, and it sounds like the project has a lack of leadership problem. I think it’d be great it Amazon could take this project over, or if they could provide similar functionality on top of its AWS SDK for Java.

As for my project, all the pieces seem to be in place; doing the whole end-to-end CloudInit bootstrap is for another day.

WebCrawler Screenshots

2012-12-18T00:00:00+00:00

The Register posted “Search engines we have known … before Google crushed them”, mentioning WebCrawler on page 3 but without a screenshot! Let’s fix that.

From Brian Pinkerton, the WebCrawler 1995 search page:

and the results:

From my archive, the 1997 front page:

Metal as a Service

2012-04-25T00:00:00+00:00

There was some hype around Canonical’s “Metal as a Service” (MAAS) a few weeks back. Coverage:

Ubuntu Press Release; full of marketing buzzwords
Metal as a Service: MAAS wiki
Ubuntu Cloud Infrastructure which describes using MAAS to deploy an OpenStack cloud
Press: The H Open Source, The Register, Linux Today

But what actually is it?

Datacenter system administrators have for years used PXE to do remote automated installs of machines. See Debian’s “Setting up a server for PXE network booting”, Ubuntu’s “PXE Install Server”, Redhat’s “Kickstarting a Machine” and Solaris “JumpStart”.

The way that works is that when a server boots:

its BIOS looks for a DHCP server, and gets some parameters like IP address and the address of a TFTP server
gets a config file from the TFTP server with a boot menu
executes a chosen selection from the menu (or a default) which then downloads the OS installer and installs it on the machine
then the machine reboots, and starts into its own OS
you can also preconfigure different configs for specific machine ethernet MAC addresses

Which is all great, but fiddly to setup and painful to maintain.

PXE itself doesn’t address OS configuration or package installation, but you can script that in various ways. Traditionally it involves making a custom installer, configuring package selection mechanisms like Debian Preseed, and executing scripts at first boot from /etc/init.d or cloudinit or similar.

Newer tools help here. From Redhat pedigree comes Cobbler (see Start Here and Cobbler: How to set up a network boot server in 10 minutes) which lets you define profiles for different types of machines or roles, and then use kickstart templates to do appropriate installs (example) to perform custom installation in the target. Ubuntu 11.10 introduced Orchestra (See Dustin Kirkland’s “Getting Started with Ubuntu Orchestra”), the predecessor of MAAS.

So, MAAS is basically a Web interface that does the PXE/DHCP server config for you, manages the install ISOs, and integrates with Juju (Docs) for role based package management on the nodes. Juju is more analogous to Chef: you can define “Nodes”, which run “charms” defining services, but it also deals with “relations” dependencies between nodes, somewhat like AWS CloudFormation. I wonder how that deals with staged provisioning and metadata sharing.

Some interesting aspects are the use of “wake on LAN” to remotely switch machines on, and the ability to use this for LXC. If you just want to test, TestingMAAS explains a VM-based setup.

I don’t support any large scale hardware deployments currently, so I’m not really in the target market for MAAS. For my EC2 customer deployments I don’t think MAAS is particularly useful; though Juju may be. For my office lab I can imagine MAAS being convenient: I could run the MAAS server in a KVM VM, then use it to install my physical lab servers for specific test projects. Definitely something for the list of weekend projects.

Ubuntu 12.04 “Precise Pangolin” has the latest support for the above and should be available form April 26th.

Some minor blog changes

2012-04-17T00:00:00+00:00

Some minor changes:

updated Jekyll, which involved a round of ruby/rvm updates, which in turn required osx-gcc-installer
made the ajax main content loading scroll back the window to the top (thanks Neil)
linkify #hashtags, @messages, and links in the twitter webclip (inspired by uudashr’s linkify)
remove the Google webclip after their whole Google Plus +1 change broke my flow.

OSX mounting the NAS

2012-04-17T00:00:00+00:00

For years I’ve accessed my Linux files from the Mac via NFS, CIFS/SMB (Samba, and Transmit’s Disk feature. When I tried the latter to stream some video on my HP MicroServer NAS I found it was too slow (presumably it pre-downloads the entire file) and it actually crashed my Airport Extreme half-way through. I’ve always wanted to setup Netatalk AFP, so this provided the perfect opportunity.

Tips and inspiration on:

Kremalicious’s HowTo: Make Ubuntu A Perfect Mac File Server And Time Machine Volume
Trollop’s OS X 10.7 Lion, Time Machine & Netatalk 2.2

I didn’t run into any particular problems with Netatalk or Avahi on this Debian Squeeze install, and can now browse the Network list and connect to the volumes (just ext3 on LVM). VideoLan happilly streams. All good. Next (at some point) is Time Machine.

I briefly considered iSCSI, but I’ve read of people hitting problems with the globalSAN ($89), and having more success with ATTO’s Xtend SAN iSCSI Initiator ($195), but that’s a little steep for just an experiment when AFP works fine.

RabbitMQ First Look

2011-08-06T00:00:00+00:00

I spent some time looking at RabbitMQ this week. It does what it says on the tin, but I encountered a couple of gotchas.

Configuration

There is a Chef cookbook for RabbitMQ which made installation easy. It uses the node[:hostname] (in the default attributes), which ends up being used by the /etc/rabbitmq/rabbitmq-env.conf and is used in the name of data directories. But I’m installing on EC2, where stopping and starting change change the hostname, which then causes problems. This is mentioned on the RabbitMQ on EC2 page, where the recommended action is to change the hostname and give it a localhost address in /etc/hosts. I’m not particularly happy about that, because it could cause problems to other parts of the system: hostname --fqdn no longer gives a fully qualified name; and other software may have started using the old name and get confused. In my case this runs on a dedicated host and I can from Chef script the hostname changing to occur before the RabbitMQ install, so it will have to do.

Clients

There are a bunch of clients available (see Client & Developer Tools). My first requirement was for Ruby, and there is a collection of clients on ruby-amqp on github, with an overview of their relations here. I initially used amqp, but switched to bunny which is easier in scripts because it’s synchronous.

For future Java integration I tried rabbitmq-java-client which worked from a simple project in IDEA.

Learning

The get started guide takes you from a simple queue to full topics, and I recommend you follow all of them. One thing that irked me was that the simple use cases turn out to be just gloss over the complex use cases. That means that what you learn in the first lesson actually misleads you (you can’t send message directly to a queue, it just looks like it), and as you try to move to more complex models it’s not always clear what smoke and mirrors was used to make the simpler ones work (Is it because you’re using the default exchange? Is it because there is special queuename/routing_key combination? etc). I would have preferred being told the full model, and have explicit declaration of those configuration items that affect routing.

When I look at a message queue, I expect it to work like a mail server: when I inject a message I expect the MQ to do its damndest to make sure that a receiver gets it. What I found that with RabbitMQ and bunny, the default configuration doesn’t: unless you deliver to a queue that has been explicitly marked durable, via an exchange that is explicitly marked durable, with a message that is explicitly marked persistent, a simple server restart can lose messages. This is documented, in tutorial 2 under “Message durability”, and in the scenarios FAQ under “Reliable persistent message delivery” and “Store-and-forward”. I just find it strange that that’s not the default.

While experimenting, I used rabbitmqctl to inspect the server state, which is useful, but the Management Plugin. looks even more promising. I had not found a tool to drink from the firehose, but it didn’t take long to make a quick-and-dirty rabbitmq-trace-logger.

Now I’m looking forward to introducing this into my system architecture and put it to real work.

Code in a blog

2011-08-06T00:00:00+00:00

My blog posts occasionally have some code, and I’ve been wondering for a while what the current/best way is to do that.

I was using embedded gists, which look good, have line-numbers, let you select and copy code without including the line numbers, let you later modify the code using git goodness, and have handy view/raw links. But, I ran into a few problems:

it only works with JavaScript enabled
the script tags didn’t get executed with my fancy DOM-modifying JavaScript
sometimes not all embedded snippets would turn up (I’ve not tried to debug that)
I don’t really want to depend on inline content hosted by third parties
I’m not wild about the “This Gist brought to you by GitHub” advertising

So for grins I spent some time evaluating some JavaScript highlighters (from this list of 9):

SyntaxHighlighter line-numbers got easily confused by long wrapping lines
Prettify which wouldn’t show up right
highlight.js which documents custom initialisation that allowed me to integrate it with my jQuery DOM manipulation

But in the end I gave up in disgust with JavaScript solutions, and went with a solution that was right under my nose all along: Jekyll supports Pygments syntax highlighting (see docs and this blog). The easy_install is trivial, I used the github syntax.css, enabled pygments in _config.yml, and that was that. It doesn’t do non-selecting line-numbers, and there’s no raw/view/download etc functionality, but that’s fine; I can always link to a gist.

Creating Machines with Chef

2011-07-28T00:00:00+00:00

After ugrading Chef to use environments, I needed to update my custom AMIs. These custom AMIs were based on Ubuntu’s Amazon EC2 Published AMIs, have some extra software pre-installed, and have a custom chef client.rb which gets configuration (chef server info, and client roles) from EC2 userdata to bootstrap itself. I then use scripts to instantiate machines from those AMIs, and pass them the appropriate userdata. This has been working great, but is not “the chef way” – the recommendation is to use knife ec2 server create, which creates a machine, and then ssh’es in to bootstrap it. In Chef 0.9 I ran into various routing and ssh timing bugs that made this approach too unreliable, but in 0.10 that appears to have been resolved. The main advantage of this approach is that you don’t need to make special AMIs; you just use the latest official Ubuntu ones, in any region/arch/store. The disadvantage of that is that you then have to wait for chef-client to install all the software, which in the case of Java and RVM/Ruby is a long time.

So the challenge is to:

make sure that the “knife ec2 server create” method produces functional machines from stock AMIs for all my roles
use custom AMIs to preload software, and use them from my existing scripts (which use ec2-run-instances) for selected roles

I also wanted to take the opportunity to upgrade OS and sanitise my Ruby install.

For the OS I wanted to switch from Ubuntu 10.4 Maverick Meerkat to 11.4 Natty Narwhal. Chef 0.10.2 includes only templates for [ubuntu10.04-apt, ubuntu10.04-gems.erb], which can be adapted for 11.4 by changing the “lucid” to “natty” (or pull the release name out of lsb_release), but then you end up with Chef 0.9, so you want to add “-0.10”.

Here I ran into an interesting issue: the apt template does a apt-get install -y chef, and then writes settings to the client.rb, and then runs chef-client for the initial bootstrap. The problem is that the install also starts the /etc/init.d/chef-client service, so that executes before the modifications to client.rb are made, and before the chef-client bootstrap runs. In my template modifications I set the node_name, and as a result the first chef-client registered the client with the default name (the host name), and the subsequent invocation failed; and I ended up with nodes in the wrong environment. I think there is actually a generic template bug here.

We’re using Ruby and RVM for applications on some machine roles, and I’ve run into various situations where there has been confusion between the system ruby, apt, RVM in /usr/local, RVM in user home directories, various gemsets, and the chef-client and our applications. To reduce that confusion I wanted to try the apt install rather than the default gem install, and limit RVM to a per-user install. [Update: there are some unique issues, such as knife not finding plugins (CHEF-2483)]

The Knife Template

Pulling it all together I ended up with this knife template ubuntu11.04-apt.erb:

#!/bin/bash
# This is a knife ec2 server create template for Ubuntu 11.4.
# It is based on the ubuntu10.04-apt.erb version in the 0.10.2 Chef distribution
# available here:
# https://github.com/opscode/chef/blob/master/chef/lib/chef/knife/bootstrap/
# with modifications to:
# - use the natty APT repository
# - install Chef 0.10.2
# - avoid starting the /etc/init.d/chef-client service until the client.rb
#   has been written
# - let a CHEF_NODE_NAME_PREFIX environment variable prefix the node name

bash -c '
# MAK: use lsb-release to pick up release name, and add -0.10 to get chef 0.10
<%= chef_server_url = Chef::Config[:chef_server_url] %>
<%= validation_client_name = Chef::Config[:validation_client_name] %>
<%= environment = Chef::Config[:environment] %>
if [ ! -f /usr/bin/chef-client ]; then
  echo "chef    chef/chef_server_url    string  <%= chef_server_url %>" \
   | debconf-set-selections
  [ -f /etc/apt/sources.list.d/opscode.list ] || \
    echo "deb http://apt.opscode.com "`lsb_release -cs`"-0.10 main" \
    > /etc/apt/sources.list.d/opscode.list
  wget -O- http://apt.opscode.com/packages@opscode.com.gpg.key | apt-key add -
fi
apt-get update

# MAK: use policy-rc.d to prevent chef-client starting and registering
# before we write client.rb
(cat <<'EOP'
#!/bin/sh
exit 101
EOP
) > /usr/sbin/policy-rc.d
chmod 755 /usr/sbin/policy-rc.d

apt-get install -y chef

# MAK: remove policy.rc
rm -f /usr/sbin/policy-rc.d

<% unless validation_client_name == "chef-validator" -%>
[  `grep -qx "validation_client_name \"<%= validation_client_name %>\"" \
    /etc/chef/client.rb` ] \
 || echo "validation_client_name \"<%= validation_client_name %>\"" \
 >> /etc/chef/client.rb
<% end -%>

(
cat <<'EOP'
<%= IO.read(Chef::Config[:validation_key]) %>
EOP
) > /tmp/validation.pem
awk NF /tmp/validation.pem > /etc/chef/validation.pem
rm /tmp/validation.pem

<% if @config[:chef_node_name] %>
[ `grep -qx "node_name \"<%= @config[:chef_node_name] %>\"" \
   /etc/chef/client.rb` ] \
 || echo "node_name \"<%= @config[:chef_node_name] %>\"" \
 >> /etc/chef/client.rb
<% end -%>

# MAK: use an environment variable to pass in a hostname prefix,
# so your node gets called e.g. web-server-i-123abc
<% if (! ENV['CHEF_NODE_NAME_PREFIX'].nil?) and
    ::File.exists?('/usr/bin/ec2metadata') %>
(
cat <<'EOP'
node_name "<%= ENV['CHEF_NODE_NAME_PREFIX'] %>`ec2metadata --instance-id`"
EOP
) >> /etc/chef/client.rb
<% end -%>

<% unless (environment == "" or environment == "_default") -%>
[  `grep -qx "environment \"<%= environment %>\"" /etc/chef/client.rb` ] \
 || echo "environment \"<%= environment %>\"" >> /etc/chef/client.rb
<% end -%>

(
cat <<'EOP'
<%= { "run_list" => @run_list }.to_json %>
EOP
) > /etc/chef/first-boot.json

/usr/bin/chef-client -j /etc/chef/first-boot.json

# MAK: start chef-client because we prevented that previously
/etc/init.d/chef-client start
'

which you can use likes this:

export CHEF_NODE_NAME_PREFIX=webserver-
knife ec2 server create -r "role[webserver]" \
  -I ami-ab16d2c2 --flavor m1.large -G webserver_demo \
  -x ubuntu --ssh-key demo-kp1 \
  --template ubuntu11.04-apt.erb \
  --environment demo

This works well for bringing up a generic instance with a given role from the command line, after which Chef kicks in and configures the machine.

The AMI

To create an AMI there are two approaches: snapshot a running instance, or build an AMI using loopback mounts and chroot. The former is somewhat easier, the latter is more secure and precise, and is recommended for public AMIs. For a discussion, see Eric Hammond’s posts on Creating Public AMIs Securely for EC2 and Building EBS Boot AMIs Using Canonical’s Downloadable EC2 Images.

For my private AMI I decided to use the simpler snapshot approach, at least initially to develop the install sequence, and I’ve split it into separate scripts for easier testing. See my github create-ami repo.

Upgrading to Chef 0.12, and using Environments

2011-07-19T00:00:00+00:00

I’ve been running Chef version 0.9 for some months, and I’ve been looking forward to trying out 0.10; in particular the “environments” feature described in OpsCode’s preview blog post, and Chef’s documentation.

The Old Environments

In Chef 0.9 I had implemented different environments as follows:

have separate machines running chef-server for each environment
have separate chef role definitions for each role × environment combination (e.g. webserver_demo, webserver_staging)
use attributes overrides in the roles to have per-environment behaviour
selectively load roles into servers (e.g. webserver_demo is not needed on staging)
assign roles at boot time from a custom client.rb (in the AMI) using JSON from instance data
the same mechanism also names the nodes to include the role and environment, e.g. webserver-demo-i-a1b2c3d4
use git to share a chef repository between environments

This has worked well, but the role × environment matrix size explosion was a headache to manage, and keeping the right roles (and only the right roles) in the right servers was error-prone.

The New Environments

In Chef 0.10 environments are a first-class concept.

Upgrading some of the servers involved a little trial and error, fresh gem installs worked fine, and upgrading the existing clients was trivial.

Because I restrict port 22 on internal machines with EC2 security groups I did immediately run into a “ssh hang” bug KNIFE_EC2-2, but the linked patch works fine.

To then migrate the configuration for existing machines:

create environment .rb files, applying the attributes from the old role files
create the environments with knife
move all nodes to the new enviroments using nodes.tranform
create new role files, which have the roles/recipes but not the attributes from the old roles
load the new roles, and apply to the relevant machines
remove the old roles
re-run chef-client and verify all roles succeed
verify normal system operation

And that was basically that.

The next step, was to review how I create machines.

Mistreated by Dell

2011-07-15T00:00:00+00:00

I’ve been a Dell hardware user for many years – I fondly remember my first Dimension XPS, and I’m happy user of their curent range of LCD displays. When I first became their customer, Dell was leading the way in on-line purchasing of computer hardware to consumers.

But, I won’t be purchasing from them any more, after the way they’ve treated me recently.

This all started when I purchased a monitor in November last year. I ordered it, entered my credit card details, and Dell shipped the product. There was a bit of hassle about price, because I wanted them to honour a advertised sales price, and after several rounds with sales and customer service, being passed from department to department over a period of weeks, I was promised a partial refund. After that I didn’t think about it again.

Then in May I received a letter from “Debt & Revenue Services - a debt collection agency” (DRS), demanding payment. Their letter design and company name gave me the initial impression this was a public body, but on closer inspection this appeard to be just a company that works with Dell. They threatened “further action” if I failed to contact them.

This was news to me.

I won’t bore you with the step-by-step interaction, but after several phonecalls and emails it was obvious that DRS does not have an adequate way of looking into the circumstances of disputed claimed oustanding payments in a timely fashion, and never actually bothers to ring you back. I turned to Dell customer service, where again it took a lot of time to find out more.

It transpires that when I provided my credit card details to Dell, they were unable to withdraw funds, apparently due to fraud prevention systems operated by my credit card issuer. It seems stupid to me that they shipped the product before receiving payment; it seems incompetent that their automated sales systems don’t immediately notify the customer in email, or alert a customer service agent to phone the customer or send a letter if there are problems with a payment.

After many more emails and phonecalls I negotiated a settlement and re-authorized the card, but even after I had contacted my bank the transactions continued to be declined. Rather than trying to fix that problem and drag things out further, I made a BACS payment on 6^th of June. Dell managed to not locate that payment, but after further phonecalls they were able to confirm they payment had been received. I thought that was the end of it.

But, on 14^th July I received a mailed “notice of action” from DRS, saying that they will be calling on or between the 18^th of July and the 18^th of August between 8am and 8pm. This is clearly not adequate notice – there are only 2 business days before the first date, and the one-month period is not adequately precise to be of any use. But obviously I should not have received this threat at all.

I rang back again, was told that if I didn’t hear anymore that would be the end of it. But I’ve been told that before, so today I rang up again, and was told that they had in fact received confirmation from Dell, but only after this notice of action was sent out on the 11^th June. The envelope did not contain a timestamp, and the notice itself was undated, but on closer inspection the address label shows “/110711/”, suggesting that it was actually sent on the 11^th of July. Clearly DRS is incapable of ensuring that mail is dispatched in a timely fashion, and is unable to track their mail.

In all, this has cost me many hours to sort out, and has been a stressfull experience. My conclusion about DRS is that they are sharks that only care about extracting money, and do not manage their interactions with debtors adequately, and are falling foul of various aspects of the Office of Fair Trading Debt collection guidance:

They are unclear in their communication, by not detailing adequately what the claimed debt is for (violating 2.1)
They continued demanding payment, even though they knew that the debt had been in dispute and after it had in fact been settled (violating 2.6h)
They completely failed to investigate when I queried and disputed the debt (violating 2.6j)
They continued collection activity after I queried and disputed the debt (violating 2.6k)

Nor do they confirm any progress (or final settlement) in writing, or even call back when they say they will. I would suggest that Dell should find a more competent company, but I fear that they’re all pretty similar, as their incentive structure does not reward good treatment of debtors.

But most of my anger is aimed squarely at Dell: sending debt collectors after customers as a first action, when Dell fails to obtain a properly authorized payment is outrageous.

Annoyingly, to the Dell and DRS beancounters this episode will be treated as a successful debt recovery statistic, and thus an incentive to continue this practice.

Review - Plugable USB Display Adapter UGA-2K-A

2011-06-30T00:00:00+00:00

For some years now I’ve been using a 3 screen workstation setup, consisting of a laptop (was Windows, now Mac) with external screen, and an additional screen for my Linux desktop, with Synergy providing cross-screen functionality.

This worked well enough, but recently I’ve found myself getting frustrated:

I use an Apple Wireless Keyboard, and the mapping of Fn/Ctrl/Option/Command on Linux is confusing
I use a Apple Magic Trackpad (highly recommended by the way), which doesn’t have 3 Linux mouse buttons obviously, and which seems to scroll too fast on Linux
Synergy occasionally crashes, and the copy-paste sometimes does not work or does not support the content
I do most my work on the Mac, and not being able to use my third screen for my Mac is unfortunate

Some of these issues can be worked around, but it’s not ideal. So I decided to hook up the third screen to the Mac too, and because it had no spare display ports, decided to try going over USB.

There are lots of options: StarTech, Diamond, EVGA, Kensington, etc. I was looking for a model that was using a modern chipset, with good support, a good company attitude, and good availability.

I purchased a Plugable USB Display Adapter UGA-2K-A from Amazon UK. This adapter uses the DisplayLink DL-195 which will do 1920x1200 and 2048x1152 on DVI, and is well known and supported. The product comes with VGI and HDMI adapters and a USB cable, and is surprisingly compact at 4cm by 9cm.

The adapter needs a driver for OSX to recognise the display. The product came with a 8cm CDROM, which my MacBook’s slot-loading drive won’t take, so I headed over to DisplayLink’s Mac Driver page and downloaded the 1.6 driver. OS 10.7 Lion will need a newer driver that is not yet available. Installation was trivial, but required a system restart. After that the display (a Dell Ultrasharp 2005FPW 20” Widescreen LCD Monitor) was recognised. System Profiler sees the USB device, but doesn’t show a Graphics/Displays entry:

At that point you can re-organise your displays as per usual in the Displays pane in the System Preferences, which says my display runs at 1680x1050, the panel’s native resolution.

In short: it works. Display quality is good. Moving windows is a little slower than on the main displays, but plenty good enough.

The hardware is passively cooled, so it’s silent, but it does get a little warm, and needs airflow. It’s bus powered, and no Extra Operating Current is displayed, so I assume power draw is OK. I couldn’t see specifications about power draw, but note they advertise using up to 6 adapters concurrently.

Some things don’t work. DisplayLink’s website says:

Please note: This driver does not support 3D acceleration. Some features of Mac OS X applications that require hardware OpenGL acceleration, such as Keynote presentations and iPhoto slideshows, will not function properly

I ran into a couple of limitations:

The DVD Player window won’t work on the display – you can move the window onto it, but it moves itself back to a main display (cute).
Command-Shift-3/4 and Take Screenshot from Preview don’t work on the display.
PinPoint draws a semi-opaque box around the cursor which is ugly and makes it hard to use:

I’ve reported this to Lagente Software, who diagnose it as lack of support for advanced alpha operations by USB adapters. I’ve requested a feature to allow a user to disable PinPoint on specific displays.

All of which I can live with; I’ll be using it mainly for web pages, email, text editting and monitoring.

Another minor usability niggle is due to the resolution differences between my monitors: when you move the pointer from a larger screen to smaller one, there are parts of the edge of the large window that fall outside the smaller screen, and won’t let you move onto it; you have to move down the edge until you get to the top of the smaller screen. Because my screens are aligned at the bottom, I tend to hit this when dragging windows, since window title bars are at the top. If this sort of thing bothers you, get two identical screens, but make sure the adapter supports your native resolution. In my case the larger screen is a Dell U2711 (which I do rather like) running at 2560x1440, which exceeds the adapter’s resolution.

What about my Linux use, I hear you ask? No, I’ve not given up on my workstation. I’ve moved the display onto another (VGA) input on the monitor, so that I can switch if needed. Most of what I do is command-line work, so I just ssh from Terminal. For GUI things I use Chicken of the VNC.

I still use Synergy too, to get to other machines, but by the time you’re scrolling to screen number 5 it does become a bit silly, and it’s not something I do often.

By the way, it looks like this product is supported under Linux too. From the manufacturer:

LINUX COMPATIBILITY: As of Linux kernel 2.6.31, this adapter has open source drivers in the kernel staging tree. As of 2.6.38, the driver was promoted to the main kernel tree. Configuration of X Windows for USB displays is still distribution and scenario dependent, however, and only for very adventurous users. Plugable is involved with Linux development work, see http://plugable.com/category/platform/linux/ for details.

which sounds good.

Conclusion: recommended.

Making Pages Faster still

2011-06-25T00:00:00+00:00

The previous results were a good improvement in terms of webpagetest numbers, but the visual latency caused by the SSL in the delayed Twitter/Google Reader loads (for initial load and repeat loads) annoyed me.

So now I’m pre-fetching those (in a If-Modified-Since friendly way), store them my own server, combine both into a single file, and load that with caching enabled. I’ve added a fallback for if the dynamic content on Greenhills is missing, and reworked the no-Javascript case so it’s nicer.

According to the new results that has dropped the first pageload from 0.8/1.7/2.6 to 0.7/1.6/1.7, and repeat page loads from 0.1/0.3/1.5 to 0.1/0.3/0.4. That’s more like it; switching pages is really fast now. But the render flicker is noticeable, and the delayed Twitter/GoogleReader loading flickers too.

So, next I switched to dynamic content loading in the DOM, combined with the HTML5 History API per Dive Into HTML5, and some jQuery animation. Now new content loads fast and smooth, old content stays undisturbed. Bookmarks and links work. This looks great in current Safari, Chrome and Firefox. IE9 does not have history API support yet.

Update: added a spinner if the ajax load takes >1 sec.

ELB traffic for the wrong host

2011-06-22T00:00:00+00:00

I’ve been using the AWS Elastic Load Balancers (ELB) for some time now. The ease of use and scalability appeal.

But I learned something unpleasant about them: you can easily end up with misdirected traffic.

When you use an ELB, your users access it via a CNAME (setup by you and served by your DNS servers), which points to a DNS name owned by Amazon (in the elb.amazonaws.com DNS zone). When your client does a lookup of that name, Amazon then returns the IP address of an ELB machine that routes the traffic to your EC2 instances. From time to time, in response to traffic conditions, those IP addresses change. To accomodate such changes, the TTL on the domain name is low (60 seconds).

The problem is that some clients will not honour the TTL, and may continue using the old IP address after it has been disassociated with your ELB, and may have been given to someone else’s ELB. Which means that your ELB may be receiving traffic for someone else, or that traffic meant for you goes to someone else.

I found out about this because my instances behind an ELB suddenly started receiving 15K requests per minute destined for ping.syndic8.com (judging by the Host header), sent by Ping-O-Matic (judging by the User-Agent header) and other clients. A DNS lookup of that name showed their ELB address, not mine. This persisted for many hours, and impacted my application.

This has been discussed in the forums at times: 28 Dec 2010, 28 Jan 2011, 21 Mar 2011. It looks like it doesn’t happen (or gets noticed) all that often, but that it’s a known issue you are meant to be aware of somehow, and you should just accept or escalate to support.

I think this is a big deal:

Clients get service errors, or have their traffic exposed to a third party. Sure, it’s “their fault” for not honouring the TTL, but depending on the kind of application that you serve this could be a common bug, or outside the control of the person running the client. I imagine this is especially likely with scripted API clients.
Being on the receiving end of misdirected traffic can affect your level of service. Sure, if you’re using AutoScaling this may help your load and latency, but at a financial cost.

I think that at the very least AWS should wait longer before recycling IP addresses. Ideally they wouldn’t recycle IPs at all – roll on IPv6. Maybe they could add an ability to associate several elastic IPs, but I imagine that is hard.

I have various other issues with ELB:

it is slow to add new servers,
you can’t have 2 ELBs pointing to the same instance (during ELB migration)
you can’t inspect traffic before it hits the ELB,
you can’t block client IP addresses before they hit your servers
your connection closes after 60 seconds of idle time
you can’t limit your instances to only receive traffic from my ELB (only all ELBs)
it’s hard to know what logic the ELB applies to your connection

but this one is pushing me over the edge – time to reconsider some of the alternatives.

Performance testing with webpagetest

2011-06-16T00:00:00+00:00

Oh wow, how did I miss this one?

www.webpagetest.org is like YSlow and Safari’s Web Inspector timeline, but as a hosted service in multiple locations, and is very comprehensive.

My results show the homepage starts to render 0.5 seconds is “document complete” in 2s, and finishes delayed loading in 3s. Total bytes in: 124KB. In repeat view it’s 0.5s, 0.8s and 1.4s, with 44KB. Page Speed 1.9, score 95/100. That’s what I expected based on my local testing, and I’m happy with that.

Going with London/Chrome config, I got completely caned by 1s DNS, and twitter fails alltogether. That will need looking at another day.

Things I want to improve in the short term:

add missing epxirations tags to my static assets
making my own twitter widget will help the most (I knew that already)
consider CloudFront for static assets for at least template images. I have a slight philosophical objection against using CDN resources as URLs; I wish you could list CDN copies and your original as a fallback.
fix their IE text centering (for the screenshots)
look into delaying jQuery loading (tricky, and maybe not worth the hassle)

Then there is of course pre-baking the currently-lazily-loaded content, but that’s going to require a bunch more coding and can come later.

Update. June 18^th:

added Expires with Lighttpd mod_expire, disabled etags with static-file.etags (fails for compressed content because of a lighttpd issue)
made my own twitter widget with jQuery.getScript. This makes inital rendering fast, but actual tweet loading slower because twitter redirects to HTTPS. I don’t know if there is a way to stop that (“Always use HTTPS” is off in Twitter’s account settings), but this issue will go away in future
moved images to S3 manually (s3cmd sync would help) and enabled CloudFront with a CNAME. Annoyingly CloudFront doesn’t do gzipping of text/css and text/javascript of S3 content; workarounds are unsatisfactory and the Custom Origin mechanism a bit of a faff. For now I just use my own copies.
fixed the IE centering with the use of netrenderer.
decided against delayed jQuery loading because of the complexity. Did combine my JS files, and delayed their loading.

The result is 0.8/1.7/2.6 (139K), and 0.1/0.3/1.5 (10K). Not much difference, but these are small margins anyway. Fixing the data flow will have more visual impact. Log analysis on my server (excluding my browser, w3c validator and webpagetest requests) shows no meaningful reduction in traffic, a reflection of the fact that it is dominated by search engines and malware bots. Heigh-ho.

WWW or bare domain?

2011-06-12T00:00:00+00:00

Should your website address have a www (e.g. www.example.org) or not (e.g. example.org)? This question came up again recently, prompting me to look into it a bit more, and write down my thoughts. Some people feel fairly strongly about this; there is even a campaign against “www”. The alternative, called bare domains, zone apex domains or root domains, is not without issues.

Some arguments against:

The www just looks “old”
www is unpleasant to spell. In conversation the short form “web” has taken over the “world wide web” phrase, so “www” is just weird and ugly
New exciting startups use bare domains: Twitter, Stack Overflow, lots of others
Typing www takes more effort, which on some mobile platforms can be inconvenient
Separate www used to be needed because you had separate machines serving as web servers; Layer 4 content switches make that a non-issue
In URLs the www is superfluous visually, because the “http://” in the URL indicates that it is a web site

Some arguments in favour:

the www looks familiar
Most popular websites use www: Facebook, Google, Zynga, most others
Typing www is hardly ever needed: browsers can prepend it for you, and you don’t often need to type URLs anyway, because you usually reach sites through existing links on other pages, bookmarks, history suggestions etc.
Separate www makes management more flexible because you can have separate addresses with different A records
The www makes a hostname visually self-identify as a web address, without needing an ugly “http://”. This is especially appealing in printed marketing materials such as business cards, TV ads etc. This may be even more important for uncommon top-level domain names (those other than .com/.org/.net), especially newly created ones and country TLDS that are used for vanity domains, which non-expert users may not recognise. Note also that some browser (e.g. Google Chrome) hide the “http://”, so it may fall further out of use.
bare domains are not allowed to have CNAMEs in the DNS, but CNAMEs are strongly recommended in some network configurations
bare domains can confuse email configurations
some DNS control panels at DNS providers don’t allow you to use bare domains
SSL configuration is easier with “www”, because that gets matched by “*.example.com” wildcard certificates.

Ultimately people seem to decide based on entrenched taste rather than actual usability or technical arguments. But I’m interested in considering the technical implications. Having the “www” has no technical drawbacks, but bare domains have two: the CNAME problem and the mail problem, which I would like to explore in some more detail.

The CNAME problem stems from the RFC1034 DNS specification rule which states that:

“If a CNAME RR is present at a node, no other data should be present; this ensures that the data for a canonical name and its aliases cannot be different. This rule also insures that a cached CNAME can be used without checking with an authoritative server for other RR types.”

A bare domain must have SOA and NS records (these are the basis of authority delegation in DNS), so if it also had a CNAME, that would violate the above rule.

So, what problems are caused by this restriction that you cannot use CNAMEs for your web server address? Surely you can simply not use CNAMEs, but use A records to locate your web server? Actually, that depends. Some CDNs use CNAMEs (RackSpace, Amazon CloudFront). Likewise, load balancers from cloud hosting providers like Amazon EC2 typically require CNAMEs. Amazon has relatively recently announced a solution: Amazon Route 53 alias resource record sets. Heroku documents DNS configuration for custom domains (duly requiring A addresses for root domains), but after a recent DDoS they recommend against root domains because they don’t allow their system administrators to update IP addresses.

Next, the email issue. To deliver internet email, a message sender consults the DNS for the MX record of the destination, then looks up the resulting DNS name, which results in an A record, listing the IP address of the mail server (MTA) to connect to. If there is no MX record on the destination, the sender does a lookup of A records on the name itself. With bare domains you can have A records pointing to the web server, and the MX record to a different server that runs the mail server (MTA). The problem comes when the MX lookup fails (or is not done), and the sender falls back to the A record – it now talks to the wrong address and fails to deliver the email. Now, wether this message sender behaviour is advisable is debatable, and not all mail sending software will work in the same way. I think that it is likely that this is less of a problem now than it was years ago. But if you weren’t using bare domains then you’d have no A records on your domain, and this wouldn’t be a problem at all.

So neither issues are show-stoppers, but it seems that having “www” is more straightforward all round.

This topic would not be complete without the obvious recommendations that you should only have one canonical URL for a resource. So if you use bare domains, you should also support www and redirect it to the bare domain, in case people type it out of habit. Or if you use “www”, you could redirect your bare domain to it.

As an aside: in the early days of the web the “web” hostname was sometimes used, and I used it myself for web.nexor.co.uk before the “www” became the norm. I still prefer it visually, and in transcription. But, the principle of least surprise directs me to continue using “www” for my own websites.

Spotify / Pinpoint Window Move Bug

2011-06-11T00:00:00+00:00

I finally found the cause of an annoying behaviour of Spotify for Mac OS X.

Spotify lets you move the main window around the screen by clicking anywhere in the window and dragging; not just in the titlebar. This is not typical Mac behaviour, so they must have implemented this themselves. Normally this doesn't cause a problem: Spotify recognises when you click in a scrollbar, and operates just the scrollbar as you expect.

I also use PinPoint which help you visually locate your mouse pointer, especially useful if you use multiple large high-resolution screens. I have it set to draw a subtle circle ("Celtica") around the pointer while it moves, and 0.5s after the pointer stops.

The problem is that this appears to cause Spotify to fail to detect when mouse moves happen in its scrollbars: it both moves the scrollbar scroller (thumb) and also moves the window, and makes the scrolling erratic. The "workaround" is not to drag in the scrollbar, but single-click in the scrollbar in the desired location, or to quite PinPoint.

Here is a screen recording that demonstrates the problem:

Spotify's FAQ says to report bugs by posting in the forums. Done.

Embedding video in blog posts

2011-06-11T00:00:00+00:00

This is how I embed a video in a blog post on my Mac, such that it shows up in current versions of Apple Safari (which wants MPEG 4), Mozilla FireFox 4 (which wants Ogg Video “Theora”), and Google Chrome (which does both). The original video format is typically from a QuickTime Screen Recording or some such in a movie.mov.

open in QuickTime Player, Save As, format iPhone, save to the Desktop. This produces a folder named movie containing a movie - iPhone.m4v and movie.jpg.
remame the movie - iPhone.m4v to movie.mp4
run ffmpeg2theora on the mp4, which produces a movie.ogv. Or use MiroVideoConverter
drag the mov into MiroVideoConverter and convert it to WebM (vp8) format
upload the movie.mov to Vimeo. Go to Settings, Custom URL, give it a name.
once available, click “embed”, copy the iframe tag from the code.

Then to embed in the blog posts:

<div class="photo_frame_center">
 <video width="480" height="360" controls preload="none"
  poster="/img/blog/movie/movie.jpg">
  <source src="/img/blog/movie/movie.mp4" type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
  <source src="/img/blog/movie/movie.ogv" type='video/ogg; codecs="theora, vorbis"'>
  <source src="/img/blog/movie/movie.webm" type='video/webm; codecs="vp8, vorbis"'>
  <iframe src="http://player.vimeo.com/video/24959362?title=0&amp;byline=0&amp;portrait=0"
   width="400" height="300"></iframe>
 </video>
</div>

Here is an example blog post.

This version works on the iPhone, but a is a bit low-quality. The movie - Computer.m4v version is much better, but if you use that then the iPhone only displays the poster, and doesn’t actually let you play the movie.

At the moment Jekyll doesn’t seem to recognise the video tag in Markdown, so as a workaround use a .html.

The .m4v to .mp4 renaming seems to work; alternatively use MiroVideoConverter or Squared 5 MPEG Streamclip for Mac OS X.

Instead of using Vimeo I could convert to FLV and use an embedded Flash player like FlowPlayer or JW Player, but that seems hardly worth the effort.

For more than you ever want to know about HTML5 video, see the Video chapter in Dive into HTML5.

Linux Containers (LXC)

2011-06-10T00:00:00+00:00

In this post I wrote up my first experience setting up LXC on Ubuntu 11.04 Natty, mainly for my own reference, but possibly of interest to others.

Background

In the past I’ve used various technologies to slice up a computer in various ways:

For security I’ve used Linux chroots and BSD jails for isolation, which works well but has limited control
For hosting web/mail servers I’ve used Xen, but got frustrated at the Dom0/DomU OS restrictions and the lack of support (then) in the mainline kernels
I switched to KVM (through libvirt), and am pleased with how well that works
I’ve experimented with OpenVZ, but found it complex to manage, and again there was (then) no mainline kernel support. LXC will eventually replace OpenVZ in Ubuntu.
For test environments with different OSes I’ve run Qemu, VirtualBox, Parallels, and VMWare Workstation, which work well with ISOs and graphical user interfaces

I have been particularly keen on trying out Linux Containers (LXC):

the cgroups controls looks very powerful (see RHEL docs) and are integrated into Linux scheduler
performance is said to be good, because there is no virtualisation overhead
it’s part of mainline kernels
the management uses command-line tools and seems reasonably straightforward
it’s actively developed so there should be community support (lxc-users, irc, bugs)
it’s not fragmented like the commercial product ranges
it’s backed by vendors (e.g. part of RHEL6, Ubuntu)
it’s free as an in beer, and free as in GPL 2

That makes it look ideal for achieving isolation for hosting purposes. But, it’s young technology, so I expect bumps in the road.

Getting Started

My main goals:

create a container, and get it talking to the network
document a set of instructions for preparing the host for LXC from scratch
document a set of instructions for creating a new LXC container
configure some containers for actual use

The test machine is running Ubuntu 11.04 (Natty).

I’ve found several resources:

but it’s not clear which are accurate, up-to-date etc. I tried with Ulli’s instructions, but that resulted in a non-booting instance (quite possibly because I did something wrong), seems to duplicate some configuration that’s built-in, and seems to have some unique external dependencies. I then tried Emanuelis’s method, but that looks incomplete, at least by comparison. So I ended up with elements from each.

There is also the question about what networking configuration to use. My hosting provider has some restrictions for fully bridged setups, and recommend Proxy ARP or internal bridge]. If you use an internal bridge, you can either configure a traditional subnet, or use point-to-point transfer links using RFC1918 addresses (see this description by Marc Haber). When I initially tried the latter, I found that I needed to specify “scope link” on the internal link (and use “pinpoint” in /etc/network/interfaces) or specify an explicit “src” on the default route to make sure that the packets were sent from the public IP rather than the internal link IP (see these notes). Then it worked well. This scheme is appealing in that you don’t waste IP addresses to the controlling host and subnet broadcast, and there’s something satisfying about plumbing explicit links. But, it does make the network layout look more complex (for things like ip addr list, ip route list, and outbound traceroute), and complicates the configuration in /etc/network/interfaces, so in the end I changed to a traditional subnet.

Preparing the controlling host

To prepare the controlling host:

mkdir /lxc
#! mount a suitable partition (local, or NFS) on /lxc
ln -s /lxc /var/lib/

apt-get install lxc debootstrap

aptitude install vlan bridge-utils python-software-properties screen libpcap-dev
aptitude install tcpdump ntp

# prepare cgroup
mkdir -p /cgroup
echo "none /cgroup cgroup defaults 0 0" >>/etc/fstab
mount /cgroup

apt-get install bridge-utils
apt-get remove network-manager network-manager-pptp

# allow ip forwarding
cat <<EOD>/etc/sysctl.d/20-lxc.conf
net.ipv4.ip_forward = 1
EOD
sysctl -w net.ipv4.ip_forward=1

# create internal bridge and allow forwarding. Adjust address for your allocation
brctl addbr br0
cat >> /etc/network/interfaces <<EOM
auto br0
iface br0 inet static
    address 46.43.55.73
    netmask 255.255.255.248
EOM
ifup br0

Creating a new container

To create a new container:

# adjust these for your purposes
NAME=natty2
THIS_IP=46.43.55.74
GW_IP=46.43.55.73
NETMASK=255.255.255.248
BITS=/29

LXCDIR=/var/lib/lxc

ROOTFS=$LXCDIR/${NAME}/rootfs
CONFIG=/root/lxc-${NAME}-config.tmp
cat > $CONFIG <<EOM
lxc.network.type = veth
lxc.network.link = br0
lxc.network.name = eth0
EOM

# create (but not start) the container
lxc-create -n $NAME -t natty -f $CONFIG

# add the root user from the controlling host to the container password file
grep root /etc/shadow > $ROOTFS/etc/shadow.new
egrep -v '^root:' $ROOTFS/etc/shadow >> $ROOTFS/etc/shadow.new
mv $ROOTFS/etc/shadow.new $ROOTFS/etc/shadow
chgrp shadow $ROOTFS/etc/shadow
chmod o-rwx $ROOTFS/etc/shadow

# copy authorized_keys
if [ -f /root/.ssh/authorized_keys ]; then
  cp -a --parents /root/.ssh/authorized_keys $ROOTFS
fi

# copy some configs from the controlling host
cp -a --parents \
  /etc/ntp.conf  \
  /etc/timezone \
  $ROOTFS

# update the resolv.conf config (lxc-create put the controlling host resolv.conf into "original")
cat $ROOTFS/etc/resolvconf/resolv.conf.d/original > $ROOTFS/etc/resolvconf/resolv.conf.d/base

# configure the network
cat > $ROOTFS/etc/network/interfaces <<EOM
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address $THIS_IP
  netmask $NETMASK
  gateway $GW_IP
EOM

# update the apt sources to include multiverse
cat <<EOD> $ROOTFS/etc/apt/sources.list
deb http://de.archive.ubuntu.com/ubuntu/ natty          main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu/ natty-updates  main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse
EOD

# remove unneeded postinstall scripts (container does not have udev or the graphical boot animation)
rm $ROOTFS/var/lib/dpkg/info/udev.postinst
rm $ROOTFS/var/lib/dpkg/info/plymouth.postinst

# copy kernel modules
export kernel=$(uname -a | awk '{print $3}')
mkdir -p $ROOTFS/lib/modules/$kernel/kernel
cp /lib/modules/$kernel/modules.dep $ROOTFS/lib/modules/$kernel/
cp -R /lib/modules/$kernel/kernel/net $ROOTFS/lib/modules/$kernel/kernel/

# add tmpfs
cat <<EOD > $ROOTFS/etc/fstab
tmpfs  /dev/shm   tmpfs  defaults  0 0
EOD

# add ipv6 localhost
cat <<EOD >> $ROOTFS/etc/hosts

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
EOD

# set locale info
cat >> $ROOTFS/etc/environment <<EOM
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"
EOM

cat > $ROOTFS/etc/default/locale <<EOM
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"
EOM

# start the container
lxc-start --name $NAME --daemon \
       --console $LXCDIR/${NAME}.console \
       --logfile=${LXCDIR}/${NAME}.log

# connect to it, log in as root. Note you can escape out with Ctrl+a q
lxc-console --name ${NAME}

export LANG=C

# test networking
ping -c 1 www.google.com

# update packages and install more
apt-get update
apt-get install -y apt-utils iptables rsyslog sudo
apt-get install -y ssh ntp lsof wget
apt-get install -y iputils-ping mtr-tiny dnsutils bind9-host
apt-get install -y ia32-libs libterm-readline-gnu-perl dialog
apt-get install -y aptitude tcpdump man less curl

# set supported locales
cat <<EOD> /var/lib/locales/supported.d/en
en_US                   ISO-8859-15
en_US.Latin1            ISO-8859-1
en_US.Latin9            ISO-8859-15
en_US.ISO-8859-1        ISO-8859-1
en_US.ISO-8859-15       ISO-8859-15
en_US.UTF-8             UTF-8
en_GB.UTF-8             UTF-8
EOD
dpkg-reconfigure locales

# remove unused init.d scripts for the hardware clock
/usr/sbin/update-rc.d -f umountfs remove
/usr/sbin/update-rc.d -f hwclock.sh remove
/usr/sbin/update-rc.d -f hwclockfirst.sh remove
rm /etc/init.d/hwclock*

# remove unused audio devices
cd /dev
rm mixer* *midi*  audio* dsp* smpte* mpu* sequencer sndstat

# remove hardware clock and boot settings
cd /etc/init
rm -f hwclock*  plymouth*

# optional: reboot to test the container comes up, and log in again
reboot

lxc-start --name $NAME --daemon \
       --console $LXCDIR/${NAME}.console \
       --logfile=${LXCDIR}/${NAME}.log

# disconnect from the console
Ctrl+a q

# ssh in
ssh -l root $THIS_IP

I imagine there will be further tweaks, but this is a good start.

LXC Commands

We’ve already seen lxc-create, lxc-start, lxc-stop, and lxc-console.

To list processes belonging to all LXC containers:

root@thunder:/$ lxc-ps --lxc
CONTAINER    PID TTY          TIME CMD
natty1     19126 ?        00:00:00 init
natty1     19181 ?        00:00:00 upstart-udev-br
natty1     19183 ?        00:00:00 sshd
natty1     19186 ?        00:00:00 udevd
natty1     19269 ?        00:00:00 udevd
natty1     19270 ?        00:00:00 udevd
natty1     19568 ?        00:00:00 upstart-socket-
natty1     19594 pts/4    00:00:00 getty
natty1     19596 pts/2    00:00:00 getty
natty1     19597 pts/3    00:00:00 getty
natty1     19613 pts/1    00:00:00 login
natty1     19614 pts/5    00:00:00 getty
natty1     19706 pts/1    00:00:00 bash

or:

root@thunder:/$ lxc-ps --lxc --forest
CONTAINER    PID TTY          TIME CMD
natty1     19126 ?        00:00:00 init
natty1     19181 ?        00:00:00  \_ upstart-udev-br
natty1     19183 ?        00:00:00  \_ sshd
natty1     19186 ?        00:00:00  \_ udevd
natty1     19269 ?        00:00:00  |   \_ udevd
natty1     19270 ?        00:00:00  |   \_ udevd
natty1     19568 ?        00:00:00  \_ upstart-socket-
natty1     19594 pts/4    00:00:00  \_ getty
natty1     19596 pts/2    00:00:00  \_ getty
natty1     19597 pts/3    00:00:00  \_ getty
natty1     19613 pts/1    00:00:00  \_ login
natty1     19706 pts/1    00:00:00  |   \_ bash
natty1     19614 pts/5    00:00:00  \_ getty

I’m going to skip lxc-ls because it’s needlessly confusing.

The recommended way of sharing part of the controlling host filesystems with the container is to use bind mounts. For example:

# on the controlling host
root@thunder:/# lxc-stop --name natty1
root@thunder:/# mkdir /mydata
root@thunder:/# mkdir /var/lib/lxc/natty1/rootfs/mydata
root@thunder:/# mount -o bind /mydata /var/lib/lxc/natty1/rootfs/mydata
root@thunder:/# lxc-start --name natty1 --daemon \
>       --console /var/lib/lxc/natty1.console \
>       --logfile=/var/lib/lxc/natty1.log

# in the container
root@natty1:/# ls -ld /mydata
drwxr-xr-x 2 root root 4096 Jun 13 11:53 /mydata
root@natty1:/# touch /mydata/hi.txt

# on the controlling host
root@thunder:/# ls /mydata
hi.txt

So far

LXC seems to work just fine. It’s fast. Documentation is lacking, and tooling seems limited. In terms of complexity it’s not dissimilar to the early days of Xen and KVM. So far so good. At this point I can see myself switching to LXC for my hosting purposes.

To do next (time permitting):

reconfigure the host and use LVM for the containers
actually use it for a while for for real work in multiple containers
try nested containers, to test the install instructions (and just because you can)
review online resources more to see what else I’m missing, and update this post accordingly
maybe do some performance tests
look into libvirt support
look into limiting CPU/IO etc through cgroups, and document that in a separate post
run some KVM guests alongside; that should just work
configure IPv6
experiment with NAT to see if I can add additional containers on RFC1918 addresses

ATOM for jekyll

2011-06-10T00:00:00+00:00

I found ATOM for jekyll, so now we have a feed.

Bytemark

2011-06-10T00:00:00+00:00

I’ve got a new premium dedicated server with Bytemark. Remote console, ipv6, good support.

The machine came with Maverick (Ubuntu 10.10) installed, so I upgraded to Natty (11.04). Note: you need to first install iptables, or do-release-upgrade fails (filed as Ubuntu Update Manager Bug #794590).

Apache Lucene Eurocon

2011-06-10T00:00:00+00:00

Just announced:

Apache Lucene EuroCon 2011, end of October in Barcelona.

Interests

2011-06-09T00:00:00+00:00

Just testing inline image handling in Jekyll. Here’s a tag cloud created with Wordle.

Embedded gists

2011-06-09T00:00:00+00:00

Testing embedded gists. Here is a .bashrc, with code to display RVM stuff in the prompt.

ALUG meet June 2011

2011-06-09T00:00:00+00:00

Location: Coach and Horses pub, Norwich. Present: Bill, David, DralaFi, katsmeat, mak, quinophonex, StillVoid, stevepdp, ViperFang

ViperFang’s Arduino Nano 7-segment display was proudly announcing the ALUG table, at least until he started “improving it” with his USB soldering iron. We admired katsmeat’s bicycle light mounting made from Shapelock, as featured on WikiPedia’s Polycaprolactone page. Beer was consumed. Various topics were discussed (Lassange, the electricity usage of bitcoin farming versus cannabis farming, C#/.NET, home schooling, Python, the car industry, Arch Linux, beer and diets, Chef, much else). Sudo privileges were duly abused for the installation of computer games, and someone felt the need to boot Linux on JavaScript PC emulator running on Firefox running on an Android phone, for no good reason at all. A good time was had.

Blogging Like a Hacker

2011-06-08T00:00:00+00:00

The old look

The greenhills.co.uk website has remained mainly unchanged for over a decade. Its main design dates back to the “homepages” of the early Web, using the most basic of HTML. The look and feel was one of simplicity and contrast, with some minor usability features (keyword high-lighting, breadcrumbs), and un-constrained resizing. For its extremely limited purpose, that has been just fine. But by today’s standards it is looking dated, is too light on content, and doesn’t suit modern screen sizes.

The implementation was based on some Perl scripts that convert XML into HTML, applying some common page elements, with some minimal CSS. The deployment used a simple Makefile and rsync, and a standard Apache server, later replaced by lighttpd. I still prefer markup-based content and command-line based management over UIs, and I still find static sites appealing, but there are better templating options available today.

So, it’s time for an updated approach, and a fresh look and feel.

I’m resisting the temptation to turn this into a play project with the latest crop of server-side technologies (I’m sure there is Node.js/MongoDB fun to be had), because that’s not conducive to either achieving a result in a reasonable timeframe, or simplifying long-term maintenance. I could give up and migrate to some hosted blog service, but I like independence. I could follow the path of least resistance with WordPress, but I’ve managed to stay away from PHP for too long to admit defeat now.

I decided to use jekyll (intro, code, wiki), and named this blog post in its honour.

Jekyll uses text markup (Markdown), is managed from the command-line, and works well with git. It has a decent templating engine. It’s a mature project, and open source. It uses ruby, which I’m familiar with. It’s a static site generator, which makes hosting easy. All good.

The new look

For the page layout I chose a standard blog design, to make it instantly recognisable: header, horizontal navigation, main content, with a sidebar. Visually, I’ve toned down the contrast, and removed the keyword highlighting and breadcrumbs, to get a more relaxed look. I’ve applied a maximum width, and use flush left / ragged-right alignment, to keep the content readable. Post pages work well with Safari’s Reader.

As I don’t expect to be blogging much, but want to give visitors some updated content, I added Twitter and Google Reader content in the sidebar. For now that uses standard webclips, which are slow to load, but that can be addressed later.

I’ve used a newer photo for accuracy, reduced it in size for modesty, and moved it to the sidebar to make the site content more personal, and to tie in with the more real-time elements from the sidebar.

The header photo is from a recent visit to San Francisco’s Baker Beach. I chose the photo because it has green(ish) hills, but with some rock faces and trees for interest, and chose this crop because it is somewhat symmetrical and draws the eye to the middle of the page.

While more tweaking is to be done, this a good start.

Lucene Revolution 2011 presentation

2011-05-25T00:00:00+00:00

Slides for my "Practical Search in the Cloud" talk at Lucene Revolution 2011 in San Francisco are now online.

Web Sequence diagrams

2011-05-19T00:00:00+00:00

Drawing sequence diagrams by hand in OmniGraffle or some such? Check out WebSequenceDiagrams, a hosted service that generates beautiful sequence diagrams based on a simple domain language.

South Norfolk Sustainable Communities Day

2010-10-02T00:00:00+00:00

I spent this Saterday at the South Norfolk Sustainable Communities Day 2010 at the Costessey Centre.

I attended three workshops, wearing my Wymondham Scouts and Guides Hall Management Committee hat.

The Grants and Funding workshop

by Meryl Harding from the South Norfolk Council.

This workshop was more about general fundraising processes rather than specific funds available from the council itself. Lots of good info about tips and pitfalls wih applications, and I look forward to reviewing the slides. Some points that stood out:

Ensure that your project is covered under the governing documents of your organisation
If you’re a single person, become a team of a few
Have policies for: Health and Safety, Equality, Protection of Children and vulnerable adults. Voluntary Norfolk has some templates
Quantify the amount of time volunteers at your organisation spend. When quantifying the value of volunteer time in monetary terms, accepted norms are £8/h per person, and £15/h for professionals.
Determine strengths in your community – even people who don’t want to commit to time on the committee may have skills to contribute. Note to self: at the next AGM invite, spell that out.
Personality types: “Anchor” is the person who takes pause to question the project being suitable/feasible, which can be viewed negatively as prohibiting progress, but also positively as a sanity check.
Obviously need a project plan, with milestones. Need a consultation. Need a communication plan, to show partners and beneficiaries what progress has been made. We do that as a matter of course, but it’s not something we normally spell out in an application; we should.
Need a delivery team, which may be distinct from the funding team.
Spell out what happens after a project. Funder like a lasting positive effect, and it’s important to show how any obligations/costs (if any) would be met.
Projects ideas/plans can run away with themselves, in a direction that’s not a good match to the organisation, or you personally.
Capital projects need legal tenure of the site. Our management committee is OK, as we act as Custodian for the owning trust.
Do you need planning permission for your project?
See the presentation for a list of advisors/helpers. For me, Voluntary Norfolk and Norfolk RCC look useful for helping to find grants.
Norfolk Pro Help is an organisation that provides professional help, from laywers, architects etc.
local government sources of funding: County Council, South Norfolk Council, Wymondham Town Council
GRANTnet. I signed up for that a while back but haven’t used it yet.
Meryl gets daily notification on funding opportunities with occasionally short deadlines. Email her specifics of your project to be notified of possible matches, and get a project plan prepared so you can execute an application quickly.
From the “Making It Happen” slide: people from the left (funders) want to see things from the column on the right (volunteers etc).
Some companies are happy to donate materials, e.g. B&Q (See B&Q Community) often for specific types of project (e.g. pond materials)
Some villages have village charities. The Parish Clerk (in my case Town Clerk) would know.
loans: often local people/organisations are prepared to extend loans. Availability is typically word of mouth, but again the Parish Clerk would know.
reporting (during and post project) is often important, and in some cases so onerous you may regret ever doing the project.
Some public companies have schemes for matched fundraising, or volunteer teams (something mentioned in committee by Tracey; was it Aviva?).
LCF companies in the area: WREN Veolia BIFFA
check out the Big Lottery Fund
South Norfolk Council Neighbourhood Fund
Before doing Lottery applications, get adivce (e.g. from Meryl)
From our region 29% of applications (not sure to who) were succesful. I guess that means I shouldn’t feel bad about an unsuccesful application

I asked a few questions:

For emergency short-term funding (like our heating), there isn’t anything available.
I asked about the typical “have you applied elsewhere” question in applications. Funders quite like to see different applications for different projects under some umbrella initiative, but they don’t like you hedging your bets by applying to different source for the same project.

Energy Efficiency for Domestic properties

Christopher Ejugbo (@cejugbo) from the Energy Saving Trust.

Topics included energy saving and renewable energies. There are still cavity and loft insulation grants for the over-70s and people on income support. There is increasing interest, and shortening payback times, in domestic Photo Voltaic, especially since the Feed-in Tariff scheme became available earlier this year.

He mentioned free solar schemes where a company essentially leases your roof, installs panels, gives you free electricity, and then make money on selling the excess to the grid. As I write these notes, the TV shows an ad for HomeSun which sounds like the scheme he referred to.

There was an audience question about an estimated figure of 10% energy use of standby mode devices. It may be fashionable to rag against unsubstantiated statistics, but for the purpose of this presentation I didn’t feel it really mattered much. Still, I had a quick look on my iPhone (thanks ubiqutous wireless) at David MacKay’s Sustainable Energy book online and found the Chapter on energy Efficiency where he quotes a 8% figure from the International Energy Agency 2001 study, see annex A2.2; they are estimates, and there’s quite a spread between countries. But I guess that’s where that figure comes from, and I would imagine the subsequent standby regulations for manufacturers have improved things.

David also nicely illustrates a personal case study where he was able to make significant savings in his own house after switching off “vampires”. See also the chapter on Gadgets. But do balance all this with chapter Every BIG helps. Anyways, I digress; back to the presentation.

There was a question about Air Source Heat Pump efficiency in practice. Christopher referred to the Heat pump field trial report and its conclusions. As a heat pump owner myself, I looked over the recommendations and re-appraised our installation:

the new part of our building is well insulated
we have it mainly for underfloor heating
we have occupation during the day
we have solar collector assist
we’re not served by the gas network
we were replacing and end-of-life boiler that relied on oil deliveries

so in our case it made a lot of sense. Of course the system is as yet untested in winter, so we’ll see.

Waste Management

This session was split in parts: an introduction by Alexandra Bone (@alexandraboneuk) Senior Environment Officer of the South Norfolk Council, a case study of the Denton Community Composting scheme by organiser Liz Cargil, and a presentation on recycling strategy from Gill Flanagan from the Norfolk County Council. Followed by general questions to Alex and Gill about mainly rubbish bin collection.

I found the Community Composting Scheme intriguing. Denton has 150 households, 61 take part, scheme has run for 7 years now. Waste is collected weekly in bags. A volunteer working party processes the material one afternoon a week. Waste is collected already separated (grass, weeds, other). Rough bits are shredded. The reception bins reduce to half size in a week, are forked over into other bins and left a year. The resulting compost is sold at 50p/bag, back to the local community only. Some gardeners buy up to 20 bags in one go. Income from bags is about £140/year, additional income comes from the council’s recycling credits. Total proceeds are ~£800. Tricky aspects:

access/location is important
green garden waste only; no kitchen/food waste
paperwork/licenses because they collect, and more because they process
seasons depend on weather
weather conditions require extra mangement (e.g. add water in dry periods)
attracting volunteers

It sounds like a successfull and well-run scheme; but also sounds like a lot of work.

Gill Flanagan explained the NCC recycling credits scheme. It is built on the idea of disposal avoidance – disposing costs the council money (£72/tonne), so if you reduce waste you save money, and get to share in the saving. Materials collected are paper, glass, cans/metals, textiles. Collection/processing is often arranged through third party recycler services. Furniture is re-used where possible, like through the Salvation Army Furniture Workshop (see also Furniture Re-Use Network. The recycling credits scheme deals with 350 organisations, 6300(?) tonnes, and £239.000 credits. More on the website.

The recycling credits are an important part of the income for my organisation, so I asked if any changes in level were foreseen. Nothing is known at this point, but given that this is a cost saving scheme, it is perhaps less likely to be affected by budget cuts.

Keynote

Prof. Tim O’Riordan Emeritus Professor of Environmental Sciences at the University of East Anglia.

This was a talk without slides, and my note-taking was lacking here so I can’t do the talk full justice, but here is a flavour. Please let me know if you have corrections.

Tim started off sharing his astonishment how the environmental agenda has been pretty much disappeared from the news and public conscious, because of the banking crises and fears over austerity, even though climate change is a problem of much more major impact than short-term economic bips. The focus now seems to be mainly on the short term, and getting back to the (unsustainable) way things were, rather than on looming crises, future generations, and humanity’s survival.

He pleaded for action at a local level, by normal people, pushing society at large and politicians in particular from the bottom up.

He proposed a South Norfolk Low Carbon Fighting Fund to part-fund local initiatives, but keep it outide of councils so the fund can be ring-fenced from budget pressures. The fund could be generated from among other things £1 on parking. I found that amusing, given the enourmeous controversy over council pay car parks in rural market towns like Wymondham and Attleborough in the last few years. Many people in the rural community can only access services in those towns by car, and there are concerns for local shops if access is impeded, with customers chosing out-of-town supermarkets with free parking instead. This is an ongoing battle, and extra fees would be deeply unpopular and politically difficult.

Another suggestion was a Norfolk Citizenship Initiative, aimed at young people, to allow them to contribute to the community at a decent wage. The idea would be to “make better people”. This sounds intriguing.

There was a mention of people in the community getting together to take advantage of ASDA bulk buying, but I know no more about that.

There was a mention about Parish Car Club schemes, and tax reduction for car-less households.

Procurement policies should include supplier requirements with regards to sustainable practises and possible revenue.

Section 106 could be another avenue for sustainable practises. This prompted one builder in the audience to complain that there is already so much in regulation and requirements that it just becomes way too expensive to actually build a house.

The Carbon Reduction Commitment was mentioned, a scheme that requires CO2 reduction in e.g. schools.

Reepham Low Carbon Communities Challenge looks like an impressive initiative, with projects right accross the community. By the way: unusual visual navigation on the front page – interesting idea, but I think it actually interferes.

Prof. O’Riordan is publishing a new paper is forthcoming in the next couple of months; hopefully that will talk about his ideas in more detail.

Other

Helene “Master Composter” Rinaldo gave me some leaflets and advice about composting (I want to put my excess building pallets to good use) and wormeries.

Jo Maddock from Voluntary Norfolk gave me some advice on grants (specifically Awards for All and Norfolk Community Foundation who do Grassroots).

A lady from the Norfolk Wildlife Trust identified my salamander as a Smooth Newt (to be confirmed).

Briefly met a fellow Dutchman: Erik Buitenhuis from Transition Norwich

Discussed some local issues with Cllr Joe Mooney (Wymondham Town Council) and Cllr Robert Savage (Norfolk County Council).

Met John Penell from Bunwell Parish Council.

Met John Heaser from Norfolk Freegle. Freegle split from FreeCycle in recent years. John is looking for funding to develop the website, using Microsoft technologies.

Had an interesting chat with a lady from the stand about emergency preparedness, business continuity, and community emergency plans. In particular I think the idea of local door-knocking in remote areas during severe weather conditions is an interesting idea, and may be something I’ll look at implementing around my home. The other thing of note is that “your local scout hall” is sometimes identified as a temporary community shelter during emergencies, which is not really something we’ve thought about at all. See Norfolk Prepared, NORMIT and the SNC Environmental Emergencies page.

Feedback

I think it was an interesting and useful event.

Location, facilities and staffing was great, and well organised. There was a good turnout, although I think it remains hard to get through to the right people who can take initiatives on a local level.

Suggestions: It was a little unclear that the workshops were repeated in three slots, and you didn’t know in which order you were planned in until the day.

I’d like presentations posted online before the event, to make it easer to refer back during the workshop, and make it easier to write up after.

A twitter hashtag would be good too; but I expect I’m the only one there who would care.

Would I come again? Quite possibly, if there wasn’t too much repetition.

Tweepy

2010-09-29T00:00:00+00:00

Twitter API for Python: tweepy (docs) supports OAuth.

I used it for a little command-line twitter client app; worked well.

7.5km

2010-09-29T00:00:00+00:00

Ran a new route:

Cambridge Geek Night

2010-09-29T00:00:00+00:00

Visited Cambridge Geek Night 5 (#cgn5) at The Cambridge Union. See also on Lanyard.

Topics included open data, scaling services, and startups.

The talks:

Lisa Evans (@objectgroup) from the Open Knowledge Foundation talked about open government data for Where Does My Money Go?. Nice visualisation of aggregate and granular data, and war stories of obtaining data from government and dealing with data compatibility issues.
Martin Kleppman (@martinkl) from Rapportive. Rapportive is a value-add for Gmail, otaining info related to items in your inbox by aggregating external data from social networks etc. It is implemented as a plugin which inserts a JavaScript tag, which then loads the JS logic from Rapportive, which then does the fetching and DOM manipulation. Rapportive uses Nginx, Ruby, RabbitMQ, Redis, S3, github, heroku. They will be looking at Hadoop, Solr, Memcached. They outsource some tech expertise. And they’re moving to San Fransisco. I asked a question about risks of reliance on gmail’s page structure and whether bizdev contract were in place with Google. They address this by lots of checks so they can partially degrade the functionality, and aggressive monitoring. So far there have not been any changes that affected them. The business model is freemium, with premium chargeable features in the pipeline, though one has to wonder if a Google buyout is the real target. I’ve heard their UI replaces the ads inside Gmail; wonder if Google will take action against that. Popular quote from the talk was “Scaling calcifies: don’t do it too early” referring to the problem getting locked into a specific scaling technology too early.
RedGate sponsored the bar. Their intent was to do some hiring, so they spent some time explaining why their company has a great culture and would be fun to work for. They handed out some swag, including copies of The Book of Red Gate 2010 which I liked quite a bit, to my surprise. I couldn’t resist doing the Clive Code Challenge, where I didn’t do too badly.
Someone mentioned a Clojure Cambridge User Group meeting Tuesday 5th October 7pm in The Punter. I’d be interested in the scaling aspects, but I may just not be Lisp enough.
IdeaSpace (@ideaspace) is a co-working space and networking community at West Cambridge Research & Development Park (not far from the MSRC office). £50/month membership. Friday afternoon they have a CamJelly, a public/social/meet thing. Should check it out if I find myself in the area.
Mat Clayton (@MatClayton) from MixCloud gave a talk about scaling an audio service: users upload/download DJ-style mixes. Fascinating and entertaining. Their main problem was not the scaling user data/access but problems with bandwidth costs, which at times threatened the company’s survival. They started off on EC2, moved content service to 100tb.com, then downgraded bitrates (from sometimes insanely high), and are now on OVH because they’re cheap. Migrating massive amounts of data between providiers alone is a challenge. Various issues around music licensing were discussed. For UK purposes they’re a radio station, and have appropriate PRS for Music licenses with track reporting. Mat works out of Cambridge, near the redgate folks, the rest of the company is in London. They managed without VC funding – impressive.
Michael Brunton-Spall (@bruntonspall) who works at The Guardian plugged UK Scale Camp.

Met various new people:

Organizer Jo Anslow (@joanslow)
Jonathan Whiteland from YTKO
Ben “tola” Francis @bfrancis from rabbitsoft
Stephen Turner (@sret) from Cytrix
Python lover Michael Brunton-Spall (@bruntonspall) from The Guardian
Xapian developer Richard Boulton @rboulton from Celestial Navigation
Oleg Podsechin (@olegpodsechin) from The Pocket Agency

Meta comments: Some parallels with the OpenTech 2010 conference a couple of weeks earlier: Lanyard is popular, people ID themselves with their twitter username these days, and live event twittering via hashtags is common and useful.

Lighting

2010-06-17T00:00:00+00:00

We’ve been spending quite a bit of time worrying about lighting for the building alterations we’re doing. It used to be simple, back in the seventies in the UK: if you wanted light you typically a 100W incandescent bulb with a Bayonet mount twist/lock fitting, and you were done. These days (2010) there’s a bewildering array of lighting technologies, lamp caps, and light colours available, and building regulations and environmental concerns to go with them. Here I’ll jot notes about my understanding of these issues and the coices we made.

Background

We’re extending a kitchen, enlarging a living/sitting room, adding a bedroom/bathroom/hallway, reworking a pantry, re-roofing a lean-to sideshed and adding a back hall. This affects the rear half of the house, the front rooms and facade remain as-is. New rooms, or rooms that change use, are subject to different building regulation requirements than existing ones.

The kitchen had 2 recessed PAR38 fittings (initially with incadescent bulbs, later with CFL), 3 recessed R80 fittings with incandescent, and 2 triple GU10 Halogen spots. The living room had 2 fittings with 6 G9 Halogen capsules. The bathroom had 7 recessed GU10 Halogen fittings.

Energy efficiency has been an important aspect of this building work: We have heavily invested in insulation (6 inch Celotex) and heating (Air source Heatpump and underfloor heating). Similarly we want to consider energy use of the lighting.

Lessons in lighting terminology

Lighting Technology

For indoor lighting technology the main choices are:

Incandescent lamps. Warm yellowish, flicker-free, instant-on light. They are not very energy efficient, typically using 100W. The bulb is round, fairly large, but not too long. Fittings are typically Bayonet (BC), or screw (ES). These are being “phased out” /banned.
Halogen Lamp. Bright yellowish light, flicker-free, instant-on light. Bulbs are small G4/G9 capsules, and GU10/MR16 multi-facetted reflectors. They get very hot, and don’t seem to last long. Typical 30-50W. These were marketed as “greener” initially, because 50W < 100W, but they’re not very efficient. We have quite a few of these: GU10 spots, 12v MR16s in recessed fittings in the bathroom, and G9 capsules in decorative lighting fixtures in the living room.
Compact Fluorescent. The long fluorescent tubes typically used in commercial or institutional buildings used to flicker and produces a cold white light. These days Fluorescent lamps are controlled by high-frequency electronic ballasts, which reduces the flickering, and they are available in compact sizes, called CFL. The first generation of CFL’s were painfully slow to warm up and reach full brightness, but the technology has improved. And crucially, they are now available in all sorts of bulb shapes and sizes, various lighting colours, and various fittings. CFLs less energy, typically between 9-18W. Lifetime up to 20.000 hours. We were using a few of these, mainly in corridors and utility rooms.
LED Lamps. The old and small LED’s produced insufficient light for general lighting, but the new high-power LED lamps produce a lot of light, and are the most energy efficient yet, between 1-3W. The LED lamp light used to be very cold and bright but these days warmer colours are available. They have also have a long life (50.000 hours)

Lamp Fittings / Caps / Bases / Socket

See My Green Lighting’s overview of the most common types, or Lightbulbs Direct’s Fittings overview for more.

Dons Bulbs is also a good resource, see for example the G24d/G24q datasheet.

Light Colour

The colour of light is called the Colour Temperature. See Lightbulbs Direct’s Colour Temperature overview or Saving-light-bulbs.

Luminous Efficacy

Light output is measured in lumens, and light power use is measured Watts. The Luminous Efficacy is lumens-per-Watt, the higher the better.

Building Regulations

Note: This is my personal interpretation – if you want guaranteed advice consult a paid professional.

In the UK, domestic lighting falls under Part L Building Regulations, in particular L1A New Dwellings and L1B Exiting Dwellings. The specifications can be downloaded freely. The 2006 Edition is current, with a 2010 Edition being valid from 1 Oct 2010.

Given that our building project involves an existing dwelling, I believe Part L1A applies, even for the new extensions.

In L1A (2006), section 43-47 deal with lighting. It demands that “reaseonable provision should be made for … efficient electric lighting”. It specifically talks about achieving 40 lumens/watt, and 1/4 light fittings needing to be energy efficient. Infamously, it says “A way of showing compliance” would be to have fittings that can only take energy efficient lamps. This has lead to development of new fittings, with some building inspectors insisting they get installed. This has turned out to be counter-productive: there is so little choice in lumaires and lamps for those restrictive fittings, that people rip them out as the building inspector’s back is turned. Furthermore, these days there are so many energy efficient lamps available for existing fittings, that there is no need special fittings. But I digress.

In the 2010 Ed, Part L1A refers to the Domestic Building Compliance Guide 2010 where it says in Section 12, Page 123, Table 40, that the new values are 45 lumen/watt and 3/4 light fittings. And interestingly, standard fittings are now OK, as long as they are supplied with low energy lamps. Also, light fittings <5W are excluded from the count, presumably targetting floor level lighting systems and kitchen under-unit lighting, and to prevent people adding LED light fittings to articilally increase the overal count. But, this appears to have an unfortunate consequence: if you’re considering say 20x 3W high output LED Lamps are your main lighting, then you get no credit for that. The workaround would probably be to choose a fitting that would take CFLs, and later switch out the CFLs for the LEDs.

It also refers to the Energy Savings Trust, who have some generic lighting info.

LED Lamp Experiments

Back in 2008 I read about the new Cree LED Lamp on RE UK and in June 2009 I bought some from ledlights4you (was solarwindpowercentre.co.uk, part of Coemi): 12V, MR16 fittings, 1W (£12), 3W (£17), and 9W (£32), in warm white and cool white, with a 180W power supply (~£40) to drive them. My conclusion was:

the 9W produce about as much light as you’d want out of an MR16 size fitting.
The smaller wattage lamps may have uses for task lighting.
the warm white in a pleasant colour.
there doesn’t seem to be a light of light spill, it is very directional, making it not a great choice for general lighting in a large kitchen, unless you use lots of them, which then defeats the energy saving purpose.
the “three dots” look is a bit strange. My wife disliked that quite a bit.
they’re expensive

So now in June 2009 we experimented with CFLs. There seem to be a lot more types on the market than a few years ago, and more light fittings that support them.

Kitchen Spots

We started with some spots, for the new extension, where a sloping roof makes recessed lights less appealing.

We went to the Norwich Lighting Centre (website is content-free), and found a single and double spot we liked: the WOFI PAZ SC SPOT SINGLE S/W 496301640000 (stock code LE199) and WOFI PAZ SC SPORT TWIN BAR 7963.0264.0000 (sock code LE200) in Matt Nickel with Alabaster glass.

These lights are also available online at KES (single, double)

The supplied box suggested it was made by ACTION, or perhaps that’s the range and it is made by WOFI (warning: horrible web site with unsollicited audio, and all content hidden in PDFs and online flash viewers).

They take 8W mini spiral CFLs with a SES fitting and were supplied with IMEX 8W/E14 2700K 240V 350 lumen, 8000 hours, which produce a warm bright light and start fast.

We’re delighted with both.

Kitchen Recessed

Next we had to tackle the recessed lights in the low, beamed ceiling.

First we evaluated GU10 CFLs (Kosnic 11W, warm white). That started orange, and took about a minute to come to brightness; not what you want in a kitchen.

Next, we found a nice recessed light fitting from JCC Lighting, the JC5081 Coral Matrix. Typical recessed lights of this type are 20-24cm, which would be a problem with the beams on our ceiling, but this one was small, at 123mm, and shallow. The construction of the fitting is well done, and the finish is nice. The ballast is in a separate metal box, which looks messy, but makes fitting quite flexible. The bulb, listed as “included” on the box, was actually not included. And we discovered that it uses a very unusal bulb type: the very short Matrix PL-T2, which appears to be only manufactured by MEGAMAN as the T3G24Q218, which is only available in limited colours, and appears hard to obtain. All in all not appealing.

Then we tried the ASD Lighting Mini Atom AT1/FSW113E. This is a commercial-style recessed fitting with integrated ballast, at about half the cost of the Coral Matrix. At 146mm it’s a little bigger than the Coral Matrix, but still much smaller than the normal 20cm+, and it’s just about 12cm deep. It comes with a white and brushed chrome bezel. Construction feels a little plasticky, but the ballast is internal and it comes pre-wired, so it’s a neat package to install. It takes a 13W TCT 4pin bulb in its GX24q-1 fitting, and comes supplied with a cool white bulb. We bought them from TLC Direct (order code GLAT113E) who were very helpful and delivered next day. We’ve been very pleased with this fitting, and installed it in the kitchen, the pantry, and the back hall.

We also got 2 with emergency pack from Tradesman Electrical Distributors. No stock, special order item with a 10 day lead time.

To match in with the spot lights, we will replace the supplied bulbs with warm white ones: General Electric Biax Long Last T/E 13w GX24q-1 4-Pin Colour: 82 - 13PLT8274PIN - F13TBX/827/A/4P - DTE132 - 900lumen, 20000h, 2700K from The Lamp Company, stock code PLT134P-82. The 830 color version (3000K) also matches well.

Living Room

In the living room, we have re-used the existing decorative G9 halogen fixture on the ceiling. In addition, we have added alternative lighting that will probably mean the ceiling lights will be little used.

We added torch-shaped alabaster glass wall lights, from the Norwich Lighting Centre, GLOBO TORCH WALL UP-LT SW SC 44100-1, order code WB532. Made by Globo-Lighting (another content-free website). It came with a 40W halogen in a E14 fitting. We replaced with a CFL (LiLUCO 08979 E14 / 9W / 3.000K.), which takes a little while to reach full brightness, which is OK in a living room. This model is a little wider and shorter than the mini spirals we use in the kitchen, to stop them protruding above the glass. These lights can be individually turned on/off.

We added some (separately switched) spotlights (the same type as in the kitchen), and one halogen picture light, which unfortunately will only take a 40W halogen.

With the wall and spot lights on, that adds up to (59) + (49) + 40 = 121W, compared to 2(640)=480W before. Let’s hope that the practice works out like that.

Bedroom

Paulman Colmar 150 4x35W, 12V Halogen, 150 VA transfo 971.82 http://www.paulmann.com UK reseller http://www.lightsourceeurope.com/ colmar 01905 24152 jessica jones lighting

5x20 colmar 105 £145 4x35 colmar 145 £145 10m run —

105 5x20 would have been ok

Upstairs Hall

http://www.screwfix.com/app/sfd/cat/products.jsp?id=95841&ts=48396 Recessed Maintained Emergency Bulkhead 8W

2xEnergy Saving Small flush fitting finished in nickel matt with opal glass diffuser http://www.thelightingsuperstore.co.uk/product.asp?productid=39168 wofi 9311.02.64.0340 suzuka 2xe14/11w

Bathroom

thelightingsuperstore http://www.thelightingsuperstore.co.uk/product.asp?productid=30536 Product Code = TP2114 LISBON BATHROOM SPOTLIGHT Description = IP44 Rated low energy modern style bathroom spotlight with 4 adjustable spots, finished in chrome with a frosted and clear glass shades.

Funny Sites

2010-04-21T00:00:00+00:00

These are funny and useful. Well, ish.

First upstream GitHub merge

2010-04-17T00:00:00+00:00

I’ve used Mercurial more than Git, but GitHub looks so nice and polished that I wanted to see it in action.

So by way of experiment I forked Io, and fixed up some documentation typos. Today these were merged back: 1882b78. How exciting. I love the network graph:

which is explained here and is html5 canvas goodness.

Later I was able to push a fix to a mailing list issue, providing a convenient diff.

Likes so far: Ah, the freedom of DVCS and open source: not having to ask permission to fork. Checkin early, checkin often. Cloning trees locally. The three-way “pull upstream, push origin” model. Pretty diff viewer built-in. And Github does make it easy.

Dislikes: Git’s complexity will take some getting used to. The GitHub Issues lists look a little too simple; I wonder if it is sufficient in practice.

vim cursor on last line

2010-04-16T00:00:00+00:00

This is useful:

vim + filename

makes vim start with the cursor on the last line. Ideal for a blog posting script…

Martijn Koster's Pages

Installing Distributed Solr 4 with Fabric

A Mini-ITX server

Requirements

Parts

Observations

Remote Management

Software Install

Conclusion

Cloning VMs with KVM

Base Image Creation

Network Preparation

Long-term VMs: Clone to raw LVM guests

Short-term VMs: Thin-provisioning with Qcow2

Closing

SuperMicro server for virtualisation

Boto with IAM roles

s3cmd with IAM roles

Creating the IAM Role

Temporary credentials in s3cmd

WebCrawler Screenshots

Metal as a Service

Some minor blog changes

OSX mounting the NAS

RabbitMQ First Look

Configuration

Clients

Learning

Code in a blog

Creating Machines with Chef

The Knife Template

The AMI

Upgrading to Chef 0.12, and using Environments

The Old Environments

The New Environments

Mistreated by Dell

Review - Plugable USB Display Adapter UGA-2K-A

Making Pages Faster still

ELB traffic for the wrong host

Performance testing with webpagetest

WWW or bare domain?

Spotify / Pinpoint Window Move Bug

Embedding video in blog posts

Linux Containers (LXC)

Background

Getting Started

Preparing the controlling host

Creating a new container

LXC Commands

Sharing a filesystem

So far

ATOM for jekyll

Bytemark

Apache Lucene Eurocon

Interests

Embedded gists

ALUG meet June 2011

Blogging Like a Hacker

Lucene Revolution 2011 presentation

Web Sequence diagrams

South Norfolk Sustainable Communities Day

The Grants and Funding workshop

Energy Efficiency for Domestic properties

Waste Management

Keynote

Other

Feedback

Tweepy

7.5km

Cambridge Geek Night

Lighting

Background

Lessons in lighting terminology

Lighting Technology

Lamp Fittings / Caps / Bases / Socket

Light Colour

Luminous Efficacy

Building Regulations

LED Lamp Experiments

Kitchen Spots