WorldTurner Blog
Creating a bootable virtual appliance, fully automated
I was looking for a way to create a VMWare image in an automated way, without actually having to start VMWare to make the image.
I can much easier integrate such a process into an automated build than if I actually have to start VMWare, and I'm not even sure if I can entirely automate an OS installation inside VMWare.
The goal is to include the building of deployable machine images into the continuous integration process, and to be able to recreate exactly the same machine configuration without having to worry what changes may have been made by hand.
Many steps turned out te be relatively straightforward since people already figured this out or even created tools for the step. However, making the image bootable using Grub 2 was a major challenge for me, and I couldn't find any resource on the Internet that described how to do this.
Steps
There are several steps involved in this process:
- Create a file that's going to serve as the disk image
- Partition the disk image file
- Create a file system on the partition
- Bootstrap the operating system, in this case that is Ubuntu
- Install additional required packages
- Configure the system using your favorite system configuration tool
- Install a boot loader
- Convert the image to VMWare vmdk format
Almost all steps turned out to be relatively simple after I found the right blog posts describing what to do. The biggest headache was the boot loader. I wanted to use Grub, and it turned out that Grub is not tailored to this use case, and would not understand which disks to write to and what should be the kernel to boot. However, I managed to get this working by using only the low-level tools available in grub: grub-mkimage and grub-setup. I'll describe that later.
Create a file that's going to serve as the disk image
I create the image file using dd, but I want to create a sparse
file: not taking up space on the physical disk for sectors that only
contain null bytes. This is much faster and takes up less real disk
space, and since I am going to convert the image to vmdk format anyway,
there is no drawback to this.
/bin/dd if=/dev/zero of=disk.img bs=1 count=0 seek=4G
This creates a 4 Gigabyte image in the file disk.img. You do need to know in advance how big you want the partition to be, and it should be at least big enough to hold the OS and everything that you want to install on it.
Partition the disk image file
I use GNU parted for this, first to label the disk image as an DOS-style partitioned disk. Secondly, I create a partition that takes up the entire disk. I'm starting the partition at sector 64, because the boot loader needs this space between the MBR and the first partition.
parted -s disk.img mklabel msdos
parted -s --align=none disk.img mkpart primary 64s 100%
Create a file system on the partition
Before it's possible to create a file system on the partition, the OS needs to have proper access to the partition. The easiest way to do this, which is necessary anyhow later when the OS is installed, is to access the image file as a block device, and then map the partitions on it to block devices.
To do this, three tools are needed: losetup, dmsetup and kpartx.
Because I'm using several values very often, I have turned them into shell variables:
LOOP_DEV=/dev/loop0
IMG_FILE=disk.img
IMG_SIZE=4 # Gibibytes = 1024^3 bytes
DISK=sda
FS_TYPE=ext4
To turn the image file into a block device, I use:
losetup $LOOP_DEV $IMG_FILE
I then want to use kpartx to create a block device for the single partition I created on the disk. kpartx works in conjunction with the device mapper tool dmsetup.
echo "0 $[IMG_SIZE*2097152] linear $LOOP_DEV 0" |
dmsetup create $DISK
This creates the block device for the entire disk /dev/mapper/sda. 2097152 is the number of 512-byte sectors in a Gibibyte.
kpartx -a /dev/mapper/$DISK
This creates a block device for the partition /dev/mapper/sda1
I perform another step that is not necessary for creating the file system, but will be necessary later to work with the grub bootloader installation. Grub doesn't understand when the block device is in the /dev/mapper subdirectory; it needs the device to be directly under the /dev directory. So we copy the device nodes that we just created to /dev under the new names /dev/msda and /dev/msda1
cp -R /dev/mapper/$DISK /dev/m$DISK
cp -R /dev/mapper/$DISK\1 /dev/m$DISK\1
Then I finally create the file syste on the partition:
mkfs -t $FS_TYPE /dev/m$DISK\1
Bootstrap the operating system
Before doing this, I mount the partition that we just created:
CHROOT=/mnt/my-mount-point
mkdir -p $CHROOT
mount /dev/m$DISK\1 $CHROOT
After that, all the work has been done by the people who created debootstrap.
UBUNTU_DISTRO=lucid
debootstrap $UBUNTU_DISTRO $CHROOT http://archive.ubuntu.com/ubuntu/
Install additional required packages
Additional work is done in a chrooted environment, so whatever tool is running believes that it is running inside the image that you are creating.
Additional work can include upgrading to the latest version of ubuntu.
To be able to boot the image, we also need a kernel and kernel modules. I combine that into one piece of script:
cp /etc/apt/sources.list $CHROOT/etc/apt/sources.list
chroot $CHROOT <<EOF
export DEBIAN_FRONTEND=noninteractive
apt-get update -y
apt-get dist-upgrade -y
apt-get install -y linux-headers-generic linux-restricted-modules-generic linux-restricted-modules-generic
apt-get install -y linux-image-generic
# Set password of root to 'root'
passwd root <<__END__
root
root
__END__
EOF
Configure the system using your favorite system configuration tool
The last part of the script sets a password for user root so that we can log into the image after booting it. This kind of task is probably better delegated to a tool like puppet. This is what I'm going to work on next: getting puppet integrated into this process.
Install a boot loader
Now here is where it gets interesting and what I spent most time on.
I want to use grub (version 2, which is now called grub, while grub 1 is now called grub-legacy). I had similar problems with grub-legacy, and didn't think it was worth resolving them for an old version of grub.
In the typical use-case for grub, the administrator runs update-grub whenever the system configuration changes with respect to installed kernels, to gather all information for the grub configuration file grub.cfg
To install the grub boot loader to the MBR, the administrator runs grub-install.
Both of these scripts don't work for disk images in files that are going to be booted in VMWare. After trying for a long while to direct these higher-level grub tools to do what I needed them to do, I gave up and decided to look at the lower-level tools behind them.
These low-level tools turned out to be grub-mkimage, which is needed to create a core image, and grub-setup, which is needed to write the boot image and the core image to the disk.
Before these two tools can be used, I need to create some configuration files that are normally automatically generated for you: device.map, load.cfg and grub.cfg
Quite a few times, I was stuck and had a hard time finding out why booting the image failed. After getting the source code for grub 2 (version 1.98), and tweaking it a little bit to let it print more diagnostic information, I found out where I omitted some steps.
In particular, in fshelp.c on line 198, I changed it from:
return grub_error (GRUB_ERR_FILE_NOT_FOUND, "file not found");
to
return grub_error (GRUB_ERR_FILE_NOT_FOUND, "file not found %s", path);
Knowing which file wasn't found made it a lot easier for me to debug things. It turned out basically that you need to copy all files for your architecture to /boot/grub:
GRUBDIR=$CHROOT/boot/grub
cp $CHROOT/usr/lib/grub/i386-pc/* $GRUBDIR/
To run grub-setup, we need a file called device.map that tells Grub which grub device is an alias for which unix device:
echo "(hd0) /dev/m$DISK" >$GRUBDIR/device.map
I found that using UUID's to identify disks is the best way to make sure that grub is talking about the same disk both when you are creating the image and when you are booting it.
The UUID of a disk can be retrieved using the command blkid, for example:
# blkid /dev/msda1
/dev/msda1: UUID="d88b3ef5-38a4-47e0-95a9-3473f3b6e674"
TYPE="ext4"
The UUID is the variable part in the files grub.cfg and load.cfg that we need to create before invoking the low-level grub tools.
The load.cfg is a simple file and can easily be created like this:
eval `blkid -o udev /dev/m$DISK\1`
export UUID=$ID_FS_UUID
echo "search.fs_uuid $UUID root " > $GRUBDIR/load.cfg
echo 'set prefix=($root)/boot/grub' >> $GRUBDIR/load.cfg
echo 'set root=(hd0,1)' >> $GRUBDIR/load.cfg
The grub.cfg was much more complicated, but I stripped it down to the bare minimum that I need:
KERNEL=2.6.32-28-generic
cat <<__EOF__
insmod ext2
set root='(/dev/sda,1)'
search --no-floppy --fs-uuid --set $UUID
linux /boot/vmlinuz-$KERNEL root=UUID=$UUID ro quiet splash
initrd /boot/initrd.img-$KERNEL
boot
__EOF__
Convert the image to VMWare vmdk format
For this, there is a great tool: qemu-img
IMG_NAME=disk
VMDK_FILE=$IMG_NAME.vmdk
qemu-img convert -O vmdk $IMG_FILE $VMDK_FILE
To be able to start this in VMWare (I'm running VMWare Fusion on a MacBook but any VMWare will work), you need a .vmx file.
Just copy an existing .vmx file, rename it to disk.vmx and make sure that you update several fields in it:
displayName = "$IMG_NAME Ubuntu $KERNEL"
guestOS = "other26xlinux-64"
nvram = "$IMG_NAME.nvram"
scsi0:0.fileName = "$IMG_NAME.vmdk"
Then copy the .vmdk and the .vmx file to a location where VMWare can find it, open the .vmx file from VMWare and start the machine!
Resources
I found the following articles and blog posts very helpful in figuring out how to do things.
http://www.linutop.com/wiki/index.php/Tutorials/Debootstrap
Using debootstrap. The part about making an image bootable using grub-legacy didn't work for me.
https://wiki.ubuntu.com/DebootstrapChroot
More about debootstrap for ubuntu.
http://ebroder.net/2009/08/04/installing-grub-onto-a-disk-image/
Installing Grub-legacy onto a disk image. This didn't work for me, unfortunately, but was very helpful in pointing out how to use dmsetup and kpartx.
It also pointed me to this Grub issue:
http://savannah.gnu.org/bugs/?27737
To which I found a partial workaround by copying the device mapper devices to /dev from /dev/mapper.
And many manpages plus the Grub2 documentation and source code.
Source code
I've sort-of made a series of scripts that work for me, but need further work to make them generally usable. You can download them in this .tar.gz file
Erwin BolwidtPosted at 11:45AM Feb 04, 2011 by Erwin Bolwidt in DevOps | Comments[0]