zomo tech

Moving disks from Xen to KVM

Sunday, 8. 04. 2012  –  Category: stash

Overview

• Moving virtual machines from Xen to KVM
• The storage for both is under LVM
• Under Xen the storage for each VM isn’t partitioned as a disk, it’s just a filesystem. GRUB under KVM will need to see a boot sector and a partition table.
• Under Xen the storage for each VM doesn’t contain a kernel. GRUB under KVM will need one to boot the VM.

Sequence

• Measure disks on Xen server, create on KVM server. Create the new root disk 10MB larger than its counterpart.¹
  
kyero@xen03 ~ $ sudo lvscan
ACTIVE '/dev/vg4/mail-root' [20.00 GB] inherit
ACTIVE '/dev/vg4/mail-swap' [1.00 GB] inherit


[kyero@vm02 ~]$ sudo lvcreate -L20490M -n mail-root vg4
[kyero@vm02 ~]$ sudo lvcreate -L2G -n mail-swap vg4

• Create a boot disk on the KVM server. We make it 4 cylinders long, roughly 32MB.²
  
[kyero@vm02 ~]$ sudo dd if=/dev/zero of=/var/boot-images/mail-boot bs=32901120 count=1

• Partition the boot disk with one partition starting at cylinder zero
  
  [kyero@vm02 ~]$ sudo sfdisk /var/boot-images/mail-boot
  last_lba(): I don't know how to handle files with mode 81a4
  Warning: /var/boot-images/mail-boot is not a block device
  Disk /var/boot-images/mail-boot: cannot get geometry

  Disk /var/boot-images/mail-boot: 4 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /var/boot-images/mail-boot: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /var/boot-images/mail-boot1 :0
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 :
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 :
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 :
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Inappropriate ioctl for device

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Format and mount the partition
  
  [kyero@vm02 ~]$ sudo kpartx -av /var/boot-images/mail-boot
  add map loop1p1 : 0 64259 linear /dev/loop1 1

  [kyero@vm02 ~]$ sudo mke2fs /dev/mapper/loop1p1
  mke2fs 1.39 (29-May-2006)
  Filesystem label=
  OS type: Linux
  Block size=1024 (log=0)
  Fragment size=1024 (log=0)
  8032 inodes, 32128 blocks
  1606 blocks (5.00%) reserved for the super user
  First data block=1
  Maximum filesystem blocks=33030144
  4 block groups
  8192 blocks per group, 8192 fragments per group
  2008 inodes per group
  Superblock backups stored on blocks:
  8193, 24577

  Writing inode tables: done
  Writing superblocks and filesystem accounting information: done

  This filesystem will be automatically checked every 34 mounts or
  180 days, whichever comes first. Use tune2fs -c or -i to override.

  [kyero@vm02 ~]$ sudo mount /dev/mapper/loop1p1 /mnt/boot/

  [kyero@vm02 ~]$ df -h /mnt/boot
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/loop1p1 31M 389K 29M 2% /mnt/boot


• Copy a kernel to it, configure GRUB’s notion of the root device and install the boot sector.
  
  [kyero@vm02 ~]$ sudo rsync -a /boot/ /mnt/boot/
  [kyero@vm02 ~]$ sudo vi /mnt/boot/grub/menu.lst
  [kyero@vm02 ~]$ sudo touch /mnt/boot/kvm-boot-image
  [kyero@vm02 ~]$ sudo grub --device-map=/dev/null

  GNU GRUB version 0.97 (640K lower / 3072K upper memory)

  [ Minimal BASH-like line editing is supported. For the first word, TAB
  lists possible command completions. Anywhere else TAB lists the possible
  completions of a device/filename.]
  grub> device (hd0) /var/boot-images/mail-boot
  device (hd0) /var/boot-images/mail-boot
  grub> find /kvm-boot-image
  find /kvm-boot-image
  (hd0,0)
  grub> root (hd0,0)
  root (hd0,0)
  Filesystem type is ext2fs, partition type 0x83
  grub> setup (hd0)
  setup (hd0)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)
  Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
  Done.
  grub> quit

  [kyero@vm02 ~]$ sudo umount /mnt/boot
  [kyero@vm02 ~]$ sudo kpartx -d /var/boot-images/mail-boot
  loop deleted : /dev/loop1

• Populate the new root disk with one cylinder of zeros and a copy of the old root disk.
  The first cylinder will contain the MBR and not much else. The padding means it’s easy to align the old root disk at cylinder one.
  Use pv so we can see the copy progress.
  The dd-lvm-out script is mostly just a sudo-blessed wrapper.
  [kyero@vm02 ~]$ ( dd if=/dev/zero bs=8225280 count=1 ; ssh xen03.local -l kyero sudo /data/system/bin/dd-lvm-out.sh mail-root ) | pv -e -r -s 21474836480 | sudo dd bs=1048577 of=/dev/vg4/mail-root
1+0 records in
1+0 records out
8225280 bytes (8.2 MB) copied, 0.03395 seconds, 242 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 611.215 s, 35.1 MB/s
0+662630 records in
0+662630 records out
21483061795 bytes (21 GB) copied, 615.249 seconds, 34.9 MB/s

• Partition the new root disk with a single partition starting at cylinder one.
  [kyero@vm02 ~]$ sudo sfdisk /dev/vg4/mail-root
  Checking that no-one is using this disk right now ...
  BLKRRPART: Invalid argument
  OK

  Disk /dev/vg4/mail-root: 2614 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /dev/vg4/mail-root: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /dev/vg4/mail-root1 :1
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 :
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 :
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 :
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Invalid argument

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Mount the new root filesystem
  [kyero@vm02 ~]$ sudo kpartx -av /dev/vg4/mail-root
  add map mail-root1 : 0 41977845 linear /dev/vg4/mail-root 16065
  add map mail-root2 : 0 16064 linear /dev/vg4/mail-root 1
  [kyero@vm02 ~]$ sudo e2fsck /dev/mapper/mail-root1
  e2fsck 1.39 (29-May-2006)
  /dev/mapper/mail-root1: clean, 435213/1310720 files, 4086961/5242880 blocks

  [kyero@vm02 ~]$ sudo mount /dev/mapper/mail-root1 /mnt/tmp


• Add the new kernel’s modules, tweak filesystem mounts, disable swap, check console.
  
  [kyero@vm02 ~]$ sudo rsync -a /lib/modules/ /mnt/tmp/lib/modules/

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/fstab

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/inittab


• Create the KVM guest, assigning the boot, root and swap storage
  [kyero@vm02 ~]$ sudo virt-install -n mail -r 1024 --vcpus 1 --os-type linux --os-variant generic26 --accelerate --disk path=/var/boot-images/mail-boot --disk path=/dev/mapper/vg4-mail--root,bus=ide,format=raw --disk path=/dev/mapper/vg4-mail--swap,bus=ide,format=raw --network bridge:br0 --network bridge:br1 --import --noreboot
  Starting install...
  Guest installation complete... you can restart your domain
  by running 'virsh start mail'

  [kyero@vm02 ~]$ sudo virsh dumpxml mail
...
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/boot-images/mail-boot'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--root'/>
<target dev='hdb' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='1'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--swap'/>
<target dev='hdd' bus='ide'/>
<address type='drive' controller='0' bus='1' unit='1'/>
</disk>
...


• Boot the KVM guest
  
[kyero@vm02 ~]$ sudo virsh start mail
[kyero@vm02 ~]$ sudo virsh autostart mail
[kyero@vm02 ~]$ sudo virsh console mail

• Log into KVM guest, partition /dev/hdd and configure it as swap, update /etc/fstab.

1. Later we align the original root filesystem at cylinder 1 on this new disk. A LVM cylinder is roughly 8MB [↩]
2. File backed disk is used rather than LVM owing to grief with GRUB and LVM (Error 22: No such partition) [↩]