Segfault > IT > How-To's > Gentoo on a USB-stick

(have as well a look at "Preparing a bootable Sabayon live/installationCD on a USB-stick")

I tried to install Gentoo/Linux not on a harddisk, but on a USB-stick, or USB-key, however you call it. This because I wanted to build a PC which consumed as less watts as possible and be as well very silent, with few moving parts.
To do this, you can follow the usual procedure mentioned in the official Gentoo installation guide, with a few remarks / changes.

Remark: Again - I wrote an additional how-to here.

1) Hardware

Choose a fast medium. The speed of your system will mostly depend on this.

I tested the write-speed of the USB-sticks I had using the following script...

dd if=/dev/zero of=2delete bs=1024 count=800000

...(which writes on the USB-stick a file of ~820MB) and I got the following results:

Brand Linux dmesg ID MB/s before sync MB/s after sync
SanDisk Cruzer Micro 8GB SanDisk U3 Cruzer Micro 16.5 7.58
SanDisk Cruzer Micro 4GB SanDisk U3 Cruzer Micro 16.3 7.37
Hama 8GB FlashPen Fancy 11.5 3.79
extreMEmory USA 4GB TinyDisk 2007-03-23 14.8 8.27

You see that Hama delivered the absolutely worst results. Especially when flushing the cached data with the sync command, the stick sometimes stops and "thinks" and a lot of seconds get lost.

extreMEmory was very good, especially with small data transfers (in these cases the transfer rate went up to 16MB/s). It's a pity I couldn't find an 8GB-stick of this brand.

SanDisk provided always very stable transfer rates, with tiny hickups which brought it up to 13MB/s.

All the sticks had a partition which I formatted with ReiserFS, as it saves a lot of space when it has to handle a lot of small files, as it is in a Linux installation.


2) Installing Gentoo

I actually followed the usual guide, including how to install the bootloader. Here are my remarks:

  • Don't format the partitions using FAT - I initially did this for the boot partition and when I tried to install grub the system didn't want to boot because during the installation I couldn't create symlinks (the symlink /boot/grub/menu.1st linked to /boot/grub/grub.conf couldn't be created).
  • When setting up the kernel, select the USB drivers to be compiled in the kernel and not as modules.
  • If grub says during boot that it cannot find the kernel file or the root filesystem, try to refer to the boot partition not with the usual /boot/kernel-yourversion, but with (hd0,0)/kernel-yourversion. I had to do it this way.
  • Add to the kernel line of grub.conf the string rootdelay=10 (or a higher time value). This so that the kernel recognizes the USB-stick BEFORE trying to mount the root partition, which is as well located on the USB-stick. If you don't do this, you'll most probably get during boot a message saying...

IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
Root-NFS: No NFS server available, giving up.
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Insert root floppy and press ENTER

...and you and the system will just stare at each other forever.


3) Don't write too much

As you know, flash memory is not good when it has to handle a lot of writes. As far as I remember it might not work anymore after already 100'000 write cycles and this might soon or later happen when e.g. software is compiled.

It is therefore a good idea to have as well a normal harddisk where to create a partition to be used as temporary space where portage can unpack files and compile the code.

I therefore created a 10GB partition on the harddisk and mapped it in /etc/fstab to /var/tmp - that's the directory that Portage uses when compiling something. I use hdparm to spin off the harddisk when it's not used (hdparm -B 40 /dev/sdb).
Eventually think about using "nilfs2" (instead of the usual filesystems like ext2/3, xfs, btrfs, etc...), which might give you better performance and much less hotspots due to (over)write cycles - have a look here.

The log files have to be taken care of as well - don't use here a harddisk as the continuous writing would force the harddisk to spin up all the time. I'm more prone to use a ramdisk in this case, but I'm not sure yet - we'll see.


4) Boot config

I have a normal harddisk in the PC on which I installed Windows. The USB-stick is recognized as sdb (1st partition contains /boot and the second one is root) and the harddisk is sda (installed Windows normally on the first partition). The configuration I use in /boot/grub/grub.conf (or /etc/boot/grub/menu.1st depending if you symlinked it to grub.conf or not) is:

default 0
timeout 5

title=Gentoo Linux 2.6.20-xen-r5
root (hd0,0)
kernel /xen-3.1.0.gz doslowusb vga=791
module /kernel-2.6.20-xen-r5 root=/dev/sdb2 rootdelay=10 doslowusb vga=791 console=tty1

title=Gentoo Linux xen OLD
root (hd0,0)
kernel /xen-3.1.0.gz.old doslowusb vga=791
module /kernel-2.6.20-xen-r5.old root=/dev/sdb2 rootdelay=10 doslowusb vga=791 console=tty1

title=Gentoo Linux 2.6.22-gentoo-r8
root (hd0,0)
kernel /kernel-2.6.22-gentoo-r8 root=/dev/sdb2 rootdelay=15 doslowusb vga=791 console=tty1

title=Windows XP
rootnoverify (hd1,0)
map (hd0) (hd1)
map (hd1) (hd0)
chainloader +1


Grub - A

If when running...

grub-install --no-floppy /dev/sd get the message...

/dev/sd1 does not have any corresponding BIOS drive

...try adding the option "--recheck" like this:

grub-install --no-floppy --recheck /dev/sd


Grub - B

If after using the above the USB-stick still does not want to boot, install the master boot record manually:

  • Mount as usual the root partition under e.g. "/mnt/memstick" and the boot partition under "/mnt/memstick/boot" and follow the usual chroot procedure:

mount -t proc none /mnt/memstick/proc
mount -o bind /dev /mnt/memstick/dev
chroot /mnt/memstick /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"
grep -v rootfs /proc/mounts > /etc/mtab

  •     Then get into grub with "grub --no-floppy" and:

root (hd<#of_your_usbstick>,0) (so e.g. "root (hd4,0)" if your usb-stick is /dev/sde)
setup (hd<#of_your_usbstick>)

  • Finally exit the chrooted environment with "exit" and unmount the stuff you mounted above:

umount /mnt/memstick/boot /mnt/memstick/dev /mnt/memstick/proc /mnt/memstick

  • If you want to be sure that you didn't mess up by mistake the MBR of the HDD of the PC you're using to set up the USB-stick just run "grub-install --no-floppy /dev/sd" to reinstall grub.


See what's going on

When booting, especially in this case with a USB-stick, you might have problems during the first attempts and see a lot of error messages in the console during the boot. Unluckily those messages might scroll beyond your screen. To see them you should:

  • raise the screen resolution.
    • To do that you see above in my grub config that I am using the parameter "vga=791" (or "vga=0x317" - e.g. "vga=0x365" is 1440x900x32bit) which sets the resolution of the screen to 1024x768 (see here or here for additional modes).
      Eventually use as well "vga=ask" to see all supported video modes - handy if you want to use some special resolution like 1680x1050 (0x369).
    • To use it you'll need to activate "framebuffer support" under "Device drivers => Graphics support => Support for frame buffer devices" in your kernel. I always choose "VESA VGA graphics support", "Enable firmware EDID" and "Enable Video Mode Handling Helpers" independently from the grahics hardware I am using.
    • Additionally I enable "Device drivers => Graphics support => Console display driver support => Framebuffer Console support".
  • Activate under "Device drivers => Graphics support => Console display driver support => Enable Scrollback Buffer in System RAM" and set for "Scrollback Buffer Size" a size of 512 and you'll be able to scroll back a few pages by pressing the keys [Shift]+[PgUp/PgDown].


My USB-stick is not sda

One of the problems I had was that my USB-stick booted fine, up to the point when it had to mount the root filesystem.

I configured /etc/fstab to find the root partition on /dev/sda3 but the normal sata-HDDs were found by the kernel earlier than my USB-stick and my root partition on the USB-stick always ended up being something like /dev/sde3 (sda, b, c and d were my normal HDDs).

To solve this problem I:

  • configured the Linux kernel to compile the HDD-drivers as modules instead of being already in the kernel.
    • In my case this means that under "Device drivers => ATA/ATAPI/MFM/RLL support" the options "Generic PCI IDE Chipset Support", "generic/default IDE chipset support" and "generic ATA/ATAPI disk support" as modules.
    • In your case, to know which options to set as modules, you can boot normally and see which drivers are used with the command "lspci -k".
    • Additionally (not sure if it's actually needed) I even entered those modules (in my case "ahci" and "ide_pci_generic") into the blacklist of modules that do not have to be loaded automatically during boot in "/etc/modprobe.d/blacklist.conf". Once the boot sequence has finished I have a small scripts which runs "modprobe " and the HDDs pop up.
  • And that was the only point :o)


Alternate OS installation?

From time to time I don't want to perform again a complete installation and compilation of Gentoo, especially if I just want to transfer the OS from HDD to USB-stick or create an OS for an identical machine. I therefore just tar the whole fs, transfer it to the new medium, unpack it and just change the options related to the new medium.

If you do like that keep an eye on:

  • the directories "sys" and "proc" should be empty on your target.
  • the directory "dev" is always mysterious. Sometimes it just works by just copying it, sometimes I get a "stage3" installation tar from one of Gentoo's mirrors and use the one in there.
  • If you transfer your OS to a new machine the timestamp of the files on the USB-stick might be in the future. As Linux complains about it, after all the error messages that appear on the new PC issue a "touch /etc" and a "find /etc/ -exec touch '{}' \;"
  • If after the boot you get weird error messages about the root fs and "df -h" shows foreign drives/partitions (not the ones of the USB-stick) mounted on root, check that your "/etc/mtab" does not contain those weird drives and eventually delete those lines by first shutting down and doing the changes with the USB-stick mounted on some other PC. 



NILFS2 seems to be pretty fast on flash memory.

See here for a few benchmarks and some problems I had when using it for my root FS.


Multiple USB-sticks

When having my OS on a USB-stick I had some total crashes when I tried to use at the same time some more USB-sticks (e.g. to transfer some files from one PC to another).

Not sure to what this is due but just try to avoid during that.


Force disk cleanup

To force the garbage collector to get rid of all old checkpoints and free up the maximum amount of disk space run "nilfs-clean /dev/[partition_you_want_to_cleanup]"