DRAFT

FreeBSD Jails

FreeBSD jails are self contained environments, which share resources on the host system but have their own ip address, configuration and programs. Essentially a jail can be used to run a complete virtual FreeBSD system (also known as virtual servers) without high resource requirements of complete virtual environments such as VMWare.

Since they are self contained environments, jails are useful when there is a need for different server configurations that do not conflict with each other or, not to mess up a system when testing development versions of ports or programs. Currently the focus of this article are jails for a development environment, and will not concentrate too much on security. The setup will be that of a full FreeBSD distribution, which has everything the default base system has.

Examples of jails as virtual servers usage

• Development versions of ports which can conflict with stable versions such as Gnome development ports.
• Different conflicting versions of servers or programs.
• Ports testing, source installs

  Where mistakes or lack of a packaging can result in a messy state. It is easy to wipe out a jail directory without affecting the host system.

• Shared hosting or sharing a physical server within a company.

  This article does not deal with security just yet to setup secure virtual servers

Creating a jail

Building and Installing

The jail(8) man page has good clear documentation on this, and this the instruction from the man page on setting up a directory tree containing an entire FreeBSD distribution.

For this example the host environment will have an ip address of 192.168.1.1 Jail directory names are usually set as IP addresses while another naming convention that can be used is hostnames.

The jail will be setup with an ip address of 192.168.1.2 and in the directory /usr/jail/192.168.1.2

The following commands should be run as the root user or by a regular user with the proper sudo(8) permissions.

For csh/tcsh:

            setenv D /here/is/the/jail

For sh/bash/ksh users:

            export D=/here/is/the/jail

On 5.X:

            cd /usr/src
            mkdir -p $D
            make world DESTDIR=$D
            cd etc
            make distribution DESTDIR=$D
            mount_devfs devfs $D/dev
            cd $D
            ln -sf dev/null kernel

On 4.X:

            cd /usr/src
            mkdir -p $D
            make world DESTDIR=$D
            cd etc
            make distribution DESTDIR=$D
            cd $D
            ln -sf dev/null kernel
            mkdir $D/stand 
            cp /stand/sysinstall $D/stand

Notes

If you have already built world, you can replace make world with make installworld on the fourth line to avoid recompiling everything again.

WARNING

With make distribution be very careful that DESTDIR environment variable is set. If it is not, it defaults to overwriting your host system's base setup (rc.conf, passwd etc.). If in doubt, type it out manually.

Setting up the host environment

Setup daemons to listen only on host ip address

On the host, any daemons that listen on all ports, need to be removed if they will affect those in the jails. This is because, jail daemons all bind onto the same network interface through use of network aliases on the host. We need to either disable some services, or configure then to listen only on the host ip address.

             inetd_flags="-wW -a 192.168.1.1"
             sendmail_enable="NO"

On 5.x

            rpcbind_enable="NO"

On 4.x

            portmap_enable="NO"

Where 192.168.1.1 is the host ip address, all services run out of inetd will now automatically only listen on that address. We have also set the sendmail_enable flag to prevent sendmail from listening on external host interfaces. This will still allow a sendmail process to start and bind to the localhost address of the host environment so as not to interupt programs and scripts which mail their output just as the standard nightly cron jobs. Lastly, the RPC port mapper has been disabled. The jail host cannot be an NFS server or client because it cannot be configured to listen to a specific IP address.

We will then need to list out any other daemons that are listening on all ip addresses.

            sockstat -l4

Any services that have *: and not a specific ip will be listening on all available ip addresses. This is not good, as it may conflict with one of the services listening in our jails.

sshd

In /etc/ssh/sshd_config, add or change the ListenAddress as below substituting the ip address with that of your host ip.

                ListenAddress 192.168.1.1

The following services are optional services that may be running on your machine though they are not enabled by default. Here is how to have them listen to a single IP address by default.

bind

/etc/nameddb/named.conf

Add the listen-on option to only listen on specific ip addresses.

               listen-on { 192.168.1.1;
                        }

Starting and setting up the jail environment

Starting the jail for the first time.

Now that the host environment has been setup it must be told to accept network traffic addressed to our jail's IP address. Once that is accomplished we can startup the jail for the first time.

            ifconfig ed0 inet alias 192.168.1.2/32
            jail /usr/jail/192.168.1.2 gnome 192.168.1.2 /bin/csh

In the ifconfig(8) command, replace the "ed0" with the name of your network interface on the host system. This will allow the host environment to accept packets that are addressed to your jail's address.

The jail command will start your jail, give it the hostname "gnome", the IP address 192.168.1.2, and drop you to a csh prompt. You will now be interacting directly with the jail host.

Configuring the jailed system.

The next few steps are the required post-install configuration that is usually done at the end of a standard installation.

• We need to create an empty /etc/fstab

touch /etc/fstab

• Run newaliases(1) to quell sendmail(8) warnings.
• Configure /etc/resolv.conf so that name resolution will work properly. For this example, set it to the host IP address where we have a dns server running.

                  nameserver 192.168.1.1
  

• In rc.conf put these variables in. Enabling SSH will make it easier for us to access and run programs on the virtual server.

                  sshd_enable="YES"
                  network_interfaces=""
  

  The following options may also be added if these services are not relevant to the operation of the jail.

                  sendmail_enable="NO"
                  inetd_flags="NO"
  

  On 5.x:

                      rpcbind_enable="NO"
  

  On 4.x:

                      portmap_enable="NO"
  

  We turned off sendmail from receiving mail from all but the localhost, in this case, the jail's own localhost. If you are not going to be providing network services from your jail you might as well turn off inetd. SSH will allow you to use SFTP for file transfers since FTP will now be shutoff. The RPC port mapper has been disabled but a jail itself MAY still use or serve NFS services if actually needed unlike the host environment.

  The last steps can be done from either the command line or through sysinstall(8).

  • Set password for root with passwd(8)
  • Add accounts for users with adduser(8)
  • Configure your timezone

Installing basic ports and packages

At this point you can begin to add programs through packages or via the ports system that are needed for the functionality of your jail. You cannot use sysinstall(8) from within a jail to add packages or install the ports tree from remote locations.

To install from packages:

For csh/tcsh:

                    setenv PACKAGEROOT ftp5.us.FreeBSD.org

For sh/bash/ksh users:

:: export PACKAGEROOT=ftp5.us.FreeBSD.org

                    pkg_add -r portupgrade
                    pkg_add -r pdksh
                    etc...

Change The PACKAGEROOT to our desired FTP mirror site. By setting that and using pkg_add(8) with the "-r" option pkg_add will download the latest package for your release.

To install from the latest ports:

                fetch ftp://ftp5.us.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.gz
                cat ports.tar.gz | tar --unlink -xpzf - -C /usr

WARNING

If you have installed packages and then decide to start installing from the latest ports, run portupgrade -uarR directly after you have created the ports tree so that your currently installed package versions will be in sync with your ports tree.

Managing Jails

Using system rc scripts to start and stop jails

In 5.X /etc/defaults/rc.conf has a well documented section on configuring jails. We can control our jails from the host system using this script after we have added the appropriate lines in /etc/rc.conf on the host system.

rc.conf

For our example we will add the following lines to /etc/rc.conf (on host)

                jail_enable="yes"    # Set to NO to disable starting of any jails
                jail_list="gnome"        # Space separated list of names of jails

                jail_gnome_rootdir="/usr/jail/192.168.1.2"      # Jail's root directory
                jail_gnome_hostname="gnome.domain.com"  # Jail's hostname
                jail_gnome_ip="192.168.1.2"                     # Jail's IP number
                jail_gnome_exec="/bin/sh /etc/rc"               # command to execute in jail
                jail_gnome_devfs_enable="YES"                   # mount devfs in the jail
                jail_gnome_fdescfs_enable="YES"         # mount fdescfs in the jail
                jail_gnome_procfs_enable="YES"          # mount procfs in jail
                #jail_gnome_devfs_ruleset="ruleset_name"        # devfs ruleset to apply to jail

                You should also do the ip aliasing automatically by adding
                this line in your rc.conf

                ifconfig_fxp0_alias0="inet 192.168.1.2 netmask 0xffffffff"

Starting the jail

Now to start the jail, you only need to issue the command. If more than one jail is defined in the startup list, all will be started.

                /etc/rc.d/jail start

To start a specific jail

                /etc/rc.d/jail start gnome

Stopping the jail

To stop all jails

                /etc/rc.d/jail stop

To stop a specific jail

                /etc/rc.d/jail stop gnome

Checking that your jail is running

To get a list of running jails use the jls(8) command

            jls

Reusing and Sharing directories

Reusing jails

The steps above to build and setup a jail can of course take a while. This is not prefferable if you are looking to do a limited test of a few specific scenarios and then discard the jail once you are done. You may find it useful to build a basic jail having gone through the steps above, make any additional configuration edits, and install any basic ports that you require. From that point you can repeatedly copy that jail in order to create new ones. To do this, cp -R will not work. Instead, do the following while the original jail is not running.

            mkdir /usr/jail/new
                cd /usr/jail/old
                tar -cpf - . | tar -C /usr/jail/new -xpf -

Sharing a filesystem between jails

Often you want to share directories over different jails, such as distfiles for ports to save disk space. One way to do it is via nullfs.

mount_nullfs(8)

            mount_nullfs /shared/distfiles /usr/jail/192.168.1.2/usr/ports/distfiles

You can automatically mount it at boot time by adding an entry in /etc/fstab on the host system.

            /shared/distfiles /usr/jail/192.168.1.2/usr/ports/distfiles nullfs rw 0 0

Network Access

FIXME

Gnome/X11 Desktop specific

Forwarding X through ssh

There are some issues with forwarding X applications via ssh. Change the entry in /etc/ssh/sshd_config (jail) and set

            PermitRootLogin yes
            X11UseLocalhost no

Notes

I still had some problems with certain applications like gvim. Also there are X authentification errors if I su and try to run an X app. If you know of a solution, please let me know. These problems may only be occuring due to having host X display, on the same box.

Gnome Desktop and GDM via secondary X Server and XDMCP

GDM and XDMCP

• ssh in as root to your virtual server and run gdmsetup. Enable XDMCP.

  

• Run gdm to start up the gdm service.
• switch to a free console and you can now login to your jail desktop on a secondary X Server.

                      X :1 -query gnome
  

• To automatically start up gdm in your jail, copy

  /usr/X11R6/etc/rc.d/gdm.sh.sample

  to

  /usr/X11R6/etc/rc.d/gdm.sh

Gnome Development testing

For those interested in helping out with FreeBSD Gnome testing, you should visit the FreeBSD Gnome Development FAQ.

There are additional jail and build scripts, available from the development cvs at marcuscom.com

Authors

• Khairil Yusof
• 4.x specific content and further editing by tmclaugh@sdf.lonestar.org