Hardware ist ein 64bit QuadCore XEON System mit CentOS 5.1. Daher können wir die auf der XEN Downloadseite angebotenen Binärpakete nicht verwenden und müssen das Source-RPM unter 64bit neu erstellen.

Nach einer Minimalinstallation von CentOS 5.1 benötigen wir folgende Pakete, um xen neu zu bauen:

error: Failed build dependencies:
        transfig is needed by xen-3.2.0-0xstom.5EL.x86_64
        libidn-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        zlib-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        texi2html is needed by xen-3.2.0-0xstom.5EL.x86_64
        SDL-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        curl-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        libX11-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        python-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        ghostscript is needed by xen-3.2.0-0xstom.5EL.x86_64
        tetex-latex is needed by xen-3.2.0-0xstom.5EL.x86_64
        ncurses-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        gtk2-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        libaio-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        /usr/include/gnu/stubs-32.h is needed by xen-3.2.0-0xstom.5EL.x86_64
        dev86 is needed by xen-3.2.0-0xstom.5EL.x86_64
        gettext is needed by xen-3.2.0-0xstom.5EL.x86_64
        gnutls-devel is needed by xen-3.2.0-0xstom.5EL.x86_64
        openssl-devel is needed by xen-3.2.0-0xstom.5EL.x86_64

Um die Abhängigkeiten alle in einer Zeile zu zeigen, um diese mit yum installieren zu können, hilft

$ rpmbuild -bp xen.spec 2>&1 | awk '{printf "%s ",$1}'

Zusätzlich fehlen jedoch noch

• /usr/include/gnu/stubs-32.h ist in glibc-devel

und

* which sowie * gnupg um dem RPM eine Signatur verpassen zu können.

Wenn diese Voraussetzungen erfüllt sind, so können die Pakete problemlos erstellt werden.

• build log xen 3.2. rpms

Die Grub Konfiguration /etc/grub.conf bzw. /boot/grub/menu.lst ist auf XEN 3.2 anzupassen:

title CentOS (2.6.18-53.1.4.el5xen) (xen3.2)
        root (hd0,0)
        kernel /boot/xen.gz-3.2 vga=text-80x50 dom0_mem=512M
        module /boot/vmlinuz-2.6.18-53.1.4.el5xen ro root=LABEL=/ vga=791
        module /boot/initrd-2.6.18-53.1.4.el5xen.img

 __  __            _____  ____    ___  
 \ \/ /___ _ __   |___ / |___ \  / _ \ 
  \  // _ \ '_ \    |_ \   __) || | | |
  /  \  __/ | | |  ___) | / __/ | |_| |
 /_/\_\___|_| |_| |____(_)_____(_)___/ 
                                       
(XEN) Xen version 3.2.0 (builder@intra.tiri.li) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) Sat Jan 19 12:53:59 CET 2008
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: vga=text-80x50 dom0_mem=512M
(XEN) Video information:
(XEN)  VGA is text mode 80x50, font 8x8
(XEN)  VBE/DDC methods: V2; EDID transfer time: 2 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 0000000000099400 (usable)
(XEN)  0000000000099400 - 00000000000a0000 (reserved)
(XEN)  00000000000eb9d0 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000dffe0000 (usable)
(XEN)  00000000dffe0000 - 00000000dffee000 (ACPI data)
(XEN)  00000000dffee000 - 00000000dfff0000 (ACPI NVS)
(XEN)  00000000dfff0000 - 00000000e0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffb00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000120000000 (usable)
(XEN) System RAM: 4095MB (4193764kB)
(XEN) Xen heap: 14MB (14960kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) Processor #0 6:15 APIC version 20
(XEN) Processor #1 6:15 APIC version 20
(XEN) Processor #2 6:15 APIC version 20
(XEN) Processor #3 6:15 APIC version 20
(XEN) IOAPIC[0]: apic_id 4, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 5, version 32, address 0xfec10000, GSI 24-47
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2133.463 MHz processor.
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz stepping 0b
(XEN) Booting processor 1/1 eip 8c000
(XEN) CPU1: Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz stepping 0b
(XEN) Booting processor 2/2 eip 8c000
(XEN) CPU2: Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz stepping 0b
(XEN) Booting processor 3/3 eip 8c000
(XEN) CPU3: Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz stepping 0b
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 234 jiffies.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 4 CPUs
(XEN) AMD IOMMU: Disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff806e2f04
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000011b000000->000000011c000000 (126976 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff806e2f04
(XEN)  Init. ramdisk: ffffffff806e3000->ffffffff80e4f200
(XEN)  Phys-Mach map: ffffffff80e50000->ffffffff80f50000
(XEN)  Start info:    ffffffff80f50000->ffffffff80f504a4
(XEN)  Page tables:   ffffffff80f51000->ffffffff80f5c000
(XEN)  Boot stack:    ffffffff80f5c000->ffffffff80f5d000
(XEN)  TOTAL:         ffffffff80000000->ffffffff81000000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Initrd len 0x76c200, start at 0xffffffff806e3000
(XEN) Scrubbing Free RAM: ...................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 100kB init memory.

xentop - 22:33:37   Xen 3.2.0
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 4193764k total, 603816k used, 3589948k free    CPUs: 4 @ 2133MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
  Domain-0 -----r         17    0.0     524288   12.5   no limit       n/a     4    0        0        0    0        0        0        0 2149961852

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     4     r-----     18.2

In unserem Setup haben wir eth0 über einen Router an den Internetprovider A und eth1 über einen Router an den Internetprovider B angeschlossen. Die Verbindung über eth0 ist dynamisch und schneller, aber weniger stabil, die andere Verbindung hat feste IP-Adressen und ist langsamer. Unsere internen Maschinen sollen via Proxy mit der Außenwelt sprechen, der Proxy soll seine Verbindung über den “günstigsten” Weg aufbauen, und wenn ein Weg nicht gehtm dann eben über den anderen Weg. Die Maschinen sollen intern über einen virtuellen Switch miteinander sprechen können.

• Router für Provider A hat IP-Adresse 192.168.0.1/24

DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
STP=off
DELAY=0
HELLO=0
IPADDR=192.168.0.2
NETMASK=255.255.255.0
NETWORK=192.168.0.0
BROADCAST=192.168.0.255

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br0
ARP=yes

• Router für Provider B hat IP-Adresse 192.168.1.1/24

DEVICE=br1
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
STP=off
DELAY=0
HELLO=0
IPADDR=192.168.1.2
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br1
ARP=yes

• Dieses Interface wird automatisch durch libvirtd erzeugt und stellt ein virtuelles Netzwerk mit DHCP Server zur Verfügung.

<network>
  <name>default</name>
  <uuid>db2df752-6d47-42f3-9f6c-70707a6d2103</uuid>
  <bridge name="virbr0" />
  <forward/>
  <ip address="192.168.122.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.122.2" end="192.168.122.254" />
    </dhcp>
  </ip>
</network>

Wir wollen ein sog. brouter-Setup. (Vgl. opensuse wiki).

(network-script 'network-virtual bridge="virbr0" bridgeip="192.168.122.1/24" brnet="192.168.122.0/24"')
(vif-script vif-bridge)

#!/bin/sh
#============================================================================
# Default Xen network start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default xenbr${vifnum}), gives it an IP address
# and the appropriate route. Then it starts the SuSEfirewall2 which should have
# the bridge device in the zone you want it.
#
# If all goes well, this should ensure that networking stays up.
# However, some configurations are upset by this, especially
# NFS roots. If the bridged setup does not meet your needs,
# configure a different script, for example using routing instead.
#
# Usage:
#
# vnet-brouter (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# bridgeip   Holds the ip address the bridge should have in the
#            the form ip/mask (10.0.0.1/24).
# brnet      Holds the network of the bridge (10.0.0.1/24).
# 
# vifnum     Virtual device number to use (default 0). Numbers >=8
#            require the netback driver to have nloopbacks set to a
#            higher value than its default of 8.
# bridge     The bridge to use (default xenbr${vifnum}).
#
# start:
# Creates the bridge
# Gives it the IP address and netmask
# Adds the routes to the routing table.
#
# stop:
# Removes all routes from the bridge
# Removes any devices on the bridge from it.
# Deletes bridge
#
# status:
# Print addresses, interfaces, routes
#
#============================================================================


dir=$(dirname "$0")
. "$dir/xen-script-common.sh"
. "$dir/xen-network-common.sh"

findCommand "$@"
evalVariables "$@"

vifnum=${vifnum:-0}
bridgeip=${bridgeip:-10.6.7.1/24}
brnet=${brnet:-10.6.7.0/24}
netmask=${netmask:-255.255.255.0}
bridge=${bridge:-xenbr${vifnum}}

##
# link_exists interface
#
# Returns 0 if the interface named exists (whether up or down), 1 otherwise.
#
link_exists()
{
    if ip link show "$1" >/dev/null 2>/dev/null
    then
        return 0
    else
        return 1
    fi
}


# Usage: create_bridge bridge
create_bridge () {
    local bridge=$1

    # Don't create the bridge if it already exists.
    if [ ! -d "/sys/class/net/${bridge}/bridge" ]; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi
    ip link set ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2
    # Don't add $dev to $bridge if it's already on a bridge.
    if ! brctl show | grep -wq ${dev} ; then
        brctl addif ${bridge} ${dev}
    fi
}

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
    local dev=$1
    local bridge=$2
    
    echo '============================================================'
    ip addr show ${dev}
    ip addr show ${bridge}
    echo ' '
    brctl show ${bridge}
    echo ' '
    ip route list
    echo ' '
    route -n
    echo '============================================================'
}

op_start () {
    if [ "${bridge}" = "null" ] ; then
        return
    fi

    create_bridge ${bridge}

    if link_exists "$bridge"; then
        ip address add dev $bridge $bridgeip
        ip link set ${bridge} up arp on
        ip route add to $brnet dev $bridge
    fi

    if [ ${antispoof} = 'yes' ] ; then
        antispoofing
    fi
    /sbin/service iptables start
}

op_stop () {
    if [ "${bridge}" = "null" ]; then
        return
    fi
    if ! link_exists "$bridge"; then
        return
    fi
    
    ip route del to $brnet dev $bridge
    ip link set ${bridge} down arp off
    ip address del dev $bridge $bridgeip
    ##FIXME: disconnect the interfaces from the bridge 1st
    brctl delbr ${bridge}
    /sbin/service iptables start
}

case "$command" in
    start)
        op_start
        ;;
    
    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
        ;;

    *)
        echo "Unknown command: $command" >&2
        echo 'Valid commands are: start, stop, status' >&2
        exit 1
esac

Guter Startpunkt ist hier, das Logging in /etc/sysconfig/iptables einzutragen (vor der REJECT - Zeile)

-A RH-Firewall-1-INPUT -j LOG --log-level info --log-prefix "_BLOCKED_"

Dann kann man schnell erkennen, was noch freigeschaltet werden muss.

• Multiple Provider, lartc mini howto, loadbalancing

• Annahme: Unser Squid in der domU hat 3 Netzwerkkarten, eth0 → br0, eth1 → br1, eth2 → virbr0
  • von innen via virbr0, nach aussen über den besten verfügbaren weg (eth0, dann eth1)

IF1=eth0 
IF2=eth1
IP1=192.168.0.11
IP2=192.168.1.11
GW1=192.168.0.1
GW2=192.168.1.1
P1NET=192.168.0.0
P2NET=192.168.1.0

• Routing Tabellen erzeugen

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
1 T1
2 T2

• Routen erzeugen

ip route add $P1NET dev $IF1 src $IP1 table T1
ip route add default via $GW1 table T1
ip route add $P2NET dev $IF2 src $IP2 table T2
ip route add default via $GW2 table T2

• Haupttabelle

ip route add $P1NET dev $IF1 src $IP1
ip route add $P2NET dev $IF2 src $IP2

• Bevorzugte Route

ip route add default via $GW1

• Regeln, die tatsächlich auswählen, welche Tabelle verwendet wird.

ip rule add from $IP1 table T1
ip rule add from $IP2 table T2

• localhost einbinden

ip route add 127.0.0.0/8 dev lo   table T1
ip route add 127.0.0.0/8 dev lo   table T2 

• Gewichtung (IF1 bevorzugen)

ip route add default scope global nexthop via $GW1 dev $IF1 weight 100 \
  nexthop via $GW2 dev $IF2 weight 1

scope link ? woran merkt er, dass das GW weg ist ?