Multipath mit RHEL4 (an DS4300)

Folgende Konstellation ist gegeben:

RedHat Enterprise Linux 4 ES U3 (Kernel 2.6.9-34.0.1ELsmp)
2x qlogic HBA Typ 2340
DS4300 mit 2 Controllern
HBA1 geht auf Controller A
HBA2 geht auf Controller B

Um Multipath einzurichten, sind nachfolgende Schritte vorzunehmen.

initrd erstellen

mkinitrd -v --with=dm-multipath /boot/initrd-2.6.9-34.0.1.ELsmp.img 2.6.9-34.0.1.ELsmp (Auszug)

Creating initramfs
Looking for deps of module scsi_mod
Looking for deps of module sd_mod        scsi_mod
Looking for deps of module mptbase
Looking for deps of module mptscsih      mptfc mptspi mptsas mptscsi mptbase scsi_mod
Looking for deps of module mptfc         mptscsi mptbase scsi_mod
Looking for deps of module mptscsi       mptbase scsi_mod
Looking for deps of module mptbase
Looking for deps of module ata_piix      libata scsi_mod
Looking for deps of module libata        scsi_mod
Looking for deps of module scsi_mod
Looking for deps of module scsi_mod
Looking for deps of module qla2300       qla2xxx scsi_transport_fc scsi_mod
Looking for deps of module qla2xxx       scsi_transport_fc scsi_mod
Looking for deps of module scsi_transport_fc     scsi_mod
Looking for deps of module scsi_mod
Looking for deps of module ide-disk
Looking for deps of module dm-mod
Looking for deps of module ext3  jbd
Looking for deps of module jbd
Looking for deps of module dm-mod
Looking for deps of module dm-mirror     dm-mod
Looking for deps of module dm-zero       dm-mod
Looking for deps of module dm-snapshot   dm-mod
Looking for deps of module dm-multipath  dm-mod
Using modules:  ./kernel/drivers/scsi/scsi_mod.ko ./kernel/drivers/scsi/sd_mod.ko ./kernel/drivers/message/fusion/mptbase.ko ./kernel/drivers/message/fusion/mptscsi.ko ./kernel/drivers/message/fusion/mptfc.ko ./kernel/drivers/message/fusion/mptspi.ko ./kernel/drivers/message/fusion/mptsas.ko ./kernel/drivers/message/fusion/mptscsih.ko ./kernel/drivers/scsi/libata.ko ./kernel/drivers/scsi/ata_piix.ko ./kernel/drivers/scsi/scsi_transport_fc.ko ./kernel/drivers/scsi/qla2xxx/qla2xxx.ko ./kernel/drivers/scsi/qla2xxx/qla2300.ko ./kernel/drivers/md/dm-mod.ko ./kernel/fs/jbd/jbd.ko ./kernel/fs/ext3/ext3.ko ./kernel/drivers/md/dm-mirror.ko ./kernel/drivers/md/dm-zero.ko ./kernel/drivers/md/dm-snapshot.ko ./kernel/drivers/md/dm-multipath.ko
/sbin/nash -> /tmp/initrd.iR6392/bin/nash
/sbin/insmod.static -> /tmp/initrd.iR6392/bin/insmod
/sbin/udev.static -> /tmp/initrd.iR6392/sbin/udev
/etc/udev/udev.conf -> /tmp/initrd.iR6392/etc/udev/udev.conf
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/scsi_mod.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/scsi_mod.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/sd_mod.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/sd_mod.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptbase.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptbase.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptscsi.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptscsi.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptfc.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptfc.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptspi.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptspi.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptsas.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptsas.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/message/fusion/mptscsih.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/mptscsih.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/libata.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/libata.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/ata_piix.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/ata_piix.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/scsi_transport_fc.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/scsi_transport_fc.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/qla2xxx/qla2xxx.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/qla2xxx.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/scsi/qla2xxx/qla2300.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/qla2300.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/md/dm-mod.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/dm-mod.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/fs/jbd/jbd.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/jbd.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/fs/ext3/ext3.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/ext3.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/md/dm-mirror.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/dm-mirror.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/md/dm-zero.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/dm-zero.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/md/dm-snapshot.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/dm-snapshot.ko(elf64-x86-64)
copy from /lib/modules/2.6.9-34.0.1.ELsmp/./kernel/drivers/md/dm-multipath.ko(elf64-x86-64) to /tmp/initrd.iR6392/lib/dm-multipath.ko(elf64-x86-64)
/sbin/lvm.static -> /tmp/initrd.iR6392/bin/lvm
/etc/lvm -> /tmp/initrd.iR6392/etc/lvm
`/etc/lvm/lvm.conf' -> `/tmp/initrd.iR6392/etc/lvm/lvm.conf'
Loading module scsi_mod
Loading module sd_mod
Loading module mptbase
Loading module mptscsi
Loading module mptfc
Loading module mptspi
Loading module mptsas
Loading module mptscsih
Loading module libata
Loading module ata_piix
Loading module scsi_transport_fc
Loading module qla2xxx
Loading module qla2300
Loading module dm-mod
Loading module jbd
Loading module ext3
Loading module dm-mirror
Loading module dm-zero
Loading module dm-snapshot
Loading module dm-multipath

dmesg

QLogic Fibre Channel HBA Driver
qla2300 0000:04:02.0: Found an ISP2312, irq 209, iobase 0xffffff0000004000
qla2300 0000:04:02.0: Configuring PCI space...
qla2300 0000:04:02.0: Configure NVRAM parameters...
qla2300 0000:04:02.0: Verifying loaded RISC code...
qla2300 0000:04:02.0: LIP reset occured (f8f7).
qla2300 0000:04:02.0: Waiting for LIP to complete...
qla2300 0000:04:02.0: LIP occured (f8f7).
qla2300 0000:04:02.0: LOOP UP detected (2 Gbps).
qla2300 0000:04:02.0: Topology - (Loop), Host Loop address 0x7d
scsi2 : qla2xxx
qla2300 0000:04:02.0: 
 QLogic Fibre Channel HBA Driver: 8.01.02-d4
  QLogic QLA2340 - 133MHz PCI-X to 2Gb FC, Single Channel
  ISP2312: PCI-X (100 MHz) @ 0000:04:02.0 hdma+, host#=2, fw=3.03.18 IPX
  Vendor: IBM       Model: 1722-600          Rev: 0520
  Type:   Direct-Access                      ANSI SCSI revision: 03
qla2300 0000:04:02.0: scsi(2:0:0:0): Enabled tagged queuing, queue depth 32.
SCSI device sdb: 853947711 512-byte hdwr sectors (437221 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 853947711 512-byte hdwr sectors (437221 MB)
SCSI device sdb: drive cache: write back
 sdb: sdb1
Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
  Vendor: IBM       Model: Universal Xport   Rev: 0520
  Type:   Direct-Access                      ANSI SCSI revision: 03
qla2300 0000:04:02.0: scsi(2:0:0:31): Enabled tagged queuing, queue depth 32.
SCSI device sdc: 40960 512-byte hdwr sectors (21 MB)
SCSI device sdc: drive cache: write through
SCSI device sdc: 40960 512-byte hdwr sectors (21 MB)
SCSI device sdc: drive cache: write through
 sdc: unknown partition table
Attached scsi disk sdc at scsi2, channel 0, id 0, lun 31
ACPI: PCI interrupt 0000:05:01.0[A] -> GSI 48 (level, low) -> IRQ 217
qla2300 0000:05:01.0: Found an ISP2312, irq 217, iobase 0xffffff0000006000
qla2300 0000:05:01.0: Configuring PCI space...
qla2300 0000:05:01.0: Configure NVRAM parameters...
qla2300 0000:05:01.0: Verifying loaded RISC code...
qla2300 0000:05:01.0: LIP reset occured (f8f7).
qla2300 0000:05:01.0: Waiting for LIP to complete...
qla2300 0000:05:01.0: LIP occured (f8f7).
qla2300 0000:05:01.0: LOOP UP detected (2 Gbps).
qla2300 0000:05:01.0: Topology - (Loop), Host Loop address 0x7d
scsi3 : qla2xxx
qla2300 0000:05:01.0: 
 QLogic Fibre Channel HBA Driver: 8.01.02-d4
  QLogic QLA2340 - 133MHz PCI-X to 2Gb FC, Single Channel
  ISP2312: PCI-X (133 MHz) @ 0000:05:01.0 hdma+, host#=3, fw=3.03.18 IPX
  Vendor: IBM       Model: 1722-600          Rev: 0520
  Type:   Direct-Access                      ANSI SCSI revision: 03
qla2300 0000:05:01.0: scsi(3:0:0:0): Enabled tagged queuing, queue depth 32.
SCSI device sdd: 853947711 512-byte hdwr sectors (437221 MB)
SCSI device sdd: drive cache: write back
SCSI device sdd: 853947711 512-byte hdwr sectors (437221 MB)
SCSI device sdd: drive cache: write back
 sdd: sdd1
Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0
  Vendor: IBM       Model: Universal Xport   Rev: 0520
  Type:   Direct-Access                      ANSI SCSI revision: 03
qla2300 0000:05:01.0: scsi(3:0:0:31): Enabled tagged queuing, queue depth 32.
SCSI device sde: 40960 512-byte hdwr sectors (21 MB)
SCSI device sde: drive cache: write through
SCSI device sde: 40960 512-byte hdwr sectors (21 MB)
SCSI device sde: drive cache: write through
 sde: unknown partition table
Attached scsi disk sde at scsi3, channel 0, id 0, lun 31

Die Platten, die für uns interessant sind, sind die 400GB LUNs (sdb,sdd).

lsscsi

RPMs sind unter http://rpms.serviert.de verfügbar.

lsscsi

[0:0:0:0]    disk    LSILOGIC 1030 IM          1000  /dev/sda
[0:0:8:0]    process IBM      25P3495a S320  1 1     -       
[2:0:0:0]    disk    IBM      1722-600         0520  /dev/sdb
[2:0:0:31]   disk    IBM      Universal Xport  0520  /dev/sdc
[3:0:0:0]    disk    IBM      1722-600         0520  /dev/sdd
[3:0:0:31]   disk    IBM      Universal Xport  0520  /dev/sde

Hier sind sdb und sdd diesselbe pysikalische LUN, die aber über unterschiedliche Pfade erreichbar sind (unterschiedliche HBA und Storage Controller Kombination). Da die device mapper block devices keine SCSI LUNs sind, werden diese nicht angezeigt.

multipath config

uids herausfinden

# /sbin/scsi_id -g -u -s /block/sdb
3600a0b800019e99000002e1234567890
# /sbin/scsi_id -g -u -s /block/sdc
3600a0b800019e99000002e6d00000000
# /sbin/scsi_id -g -u -s /block/sdd
3600a0b800019e99000002e1234567890

D.h. sdb und sdd sind identisch.

/etc/multipath.conf

defaults {
        udev_dir                /dev
        multipath_tool          "/sbin/multipath -v0"
        polling_interval        60
        selector                "round-robin 0"
        path_grouping_policy    failover
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            /bin/true
        path_checker            tur
        rr_min_io               100
        rr_weight               priorities
        failback                immediate
        no_path_retry           fail
        user_friendly_name      yes
}
devnode_blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*"
        devnode "^sda"
}
devices {
        device {
                vendor                  "IBM      "
                product                 "1722-600         "
                path_grouping_policy    group_by_serial
                path_checker tur
        }
        device {
                vendor "DataCore"
                product "SAN*"
                path_checker tur
                failback 10
        }
}

Erklärung

vendor → LUNs, die DataCore oder IBM als vendor in der SCSI INQUIRY data
product → LUNs, die “SAN*” oder “1722-600” als product-string enthalten
path_checker tur → Nach einem Pfad-Fehler wird der “failed Path” mit SCSI TEST UNIT READY geprüft
failback 10 → Wenn ein “failed-path” wieder “healthy” ist, warte 10 Sekunden vor dem FailBack
polling_interval 60 → Wenn ein Pfad defekt ist, prüfe alle 60 Seconden, ob er wieder OK ist

multipath -v2 -d

create: 3600a0b800019e99000002e1234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 
 \_ 2:0:0:0  sdb 8:16 [ready]
\_ round-robin 0 
 \_ 3:0:0:0  sdd 8:48 [ready]

create: 3600a0b800019e99000002e6d00000000
[size=20 MB][features="0"][hwhandler="0"]
\_ round-robin 0 
 \_ 2:0:0:31 sdc 8:32 [ready]

create: 3600a0b800019eac20000383600000000
[size=20 MB][features="0"][hwhandler="0"]
\_ round-robin 0 
 \_ 3:0:0:31 sde 8:64 [ready]

ls -al /dev/mapper

brw-rw----   1 root disk 253,  5 Jun 27 09:20 3600a0b800019e99000002e1234567890
brw-rw----   1 root disk 253,  8 Jun 27 09:20 3600a0b800019e99000002e1234567890p1
brw-rw----   1 root disk 253,  6 Jun 27 09:20 3600a0b800019e99000002e6d00000000
brw-rw----   1 root disk 253,  7 Jun 27 09:20 3600a0b800019eac20000383600000000

Mountpoint anpassen (vorher /dev/sdb1, jetzt /dev/mapper), Mounten nach LABEL=XYZ ist nicht möglich, da durch die Mehrfachpfade die Fehlermeldung mount: LABEL=XYZ duplicate - not mounted beim mount erscheint.

/etc/fstab

/dev/mapper/3600a0b800019e99000002e1234567890p1 /data ext3 defaults 0 0

Vorbereitung reboot

chkconfig multipathd on

Da beim Systemstart die multipath-Devices erzeugt werden, und der multipath.static auf das Dateisystem schreiben muss, ist die Datei /etc/rc.sysinit anzupassen.

/etc/rc.sysinit (excerpt - nach LVM initialization)

# Multipath initialization
if [ -x /sbin/multipath.static ]; then
    if ! LC_ALL=C fgrep -q "dm_multipath" /proc/modules 2>/dev/null ; then
        modprobe dm-multipath >/dev/null 2>&1
    fi
    if [ -x /sbin/multipath.static ]; then
        if /sbin/multipath.static -v0 > /dev/null 2>&1 ; then
            action $"Setting up Multipathing:" /sbin/multipath.static -v0
        fi
    fi
fi

Reboot (dmesg)

/var/log/messages

Jun 28 07:19:38 ds4k kernel: QLogic Fibre Channel HBA Driver
Jun 28 07:19:38 ds4k kernel: ACPI: PCI interrupt 0000:04:02.0[A] -> GSI 24 (level, low) -> IRQ 209
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Found an ISP2312, irq 209, iobase 0xffffff0000004000
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Configuring PCI space...
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Configure NVRAM parameters...
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Verifying loaded RISC code...
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: LIP reset occured (f8f7).
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Waiting for LIP to complete...
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: LIP occured (f8f7).
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: LOOP UP detected (2 Gbps).
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: Topology - (Loop), Host Loop address 0x7d
Jun 28 07:19:38 ds4k kernel: scsi2 : qla2xxx
Jun 28 07:19:38 ds4k kernel: qla2300 0000:04:02.0: 
Jun 28 07:19:39 ds4k kernel:  QLogic Fibre Channel HBA Driver: 8.01.02-d4
Jun 28 07:19:39 ds4k kernel:   QLogic QLA2340 - 133MHz PCI-X to 2Gb FC, Single Channel
Jun 28 07:19:39 ds4k kernel:   ISP2312: PCI-X (100 MHz) @ 0000:04:02.0 hdma+, host#=2, fw=3.03.18 IPX
Jun 28 07:19:39 ds4k kernel:   Vendor: IBM       Model: 1722-600          Rev: 0520
Jun 28 07:19:39 ds4k kernel:   Type:   Direct-Access                      ANSI SCSI revision: 03
Jun 28 07:19:39 ds4k kernel: qla2300 0000:04:02.0: scsi(2:0:0:0): Enabled tagged queuing, queue depth 32.
Jun 28 07:19:39 ds4k kernel: SCSI device sdb: 853947711 512-byte hdwr sectors (437221 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdb: drive cache: write back
Jun 28 07:19:39 ds4k kernel: SCSI device sdb: 853947711 512-byte hdwr sectors (437221 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdb: drive cache: write back
Jun 28 07:19:39 ds4k kernel:  sdb: sdb1
Jun 28 07:19:39 ds4k kernel: Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
Jun 28 07:19:39 ds4k kernel:   Vendor: IBM       Model: Universal Xport   Rev: 0520
Jun 28 07:19:39 ds4k kernel:   Type:   Direct-Access                      ANSI SCSI revision: 03
Jun 28 07:19:39 ds4k kernel: qla2300 0000:04:02.0: scsi(2:0:0:31): Enabled tagged queuing, queue depth 32.
Jun 28 07:19:39 ds4k kernel: SCSI device sdc: 40960 512-byte hdwr sectors (21 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdc: drive cache: write through
Jun 28 07:19:39 ds4k kernel: SCSI device sdc: 40960 512-byte hdwr sectors (21 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdc: drive cache: write through
Jun 28 07:19:39 ds4k kernel:  sdc: unknown partition table
Jun 28 07:19:39 ds4k kernel: Attached scsi disk sdc at scsi2, channel 0, id 0, lun 31
Jun 28 07:19:39 ds4k kernel: ACPI: PCI interrupt 0000:05:01.0[A] -> GSI 48 (level, low) -> IRQ 217
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Found an ISP2312, irq 217, iobase 0xffffff0000006000
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Configuring PCI space...
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Configure NVRAM parameters...
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Verifying loaded RISC code...
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: LIP reset occured (f8f7).
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Waiting for LIP to complete...
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: LIP occured (f8f7).
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: LOOP UP detected (2 Gbps).
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: Topology - (Loop), Host Loop address 0x7d
Jun 28 07:19:39 ds4k kernel: scsi3 : qla2xxx
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: 
Jun 28 07:19:39 ds4k kernel:  QLogic Fibre Channel HBA Driver: 8.01.02-d4
Jun 28 07:19:39 ds4k kernel:   QLogic QLA2340 - 133MHz PCI-X to 2Gb FC, Single Channel
Jun 28 07:19:39 ds4k kernel:   ISP2312: PCI-X (133 MHz) @ 0000:05:01.0 hdma+, host#=3, fw=3.03.18 IPX
Jun 28 07:19:39 ds4k kernel:   Vendor: IBM       Model: 1722-600          Rev: 0520
Jun 28 07:19:39 ds4k kernel:   Type:   Direct-Access                      ANSI SCSI revision: 03
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: scsi(3:0:0:0): Enabled tagged queuing, queue depth 32.
Jun 28 07:19:39 ds4k kernel: SCSI device sdd: 853947711 512-byte hdwr sectors (437221 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdd: drive cache: write back
Jun 28 07:19:39 ds4k kernel: SCSI device sdd: 853947711 512-byte hdwr sectors (437221 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sdd: drive cache: write back
Jun 28 07:19:39 ds4k kernel:  sdd: sdd1
Jun 28 07:19:39 ds4k kernel: Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0
Jun 28 07:19:39 ds4k kernel:   Vendor: IBM       Model: Universal Xport   Rev: 0520
Jun 28 07:19:39 ds4k kernel:   Type:   Direct-Access                      ANSI SCSI revision: 03
Jun 28 07:19:39 ds4k kernel: qla2300 0000:05:01.0: scsi(3:0:0:31): Enabled tagged queuing, queue depth 32.
Jun 28 07:19:39 ds4k kernel: SCSI device sde: 40960 512-byte hdwr sectors (21 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sde: drive cache: write through
Jun 28 07:19:39 ds4k kernel: SCSI device sde: 40960 512-byte hdwr sectors (21 MB)
Jun 28 07:19:39 ds4k kernel: SCSI device sde: drive cache: write through
Jun 28 07:19:39 ds4k kernel:  sde: unknown partition table
Jun 28 07:19:39 ds4k kernel: Attached scsi disk sde at scsi3, channel 0, id 0, lun 31
Jun 28 07:19:39 ds4k kernel: device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-devel@redhat.com
Jun 28 07:19:39 ds4k kernel: device-mapper: dm-multipath version 1.0.4 loaded

Failover Test

Hierzu wird die LUN über den eindeutigen /dev/mapper/-Devicenamen gemountet und sowohl gelesen als auch geschrieben. Während dieses Vorgangs wird zunächst der eine Pfad entfernt und anschließend wieder angeschlossen und dann der andere Pfad entfernt und wieder angeschlossen.

Test HBA1

Ziehen des FibreChannel Kabels von HBA1 und anschließendes wiederaufstecken. Nachdem der LOOP DOWN erkannt wird setzt der multipathd den einen Pfad auf failed faulty. Sobald der Link wieder vorhanden ist, wechselt der Status von failed ready auf active ready. Jetzt sind alle zwei Pfade wieder vorhanden.

HBA1: multipath -ll

watch multipath -ll

16:04:53
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:04:55
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [failed][faulty]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:06:06
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [enabled]
 \_ 2:0:0:0 sdb 8:16 [failed][faulty]
\_ round-robin 0 [active]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:06:48
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [enabled]
 \_ 2:0:0:0 sdb 8:16 [failed][faulty]
\_ round-robin 0 [active]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:06:51
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [enabled]
 \_ 2:0:0:0 sdb 8:16 [failed][ready]
\_ round-robin 0 [active]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:06:58
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [enabled]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:01
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][faulty]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:04
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

HBA1: syslog

/var/log/messages

Jun 28 16:04:55 ds4k kernel: qla2300 0000:04:02.0: LIP reset occured (f8ef).
Jun 28 16:04:56 ds4k kernel: qla2300 0000:04:02.0: LOOP DOWN detected (2).
Jun 28 16:05:55 ds4k kernel: SCSI error : <2 0 0 0> return code = 0x10000
Jun 28 16:05:55 ds4k kernel: end_request: I/O error, dev sdb, sector 135849247
Jun 28 16:05:55 ds4k kernel: end_request: I/O error, dev sdb, sector 135849255
Jun 28 16:05:55 ds4k kernel: device-mapper: dm-multipath: Failing path 8:16.
Jun 28 16:05:55 ds4k kernel: SCSI error : <2 0 0 0> return code = 0x10000
...
Jun 28 16:05:55 ds4k kernel: end_request: I/O error, dev sdb, sector 269927031
Jun 28 16:05:55 ds4k kernel: SCSI error : <2 0 0 0> return code = 0x10000
Jun 28 16:05:55 ds4k multipathd: 8:16: tur checker reports path is down
Jun 28 16:05:55 ds4k kernel: end_request: I/O error, dev sdb, sector 269929095
Jun 28 16:05:55 ds4k multipathd: checker failed path 8:16 in map 3600a0b80001773890000001234567890
Jun 28 16:05:55 ds4k multipathd: 3600a0b80001773890000001234567890: remaining active paths: 1
Jun 28 16:05:55 ds4k kernel: end_request: I/O error, dev sdb, sector 269929103
Jun 28 16:06:02 ds4k kernel: SCSI error : <2 0 0 0> return code = 0x10000
...
Jun 28 16:06:02 ds4k kernel: end_request: I/O error, dev sdb, sector 270066631
Jun 28 16:06:02 ds4k kernel: end_request: I/O error, dev sdb, sector 270066639
Jun 28 16:06:08 ds4k multipathd: 3600a0b80001773890000001234567890: switch to path group #1
Jun 28 16:06:50 ds4k kernel: qla2300 0000:04:02.0: LIP occured (f8ef).
Jun 28 16:06:50 ds4k kernel: qla2300 0000:04:02.0: LOOP UP detected (2 Gbps).
Jun 28 16:06:59 ds4k multipathd: 8:16: tur checker reports path is up
Jun 28 16:06:59 ds4k multipathd: 8:16: reinstated
Jun 28 16:06:59 ds4k multipathd: 3600a0b80001773890000001234567890: remaining active paths: 2
Jun 28 16:06:59 ds4k multipathd: 3600a0b80001773890000001234567890: switch to path group #1
Jun 28 16:06:59 ds4k multipathd: 3600a0b80001773890000001234567890: switch to path group #1

Test HBA2

Ziehen des FibreChannel Kabels von HBA2

HBA2: multipath -ll

watch multipath -ll

16:07:06
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:08
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:10
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:12
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:14
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:17
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:19
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:21
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:23
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:26
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:28
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:31
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:33
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:36
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:38
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:41
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:43
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

16:07:46
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [failed][faulty]

16:08:47
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [failed][faulty]

16:09:13
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [failed][faulty]

16:09:16
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [failed][ready]

16:09:26
3600a0b80001773890000001234567890
[size=407 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:0:0 sdb 8:16 [active][ready]
\_ round-robin 0 [enabled]
 \_ 3:0:0:0 sdc 8:32 [active][ready]

HBA2: syslog

/var/log/messages

Jun 28 16:07:46 ds4k kernel: qla2300 0000:05:01.0: LIP reset occured (f8e4).
Jun 28 16:07:47 ds4k kernel: qla2300 0000:05:01.0: LOOP DOWN detected (2).
Jun 28 16:08:45 ds4k kernel: device-mapper: dm-multipath: Failing path 8:32.
Jun 28 16:08:45 ds4k multipathd: 8:32: tur checker reports path is down
Jun 28 16:08:45 ds4k multipathd: checker failed path 8:32 in map 3600a0b80001773890000001234567890
Jun 28 16:08:45 ds4k multipathd: 3600a0b80001773890000001234567890: remaining active paths: 1
Jun 28 16:09:15 ds4k kernel: qla2300 0000:05:01.0: LIP occured (f8e4).
Jun 28 16:09:15 ds4k kernel: qla2300 0000:05:01.0: LOOP UP detected (2 Gbps).
Jun 28 16:09:25 ds4k multipathd: 8:32: tur checker reports path is up
Jun 28 16:09:25 ds4k multipathd: 8:32: reinstated
Jun 28 16:09:25 ds4k multipathd: 3600a0b80001773890000001234567890: remaining active paths: 2

dmsetup info

dmsetup info 3600a0b80001773890000001234567890

Name:              3600a0b80001773890000001234567890
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      1
Major, minor:      253, 5
Number of targets: 1
UUID: mpath-3600a0b80001773890000001234567890

dmsetup info 3600a0b80001773890000001234567890p1

Name:              3600a0b80001773890000001234567890p1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 8
Number of targets: 1
UUID: part1-mpath-3600a0b80001773890000001234567890

bzw.

dmsetup info -c --noheadings | grep mpath

3600a0b80001773890000001234567890:253:5:L--w:1:1:8:mpath-3600a0b80001773890000001234567890
3600a0b80001773890000001234567890p1:253:6:L--w:1:1:0:part1-mpath-3600a0b80001773890000001234567890

/sbin/dmsetup ls --target multipath

3600a0b800017738900000008437c0944       (253, 5)

Multipath mit RHEL4 (an DS4300)

initrd erstellen

dmesg

lsscsi

multipath config

Vorbereitung reboot

Reboot (dmesg)

Failover Test

Test HBA1

HBA1: multipath -ll

HBA1: syslog

Test HBA2

HBA2: multipath -ll

HBA2: syslog

dmsetup info

Navigation

wissen:ibm:blade:total-storage

aktuelle_ebene