Dotaz: FC SAN multipath, Debian a Ubuntu vidi ruzna zarizeni pri stejnem multipath.conf

[aldrtik]

Zdravim

na strojich s OS Debian 7-9 nam chodi multipath ke spokojenosti. Ovladac karet se pouziva z balicku firmware-qlogic.

Na jednom stroji, kde byl drive Debian je ted z duvodu prechodu na LXD Ubuntu 20.04. Identicky ovladac tam neni, je tam linux-firmware. multipath.conf je stejny jako predtim v Debianu.

Ubuntu vidi svazky (lsscsi, lvs) po jedne ceste, ale vystup multipath -ll je oproti Debianu jen zlomek seznamu zarizeni/cest, stejne jako zlomek zarizeni v /dev/mapper.

Proces bezi:

root 4325 1 0 14:29 ? 00:00:01 /sbin/multipathd -d -s

multipathd -k

multipathd> list multipaths status
name        failback  queueing paths dm-st  write_prot
LCSM100_F_1 immediate 30 chk   2     active rw        
LCSM100_F_2 immediate 30 chk   2     active rw        



multipathd> show paths
hcil     dev  dev_t  pri dm_st  chk_st dev_st  next_check     
7:0:0:0  sdb  8:16   6   undef  undef  unknown orphan         
7:0:1:0  sdf  8:80   50  undef  undef  unknown orphan         
7:0:1:1  sdj  8:144  10  undef  undef  unknown orphan         
7:0:1:2  sdk  8:160  50  undef  undef  unknown orphan         
7:0:1:3  sdm  8:192  10  undef  undef  unknown orphan         
7:0:1:4  sdo  8:224  50  undef  undef  unknown orphan         
7:0:8:0  sdbj 67:208 50  undef  undef  unknown orphan         
7:0:8:1  sdbm 68:0   50  undef  undef  unknown orphan         
7:0:8:2  sdbo 68:32  50  undef  undef  unknown orphan         
7:0:8:3  sdbr 68:80  10  undef  undef  unknown orphan         
7:0:8:4  sdbs 68:96  50  undef  undef  unknown orphan         
7:0:2:0  sdq  65:0   10  undef  undef  unknown orphan         
7:0:2:1  sdt  65:48  10  undef  undef  unknown orphan         
7:0:2:2  sdu  65:64  10  undef  undef  unknown orphan         
7:0:3:0  sdw  65:96  50  undef  undef  unknown orphan         
7:0:3:1  sdab 65:176 50  undef  undef  unknown orphan         
7:0:3:2  sdad 65:208 50  undef  undef  unknown orphan         
7:0:3:3  sdae 65:224 10  undef  undef  unknown orphan         
7:0:3:4  sdag 66:0   50  undef  undef  unknown orphan         
7:0:4:0  sdai 66:32  10  undef  undef  unknown orphan         
7:0:4:1  sdal 66:80  10  undef  undef  unknown orphan         
7:0:4:2  sdam 66:96  10  undef  undef  unknown orphan         
7:0:5:0  sdan 66:112 50  undef  undef  unknown orphan         
7:0:5:1  sdap 66:144 50  undef  undef  unknown orphan         
7:0:5:2  sdar 66:176 50  undef  undef  unknown orphan         
7:0:5:3  sdat 66:208 10  undef  undef  unknown orphan         
7:0:5:4  sdav 66:240 50  undef  undef  unknown orphan         
7:0:6:0  sday 67:32  6   active ready  running .......... 1/20
7:0:6:1  sdbb 67:80  6   active ready  running .......... 1/20
7:0:7:0  sdbd 67:112 10  undef  undef  unknown orphan         
7:0:7:1  sdbg 67:160 10  undef  undef  unknown orphan         
7:0:7:2  sdbh 67:176 10  undef  undef  unknown orphan         
7:0:9:0  sdbt 68:112 50  undef  undef  unknown orphan         
7:0:9:1  sdbw 68:160 50  undef  undef  unknown orphan         
7:0:9:2  sdbx 68:176 50  undef  undef  unknown orphan         
7:0:9:3  sdby 68:192 10  undef  undef  unknown orphan         
7:0:9:4  sdbz 68:208 50  undef  undef  unknown orphan         
7:0:10:0 sdca 68:224 10  undef  undef  unknown orphan         
7:0:10:1 sdcb 68:240 10  undef  undef  unknown orphan         
7:0:10:2 sdcc 69:0   10  undef  undef  unknown orphan         
9:0:0:0  sdc  8:32   6   active ready  running .......... 1/20
9:0:0:1  sdd  8:48   6   active ready  running .......... 1/20
9:0:3:0  sds  65:32  1   undef  undef  unknown orphan         
9:0:6:0  sdaj 66:48  10  undef  undef  unknown orphan         
9:0:6:1  sdak 66:64  10  undef  undef  unknown orphan         
9:0:6:2  sdao 66:128 10  undef  undef  unknown orphan         
9:0:6:3  sdaq 66:160 50  undef  undef  unknown orphan         
9:0:6:4  sdas 66:192 10  undef  undef  unknown orphan         
9:0:2:0  sdi  8:128  10  undef  undef  unknown orphan         
9:0:2:1  sdl  8:176  50  undef  undef  unknown orphan         
9:0:2:2  sdn  8:208  10  undef  undef  unknown orphan         
9:0:2:3  sdp  8:240  50  undef  undef  unknown orphan         
9:0:2:4  sdr  65:16  10  undef  undef  unknown orphan         
9:0:5:0  sdac 65:192 50  undef  undef  unknown orphan         
9:0:5:1  sdaf 65:240 50  undef  undef  unknown orphan         
9:0:5:2  sdah 66:16  50  undef  undef  unknown orphan         
9:0:7:0  sdau 66:224 10  undef  undef  unknown orphan         
9:0:7:1  sdaw 67:0   10  undef  undef  unknown orphan         
9:0:7:2  sdax 67:16  10  undef  undef  unknown orphan         
9:0:7:3  sdaz 67:48  50  undef  undef  unknown orphan         
9:0:7:4  sdba 67:64  10  undef  undef  unknown orphan         
9:0:1:0  sde  8:64   50  undef  undef  unknown orphan         
9:0:1:1  sdg  8:96   50  undef  undef  unknown orphan         
9:0:1:2  sdh  8:112  50  undef  undef  unknown orphan         
9:0:4:0  sdv  65:80  10  undef  undef  unknown orphan         
9:0:4:1  sdx  65:112 10  undef  undef  unknown orphan         
9:0:4:2  sdy  65:128 10  undef  undef  unknown orphan         
9:0:4:3  sdz  65:144 50  undef  undef  unknown orphan         
9:0:4:4  sdaa 65:160 10  undef  undef  unknown orphan         
9:0:9:0  sdbl 67:240 50  undef  undef  unknown orphan         
9:0:9:1  sdbn 68:16  50  undef  undef  unknown orphan         
9:0:9:2  sdbp 68:48  50  undef  undef  unknown orphan         
9:0:8:0  sdbc 67:96  10  undef  undef  unknown orphan         
9:0:8:1  sdbe 67:128 10  undef  undef  unknown orphan         
9:0:8:2  sdbf 67:144 10  undef  undef  unknown orphan         
9:0:8:3  sdbi 67:192 50  undef  undef  unknown orphan         
9:0:8:4  sdbk 67:224 10  undef  undef  unknown orphan         
9:0:10:0 sdbq 68:64  50  undef  undef  unknown orphan         
9:0:10:1 sdbu 68:128 50  undef  undef  unknown orphan         
9:0:10:2 sdbv 68:144 50  undef  undef  unknown orphan         


multipath -ll 
LCSM100_F_2 (3600a0b800075fb520000025851c90f12) dm-5 SUN,LCSM100_F
size=1.1T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
`-+- policy='round-robin 0' prio=6 status=active
  |- 7:0:6:1  sdbb 67:80  active ready running
  `- 9:0:0:1  sdd  8:48   active ready running
LCSM100_F_1 (3600a0b800075fb520000023d51bfd912) dm-4 SUN,LCSM100_F
size=1.1T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
`-+- policy='round-robin 0' prio=6 status=active
  |- 7:0:6:0  sday 67:32  active ready running
  `- 9:0:0:0  sdc  8:32   active ready running

Jestli jste se s tim uz nekdo nesetkal pri prechodu, resp. jaky by mohl byt postup jak to uchodit, aby videl svazky jako predtim Debian?

Odpovědi

[pupu]

Doporucoval bych upravit konfiguraci multipathu podle doporuceni vyrobce protejsku, neboli pole. U nas jsme museli kvuli novemu poli prekopaval pomerne dost parametru... Jinak nasledne bych asi zkontroloval, ze nemam zarizeni omylem blacklistovana, a udelal multipath -v 2 , abych videl, co a jak to detekuje.

[aldrtik]

Diky za reakci.

Ze zoufalstvi jsem vygeneroval multipath.conf pomoci:

multipath -T > multipath.conf

a prislusna sekce zarizeni s konfiguraci vypadla:

	device {
		vendor "SUN"
		product "SUN_6180"
		product_blacklist "Universal Xport"
		path_grouping_policy "group_by_prio"
		path_checker "rdac"
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio "rdac"
		failback "immediate"
		no_path_retry 30
	}
	
multipaths {
	multipath {	
		wwid "360080e50001808ce0000113051c03580"
        alias SUN_6180
	}

Ale Ubuntu i v posledni verzi 20.10 se na diskove pole tvari porad nepratelsky:

root@blade2:~# multipath -v 2
Mar 12 14:25:13 | SUN_6180: ignoring map
Mar 12 14:25:14 | SUN_6180: ignoring map


~# multipath -d
: SUN_6180 (360080e50001808ce0000113051c03580) undef SUN,SUN_6180
size=1.0T features='3 pg_init_retries 50 queue_if_no_path' hwhandler='1 rdac' wp=undef
|-+- policy='service-time 0' prio=6 status=undef
| `- 7:0:0:0  sdb  8:16   undef ready running
`-+- policy='service-time 0' prio=1 status=undef
  `- 9:0:1:0  sdf  8:80   undef ghost running

V sekci blacklist neni.

Pritom velmi podobne pole od stejneho vyrobce:

SUN STK2540 v pohode detekuje jako LCSM100 a zadne potize nejsou.

Zatim netusim, jakou direktivu ohnout, aby ho Ubuntu vzalo, resp jaky postup zvolit, aby se to dalo odvsivit.

[pupu]

Moc dobre to nevypada. Chvilka Googleni (https://access.redhat.com/solutions/36232 )...

Resolution Device is unsupported..? Please use SUN supplied RDAC or MPIO multipath.

Root Cause There is no default configurations for SUN StorageTEK 6180 SAN devices for device-mapper-mulitpath on Red Hat Enterprise Linux 5. SUN_6180 is not in the supported scsi_dh_rdac.ko list of SAN's, so the module cannot be used with this device

It is unknown if device-mapper-multipath is supported by SUN on SUN StorageTek 6180 at this stage. Not listed here: http://lu.sun.com/products/disk/6180/specs.xml#Software

No defaults for SUN 6180 in upstream multipath-tools either: http://git.kernel.org/gitweb.cgi?p=linux%2Fstorage%2Fmultipath-tools%2F.git&a=search&h=HEAD&st=commit&s=6180

No support for SUN 6180 in upstream scsi_dh_rdac either.

Diagnostic Steps When trying to use device-mapper-multipath with SUN StorageTEK 6180, paths appear as failed

If you try to use a similar RDAC config from another SUN device (such as SUN,CSM200_R), tThe devices sometimes don't show up correctly (some paths missing, some multipath devices completely missing) even though all the paths are discovered. Even after building initrd with --preload=scsi_dh_rdac, the scsi_dh_rdac module is not loaded on boot:

This is due to there being no support for SUN_6180 in the scsi_dh_rdac module.

Ja bych tu vykopavku zahodil nebo nechal pro pouziti se Solarisem

[aldrtik]

Diky za postreh. Jen se podivuji nad tim, ze Debian prokazatelne ve verzich 7-9 s tim nema vubec zadny problem. Tu verzi 10 jeste jednou proverim.

[j]

Proste slohni ten balicek primo z debianu, to by mohlo i fungovat, alternativne se porozhlidni po strankach vyrobce tech karet.

https://packages.debian.org/sid/all/firmware-qlogic/download

Mozna si nekde puc jiny FC karty. Protoze IMO nebude problem s polem, ale prave snima. Respektive prave s tim ze pouzivas nejaky obecny balicek ktery si stema co mas asi ne uplne rozumi.

Jinak tohle je typicka ukazka toho, proc se zasadne podobny veci neaktualizujou. Pokud dobre vidim, je to 7+ let stary HW.

Jinak to v principu muzes jeste otestovat bez pole, s jinym serverem. To kdybys chtel 100% eliminovat vliv pole. Jednoduse z toho na kterym vis ze to funguje udelas jakopole ;D.

---

Dete s tim guuglem dopice!

[drunkezz]

JJ to bude imo ono

hod tu lspci tych kariet, som zvedavy aky je tam fw a driver. Je dost mozne ze to tie 4G karty uz prosto nebudu mat drive , rhel to vyhodil uz davno, debian taky tak

D.

[aldrtik]

lspci vypisy jsou tyto:

Debian 9

lspci  -mm | grep ibre
09:00.0 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
09:00.1 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
0f:00.0 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
0f:00.1 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"



Ubuntu 20.10

lspci  -mm | grep ibre
09:00.0 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
09:00.1 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
0f:00.0 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"
0f:00.1 "Fibre Channel" "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA" -r02 "QLogic Corp." "ISP2532-based 8Gb Fibre Channel to PCI Express HBA"

[R]

Pises nezmysly.

# dmesg | head -n 1
[    0.000000] Linux version 5.10.0-0.bpo.3-amd64 (debian-kernel@lists.debian.org) (gcc-8 (Debian 8.3.0-6) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP Debian 5.10.13-1~bpo10+1 (2021-02-11)
# dmesg | egrep qla\|emc
[    1.866854] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 10.02.00.103-k.
[    1.867030] qla2xxx [0000:00:00.0]-001d: : Found an ISP2432 irq 16 iobase 0x(____ptrval____).
[    1.924324] qla2xxx [0000:0c:00.0]-ffff:3: FC4 priority set to FCP
[    1.947884] qla2xxx 0000:0c:00.0: firmware: direct-loading firmware ql2400_fw.bin
[    2.299809] scsi host3: qla2xxx
[    2.301632] qla2xxx [0000:0c:00.0]-00fb:3: QLogic QLE2460 - PCI-Express Single Channel 4Gb Fibre Channel HBA.
[    2.301640] qla2xxx [0000:0c:00.0]-00fc:3: ISP2432: PCIe (2.5GT/s x4) @ 0000:0c:00.0 hdma+ host#=3 fw=8.07.00 (9496).
[    2.301817] qla2xxx [0000:00:00.0]-001d: : Found an ISP2432 irq 16 iobase 0x(____ptrval____).
[    2.359000] qla2xxx [0000:0e:00.0]-ffff:4: FC4 priority set to FCP
[    2.719758] scsi host4: qla2xxx
[    2.721561] qla2xxx [0000:0e:00.0]-00fb:4: QLogic QLE2460 - PCI-Express Single Channel 4Gb Fibre Channel HBA.
[    2.721572] qla2xxx [0000:0e:00.0]-00fc:4: ISP2432: PCIe (2.5GT/s x4) @ 0000:0e:00.0 hdma+ host#=4 fw=8.07.00 (9496).
[    2.902861] qla2xxx [0000:0c:00.0]-500a:3: LOOP UP detected (4 Gbps).
[    3.301919] qla2xxx [0000:0e:00.0]-500a:4: LOOP UP detected (4 Gbps).
[    5.046378] emc: device handler registered
[    5.105658] sd 3:0:0:0: emc: detected Clariion CX3-10c, flags 0
[    5.105842] sd 3:0:0:0: emc: connected to SP A Port 2 (owned, default SP A)
[    5.105855] sd 4:0:0:0: emc: detected Clariion CX3-10c, flags 0
[    5.106016] sd 4:0:0:0: emc: connected to SP A Port 2 (owned, default SP A)
[    5.106028] sd 3:0:1:0: emc: detected Clariion CX3-10c, flags 0
[    5.106195] sd 3:0:1:0: emc: connected to SP B Port 3 (bound, default SP A)
[    5.106206] sd 4:0:1:0: emc: detected Clariion CX3-10c, flags 0
[    5.106371] sd 4:0:1:0: emc: connected to SP B Port 3 (bound, default SP A)
[    5.192009] sd 3:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[    5.192189] sd 4:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[    5.210445] sd 3:0:1:1: emc: detected Clariion CX3-10c, flags 0
[    5.210618] sd 3:0:1:1: emc: connected to SP B Port 3 (owned, default SP B)
[    5.210631] sd 4:0:1:1: emc: detected Clariion CX3-10c, flags 0
[    5.210793] sd 4:0:1:1: emc: connected to SP B Port 3 (owned, default SP B)
[    5.210807] sd 3:0:0:1: emc: detected Clariion CX3-10c, flags 0
[    5.210975] sd 3:0:0:1: emc: connected to SP A Port 2 (bound, default SP B)
[    5.210987] sd 4:0:0:1: emc: detected Clariion CX3-10c, flags 0
[    5.211149] sd 4:0:0:1: emc: connected to SP A Port 2 (bound, default SP B)
[    5.279864] sd 3:0:1:1: emc: at SP B Port 3 (owned, default SP B)
[    5.280046] sd 4:0:1:1: emc: at SP B Port 3 (owned, default SP B)
[    5.301237] sd 3:0:1:1: emc: at SP B Port 3 (owned, default SP B)
[    5.301435] sd 4:0:1:1: emc: at SP B Port 3 (owned, default SP B)
[    5.304488] sd 3:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[    5.304666] sd 4:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[   17.158473] sd 3:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[   17.158670] sd 4:0:0:0: emc: at SP A Port 2 (owned, default SP A)
[   17.163908] sd 3:0:1:1: emc: at SP B Port 3 (owned, default SP B)
[   17.164094] sd 4:0:1:1: emc: at SP B Port 3 (owned, default SP B)

[aldrtik]

Zkusil jsem umaznout linux-firmware z Ubuntu a strcit tam balicek z Debianu 9 firmware-qlogic, ale zadna zmena k lepsimu se nekonala.

Debiansky

dpkg -L firmware-qlogic | grep ql25
/lib/firmware/ql2500_fw.bin

COPYRIGHT 2015 QLOGIC CORPORATION   ISP25xx Firmware   Version   8.03.00 



Ubunti
dpkg -L linux-firmware  | grep ql25
/lib/firmware/ql2500_fw.bin

COPYRIGHT 2017 QLOGIC CORPORATION   ISP25xx Firmware   Version   8.07.00

Nechapu, proc by alespon podle verze novejsi verze FW mela ignorovat novejsi SUN/STK 6180 pole a starsi STK 2540 tam je videt.

[Ruža Becelin]

Prosel bych tohle, ale podle tohohle to vypada s podporou dost bidne. Nechal bych tam Debian 9, nic novejsiho asi nehrozi.

[aldrtik]

Jo, tohle jsme probadali uz pri zprovoznovani Debian 7.

[R]

Chybaju tu zakladne udaje - dmesg z oboch pripadov (funkcny aj nefunkcny stav).

SUN_6180 je v kerneli normalne podporovany, nejake nezmyselne linky na 100 rokov stare dokumenty u RedHatu a Oraclu neznamenaju nic: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/scsi/scsi_dh.c

[aldrtik]

Příloha:

• dmesg_Ubu20.10.txt (190012 bytů)

Posilam oba dmesg vystupy. Jasne ze je to stary kus diskoveho pole, ale ma jednu nespornou vlastnost - pouziva jeste stare FC disky a u tech jsem od roku 1997, kdy jsem je mel poprve v jednom z prvnich FC poli, doted nevidel zadny vadny, narozdil od spousty spotrebnich SATA. dmesg Debian 9 pole vidi, Ubuntu 20.10 pole odmita.

[aldrtik]

Příloha:

• dmesg_Deb9.txt (193177 bytů)

Nepodarilo se mi dostat do jednoho prispevku najednou oba soubory, takze jeste druhy.

[R]

Chyba tam zaciatok (v oboch suboroch) - asi sa nevosiel do kernel bufferu. Cele by to mohlo byt vo /var/log/dmesg.

Ale predbezne vidiet, ze aj v Ubuntu sa aktivuje rdac device handler.

Debian 10 si neskusal? Pripadne Debian Testing.