Dotaz: Disk vypadává ze ZFS

[Michal]

Mám nějakou dobu pole ze 4 SSD (2 mirrory, tedy "RAID10").

To si vcera v noci vesele degradovalo kvuli tohodle:

[5777221.615840] mpt2sas_cm0: log_info(0x31110d00): originator(PL), code(0x11), sub_code(0x0d00)
[5777221.629251] sd 1:0:0:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.629274] sd 1:0:0:0: [sde] tag#0 Sense Key : Not Ready [current]
[5777221.629279] sd 1:0:0:0: [sde] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[5777221.629284] sd 1:0:0:0: [sde] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[5777221.629308] print_req_error: I/O error, dev sde, sector 0
[5777221.630620] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.630646] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5777221.630652] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5777221.630661] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 00 00 09 10 00 00 10 00
[5777221.630679] print_req_error: I/O error, dev sde, sector 2320
[5777221.630778] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.630783] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5777221.630788] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5777221.630792] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 00 00 0b 10 00 00 10 00
[5777221.630795] print_req_error: I/O error, dev sde, sector 2832
[5777221.630975] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.630977] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5777221.630979] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5777221.630981] sd 1:0:0:0: [sde] tag#12 CDB: Read(10) 28 00 00 00 0a 10 00 00 10 00
[5777221.630983] print_req_error: I/O error, dev sde, sector 2576
[5777221.631004] sd 1:0:0:0: [sde] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.631006] sd 1:0:0:0: [sde] tag#13 Sense Key : Not Ready [current]
[5777221.631008] sd 1:0:0:0: [sde] tag#13 Add. Sense: Logical unit not ready, cause not reportable
[5777221.631010] sd 1:0:0:0: [sde] tag#13 CDB: Read(10) 28 00 3a 38 1c 10 00 00 10 00
[5777221.631011] print_req_error: I/O error, dev sde, sector 976755728
[5777221.631021] sd 1:0:0:0: [sde] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.631023] sd 1:0:0:0: [sde] tag#14 Sense Key : Not Ready [current]
[5777221.631025] sd 1:0:0:0: [sde] tag#14 Add. Sense: Logical unit not ready, cause not reportable
[5777221.631026] sd 1:0:0:0: [sde] tag#14 CDB: Read(10) 28 00 3a 38 1e 10 00 00 10 00
[5777221.631028] print_req_error: I/O error, dev sde, sector 976756240
[5777221.631073] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.631076] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5777221.631078] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5777221.631080] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 3a 38 1f 10 00 00 10 00
[5777221.631082] print_req_error: I/O error, dev sde, sector 976756496
[5777221.631090] sd 1:0:0:0: [sde] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.631094] sd 1:0:0:0: [sde] tag#13 Sense Key : Not Ready [current]
[5777221.631096] sd 1:0:0:0: [sde] tag#13 Add. Sense: Logical unit not ready, cause not reportable
[5777221.631108] sd 1:0:0:0: [sde] tag#13 CDB: Write(10) 2a 00 3a 38 1d 10 00 00 10 00
[5777221.631109] print_req_error: I/O error, dev sde, sector 976755984
[5777221.641935] sd 1:0:0:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.641941] sd 1:0:0:0: [sde] tag#0 Sense Key : Not Ready [current]
[5777221.641945] sd 1:0:0:0: [sde] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[5777221.641959] sd 1:0:0:0: [sde] tag#0 CDB: Read(10) 28 00 00 00 0a 10 00 00 10 00
[5777221.641963] print_req_error: I/O error, dev sde, sector 2576
[5777221.641987] sd 1:0:0:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5777221.641989] sd 1:0:0:0: [sde] tag#1 Sense Key : Not Ready [current]
[5777221.641992] sd 1:0:0:0: [sde] tag#1 Add. Sense: Logical unit not ready, cause not reportable
[5777221.641995] sd 1:0:0:0: [sde] tag#1 CDB: Read(10) 28 00 3a 38 1c 10 00 00 10 00
[5777221.641997] print_req_error: I/O error, dev sde, sector 976755728
[5777796.527580] userif-3: sent link down event.
[5777796.527585] userif-3: sent link up event.
[5778018.892616] sd 1:0:0:0: Power-on or device reset occurred

smart + short offline test -> pry ok, dd z toho disku do dev null -> ok

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       24626
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       45
177 Wear_Leveling_Count     0x0013   097   097   000    Pre-fail  Always       -       54
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   064   052   000    Old_age   Always       -       36
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       22
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       36027669532

SMART Error Log Version: 1
No Errors Logged

Warning! SMART Self-Test Log Structure error: invalid SMART checksum.
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     24624         -

Tak jsem udelal detach a attach, tesne pred koncem resilveru:

[5836666.142170] mpt2sas_cm0: log_info(0x31110d00): originator(PL), code(0x11), sub_code(0x0d00)
[5836666.158077] scsi_io_completion: 1 callbacks suppressed
[5836666.158086] sd 1:0:0:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.158093] sd 1:0:0:0: [sde] tag#0 Sense Key : Not Ready [current]
[5836666.158097] sd 1:0:0:0: [sde] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[5836666.158102] sd 1:0:0:0: [sde] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[5836666.158110] print_req_error: 1 callbacks suppressed
[5836666.158112] print_req_error: I/O error, dev sde, sector 0
[5836666.159396] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159401] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5836666.159405] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159409] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 00 00 09 e0 00 00 10 00
[5836666.159412] print_req_error: I/O error, dev sde, sector 2528
[5836666.159571] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159578] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5836666.159588] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159591] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 00 00 0b e0 00 00 10 00
[5836666.159593] print_req_error: I/O error, dev sde, sector 3040
[5836666.159662] sd 1:0:0:0: [sde] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159671] sd 1:0:0:0: [sde] tag#12 Sense Key : Not Ready [current]
[5836666.159676] sd 1:0:0:0: [sde] tag#12 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159679] sd 1:0:0:0: [sde] tag#12 CDB: Write(10) 2a 00 3a 38 1d e0 00 00 10 00
[5836666.159681] print_req_error: I/O error, dev sde, sector 976756192
[5836666.159736] sd 1:0:0:0: [sde] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159744] sd 1:0:0:0: [sde] tag#13 Sense Key : Not Ready [current]
[5836666.159750] sd 1:0:0:0: [sde] tag#13 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159757] sd 1:0:0:0: [sde] tag#13 CDB: Read(10) 28 00 00 00 0a 10 00 00 10 00
[5836666.159762] print_req_error: I/O error, dev sde, sector 2576
[5836666.159797] sd 1:0:0:0: [sde] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159799] sd 1:0:0:0: [sde] tag#14 Sense Key : Not Ready [current]
[5836666.159801] sd 1:0:0:0: [sde] tag#14 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159803] sd 1:0:0:0: [sde] tag#14 CDB: Read(10) 28 00 3a 38 1c 10 00 00 10 00
[5836666.159805] print_req_error: I/O error, dev sde, sector 976755728
[5836666.159812] sd 1:0:0:0: [sde] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159814] sd 1:0:0:0: [sde] tag#15 Sense Key : Not Ready [current]
[5836666.159816] sd 1:0:0:0: [sde] tag#15 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159818] sd 1:0:0:0: [sde] tag#15 CDB: Read(10) 28 00 3a 38 1e 10 00 00 10 00
[5836666.159821] print_req_error: I/O error, dev sde, sector 976756240
[5836666.159868] sd 1:0:0:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.159874] sd 1:0:0:0: [sde] tag#0 Sense Key : Not Ready [current]
[5836666.159875] sd 1:0:0:0: [sde] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[5836666.159877] sd 1:0:0:0: [sde] tag#0 CDB: Write(10) 2a 00 3a 38 1f e0 00 00 10 00
[5836666.159879] print_req_error: I/O error, dev sde, sector 976756704
[5836666.165482] sd 1:0:0:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.165484] sd 1:0:0:0: [sde] tag#0 Sense Key : Not Ready [current]
[5836666.165486] sd 1:0:0:0: [sde] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[5836666.165487] sd 1:0:0:0: [sde] tag#0 CDB: Read(10) 28 00 00 00 0a 10 00 00 10 00
[5836666.165488] print_req_error: I/O error, dev sde, sector 2576
[5836666.165503] sd 1:0:0:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[5836666.165504] sd 1:0:0:0: [sde] tag#1 Sense Key : Not Ready [current]
[5836666.165505] sd 1:0:0:0: [sde] tag#1 Add. Sense: Logical unit not ready, cause not reportable
[5836666.165516] sd 1:0:0:0: [sde] tag#1 CDB: Read(10) 28 00 3a 38 1c 10 00 00 10 00
[5836666.165517] print_req_error: I/O error, dev sde, sector 976755728

Nejaky napad?

Odpovědi

[Max]

Stávalo se mi něco podobného a problém jsem měl v řadiči. Server fungoval vždy nějakou dobu plně ok a po třeba měsíci tohle. Testy disků a vše okolo plně ok. Ve finále se mi stávalo, že to řadič vzdával kompletně (= server vytunul a přístup na všechny disky nefungoval).
V mým případě to byl Areca řadič (v passthrough režimu a nepomáhaly ani novější verze fw), ke kterému jsem měl připojený 4TB SATA disky WD RE4. Po výměně za "LSI SAS 9300-8i SGL" vše plně ok.
Jinak problém vždy nastával u velkého zatížení pole.
Zdar Max

[Vantomas]

Stejná zkušenost s Areca řadičem v Supermicru. Po výměně za LSI v pohodě.

[Michal]

Diky za napad.

/sys/block/sde -> ../devices/pci0000:00/0000:00:03.0/0000:04:00.0/host1/port-1:0/end_device-1:0/target1:0:0/1:0:0:0/block/sde

nevim, proc tam jsou 2 cisla ale predpokladam, ze relevantni je to druhe pak: lspci

04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

No, skusim vymenit napajeci kablik. sata kablik je spolecny pro 4 zarizeni ten tady asi nahradni nemam

[dustin]

Jedná se o kabel splitter ze SFF-8087 na 4x SATA? S těmi mám horší zkušenost, SATA nemají úplně dobré kontakty, nemívají ani takové ty zámky na SATA. S nimi mi také občas disk z pole vypadnul. Po přechodu na pořádný case se šuplíky a backplane se SFF-8087 je pokoj.

[Michal]

Sprej na cisteni kontaktu, trocha cisteni sexualni metodou (znate to: zasunout, vysunout,...) a zda se, ze to pomohlo.

[Max]

Pokud je to tak, tak to dlouho asi nevydrží.
Zdar Max

[j]

Viz Max, konektory by mely predevsim fakt pomerne pevne sedet, a pokud sou volny, a daji se lehkym tahem sundat, tak je to spatne a dlouho to fungovat nebude.

Mimochodem, tyhle spreje jsou vetsinou docela svinstvo, a specilne elektronice rozhodne dobre nedelaji. Vetsinou jsou totiz urceny spis na veci v autech a podobne. Elektronicky konektory (predevsim ty na plosnacich) se nejlip cistej obycejnou gumovaci gumou. A z druhy strany by to melo byt pozlaceny.

[Pavel 'TIGER' Růžička]

Když nezabere pouhý vzduch, přidám isopropilalkohol, když nezabere ani to, stříknu Brunox Turbo Spray a následně isopropilalkohol. Avšak kabely je lepší měnit, protože ty se šidí víc a víc.