Portál AbcLinuxu, 13. července 2025 16:26
Dobrý den,
Prosím o pomoc s následujícím problémem, mám server HP proliant, OS Fedora7, 2 disky v sw raid mirror, a z neznámých příčin se poškodily některé soubory v /bin (výpis jen inkriminovaných souborů, zbytek zdá se ok)ls -l celkem 8090352185 c-----x-w- 8224 1634869280 1769349130 51, 55 2000-06-02 06:04 arch ?--S--xr-t 11565 842345773 959917357 757935405 1994-01-07 10:36 dd ?---r----T 8224 757936176 757932064 221261868 1987-04-04 04:43 dmesg cr-Sr-sr-x 29300 986197293 538996560 10, 13 2030-10-26 13:16 dumpkeys c---r----- 28534 757932064 757952372 49, 32 1975-05-06 20:27 kbd_mode ?---rw-r-t 12852 824195383 741684023 221395000 1995-08-16 00:30 login br-xr-xrw- 13856 1260396576 1109404982 32, 10 1995-08-12 06:38 mktemp ?r-Srws--x 11565 539893770 776809773 808464419 1977-01-05 16:50 mount ?---rw-r-x 8224 807416876 825237536 537529656 1987-01-30 04:38 pgawk cr-Sr-sr-x 28462 943262221 842560046 108, 236 2026-01-25 19:31 rpm ?r-xr-sr-- 28532 1986081056 1852404716 221263408 2029-03-05 08:43 setfont ?--Srwx--x 20234 808202808 221252914 538970637 1997-03-19 19:28 setserial c--SrwSrwx 8224 168637489 538976288 80, 32 1987-01-30 04:38 su c---r--r-x 8313 538976288 538996329 55, 44 2029-03-06 21:08 sync ?---rw-r-- 14368 538970637 1346641952 1867522097 1987-01-30 04:59 touch c---r----- 13600 538976288 538999150 48, 57 2031-11-26 10:34 true c---r----- 12576 1953436704 1634885992 32, 121 2012-01-08 22:40 ulockmgr_server c---rw---x 3376 541618000 624245036 32, 32 1987-01-30 04:38 unameNěkteré mají na začátku práv ?, ty mají také nesmyslnou délku (viz touch 1,7GB) jiné se tváří jako soubory z /dev (blokové) moje idea byla je smazat a nahradit kopií z jiného systému ale na všechny moje pokusy mám permission denied. zkoušel jsem mazat podle inode viz
#find . -inum 34734123 -exec rm -i {} \;
rm: remove write-protected character special file `./su'? y
rm: cannot remove `./su': Operace není povolena
# id
uid=0(root) gid=0(root) skupiny=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
Restart jsem nezkoušel protože je poškozen login, mám obavy že server nenaběhne a jeho provoz je kritický.
Přihlašuji se pomocí ssh.
V logu jsem nic nenašel jen chyby disku, ale myslím, že ty souvisí s poškozenými soubory ne se samotným HW.
cat /var/log/messages
Dec 29 12:58:09 datav kernel: attempt to access beyond end of device Dec 29 12:58:09 datav kernel: md0: rw=0, want=4311820720, limit=484198912 Dec 29 12:58:24 datav kernel: attempt to access beyond end of device Dec 29 12:58:24 datav kernel: md0: rw=0, want=4312220256, limit=484198912 Dec 29 13:00:24 datav kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen Dec 29 13:00:24 datav kernel: ata2.00: cmd 35/00:10:e5:fa:cf/00:00:10:00:00/e0 tag 0 cdb 0x0 data 8192 out Dec 29 13:00:24 datav kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Dec 29 13:00:24 datav kernel: ata2: port is slow to respond, please be patient (Status 0xd0) Dec 29 13:00:24 datav kernel: ata2: device not ready (errno=-16), forcing hardreset Dec 29 13:00:24 datav kernel: ata2: soft resetting port Dec 29 13:00:24 datav kernel: ata2.00: configured for UDMA/100 Dec 29 13:00:24 datav kernel: ata2: EH complete Dec 29 13:00:25 datav kernel: sd 1:0:0:0: [sdb] 488397168 512-byte hardware sectors (250059 MB) Dec 29 13:00:25 datav kernel: sd 1:0:0:0: [sdb] Write Protect is off Dec 29 13:00:25 datav kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Dec 29 13:03:08 datav kernel: ata2.00: limiting speed to UDMA/33:PIO4 Dec 29 13:03:18 datav kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen Dec 29 13:03:19 datav kernel: ata2.00: cmd 35/00:10:e5:fa:cf/00:00:10:00:00/e0 tag 0 cdb 0x0 data 8192 out Dec 29 13:03:19 datav kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Dec 29 13:03:19 datav kernel: ata2: port is slow to respond, please be patient (Status 0xd0) Dec 29 13:03:19 datav kernel: ata2: device not ready (errno=-16), forcing hardreset Dec 29 13:03:19 datav kernel: ata2: soft resetting port Dec 29 13:03:19 datav kernel: ata2.00: configured for UDMA/33 Dec 29 13:03:19 datav kernel: ata2: EH complete Dec 29 13:03:19 datav kernel: sd 1:0:0:0: [sdb] 488397168 512-byte hardware sectors (250059 MB) Dec 29 13:03:19 datav kernel: sd 1:0:0:0: [sdb] Write Protect is off Dec 29 13:03:19 datav kernel: sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUAhlášení se opakuje nepravidelně dokola, zřejmě jen když používám konzoli. První chyba (poškození souborů) se stala dřív než hlášení kernelu do logu, neprovedly se zálohy - nepřipojil se disk (poškozený mount). RAID běží zdá se ok,
mdadm -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Thu Oct 16 13:47:36 2008
Raid Level : raid1
Array Size : 242099456 (230.88 GiB 247.91 GB)
Used Dev Size : 242099456 (230.88 GiB 247.91 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Dec 30 00:09:31 2009
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 0dd63790:02b414c0:794694bf:d8958e12
Events : 0.284676
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
1 8 2 1 active sync /dev/sda2
Server zatím běží, nejsou aktuální zálohy neboť není mount (cca 45GB)
Prosím o jakýkoliv nápad na řešení jak se zbavit chybných souborů
Děkuji za jakoukoliv přínosnou pomoc.
Ja si naopak myslim, ze mate HW problem, ktery s nejvetsi pravdepodobnosti zpusobuje problemy s filesystemem. Rozhodne bych se podival na podrobnosti ve SMARTu disku.
Tak si mount a dalsi utility nakopirujte z jineho stroje treba do /root/ a spoustejte je tam.
Soubory nejspis nemuzete smazat proto, ze se korenovy filesystem automaticky premountoval jako read-only po zaznamenani chyb.
Děkuji,
máte pravdu, opravdu je vadný druhý disk v raidu sdb2
# smartctl --all /dev/sdb
smartctl version 5.37 [i386-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Device Model: GB0250C8045
Serial Number: 9SF09ZAA
Firmware Version: HPG1
User Capacity: 250 059 350 016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 4a
Local Time is: Wed Dec 30 19:50:18 2009 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 634) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 57) minutes.
Conveyance self-test routine
recommended polling time: ( 3) minutes.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 081 063 044 Pre-fail Always - 143583136
3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 34
5 Reallocated_Sector_Ct 0x0033 085 085 036 Pre-fail Always - 309
7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 4512812337
9 Power_On_Hours 0x0032 088 088 000 Old_age Always - 10578
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 34
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Unknown_Attribute 0x0032 094 094 000 Old_age Always - 6
188 Unknown_Attribute 0x0032 100 078 000 Old_age Always - 2370858123821
189 Unknown_Attribute 0x003a 100 100 000 Old_age Always - 0
190 Temperature_Celsius 0x0022 079 065 045 Old_age Always - 588316693
194 Temperature_Celsius 0x0022 021 040 000 Old_age Always - 21 (Lifetime Min/Max 0/16)
195 Hardware_ECC_Recovered 0x001a 052 031 000 Old_age Always - 143583136
196 Reallocated_Event_Count 0x0033 085 085 036 Pre-fail Always - 309
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 16 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 16 occurred at disk power-on lifetime: 1991 hours (82 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 d5 d3 40 00 Error: UNC at LBA = 0x0040d3d5 = 4248533
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 f8 75 d3 40 e0 00 00:35:47.184 READ DMA
ec 00 00 00 00 00 a0 00 00:35:47.181 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 00:35:47.178 SET FEATURES [Set transfer mode]
ec 00 00 00 00 00 a0 00 00:35:47.083 IDENTIFY DEVICE
c8 00 f8 75 d3 40 e0 00 00:35:44.227 READ DMA
Error 15 occurred at disk power-on lifetime: 1991 hours (82 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 d5 d3 40 00 Error: UNC at LBA = 0x0040d3d5 = 4248533
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 f8 75 d3 40 e0 00 00:35:44.227 READ DMA
ec 00 00 00 00 00 a0 00 00:35:44.224 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 00:35:44.221 SET FEATURES [Set transfer mode]
ec 00 00 00 00 00 a0 00 00:35:44.126 IDENTIFY DEVICE
c8 00 f8 75 d3 40 e0 00 00:35:41.271 READ DMA
Error 14 occurred at disk power-on lifetime: 1991 hours (82 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 d5 d3 40 00 Error: UNC at LBA = 0x0040d3d5 = 4248533
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 f8 75 d3 40 e0 00 00:35:41.271 READ DMA
ec 00 00 00 00 00 a0 00 00:35:41.268 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 00:35:41.265 SET FEATURES [Set transfer mode]
ec 00 00 00 00 00 a0 00 00:35:41.186 IDENTIFY DEVICE
c8 00 f8 75 d3 40 e0 00 00:35:38.333 READ DMA
Error 13 occurred at disk power-on lifetime: 1991 hours (82 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 d5 d3 40 00 Error: UNC at LBA = 0x0040d3d5 = 4248533
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 f8 75 d3 40 e0 00 00:35:38.333 READ DMA
ec 00 00 00 00 00 a0 00 00:35:38.330 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 00:35:38.327 SET FEATURES [Set transfer mode]
ec 00 00 00 00 00 a0 00 00:35:38.213 IDENTIFY DEVICE
c8 00 f8 75 d3 40 e0 00 00:35:35.320 READ DMA
Error 12 occurred at disk power-on lifetime: 1991 hours (82 days + 23 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 d5 d3 40 00 Error: UNC at LBA = 0x0040d3d5 = 4248533
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 f8 75 d3 40 e0 00 00:35:35.320 READ DMA
ec 00 00 00 00 00 a0 00 00:35:35.317 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 00:35:35.314 SET FEATURES [Set transfer mode]
ec 00 00 00 00 00 a0 00 00:35:35.181 IDENTIFY DEVICE
c8 00 f8 75 d3 40 e0 00 00:35:32.352 READ DMA
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
sda2 se zdá být OK
# smartctl --all /dev/sda
smartctl version 5.37 [i386-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Device Model: Hitachi HDP725025GLA380
Serial Number: GEK234RBTDPLDA
Firmware Version: GM2OA52A
User Capacity: 250 059 350 016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: Not recognized. Minor revision code: 0x29
Local Time is: Wed Dec 30 19:50:14 2009 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (3981) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 66) minutes.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 132 132 054 Pre-fail Offline - 140
3 Spin_Up_Time 0x0007 104 104 024 Pre-fail Always - 198 (Average 196)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 18
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 136 136 020 Pre-fail Offline - 27
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 10564
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 18
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 56
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 56
194 Temperature_Celsius 0x0002 253 253 000 Old_age Always - 18 (Lifetime Min/Max 14/38)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Takže to bychom měli proč, nyní jsem tedy "rozbil" raid a zkontroloval jestli opravdu :)
# mdadm /dev/md0 -f /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md0
# mdadm -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Thu Oct 16 13:47:36 2008
Raid Level : raid1
Array Size : 242099456 (230.88 GiB 247.91 GB)
Used Dev Size : 242099456 (230.88 GiB 247.91 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Dec 30 19:52:06 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
UUID : 0dd63790:02b414c0:794694bf:d8958e12
Events : 0.284686
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 2 1 active sync /dev/sda2
2 8 18 - faulty spare /dev/sdb2
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sdb2[2](F) sda2[1]
242099456 blocks [2/1] [_U]
unused devices: none
odstranil disk z konfigurace raid
# mdadm /dev/md0 -r /dev/sdb2
mdadm: hot removed /dev/sdb2
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda2[1]
242099456 blocks [2/1] [_U]
unused devices: none
vytvořil zálohy pomocí nakopírování mount do root jak říkáte, děkuji,
disky, tedy teď již jen sda2, je RW, do adresáře /bin můžu zapisovat/mazat, ale stále mě trápí ty nesmyslné práva, délky u výše zmiňovaných souborů a stále nevím jak se jich zbavit. Smazat stále nejdou, znovu zkouším mazat podle i-node. Potřeboval bych se jich zbavit před opravou raid, abych nekopíroval vadné soubory.
Tiskni
Sdílej:
ISSN 1214-1267, (c) 1999-2007 Stickfish s.r.o.