Dotaz: Ext3 - chyby ve FS i po fsck -af...

[toxin]

Ahoj. Jedu na Gentoo 2.6.24-gentoo-r4 64bit 2SMP. MB od Intelu s P35, 2GB DDR2, C2D 2.33GHz, 2x400GB Seagate ES v SW md RAIDu 1. Na stroji jede mailový systém Exim, FTP ProFTPD, webserver Apache a Lighttpd. Vše pro cca 30 uživatelů. Na discích je vytvořen klasický md raid 1. Výpis oddílů:

df -lh
Filesystem            Size  Used Avail Use% Mounted on
/dev/md/3              94G  8.1G   86G   9% /
udev                   10M  196K  9.9M   2% /dev
/dev/md/4             273G  6.1G  254G   3% /home
shm                   1.9G     0  1.9G   0% /dev/shm

A teď k problému. Vše začalo tak, že mi uživatel hlásil, že se nemůže připojit na IMAP účet. Začal jsem tedy pátrat a zjistil jsem, že na FS md3,4 jsou chyby. Dal jsem je tedy na odpojených svazcích opravit a vše zase jelo OK. Problém je v tom, že ty chyby se tvoří stále. Čím déle to běží bez FSCK tím je to horší :(. Přitom HW je fyzicky v pořádku - testováno na jiné desce, cpu, ram, hdd. Pořád vykazuje chyby :(. Výpis s fsck...

fsck -n /dev/md3
fsck 1.40.6 (09-Feb-2008)
fsck.jfs version 1.1.8, 03-May-2005
processing started: 4/22/2008 10.38.51
Filesystem is currently mounted.
WARNING: Checking a mounted filesystem does not produce dependable results.
The current device is:  /dev/md3
Block size in bytes:  4096
Filesystem size in blocks:  24416768
**Phase 1 - Check Blocks, Files/Directories, and  Directory Entries
**Phase 2 - Count links
Incorrect link counts detected in the aggregate.
**Phase 3 - Duplicate Block Rescan and Directory Connectedness
Directory entries for unallocated files have been detected.
**Phase 4 - Report Problems
File system object FF86059 is linked as: /var/spool/exim/scan/1JoE1j-0001gD-OV
The path(s) refer to an unallocated file.
File system object FF106534 is linked as: /var/spool/exim/scan/1JoE1k-0001gF-RA
The path(s) refer to an unallocated file.
**Phase 5 - Check Connectivity
**Phase 6 - Perform Approved Corrections
**Phase 7 - Verify File/Directory Allocation Maps
Errors detected in the Fileset File/Directory Allocation Map control information. (F)
Errors detected in the Fileset File/Directory Allocation Map. (F)
**Phase 8 - Verify Disk Allocation Maps
Incorrect data detected in disk allocation structures.
Incorrect data detected in disk allocation control structures.
 97667072 kilobytes total disk space.
   198024 kilobytes in 42216 directories.
  8203764 kilobytes in 435944 user files.
        0 kilobytes in extended attributes
   480572 kilobytes reserved for system use.
 89180760 kilobytes are available for use.
File system checked READ ONLY.
ERRORS HAVE BEEN DETECTED.  Run fsck with the -f parameter to repair.
Filesystem is dirty.
Filesystem is dirty but is marked clean.  In its present state,
the results of accessing /dev/md3 (except by this utility) are undefined.

fsck -n /dev/md4
fsck 1.40.6 (09-Feb-2008)
e2fsck 1.40.6 (09-Feb-2008)
Warning!  /dev/md4 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/md4 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (69960966, counted=69960944).
Fix? no
Free inodes count wrong (18087044, counted=18087043).
Fix? no
/dev/md4: ********** WARNING: Filesystem still has errors **********
/dev/md4: 91004/18178048 files (0.3% non-contiguous), 2733130/72694096 blocks

Po opravě jsou FS jako clean, ale stačí počkat hodinku a jsou tam zase chyby :(. Nevíte někdo kde by mohl být problém? Je to docela kritická věc, protože se jedná o produkční server :(... Díky za jakékoli podněty...

Odpovědi

[Michal Čihař]

Co třeba ten filesystém nejdřív odpojit a pak až kontrolovat? Proč ten fsck asi píše

WARNING: Checking a mounted filesystem does not produce dependable results.

[toxin]

OK. Udělám fsck -avf na nenamountované partition a pošlu výsledky. Díky.

[toxin]

Tak tady je report z fsck pri odmountovanem FS.

The current device is:  /dev/md3
Open(...READ/WRITE EXCLUSIVE...) returned rc = 0
Primary superblock is valid.
The type of file system for the device is JFS.
Block size in bytes:  4096
Filesystem size in blocks:  24416768
**Phase 0 - Replay Journal Log
LOGREDO:  Allocating for ReDoPage:  (d) 4096 bytes
LOGREDO:  Allocating for NoDoFile:  (d) 4096 bytes
LOGREDO:  Allocating for BMap:  (d) 96264 bytes
LOGREDO:  Allocating for IMap:  (d) 21280 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Log record for Sync Point at:    0x0196f1d0
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for DoBLk:  (d) 4096 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Allocating for IMap:  (d) 2048 bytes
LOGREDO:  Beginning to update the Inode Allocation Map.
LOGREDO:  Done updating the Inode Allocation Map.
LOGREDO:  Beginning to update the Block Map.
LOGREDO:  Done updating the Block Map.
LOGREDO:  End of log found at logend = 0x01972fd4
LOGREDO:  Synch point record number:  0x0196f1d0
LOGREDO:  Synch point record address:  0x01959b10
LOGREDO:  Number of log records:    (d) 730
LOGREDO:  Number of Do blocks:    (d) 114
LOGREDO:  Number of NoDo blocks:    (d) 45
logredo returned rc = 0
**Phase 1 - Check Blocks, Files/Directories, and  Directory Entries
File system object FF188471 has corrupt data (9).
**Phase 2 - Count links
Inode F106509 has incorrect link count.
Incorrect link counts have been detected. Will correct.
**Phase 3 - Duplicate Block Rescan and Directory Connectedness
**Phase 4 - Report Problems
File system object FF188471 is linked as: /var/lib/postgresql/data/pg_subtrans/00B8
cannot repair the data format error(s) in this file.
cannot repair FF188471.  Will release.
**Phase 5 - Check Connectivity
**Phase 6 - Perform Approved Corrections
Superblock marked dirty because repairs are about to be written.
Link count for inode F106509 has been adjusted/corrected.
Directory inode F188451 entry reference to inode F188471 removed.
Storage allocated to inode F188471 has been cleared.
**Phase 7 - Rebuild File/Directory Allocation Maps
**Phase 8 - Rebuild Disk Allocation Maps
Filesystem Summary:
Blocks in use for inodes:  68500
Inode count:  548000
File count:  435816
Directory count:  42213
Block count:  24416768
Free block count:  22331913
 97667072 kilobytes total disk space.
   197998 kilobytes in 42213 directories.
  8057632 kilobytes in 435816 user files.
        0 kilobytes in extended attributes
        0 kilobytes in access control lists
   479786 kilobytes reserved for system use.
 89327652 kilobytes are available for use.
Filesystem is clean.
All observed inconsistencies have been repaired.
Filesystem has been marked clean.
**** Filesystem was modified. ****
processing terminated:  4/22/2008 12:11:18  with return code: 0  exit code: 1.

fsck.ext3 -pfv /dev/md4

   91056 inodes used (0.50%)
     274 non-contiguous inodes (0.3%)
         # of inodes with ind/dind/tind blocks: 3681/211/0
 2743777 blocks used (3.77%)
       0 bad blocks
       1 large file

   86257 regular files
    4786 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       4 symbolic links (4 fast symbolic links)
       0 sockets
--------
   91047 files