Administrace komentářů

Už delší dobu se potýkám na zálohovacím stroji s nepředvídatelnými pády kernelu. Napřed jsem křivě "obviňoval" BTRFS, ale na EXT4 se problém časem objevil také. Systém prostě čas od času (jednou týdně, někdy častěji) vytuhne na kernel panic anebo alespoň kernel oops. Jedná se o aktualizovaný Debian Squeeze, běží zde prakticky jen rsync a rdiff-backup. Podotýkám, že server má 8GB paměti a nemá zapnutý swap. Většinu paměti prakticky trvale využívá pouze jako cache. Data se ukládají na 4 SATA disky v RAID10.

Měl by někdo nápad, kde je zakopaný pes?

Apr 29 10:15:01 backer /USR/SBIN/CRON[28245]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 10:15:49 backer kernel: [1111765.028106] ------------[ cut here ]------------
Apr 29 10:15:49 backer kernel: [1111765.028137] kernel BUG at /build/buildd-linux-2.6_2.6.32-48squeeze1-amd64-qu4MIV/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/buffer.c:3290!
Apr 29 10:15:49 backer kernel: [1111765.028192] invalid opcode: 0000 [#2] SMP 
Apr 29 10:15:49 backer kernel: [1111765.028220] last sysfs file: /sys/kernel/mm/ksm/run
Apr 29 10:15:49 backer kernel: [1111765.028245] CPU 3 
Apr 29 10:15:49 backer kernel: [1111765.028267] Modules linked in: fuse 8021q garp bridge stp bonding loop snd_pcm snd_timer radeon ttm snd drm_kms_helper soundcore drm i2c_algo_bit i2c_i801 snd_page_alloc parport_pc psmouse parport i2c_core serio_raw pcspkr joydev i3200_edac evdev video container output shpchp button pci_hotplug edac_core processor ext4 mbcache jbd2 crc16 dm_mod raid10 raid1 md_mod btrfs zlib_deflate crc32c libcrc32c usbhid hid sg sr_mod sd_mod crc_t10dif cdrom ata_generic uhci_hcd ahci it8213 libata ide_core floppy scsi_mod ehci_hcd e1000e thermal usbcore thermal_sys nls_base [last unloaded: scsi_wait_scan]
Apr 29 10:15:49 backer kernel: [1111765.028593] Pid: 47, comm: kswapd0 Tainted: G      D    2.6.32-5-amd64 #1 X7SBi
Apr 29 10:15:49 backer kernel: [1111765.028637] RIP: 0010:[ffffffff8110d4e5]  [ffffffff8110d4e5] free_buffer_head+0x11/0x3d
Apr 29 10:15:49 backer kernel: [1111765.028688] RSP: 0018:ffff88022e1cbbc0  EFLAGS: 00010206
Apr 29 10:15:49 backer kernel: [1111765.028715] RAX: ffff88015f1440b8 RBX: ffff88015f144070 RCX: ffff88015f144070
Apr 29 10:15:49 backer kernel: [1111765.028758] RDX: ffff88015f144070 RSI: ffff88022e1cbbd8 RDI: ffff88015f144070
Apr 29 10:15:49 backer kernel: [1111765.028802] RBP: 0000000000000001 R08: 2000000000000000 R09: ffff880149206478
Apr 29 10:15:49 backer kernel: [1111765.028845] R10: ffff880149206478 R11: ffffffffa025be5f R12: ffff880149206478
Apr 29 10:15:49 backer kernel: [1111765.028889] R13: 0000000000001830 R14: 000000000000182f R15: ffff88022e1cbc40
Apr 29 10:15:49 backer kernel: [1111765.028933] FS:  0000000000000000(0000) GS:ffff880008d80000(0000) knlGS:0000000000000000
Apr 29 10:15:49 backer kernel: [1111765.028978] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Apr 29 10:15:49 backer kernel: [1111765.029005] CR2: 00007fdb09846c30 CR3: 0000000122f33000 CR4: 00000000000006e0
Apr 29 10:15:49 backer kernel: [1111765.029049] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 29 10:15:49 backer kernel: [1111765.029503] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 29 10:15:49 backer kernel: [1111765.029503] Process kswapd0 (pid: 47, threadinfo ffff88022e1ca000, task ffff88022f250e20)
Apr 29 10:15:49 backer kernel: [1111765.029503] Stack:
Apr 29 10:15:49 backer kernel: [1111765.029503]  0000000000000001 ffffffff8110d83d ffff880149206478 ffff88015f144070
Apr 29 10:15:49 backer kernel: [1111765.029503] 0 ffffea00045f8548 ffff880149206478 ffff88022e1cbc68 ffffffff810bd321
Apr 29 10:15:49 backer kernel: [1111765.029503] 0 000000000000182f ffffea00045f8548 0000000000000003 ffffffff810bd3b5
Apr 29 10:15:49 backer kernel: [1111765.029503] Call Trace:
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff8110d83d] ? try_to_free_buffers+0x81/0x96
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810bd321] ? invalidate_inode_page+0x5c/0x87
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810bd3b5] ? invalidate_mapping_pages+0x69/0xdb
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff81100f47] ? shrink_icache_memory+0xfc/0x228
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810bfa09] ? shrink_slab+0xe0/0x153
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810c02ac] ? kswapd+0x4d9/0x686
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810bd923] ? isolate_pages_global+0x0/0x20f
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810652b2] ? autoremove_wake_function+0x0/0x2e
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff8103aa66] ? __wake_up_common+0x44/0x72
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff810bfdd3] ? kswapd+0x0/0x686
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff81064fe5] ? kthread+0x79/0x81
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff81011baa] ? child_rip+0xa/0x20
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff81064f6c] ? kthread+0x0/0x81
Apr 29 10:15:49 backer kernel: [1111765.029503]  [ffffffff81011ba0] ? child_rip+0x0/0x20
Apr 29 10:15:49 backer kernel: [1111765.029503] Code: 89 c2 7c d5 31 c0 3b 1d f2 40 5a 00 0f 9f c0 89 05 d1 40 5a 00 5b 5d 41 5c c3 48 83 ec 08 48 8d 47 48 48 39 47 48 48 89 fa 74 04 0f 0b eb fe 48 8b 3d c0 40 5a 00 48 89 d6 e8 d3 a1 fd ff 65 48 
Apr 29 10:15:49 backer kernel: [1111765.029503] RIP  [ffffffff8110d4e5] free_buffer_head+0x11/0x3d
Apr 29 10:15:49 backer kernel: [1111765.029503]  RSP ffff88022e1cbbc0
Apr 29 10:15:49 backer kernel: [1111765.036333] ---[ end trace 1277b17bea4c4d33 ]---
Apr 29 10:17:01 backer /USR/SBIN/CRON[28262]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr 29 10:25:01 backer /USR/SBIN/CRON[28314]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 10:35:01 backer /USR/SBIN/CRON[28384]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 10:45:01 backer /USR/SBIN/CRON[28449]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 10:55:01 backer /USR/SBIN/CRON[28520]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 11:05:01 backer /USR/SBIN/CRON[28585]: (root) CMD (command -v debian-sa1  /dev/null && debian-sa1 1 1)
Apr 29 11:14:33 backer kernel: [1115289.501382] ------------[ cut here ]------------
Apr 29 11:14:33 backer kernel: [1115289.501429] kernel BUG at /build/buildd-linux-2.6_2.6.32-48squeeze1-amd64-qu4MIV/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/buffer.c:3290!
Apr 29 11:14:33 backer kernel: [1115289.501502] invalid opcode: 0000 [#3] SMP 
Apr 29 11:14:33 backer kernel: [1115289.501531] last sysfs file: /sys/kernel/mm/ksm/run
Apr 29 11:14:33 backer kernel: [1115289.501556] CPU 3 
Apr 29 11:14:33 backer kernel: [1115289.501579] Modules linked in: fuse 8021q garp bridge stp bonding loop snd_pcm snd_timer radeon ttm snd drm_kms_helper soundcore drm i2c_algo_bit i2c_i801 snd_page_alloc parport_pc psmouse parport i2c_core serio_raw pcspkr joydev i3200_edac evdev video container output shpchp button pci_hotplug edac_core processor ext4 mbcache jbd2 crc16 dm_mod raid10 raid1 md_mod btrfs zlib_deflate crc32c libcrc32c usbhid hid sg sr_mod sd_mod crc_t10dif cdrom ata_generic uhci_hcd ahci it8213 libata ide_core floppy scsi_mod ehci_hcd e1000e thermal usbcore thermal_sys nls_base [last unloaded: scsi_wait_scan]
Apr 29 11:14:33 backer kernel: [1115289.502022] Pid: 28082, comm: ssh Tainted: G      D    2.6.32-5-amd64 #1 X7SBi
Apr 29 11:14:33 backer kernel: [1115289.502099] RIP: 0010:[ffffffff8110d4e5]  [ffffffff8110d4e5] free_buffer_head+0x11/0x3d
Apr 29 11:14:33 backer kernel: [1115289.502189] RSP: 0018:ffff8801940075f8  EFLAGS: 00010206
Apr 29 11:14:33 backer kernel: [1115289.502237] RAX: ffff88015f144b38 RBX: ffff88015f144af0 RCX: ffff88015f144af0
Apr 29 11:14:33 backer kernel: [1115289.502318] RDX: ffff88015f144af0 RSI: ffff880194007610 RDI: ffff88015f144af0
Apr 29 11:14:33 backer kernel: [1115289.502399] RBP: 0000000000000001 R08: 0000000000000009 R09: ffff880069766850
Apr 29 11:14:33 backer kernel: [1115289.502470] R10: ffffea000638f3e0 R11: ffffffffa025be5f R12: ffff880069766850
Apr 29 11:14:33 backer kernel: [1115289.502521] R13: ffff880069766850 R14: dead000000100101 R15: ffff880194007a08
Apr 29 11:14:33 backer kernel: [1115289.502575] FS:  00007f1eaaabe720(0000) GS:ffff880008d80000(0000) knlGS:0000000000000000
Apr 29 11:14:33 backer kernel: [1115289.502658] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 29 11:14:33 backer kernel: [1115289.502714] CR2: 00007f2613f86c30 CR3: 0000000181de2000 CR4: 00000000000006e0
Apr 29 11:14:33 backer kernel: [1115289.502795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 29 11:14:33 backer kernel: [1115289.502876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 29 11:14:33 backer kernel: [1115289.502957] Process ssh (pid: 28082, threadinfo ffff880194006000, task ffff88022b83f100)
Apr 29 11:14:33 backer kernel: [1115289.503040] Stack:
Apr 29 11:14:33 backer kernel: [1115289.503076]  0000000000000001 ffffffff8110d83d ffffea000638f418 ffff88015f144af0
Apr 29 11:14:33 backer kernel: [1115289.503138] 0 ffffea000638f3f0 ffffea000638f418 0000000000000001 ffffffff810bea11
Apr 29 11:14:33 backer kernel: [1115289.503229] 0 0000000000000042 ffff8801940076f8 ffff88022b83f100 ffff88022b83f100
Apr 29 11:14:33 backer kernel: [1115289.503349] Call Trace:
Apr 29 11:14:33 backer kernel: [1115289.503389]  [ffffffff8110d83d] ? try_to_free_buffers+0x81/0x96
Apr 29 11:14:33 backer kernel: [1115289.503442]  [ffffffff810bea11] ? shrink_page_list+0x48d/0x623
Apr 29 11:14:33 backer kernel: [1115289.503494]  [ffffffff810bdac3] ? isolate_pages_global+0x1a0/0x20f
Apr 29 11:14:33 backer kernel: [1115289.503548]  [ffffffff810bf2dc] ? shrink_list+0x45c/0x767
Apr 29 11:14:33 backer kernel: [1115289.503600]  [ffffffff810482ef] ? finish_task_switch+0x3a/0xaf
Apr 29 11:14:33 backer kernel: [1115289.503652]  [ffffffff812fc818] ? thread_return+0x79/0xe0
Apr 29 11:14:33 backer kernel: [1115289.503702]  [ffffffff810bf867] ? shrink_zone+0x280/0x342
Apr 29 11:14:33 backer kernel: [1115289.503754]  [ffffffff810168ec] ? read_tsc+0xa/0x20
Apr 29 11:14:33 backer kernel: [1115289.503790]  [ffffffff810c092e] ? try_to_free_pages+0x232/0x38e
Apr 29 11:14:33 backer kernel: [1115289.503824]  [ffffffff810bd923] ? isolate_pages_global+0x0/0x20f
Apr 29 11:14:33 backer kernel: [1115289.503858]  [ffffffff810ba9b8] ? __alloc_pages_nodemask+0x3d4/0x5fc
Apr 29 11:14:33 backer kernel: [1115289.503913]  [ffffffff812452db] ? sock_alloc_send_pskb+0xdd/0x2f6
Apr 29 11:14:33 backer kernel: [1115289.503966]  [ffffffff810e706a] ? kmalloc_large_node+0x5d/0x9b
Apr 29 11:14:33 backer kernel: [1115289.504017]  [ffffffff81249bec] ? __alloc_skb+0x69/0x15a
Apr 29 11:14:33 backer kernel: [1115289.504068]  [ffffffff810fd478] ? pollwake+0x0/0x58
Apr 29 11:14:33 backer kernel: [1115289.504117]  [ffffffff812452db] ? sock_alloc_send_pskb+0xdd/0x2f6
Apr 29 11:14:33 backer kernel: [1115289.504169]  [ffffffff81244ab2] ? release_sock+0x13/0xa0
Apr 29 11:14:33 backer kernel: [1115289.504219]  [ffffffff8103aa66] ? __wake_up_common+0x44/0x72
Apr 29 11:14:33 backer kernel: [1115289.504272]  [ffffffff812b4692] ? unix_stream_sendmsg+0x13b/0x2c8
Apr 29 11:14:33 backer kernel: [1115289.504325]  [ffffffff81241d8c] ? sock_aio_write+0xb1/0xbc
Apr 29 11:14:33 backer kernel: [1115289.504375]  [ffffffff810ef21e] ? do_sync_write+0xce/0x113
Apr 29 11:14:33 backer kernel: [1115289.504426]  [ffffffff8100f79c] ? __switch_to+0x285/0x297
Apr 29 11:14:33 backer kernel: [1115289.504474]  [ffffffff810652b2] ? autoremove_wake_function+0x0/0x2e
Apr 29 11:14:33 backer kernel: [1115289.504526]  [ffffffff810efb83] ? vfs_write+0xbc/0x102
Apr 29 11:14:33 backer kernel: [1115289.504572]  [ffffffff810efc85] ? sys_write+0x45/0x6e
Apr 29 11:14:33 backer kernel: [1115289.504618]  [ffffffff81010b42] ? system_call_fastpath+0x16/0x1b
Apr 29 11:14:33 backer kernel: [1115289.504667] Code: 89 c2 7c d5 31 c0 3b 1d f2 40 5a 00 0f 9f c0 89 05 d1 40 5a 00 5b 5d 41 5c c3 48 83 ec 08 48 8d 47 48 48 39 47 48 48 89 fa 74 04 0f 0b eb fe 48 8b 3d c0 40 5a 00 48 89 d6 e8 d3 a1 fd ff 65 48 
Apr 29 11:14:33 backer kernel: [1115289.504938] RIP  [ffffffff8110d4e5] free_buffer_head+0x11/0x3d
Apr 29 11:14:33 backer kernel: [1115289.504986]  RSP ffff8801940075f8
Apr 29 11:14:33 backer kernel: [1115289.505329] ---[ end trace 1277b17bea4c4d34 ]---