Dotaz: Kernel panic - not syncing: Watchdog detected hard LOCKUP

[vasek]

Příloha:

• IMAG0100.jpg (193544 bytů)

Ahoj. Provozuji virtuální stroje pod CentOS. Přešel jsem z XEN v Centos 5 na KVM na Centos 6. Běží mi tam 4 virtuální linuxové stroje. Dnes jsem musel hostitelský stroj natvrdo restartovat, protože zatuhnul. Na monitoru se objevilo "Kernel panic - not syncing: Watchdog detected hard LOCKUP" pro všechna CPU, která ten počítač má (viz. příloha). Žádné další zprávy mi kernel neposkytnul. Problém se vyskytnul po asi týdnu provozu. cat /proc/cpuinfo

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 4999.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5000.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 4999.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 4999.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
cpu MHz		: 2499.594
cache size	: 6144 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dts tpr_shadow vnmi flexpriority
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

cat /proc/meminfo

MemTotal:       16331516 kB
MemFree:        14312348 kB
Buffers:            9708 kB
Cached:            74180 kB
SwapCached:            0 kB
Active:          1531500 kB
Inactive:          59356 kB
Active(anon):    1507252 kB
Inactive(anon):       20 kB
Active(file):      24248 kB
Inactive(file):    59336 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1023992 kB
SwapFree:        1023992 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       1507072 kB
Mapped:            17072 kB
Shmem:               312 kB
Slab:             123300 kB
SReclaimable:      14852 kB
SUnreclaim:       108448 kB
KernelStack:        2160 kB
PageTables:         8888 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     9189748 kB
Committed_AS:   13752484 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       82608 kB
VmallocChunk:   34359640200 kB
HardwareCorrupted:     0 kB
AnonHugePages:    403456 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7488 kB
DirectMap2M:    16769024 kB

K zatuhnutí došlo někdy po 23.55 včera (podle logu jednoho guesta). 4 provozované guesty dělají: 1 - dhcp server, 2 - NX server (x2go), 3 - web server, 4 - monitoring (observium + zabbix). Neměl jste někdo někdy podobný problém? Co mám zkusit udělat? Těžko můžu udělat bug report z toho trochu textu, co mi kernel vypsal předtím než zkapal.

Odpovědi

[jekub]

možná náhoda, ale včera se posunoval čas.

[vasek]

Teoreticky je to až moc velká náhoda, ale na druhou stranu ten čas se posouvá v 2.00 SELČ a poslední log mám z 23.50 SELČ.

[vasek]

Teda 23.55. Abych to ještě upřesnil, loguje mi CRON po 5 minutách, takže problém musel nastat mezi 23.55 - 0.00.

[DaBler]

Ve 23:59:59, přečti si ten odkaz.

[Jendа]

UTC, přečti si ten komentář.

[Roman DAVID]

Stejny problem !!!

V sobotu rano v 4:38 je posledni zaznam v logu.

Na obrazovce stejna hlaska. Stroj jsem restartnul a za chvili se odporoucel znovu.

Po dalším restartu uz jede.

Postizeny system: Centos 6.2 64bit a na nem bezici KVM.

Sazim na to, ze to byl problem s prestupnou sekundou. Puvodne jsem si myslel, ze je vadny HW.

Jine verze centosu to prezily bez uhony (6.2 32bit, 5.8 32bit)

[ET]

fyi: https://access.redhat.com/knowledge/articles/15145

[michich]

Vypadá to, že RHEL6 skutečně má chybu související s přestupnou sekundou: https://access.redhat.com/knowledge/solutions/154713

A stejné problémy pozorovány s Debian Squeeze: http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second