Starting March 26th, 2 different RHEL 5 servers have been crashing.
Server 1 has crashed 1 time with Kernel 2.6.18-53.1.13
Server 2 has crashed 1 time with Kernel 2.6.18-53.1.13PAE and 1 time with Kernel 2.6.18-53.1.14PAE
Before the crashes, there are a few EXT3 messages which I include at the end, these EXT3 errors DO NOT show up on any logs going weeks back. These are brand new servers ordered in January and February, and they are distinct in data and hardware, so thinking this is a hardware or data corruption issue would be barking to the wrong tree in my humble opinion.
I have seen these type of EXT3 error messages on RHEL4 Kernel Crashes before. Typically going back to an older kernel solves the problem for a while, until Red Hat issues a new kernel that fixes the issue.
I'll appreciate in someone could help to debug the issue and share intructions on going back to an older RHEL 5 Kernel. I would do it myself, but the servers are so new they only have these newer kernels. I know about the local root exploit issue, but I don't issue SSH accounts to my customers, have compilers on only for root and my PHP config is pretty fortified, so that might not be as important.
Many thanks!
Details of errors:
Server 1 Crash 1
Mar 26 11:18:04 linux kernel: EXT3-fs warning (device sda8): dx_probe: Unrecognised inode hash code 56
Mar 26 11:35:01 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9569664), 0
Mar 26 11:54:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577963), 0
Mar 26 11:55:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577995), 0
Mar 26 12:01:08 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578025), 0
Mar 26 12:01:08 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578042), 0
Mar 26 12:01:08 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568702), 0
Mar 26 12:02:15 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578031), 0
Mar 26 12:13:53 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577856), 0
Mar 26 12:15:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568710), 0
Mar 26 12:15:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568984), 0
Mar 26 12:18:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568934), 0
Mar 26 12:18:40 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578081), 0
Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578051), 0
Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578073), 0
Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578089), 0
Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568695), 0
Mar 26 12:25:37 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577865), 0
Mar 26 12:25:52 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578040), 0
Mar 26 12:25:52 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568705), 0
Mar 26 12:30:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577998), 0
Mar 26 12:40:02 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577996), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568652), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578053), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578054), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568690), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578069), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568689), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577997), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578003), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9569016), 0
Mar 26 12:54:49 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577876), 0
Mar 26 12:54:53 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568703), 0
Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577905), 0
Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568530), 0
Mar 26 18:04:15 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Mar 26 18:04:15 linux kernel: EXT3-fs: write access will be enabled during recovery.
Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: 3 orphan inodes deleted
Mar 26 18:04:16 linux kernel: EXT3-fs: recovery complete.
Mar 26 18:04:16 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda5, internal journal
Mar 26 18:04:17 linux kernel: EXT3 FS on sdb1, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda1, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda8, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda6, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda3, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda2, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Server 2 Crash 1
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233736), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233765), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233771), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233748), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233749), 0
Mar 27 10:32:49 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Mar 27 10:32:49 linux kernel: EXT3-fs: write access will be enabled during recovery.
Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: 1 orphan inode deleted
Mar 27 10:32:49 linux kernel: EXT3-fs: recovery complete.
Mar 27 10:32:49 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda5, internal journal
Mar 27 10:32:50 linux kernel: EXT3 FS on sda8, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sdb1, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda6, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda3, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda2, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda1, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Server 2 Crash 2
Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): htree_dirblock_to_tree: bad entry in directory #21233720: rec_len % 4 != 0 - offset=0, inode=1970562386, rec_len=28274, name_len=45
Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): ext3_journal_start_sb: Detected aborted journal
Mar 28 01:10:33 linux kernel: EXT3-fs error (device sda8) in ext3_reserve_inode_write: Journal has aborted
Mar 28 04:00:18 linux kernel: BUG: soft lockup detected on CPU#1!
Mar 28 04:00:18 linux kernel: [<c044b2a4>] softlockup_tick+0x96/0xa4
Mar 28 04:00:18 linux kernel: [<c042e214>] update_process_times+0x39/0x5c
Mar 28 04:00:18 linux kernel: [<c04196ff>] smp_apic_timer_interrupt+0x5b/0x6c
Mar 28 04:00:18 linux kernel: [<c04059bf>] apic_timer_interrupt+0x1f/0x24
Mar 28 04:00:18 linux kernel: [<c0497061>] dqput+0x6b/0x15d
Mar 28 04:00:18 linux kernel: [<c0498556>] vfs_quota_sync+0x9b/0x131
Mar 28 04:00:18 linux kernel: [<c049a2dd>] quota_sync_sb+0x11/0xcc
Mar 28 04:00:18 linux kernel: [<c0438d8d>] down_read+0x8/0x11
Mar 28 04:00:18 linux kernel: [<c049a94c>] sys_quotactl+0x4c7/0x5f3
Mar 28 04:00:18 linux kernel: [<c0464cc5>] free_pages_and_swap_cache+0x6b/0x7f
Mar 28 04:00:18 linux kernel: [<c0605f6e>] do_page_fault+0x2a5/0x5d3
Mar 28 04:00:18 linux kernel: [<c0605fe7>] do_page_fault+0x31e/0x5d3
Mar 28 04:00:18 linux kernel: [<c0448c60>] audit_syscall_entry+0x11c/0x14e
Mar 28 04:00:18 linux kernel: [<c0407ee7>] do_syscall_trace+0xab/0xb1
Mar 28 04:00:18 linux kernel: [<c0404eff>] syscall_call+0x7/0xb