1.24.2 and 1.24.3 crash Ubuntu 10.04

[clint09]

Background:
I have 10 IP cameras, a mixture of Acti and Vivoteks all running Mjpeg. Zoneminder was installed using the APT 1.24.2 package. The system is a Core 2 Quad 2.5ghz, 4gb ram. I've been using Ubuntu 10.04 x64. When the system is running, it runs at a nice 1.8 load with room to spare.

Just recently the system has began to crash/panic/lock-up the whole system. In the sys log I see messages like this

Feb 17 15:39:47 ubuntu kernel: [ 3668.735547] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Feb 17 15:39:47 ubuntu kernel: [ 3668.735984] IP: [<ffffffff814698b6>] skb_release_head_state+0x76/0x120
Feb 17 15:39:47 ubuntu kernel: [ 3668.736291] PGD 23437067 PUD ed126067 PMD 0
Feb 17 15:39:47 ubuntu kernel: [ 3668.736626] Oops: 0002 [#1] SMP
Feb 17 15:39:47 ubuntu kernel: [ 3668.736867] last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:05:00.0/irq
Feb 17 15:39:47 ubuntu kernel: [ 3668.742611] CPU 0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Modules linked in: xfs exportfs fbcon tileblit font bitblit softcursor vga16fb vgastate intel_agp lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 pata_via multipath sky2 ohci1394 ahci r8169 mii ieee1394 linear
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Pid: 28237, comm: zmc Not tainted 2.6.32-28-server #55-Ubuntu To Be Filled By O.E.M.
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RIP: 0010:[<ffffffff814698b6>] [<ffffffff814698b6>] skb_release_head_state+0x76/0x120
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RSP: 0018:ffff880037407b68 EFLAGS: 00010202
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RAX: ffff8800eaf0a080 RBX: ffff8800594b4000 RCX: 0000000000000728
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RDX: ffffffff8182a480 RSI: 0000000000000000 RDI: 0000000000000008
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RBP: ffff880037407b78 R08: 0000000065640327 R09: 0000000000000000
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] R10: ffff8800eaf0a080 R11: 00000000000005a8 R12: 0000000000000000
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] R13: 0000000000000000 R14: ffff8800eaf0a130 R15: ffff8800eaf0a4dc
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] FS: 00007f22f882b720(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] CR2: 0000000000000008 CR3: 00000000f369b000 CR4: 00000000000406f0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Process zmc (pid: 28237, threadinfo ffff880037406000, task ffff880103918000)
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Stack:
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] 0000000000000000 ffff8800594b4000 ffff880037407b98 ffffffff81469556
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] <0> 0000000000000000 ffff8800594b4000 ffff880037407ca8 ffffffff814b3678
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] <0> ffff880037407d28 ffffffff81055044 00000000000005a8 ffff8800eaf0a080
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Call Trace:
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81469556>] __kfree_skb+0x16/0xa0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff814b3678>] tcp_recvmsg+0xda8/0xe90
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81055044>] ? find_busiest_group+0x234/0x990
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81464869>] sock_common_recvmsg+0x39/0x50
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81464459>] sock_aio_read+0x159/0x160
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81057a90>] ? finish_task_switch+0x50/0xe0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff8114422a>] do_sync_read+0xfa/0x140
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81465301>] ? lock_sock_nested+0xb1/0xc0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81084400>] ? autoremove_wake_function+0x0/0x40
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff814d4380>] ? inet_ioctl+0x30/0xb0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81252b06>] ? security_file_permission+0x16/0x20
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81144be1>] vfs_read+0x181/0x1a0
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff81144cd1>] sys_read+0x51/0x80
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] Code: 8b 14 25 c8 cb 00 00 48 63 92 44 e0 ff ff f7 c2 00 00 ff 03 0f 85 95 00 00 00 48 89 df ff d0 48 8b bb 88 00 00 00 48 85 ff 74 0a <f0> ff 0f 0f 94 c0 84 c0 75 70 48 8b bb 90 00 00 00 48 85 ff 74
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RIP [<ffffffff814698b6>] skb_release_head_state+0x76/0x120
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] RSP <ffff880037407b68>
Feb 17 15:39:47 ubuntu kernel: [ 3668.745337] CR2: 0000000000000008
Feb 17 15:39:47 ubuntu kernel: [ 3669.160747] ---[ end trace e9d09d7008bf160c ]---

And also

Feb 15 19:11:21 ubuntu kernel: [25040.742004] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b0
Feb 15 19:11:21 ubuntu kernel: [25040.742456] IP: [<ffffffff81464488>] sock_rfree+0x28/0x40
Feb 15 19:11:21 ubuntu kernel: [25040.742738] PGD f2acd067 PUD f2acc067 PMD 0
Feb 15 19:11:21 ubuntu kernel: [25040.743082] Oops: 0000 [#1] SMP
Feb 15 19:11:21 ubuntu kernel: [25040.743329] last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:05:00.0/irq
Feb 15 19:11:21 ubuntu kernel: [25040.749166] CPU 0
Feb 15 19:11:21 ubuntu kernel: [25040.752152] Modules linked in: xfs exportfs fbcon tileblit font bitblit softcursor vga16fb vgastate intel_agp lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 ohci1394 sky2 r8169 multipath mii ieee1394 pata_via ahci linear
Feb 15 19:11:21 ubuntu kernel: [25040.765252] Pid: 1976, comm: zmc Not tainted 2.6.32-28-server #55-Ubuntu To Be Filled By O.E.M.
Feb 15 19:11:21 ubuntu kernel: [25040.772515] RIP: 0010:[<ffffffff81464488>] [<ffffffff81464488>] sock_rfree+0x28/0x40
Feb 15 19:11:21 ubuntu kernel: [25040.793181] RSP: 0018:ffff8800f2167b58 EFLAGS: 00010297
Feb 15 19:11:21 ubuntu kernel: [25040.800103] RAX: ffff8800f05a1380 RBX: ffff88002d82ab00 RCX: 0000000000000728
Feb 15 19:11:21 ubuntu kernel: [25040.804037] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88002d82ab00
Feb 15 19:11:21 ubuntu kernel: [25040.813763] RBP: ffff8800f2167b58 R08: 00000000037cbfb2 R09: 0000000000000000
Feb 15 19:11:21 ubuntu kernel: [25040.821260] R10: ffff8800f05e1380 R11: 00000000000004bf R12: 0000000000000000
Feb 15 19:11:21 ubuntu kernel: [25040.824447] R13: 0000000000000000 R14: ffff8800f05e1430 R15: ffff8800f05e17dc
Feb 15 19:11:21 ubuntu kernel: [25040.837422] FS: 00007f8d7c1bf720(0000) GS:ffff880005a00000(0000) knlGS:0000000000000000
Feb 15 19:11:21 ubuntu kernel: [25040.853759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 15 19:11:21 ubuntu kernel: [25040.865432] CR2: 00000000000000b0 CR3: 00000000f2aca000 CR4: 00000000000406f0
Feb 15 19:11:21 ubuntu kernel: [25040.873757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 15 19:11:21 ubuntu kernel: [25040.885707] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 15 19:11:21 ubuntu kernel: [25040.893757] Process zmc (pid: 1976, threadinfo ffff8800f2166000, task ffff8800f0560000)
Feb 15 19:11:21 ubuntu kernel: [25040.919031] Stack:
Feb 15 19:11:21 ubuntu kernel: [25040.928823] ffff8800f2167b78 ffffffff814698aa 0000000000000000 ffff88002d82ab00
Feb 15 19:11:21 ubuntu kernel: [25040.939241] <0> ffff8800f2167b98 ffffffff81469556 0000000000000000 ffff88002d82ab00
Feb 15 19:11:21 ubuntu kernel: [25040.960831] <0> ffff8800f2167ca8 ffffffff814b3678 ffff8800f2167d28 ffffffff81055044
Feb 15 19:11:21 ubuntu kernel: [25040.992527] Call Trace:
Feb 15 19:11:21 ubuntu kernel: [25041.004358] [<ffffffff814698aa>] skb_release_head_state+0x6a/0x120
Feb 15 19:11:21 ubuntu kernel: [25041.013785] [<ffffffff81469556>] __kfree_skb+0x16/0xa0
Feb 15 19:11:21 ubuntu kernel: [25041.023759] [<ffffffff814b3678>] tcp_recvmsg+0xda8/0xe90
Feb 15 19:11:21 ubuntu kernel: [25041.033759] [<ffffffff81055044>] ? find_busiest_group+0x234/0x990
Feb 15 19:11:21 ubuntu kernel: [25041.043757] [<ffffffff81464869>] sock_common_recvmsg+0x39/0x50
Feb 15 19:11:21 ubuntu kernel: [25041.053761] [<ffffffff81464459>] sock_aio_read+0x159/0x160
Feb 15 19:11:21 ubuntu kernel: [25041.065757] [<ffffffff81057a90>] ? finish_task_switch+0x50/0xe0
Feb 15 19:11:21 ubuntu kernel: [25041.074795] [<ffffffff8114422a>] do_sync_read+0xfa/0x140
Feb 15 19:11:21 ubuntu kernel: [25041.086031] [<ffffffff81465301>] ? lock_sock_nested+0xb1/0xc0
Feb 15 19:11:21 ubuntu kernel: [25041.093760] [<ffffffff81084400>] ? autoremove_wake_function+0x0/0x40
Feb 15 19:11:21 ubuntu kernel: [25041.104519] [<ffffffff814d4380>] ? inet_ioctl+0x30/0xb0
Feb 15 19:11:21 ubuntu kernel: [25041.115240] [<ffffffff81252b06>] ? security_file_permission+0x16/0x20
Feb 15 19:11:21 ubuntu kernel: [25041.124337] [<ffffffff81144be1>] vfs_read+0x181/0x1a0
Feb 15 19:11:21 ubuntu kernel: [25041.133856] [<ffffffff81144cd1>] sys_read+0x51/0x80
Feb 15 19:11:21 ubuntu kernel: [25041.142507] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
Feb 15 19:11:21 ubuntu kernel: [25041.151602] Code: fe ff ff 55 48 89 e5 0f 1f 44 00 00 48 8b 47 10 8b 97 e0 00 00 00 f0 29 90 9c 00 00 00 48 8b 47 10 8b 8f e0 00 00 00 48 8b 50 30 <48> 83 ba b0 00 00 00 00 74 06 01 88 fc 00 00 00 c9 c3 66 0f 1f
Feb 15 19:11:21 ubuntu kernel: [25041.181994] RIP [<ffffffff81464488>] sock_rfree+0x28/0x40
Feb 15 19:11:21 ubuntu kernel: [25041.191268] RSP <ffff8800f2167b58>
Feb 15 19:11:21 ubuntu kernel: [25041.199311] CR2: 00000000000000b0
Feb 15 19:11:21 ubuntu kernel: [25041.216467] ---[ end trace 3ec515579cd1a3b2 ]---

When it crashes, the system will semi be available, but it reports the ZMC process as a zombie and ZM wont shutdown, nor will the system.

I have changed the following: CPU, Motherboard, Ram,network card, and reinstalled the OS several times. I've tried several kernels, by updating the system and running the system on the default kernel from the disk. Today I compiled and installed the 1.24.3 preview with the same results. This seems to be completely random. It happens with all 10 cameras enabled, and as little as 3 enabled. All cameras are set to "Record".

Anyone have any ideas? Thanks in advance.

[wkulecz]

I set up a similar quad core Ubuntu 10.04 zm 1.24.2 system last weekend. It locked up Monday morning -- first hard crash to the point even caps lock key wouldn't toggle the keyboard LED I've ever had with Ubuntu. This machine has been 100% stable as a file server since 10.04 came out -- more reliable than the power company in fact.

Looks like it crashed again today as my wife called and couldn't start it from the web browser on her machine before she left

I'm using six cameras and three V4l2 capture cards that have proven reliable over the years. Two are BT8x8 based the other is SAA713x based. No netcams yet, although adding two was on the agenda for this weekend under the hopeful assumption the lock up was caused by all my playing around setting things up. Looks like a real issue here.

Where is the system log files you are dumping? I'll take a look at mine when I get home.

If there is no quick resolution for this, I'll have to try and repair the old Windows 2000 system that has been running GeoVision these past 10 years or buy a standalone security DVR box from Fry's

[mastertheknife]

Background:
I have 10 IP cameras, a mixture of Acti and Vivoteks all running Mjpeg. Zoneminder was installed using the APT 1.24.2 package. The system is a Core 2 Quad 2.5ghz, 4gb ram. I've been using Ubuntu 10.04 x64. When the system is running, it runs at a nice 1.8 load with room to spare.

Just recently the system has began to crash/panic/lock-up the whole system. In the sys log I see messages like this



When it crashes, the system will semi be available, but it reports the ZMC process as a zombie and ZM wont shutdown, nor will the system.

I have changed the following: CPU, Motherboard, Ram,network card, and reinstalled the OS several times. I've tried several kernels, by updating the system and running the system on the default kernel from the disk. Today I compiled and installed the 1.24.3 preview with the same results. This seems to be completely random. It happens with all 10 cameras enabled, and as little as 3 enabled. All cameras are set to "Record".

Anyone have any ideas? Thanks in advance.

Hi,
The crash happens inside networking code (socket buffers to be precise) so try playing with your network configuration, drivers and modules. and try unloading modules from your kernel, its possible that a module is causing memory corruption.
If more data was available, i could know what syscall from zmc is the one causing the crash. Try enabling debugging level 9 and upload the logfile for zmc when it crashes again.

mastertheknife.

[clint09]

Background:
I have 10 IP cameras, a mixture of Acti and Vivoteks all running Mjpeg. Zoneminder was installed using the APT 1.24.2 package. The system is a Core 2 Quad 2.5ghz, 4gb ram. I've been using Ubuntu 10.04 x64. When the system is running, it runs at a nice 1.8 load with room to spare.

Just recently the system has began to crash/panic/lock-up the whole system. In the sys log I see messages like this



When it crashes, the system will semi be available, but it reports the ZMC process as a zombie and ZM wont shutdown, nor will the system.

I have changed the following: CPU, Motherboard, Ram,network card, and reinstalled the OS several times. I've tried several kernels, by updating the system and running the system on the default kernel from the disk. Today I compiled and installed the 1.24.3 preview with the same results. This seems to be completely random. It happens with all 10 cameras enabled, and as little as 3 enabled. All cameras are set to "Record".

Anyone have any ideas? Thanks in advance.

Hi,
The crash happens inside networking code (socket buffers to be precise) so try playing with your network configuration, drivers and modules. and try unloading modules from your kernel, its possible that a module is causing memory corruption.
If more data was available, i could know what syscall from zmc is the one causing the crash. Try enabling debugging level 9 and upload the logfile for zmc when it crashes again.

mastertheknife.

I pulled the Realtek and the Marvell network cards and installed an Intel dual port gigabit card and it still crashed. This time it was a full kernel panic and didn't write anything to the log. On the monitor it still listed ZMC as the process that crashed. I'll try and see if I can pull some info out of the debug log when it happens again.

Here is a list of the modules that are currently loaded:

Code: Select all

Module                  Size  Used by
xfs                   541594  1
exportfs                4202  1 xfs
fbcon                  39270  71
tileblit                2487  1 fbcon
font                    8053  1 fbcon
bitblit                 5811  1 fbcon
softcursor              1565  1 bitblit
vga16fb                12757  1
vgastate                9857  1 vga16fb
intel_agp              29287  0
lp                      9336  0
parport                37160  1 lp
raid10                 21450  0
raid456                54784  1
async_pq                3891  1 raid456
async_xor               3111  2 raid456,async_pq
xor                     4685  1 async_xor
async_memcpy            1537  1 raid456
async_raid6_recov       1816  1 raid456
raid6_pq               80147  2 async_pq,async_raid6_recov
async_tx                2545  5 raid456,async_pq,async_xor,async_memcpy,async_raid6_recov
raid1                  22514  0
raid0                   6778  0
ohci1394               30260  0
multipath               7181  0
e1000                 108809  0
ieee1394               94771  1 ohci1394
pata_via                9072  0
ahci                   37870  9
linear                  4158  0

[clint09]

Background:
I have 10 IP cameras, a mixture of Acti and Vivoteks all running Mjpeg. Zoneminder was installed using the APT 1.24.2 package. The system is a Core 2 Quad 2.5ghz, 4gb ram. I've been using Ubuntu 10.04 x64. When the system is running, it runs at a nice 1.8 load with room to spare.

Just recently the system has began to crash/panic/lock-up the whole system. In the sys log I see messages like this



When it crashes, the system will semi be available, but it reports the ZMC process as a zombie and ZM wont shutdown, nor will the system.

I have changed the following: CPU, Motherboard, Ram,network card, and reinstalled the OS several times. I've tried several kernels, by updating the system and running the system on the default kernel from the disk. Today I compiled and installed the 1.24.3 preview with the same results. This seems to be completely random. It happens with all 10 cameras enabled, and as little as 3 enabled. All cameras are set to "Record".

Anyone have any ideas? Thanks in advance.

Hi,
The crash happens inside networking code (socket buffers to be precise) so try playing with your network configuration, drivers and modules. and try unloading modules from your kernel, its possible that a module is causing memory corruption.
If more data was available, i could know what syscall from zmc is the one causing the crash. Try enabling debugging level 9 and upload the logfile for zmc when it crashes again.

mastertheknife.

I installed Debian 6.0 today and 1.24.3 from source on the same hardware. Seems to be same results, this time ZMA though...

Code: Select all

munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Feb 21 16:35:32 debian kernel: [ 9655.170732] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
Feb 21 16:35:32 debian kernel: [ 9655.170737] IP: [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:35:32 debian kernel: [ 9655.170745] PGD 7ea67067 PUD 68222067 PMD 0 
Feb 21 16:35:32 debian kernel: [ 9655.170748] Oops: 0000 [#1] SMP 
Feb 21 16:35:32 debian kernel: [ 9655.170751] last sysfs file: /sys/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/host16/target16:0:0/16:0:0:0/model
Feb 21 16:35:32 debian kernel: [ 9655.170754] CPU 3 
Feb 21 16:35:32 debian kernel: [ 9655.170755] Modules linked in: ip6table_filter ip6_tables xfs exportfs iptable_filter ip_tables x_tables parport_pc ppdev lp parport ext3 jbd loop firewire_sbp2 pcspkr evdev i2c_i801 i2c_core button processor ext4 mbcache jbd2 crc16 dm_mod raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sg sd_mod crc_t10dif uhci_hcd ahci e1000 pata_via thermal thermal_sys firewire_ohci firewire_core ata_generic ehci_hcd crc_itu_t libata scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
Feb 21 16:35:32 debian kernel: [ 9655.170789] Pid: 47, comm: kswapd0 Not tainted 2.6.32-5-amd64 #1 To Be Filled By O.E.M.
Feb 21 16:35:32 debian kernel: [ 9655.170791] RIP: 0010:[<ffffffff81116a7e>]  [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:35:32 debian kernel: [ 9655.170795] RSP: 0018:ffff880106a87c80  EFLAGS: 00010202
Feb 21 16:35:32 debian kernel: [ 9655.170797] RAX: 0000000000000020 RBX: ffff8800db56b2b8 RCX: ffff880106a87d00
Feb 21 16:35:32 debian kernel: [ 9655.170799] RDX: 0000000000000006 RSI: ffff8800db56b4d0 RDI: ffff880106a87c80
Feb 21 16:35:32 debian kernel: [ 9655.170801] RBP: ffff8800db56b2b8 R08: 0000000000000000 R09: 8000000000000000
Feb 21 16:35:32 debian kernel: [ 9655.170803] R10: ffff880106a87c50 R11: ffffffff81152c15 R12: 0000000000000007
Feb 21 16:35:32 debian kernel: [ 9655.170805] R13: ffff880106a87d00 R14: 0000000000000080 R15: ffff880106a87d00
Feb 21 16:35:32 debian kernel: [ 9655.170808] FS:  0000000000000000(0000) GS:ffff880005580000(0000) knlGS:0000000000000000
Feb 21 16:35:32 debian kernel: [ 9655.170810] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Feb 21 16:35:32 debian kernel: [ 9655.170812] CR2: 0000000000000020 CR3: 000000006bb56000 CR4: 00000000000406e0
Feb 21 16:35:32 debian kernel: [ 9655.170814] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 21 16:35:32 debian kernel: [ 9655.170816] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 21 16:35:32 debian kernel: [ 9655.170818] Process kswapd0 (pid: 47, threadinfo ffff880106a86000, task ffff880107bec6a0)
Feb 21 16:35:32 debian kernel: [ 9655.170820] Stack:
Feb 21 16:35:32 debian kernel: [ 9655.170821]  ffff880106a87c80 ffff880106a87c80 ffff8800db56b2b8 ffff8800db56b2c8
Feb 21 16:35:32 debian kernel: [ 9655.170824] <0> 0000000000000007 ffffffff8110009d ffff8800db56b2b8 ffffffff811000f3
Feb 21 16:35:32 debian kernel: [ 9655.170827] <0> ffff8800db56b2b8 ffffffff811001fb 000000000000007f ffff8800db569ba8
Feb 21 16:35:32 debian kernel: [ 9655.170831] Call Trace:
Feb 21 16:35:32 debian kernel: [ 9655.170835]  [<ffffffff8110009d>] ? __destroy_inode+0x21/0x6e
Feb 21 16:35:32 debian kernel: [ 9655.170838]  [<ffffffff811000f3>] ? destroy_inode+0x9/0x37
Feb 21 16:35:32 debian kernel: [ 9655.170841]  [<ffffffff811001fb>] ? dispose_list+0xda/0x10a
Feb 21 16:35:32 debian kernel: [ 9655.170844]  [<ffffffff8110041d>] ? shrink_icache_memory+0x1f2/0x228
Feb 21 16:35:32 debian kernel: [ 9655.170848]  [<ffffffff810bf1a5>] ? shrink_slab+0xe0/0x153
Feb 21 16:35:32 debian kernel: [ 9655.170851]  [<ffffffff810bfa48>] ? kswapd+0x4d9/0x686
Feb 21 16:35:32 debian kernel: [ 9655.170854]  [<ffffffff810bd0bf>] ? isolate_pages_global+0x0/0x20f
Feb 21 16:35:32 debian kernel: [ 9655.170857]  [<ffffffff81064d2a>] ? autoremove_wake_function+0x0/0x2e
Feb 21 16:35:32 debian kernel: [ 9655.170861]  [<ffffffff8103a9dd>] ? __wake_up_common+0x44/0x72
Feb 21 16:35:32 debian kernel: [ 9655.170864]  [<ffffffff810bf56f>] ? kswapd+0x0/0x686
Feb 21 16:35:32 debian kernel: [ 9655.170866]  [<ffffffff81064a5d>] ? kthread+0x79/0x81
Feb 21 16:35:32 debian kernel: [ 9655.170870]  [<ffffffff81011baa>] ? child_rip+0xa/0x20
Feb 21 16:35:32 debian kernel: [ 9655.170872]  [<ffffffff810649e4>] ? kthread+0x0/0x81
Feb 21 16:35:32 debian kernel: [ 9655.170874]  [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Feb 21 16:35:32 debian kernel: [ 9655.170876] Code: 85 c0 48 89 02 74 04 48 89 50 08 48 c7 41 18 00 00 00 00 48 c7 41 20 00 00 00 00 48 8d 41 04 f0 ff 41 04 48 89 f0 48 85 c0 74 2a <48> 8b 30 48 8d 48 e8 48 8d 50 28 48 8b 04 24 48 89 50 08 48 89 
Feb 21 16:35:32 debian kernel: [ 9655.170901] RIP  [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:35:32 debian kernel: [ 9655.170904]  RSP <ffff880106a87c80>
Feb 21 16:35:32 debian kernel: [ 9655.170905] CR2: 0000000000000020
Feb 21 16:35:32 debian kernel: [ 9655.170908] ---[ end trace d131014c3f547163 ]---

Code: Select all

BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
Feb 21 16:36:30 debian kernel: [ 9713.225398] IP: [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:36:30 debian kernel: [ 9713.225406] PGD 104b4f067 PUD 1066a0067 PMD 0 
Feb 21 16:36:30 debian kernel: [ 9713.225409] Oops: 0000 [#2] SMP 
Feb 21 16:36:30 debian kernel: [ 9713.225411] last sysfs file: /sys/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/host16/target16:0:0/16:0:0:0/model
Feb 21 16:36:30 debian kernel: [ 9713.225414] CPU 3 
Feb 21 16:36:30 debian kernel: [ 9713.225416] Modules linked in: ip6table_filter ip6_tables xfs exportfs iptable_filter ip_tables x_tables parport_pc ppdev lp parport ext3 jbd loop firewire_sbp2 pcspkr evdev i2c_i801 i2c_core button processor ext4 mbcache jbd2 crc16 dm_mod raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sg sd_mod crc_t10dif uhci_hcd ahci e1000 pata_via thermal thermal_sys firewire_ohci firewire_core ata_generic ehci_hcd crc_itu_t libata scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
Feb 21 16:36:30 debian kernel: [ 9713.225450] Pid: 5269, comm: zma Tainted: G      D    2.6.32-5-amd64 #1 To Be Filled By O.E.M.
Feb 21 16:36:30 debian kernel: [ 9713.225452] RIP: 0010:[<ffffffff81116a7e>]  [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:36:30 debian kernel: [ 9713.225456] RSP: 0018:ffff88007e8c18a8  EFLAGS: 00010202
Feb 21 16:36:30 debian kernel: [ 9713.225458] RAX: 0000000000000020 RBX: ffff8800d8c632b8 RCX: ffff88007e8c1928
Feb 21 16:36:30 debian kernel: [ 9713.225460] RDX: 0000000000000006 RSI: ffff8800d8c634d0 RDI: ffff88007e8c18a8
Feb 21 16:36:30 debian kernel: [ 9713.225462] RBP: ffff8800d8c632b8 R08: 0000000000000000 R09: 8000000000000000
Feb 21 16:36:30 debian kernel: [ 9713.225464] R10: ffff88007e8c1878 R11: ffffffff81152c15 R12: 0000000000000004
Feb 21 16:36:30 debian kernel: [ 9713.225466] R13: ffff88007e8c1928 R14: 0000000000000080 R15: ffff88007e8c1928
Feb 21 16:36:30 debian kernel: [ 9713.225469] FS:  00007fa52673e720(0000) GS:ffff880005580000(0000) knlGS:0000000000000000
Feb 21 16:36:30 debian kernel: [ 9713.225471] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 21 16:36:30 debian kernel: [ 9713.225473] CR2: 0000000000000020 CR3: 000000000e34f000 CR4: 00000000000406e0
Feb 21 16:36:30 debian kernel: [ 9713.225475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 21 16:36:30 debian kernel: [ 9713.225477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 21 16:36:30 debian kernel: [ 9713.225479] Process zma (pid: 5269, threadinfo ffff88007e8c0000, task ffff8800f409b170)
Feb 21 16:36:30 debian kernel: [ 9713.225481] Stack:
Feb 21 16:36:30 debian kernel: [ 9713.225482]  ffff88007e8c18a8 ffff88007e8c18a8 ffff8800d8c632b8 ffff8800d8c632c8
Feb 21 16:36:30 debian kernel: [ 9713.225485] <0> 0000000000000004 ffffffff8110009d ffff8800d8c632b8 ffffffff811000f3
Feb 21 16:36:30 debian kernel: [ 9713.225488] <0> ffff8800d8c632b8 ffffffff811001fb 000000000000006a ffff8800d8c7c870
Feb 21 16:36:30 debian kernel: [ 9713.225492] Call Trace:
Feb 21 16:36:30 debian kernel: [ 9713.225496]  [<ffffffff8110009d>] ? __destroy_inode+0x21/0x6e
Feb 21 16:36:30 debian kernel: [ 9713.225499]  [<ffffffff811000f3>] ? destroy_inode+0x9/0x37
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff811001fb>] ? dispose_list+0xda/0x10a
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff8110041d>] ? shrink_icache_memory+0x1f2/0x228
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810bf1a5>] ? shrink_slab+0xe0/0x153
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810c0102>] ? try_to_free_pages+0x26a/0x38e
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810bd0bf>] ? isolate_pages_global+0x0/0x20f
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810ba159>] ? __alloc_pages_nodemask+0x3cd/0x5f4
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810b42ce>] ? grab_cache_page_write_begin+0x55/0x97
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff8110eac2>] ? block_write_begin+0x3a/0xc7
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffffa026e018>] ? xfs_vm_write_begin+0x22/0x27 [xfs]
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffffa026d144>] ? xfs_get_blocks+0x0/0xe [xfs]
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810b4b44>] ? generic_file_buffered_write+0x118/0x278
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffffa0273ac6>] ? xfs_write+0x4df/0x6ea [xfs]
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810ba04f>] ? __alloc_pages_nodemask+0x2c3/0x5f4
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810ee6f6>] ? do_sync_write+0xce/0x113
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff81064d2a>] ? autoremove_wake_function+0x0/0x2e
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810cc807>] ? handle_mm_fault+0x35c/0x80f
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810ef06e>] ? vfs_write+0xa9/0x102
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff810ef183>] ? sys_write+0x45/0x6e
Feb 21 16:36:30 debian kernel: [ 9713.225502]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Feb 21 16:36:30 debian kernel: [ 9713.225502] Code: 85 c0 48 89 02 74 04 48 89 50 08 48 c7 41 18 00 00 00 00 48 c7 41 20 00 00 00 00 48 8d 41 04 f0 ff 41 04 48 89 f0 48 85 c0 74 2a <48> 8b 30 48 8d 48 e8 48 8d 50 28 48 8b 04 24 48 89 50 08 48 89 
Feb 21 16:36:30 debian kernel: [ 9713.225502] RIP  [<ffffffff81116a7e>] fsnotify_clear_marks_by_inode+0x5c/0xcd
Feb 21 16:36:30 debian kernel: [ 9713.225502]  RSP <ffff88007e8c18a8>
Feb 21 16:36:30 debian kernel: [ 9713.225502] CR2: 0000000000000020
Feb 21 16:36:30 debian kernel: [ 9713.225629] ---[ end trace d131014c3f547164 ]---

[/code]

[ruehlchr]

Hi Clint,

can you give me the commands:
# sysctl -a | grep kernel.shm

# free (when its happen)

Please create a partition using ext3/4 and mount it
/var/cache/zoneminder

I'm not sure if your XFS filesystem is involved with this bug (maybe a work around for you)

-Chris

[wkulecz]

Looks like it crashed again today as my wife called and couldn't start it from the web browser on her machine before she left

My problem appears to have just been a web interface/apache2 temporary lockup, not a crash. It seems to have resolved itself a few hours later by the time I got home. Seems to be working fine although I do see web interface lockups from time to time when messing with the configuration options or camera settings.

[clint09]

Hi Clint,

can you give me the commands:
# sysctl -a | grep kernel.shm

# free (when its happen)

Please create a partition using ext3/4 and mount it
/var/cache/zoneminder

I'm not sure if your XFS filesystem is involved with this bug (maybe a work around for you)

-Chris

I changed to an EXT4 filesystem again (it had crashed on EXT4 before switch to XFS) and same results. This time it was ZMC with only 6 IP cameras all set to record.

Code: Select all

            total       used       free     shared    buffers     cached
Mem:       4061060    4015996      45064          0      81428    3412404
-/+ buffers/cache:     522164    3538896
Swap:      4881400        780    4880620

Code: Select all

debian:/var/cache# sysctl -a | grep kernel.shm
error: permission denied on key 'net.ipv4.route.flush'
error: permission denied on key 'net.ipv6.route.flush'
kernel.shmmax = 33554432
kernel.shmall = 2097152
kernel.shmmni = 4096

Code: Select all

Feb 22 10:04:24 debian kernel: [ 4428.865706] BUG: unable to handle kernel NULL pointer dereference at 0000000000000007
Feb 22 10:04:24 debian kernel: [ 4428.865711] IP: [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4428.865719] PGD 5e13c067 PUD 5efaf067 PMD 0 
Feb 22 10:04:24 debian kernel: [ 4428.865722] Oops: 0000 [#1] SMP 
Feb 22 10:04:24 debian kernel: [ 4428.865724] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host9/target9:0:0/9:0:0:0/model
Feb 22 10:04:24 debian kernel: [ 4428.865728] CPU 1 
Feb 22 10:04:24 debian kernel: [ 4428.865729] Modules linked in: parport_pc ppdev lp parport ext3 jbd loop firewire_sbp2 i2c_i801 i2c_core processor evdev pcspkr button ext4 mbcache jbd2 crc16 dm_mod raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sg firewire_ohci sd_mod crc_t10dif uhci_hcd ahci e1000 firewire_core sata_sil24 crc_itu_t pata_via ata_generic libata ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base [last unloaded: scsi_wait_scan]
Feb 22 10:04:24 debian kernel: [ 4428.865759] Pid: 4403, comm: zmc Not tainted 2.6.32-5-amd64 #1 To Be Filled By O.E.M.
Feb 22 10:04:24 debian kernel: [ 4428.865761] RIP: 0010:[<ffffffff810e7008>]  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4428.865765] RSP: 0018:ffff880076f11be8  EFLAGS: 00010082
Feb 22 10:04:24 debian kernel: [ 4428.865767] RAX: 0000000000000001 RBX: 0000000000000282 RCX: ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4428.865769] RDX: 00000000000005a8 RSI: ffffffffffffffff RDI: ffffea0001f17e00
Feb 22 10:04:24 debian kernel: [ 4428.865771] RBP: ffff88008e245000 R08: ffff880104d0ee80 R09: ffffffff8144e980
Feb 22 10:04:24 debian kernel: [ 4428.865773] R10: 000000000000038c R11: ffff880104d0ee80 R12: ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4428.865775] R13: 00000000000005a8 R14: 0000000000000000 R15: 00000000000056f4
Feb 22 10:04:24 debian kernel: [ 4428.865777] FS:  00007f19874fc720(0000) GS:ffff880005480000(0000) knlGS:0000000000000000
Feb 22 10:04:24 debian kernel: [ 4428.865780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 22 10:04:24 debian kernel: [ 4428.865781] CR2: 0000000000000007 CR3: 000000005efad000 CR4: 00000000000406e0
Feb 22 10:04:24 debian kernel: [ 4428.865783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 22 10:04:24 debian kernel: [ 4428.865785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 22 10:04:24 debian kernel: [ 4428.865788] Process zmc (pid: 4403, threadinfo ffff880076f10000, task ffff8800ef973880)
Feb 22 10:04:24 debian kernel: [ 4428.865789] Stack:
Feb 22 10:04:24 debian kernel: [ 4428.865791]  ffff880090b95300 ffff880104d0ee80 ffff880090b95300 ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4428.865794] <0> 00000000000005a8 ffffffff8127e32f 0000000000015788 0000000000015780
Feb 22 10:04:24 debian kernel: [ 4428.865797] <0> 0000000000000008 ffff8800ef973880 ffff8800ef973880 ffff8800ef973880
Feb 22 10:04:24 debian kernel: [ 4428.865801] Call Trace:
Feb 22 10:04:24 debian kernel: [ 4428.865805]  [<ffffffff812472c2>] ? __kfree_skb+0x11/0x7d
Feb 22 10:04:24 debian kernel: [ 4428.865809]  [<ffffffff8127e32f>] ? tcp_recvmsg+0x7c3/0xa9e
Feb 22 10:04:24 debian kernel: [ 4428.865813]  [<ffffffff812425fa>] ? sock_common_recvmsg+0x30/0x45
Feb 22 10:04:24 debian kernel: [ 4428.865815]  [<ffffffff812405f8>] ? sock_aio_read+0xb9/0xc4
Feb 22 10:04:24 debian kernel: [ 4428.865819]  [<ffffffff810ee809>] ? do_sync_read+0xce/0x113
Feb 22 10:04:24 debian kernel: [ 4428.865822]  [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Feb 22 10:04:24 debian kernel: [ 4428.865825]  [<ffffffff81064d2a>] ? autoremove_wake_function+0x0/0x2e
Feb 22 10:04:24 debian kernel: [ 4428.865828]  [<ffffffff8124037d>] ? sock_ioctl+0x1f2/0x216
Feb 22 10:04:24 debian kernel: [ 4428.865830]  [<ffffffff810ef265>] ? vfs_read+0xb9/0xff
Feb 22 10:04:24 debian kernel: [ 4428.865833]  [<ffffffff810ef367>] ? sys_read+0x45/0x6e
Feb 22 10:04:24 debian kernel: [ 4428.865836]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Feb 22 10:04:24 debian kernel: [ 4428.865838] Code: 8b 4c 24 18 4c 8b 4f 10 9c 58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b b4 c1 f0 02 00 00 <48> 3b 7e 08 75 16 83 7e 10 00 78 10 8b 56 14 48 8b 06 48 89 44 
Feb 22 10:04:24 debian kernel: [ 4428.865863] RIP  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4428.865866]  RSP <ffff880076f11be8>
Feb 22 10:04:24 debian kernel: [ 4428.865867] CR2: 0000000000000007
Feb 22 10:04:24 debian kernel: [ 4428.865870] ---[ end trace f95bb4be205daf9a ]---

Code: Select all

Feb 22 10:04:24 debian kernel: [ 4429.180172] BUG: unable to handle kernel NULL pointer dereference at 0000000000000007
Feb 22 10:04:24 debian kernel: [ 4429.180178] IP: [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4429.180186] PGD efbc6067 PUD 739c7067 PMD 0 
Feb 22 10:04:24 debian kernel: [ 4429.180189] Oops: 0000 [#2] SMP 
Feb 22 10:04:24 debian kernel: [ 4429.180192] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host9/target9:0:0/9:0:0:0/model
Feb 22 10:04:24 debian kernel: [ 4429.180195] CPU 1 
Feb 22 10:04:24 debian kernel: [ 4429.180197] Modules linked in: parport_pc ppdev lp parport ext3 jbd loop firewire_sbp2 i2c_i801 i2c_core processor evdev pcspkr button ext4 mbcache jbd2 crc16 dm_mod raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sg firewire_ohci sd_mod crc_t10dif uhci_hcd ahci e1000 firewire_core sata_sil24 crc_itu_t pata_via ata_generic libata ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base [last unloaded: scsi_wait_scan]
Feb 22 10:04:24 debian kernel: [ 4429.180227] Pid: 4769, comm: zmc Tainted: G      D    2.6.32-5-amd64 #1 To Be Filled By O.E.M.
Feb 22 10:04:24 debian kernel: [ 4429.180229] RIP: 0010:[<ffffffff810e7008>]  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4429.180233] RSP: 0018:ffff880076ee7be8  EFLAGS: 00010082
Feb 22 10:04:24 debian kernel: [ 4429.180234] RAX: 0000000000000001 RBX: 0000000000000282 RCX: ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4429.180237] RDX: 00000000000005a8 RSI: ffffffffffffffff RDI: ffffea0001f17e00
Feb 22 10:04:24 debian kernel: [ 4429.180239] RBP: ffff88008e242000 R08: ffff880104dfe180 R09: ffffffff8144e980
Feb 22 10:04:24 debian kernel: [ 4429.180241] R10: 000000000000017b R11: ffff880104dfe180 R12: ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4429.180243] R13: 00000000000005a8 R14: 0000000000000000 R15: 0000000000002bc5
Feb 22 10:04:24 debian kernel: [ 4429.180245] FS:  00007f433b374720(0000) GS:ffff880005480000(0000) knlGS:0000000000000000
Feb 22 10:04:24 debian kernel: [ 4429.180247] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 22 10:04:24 debian kernel: [ 4429.180249] CR2: 0000000000000007 CR3: 0000000070bbb000 CR4: 00000000000406e0
Feb 22 10:04:24 debian kernel: [ 4429.180251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 22 10:04:24 debian kernel: [ 4429.180253] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 22 10:04:24 debian kernel: [ 4429.180256] Process zmc (pid: 4769, threadinfo ffff880076ee6000, task ffff880104063170)
Feb 22 10:04:24 debian kernel: [ 4429.180257] Stack:
Feb 22 10:04:24 debian kernel: [ 4429.180258]  ffff8800cb432a00 ffff880104dfe180 ffff8800cb432a00 ffffffff812472c2
Feb 22 10:04:24 debian kernel: [ 4429.180261] <0> 00000000000005a8 ffffffff8127e32f 0000000000015788 0000000000015780
Feb 22 10:04:24 debian kernel: [ 4429.180265] <0> 0000000000000008 ffff880104063170 ffff880104063170 ffff880104063170
Feb 22 10:04:24 debian kernel: [ 4429.180268] Call Trace:
Feb 22 10:04:24 debian kernel: [ 4429.180272]  [<ffffffff812472c2>] ? __kfree_skb+0x11/0x7d
Feb 22 10:04:24 debian kernel: [ 4429.180276]  [<ffffffff8127e32f>] ? tcp_recvmsg+0x7c3/0xa9e
Feb 22 10:04:24 debian kernel: [ 4429.180280]  [<ffffffff812425fa>] ? sock_common_recvmsg+0x30/0x45
Feb 22 10:04:24 debian kernel: [ 4429.180283]  [<ffffffff812405f8>] ? sock_aio_read+0xb9/0xc4
Feb 22 10:04:24 debian kernel: [ 4429.180286]  [<ffffffff810ee809>] ? do_sync_read+0xce/0x113
Feb 22 10:04:24 debian kernel: [ 4429.180289]  [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Feb 22 10:04:24 debian kernel: [ 4429.180293]  [<ffffffff81064d2a>] ? autoremove_wake_function+0x0/0x2e
Feb 22 10:04:24 debian kernel: [ 4429.180295]  [<ffffffff8124037d>] ? sock_ioctl+0x1f2/0x216
Feb 22 10:04:24 debian kernel: [ 4429.180298]  [<ffffffff810ef265>] ? vfs_read+0xb9/0xff
Feb 22 10:04:24 debian kernel: [ 4429.180301]  [<ffffffff810ef367>] ? sys_read+0x45/0x6e
Feb 22 10:04:24 debian kernel: [ 4429.180304]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Feb 22 10:04:24 debian kernel: [ 4429.180305] Code: 8b 4c 24 18 4c 8b 4f 10 9c 58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b b4 c1 f0 02 00 00 <48> 3b 7e 08 75 16 83 7e 10 00 78 10 8b 56 14 48 8b 06 48 89 44 
Feb 22 10:04:24 debian kernel: [ 4429.180330] RIP  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:24 debian kernel: [ 4429.180333]  RSP <ffff880076ee7be8>
Feb 22 10:04:24 debian kernel: [ 4429.180335] CR2: 0000000000000007
Feb 22 10:04:24 debian kernel: [ 4429.180337] ---[ end trace f95bb4be205daf9b ]---
Feb 22 10:04:26 debian zma_m23[4466]: INF [Front-Outside: 15000 - Processing at 3.98 fps]
Feb 22 10:04:33 debian kernel: [ 4438.129798] BUG: unable to handle kernel NULL pointer dereference at 0000000000000007
Feb 22 10:04:33 debian kernel: [ 4438.129803] IP: [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:33 debian kernel: [ 4438.129810] PGD 6ece4067 PUD 5e161067 PMD 0 
Feb 22 10:04:33 debian kernel: [ 4438.129813] Oops: 0000 [#3] SMP 
Feb 22 10:04:33 debian kernel: [ 4438.129816] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host9/target9:0:0/9:0:0:0/model
Feb 22 10:04:33 debian kernel: [ 4438.129819] CPU 1 
Feb 22 10:04:33 debian kernel: [ 4438.129820] Modules linked in: parport_pc ppdev lp parport ext3 jbd loop firewire_sbp2 i2c_i801 i2c_core processor evdev pcspkr button ext4 mbcache jbd2 crc16 dm_mod raid456 md_mod async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx sg firewire_ohci sd_mod crc_t10dif uhci_hcd ahci e1000 firewire_core sata_sil24 crc_itu_t pata_via ata_generic libata ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base [last unloaded: scsi_wait_scan]
Feb 22 10:04:33 debian kernel: [ 4438.129850] Pid: 4399, comm: zmc Tainted: G      D    2.6.32-5-amd64 #1 To Be Filled By O.E.M.
Feb 22 10:04:33 debian kernel: [ 4438.129852] RIP: 0010:[<ffffffff810e7008>]  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:33 debian kernel: [ 4438.129856] RSP: 0018:ffff880062cf5be8  EFLAGS: 00010082
Feb 22 10:04:33 debian kernel: [ 4438.129857] RAX: 0000000000000001 RBX: 0000000000000282 RCX: ffffffff812472c2
Feb 22 10:04:33 debian kernel: [ 4438.129860] RDX: 00000000000005a8 RSI: ffffffffffffffff RDI: ffffea0001f17e00
Feb 22 10:04:33 debian kernel: [ 4438.129862] RBP: ffff88008e247000 R08: ffff880037b44780 R09: ffffffff8144e980
Feb 22 10:04:33 debian kernel: [ 4438.129864] R10: 00000000ffffffff R11: ffff880037b44780 R12: ffffffff812472c2
Feb 22 10:04:33 debian kernel: [ 4438.129866] R13: 00000000000005a8 R14: 0000000000000000 R15: 00000000000047a8
Feb 22 10:04:33 debian kernel: [ 4438.129868] FS:  00007f3afcf27720(0000) GS:ffff880005480000(0000) knlGS:0000000000000000
Feb 22 10:04:33 debian kernel: [ 4438.129870] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 22 10:04:33 debian kernel: [ 4438.129872] CR2: 0000000000000007 CR3: 000000006cd3e000 CR4: 00000000000406e0
Feb 22 10:04:33 debian kernel: [ 4438.129874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 22 10:04:33 debian kernel: [ 4438.129876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb 22 10:04:33 debian kernel: [ 4438.129878] Process zmc (pid: 4399, threadinfo ffff880062cf4000, task ffff88010365dbd0)
Feb 22 10:04:33 debian kernel: [ 4438.129880] Stack:
Feb 22 10:04:33 debian kernel: [ 4438.129881]  ffff880103f5a400 ffff880037b44780 ffff880103f5a400 ffffffff812472c2
Feb 22 10:04:33 debian kernel: [ 4438.129884] <0> 00000000000005a8 ffffffff8127e32f 0000000000015788 0000000000015780
Feb 22 10:04:33 debian kernel: [ 4438.129888] <0> 0000000000000008 ffff88010365dbd0 ffff88010365dbd0 ffff88010365dbd0
Feb 22 10:04:33 debian kernel: [ 4438.129891] Call Trace:
Feb 22 10:04:33 debian kernel: [ 4438.129895]  [<ffffffff812472c2>] ? __kfree_skb+0x11/0x7d
Feb 22 10:04:33 debian kernel: [ 4438.129899]  [<ffffffff8127e32f>] ? tcp_recvmsg+0x7c3/0xa9e
Feb 22 10:04:33 debian kernel: [ 4438.129902]  [<ffffffff812425fa>] ? sock_common_recvmsg+0x30/0x45
Feb 22 10:04:33 debian kernel: [ 4438.129905]  [<ffffffff812405f8>] ? sock_aio_read+0xb9/0xc4
Feb 22 10:04:33 debian kernel: [ 4438.129909]  [<ffffffff810ee809>] ? do_sync_read+0xce/0x113
Feb 22 10:04:33 debian kernel: [ 4438.129912]  [<ffffffff812431e2>] ? release_sock+0x13/0xa0
Feb 22 10:04:33 debian kernel: [ 4438.129915]  [<ffffffff81064d2a>] ? autoremove_wake_function+0x0/0x2e
Feb 22 10:04:33 debian kernel: [ 4438.129918]  [<ffffffff8124037d>] ? sock_ioctl+0x1f2/0x216
Feb 22 10:04:33 debian kernel: [ 4438.129921]  [<ffffffff810ef265>] ? vfs_read+0xb9/0xff
Feb 22 10:04:33 debian kernel: [ 4438.129923]  [<ffffffff810ef367>] ? sys_read+0x45/0x6e
Feb 22 10:04:33 debian kernel: [ 4438.129926]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Feb 22 10:04:33 debian kernel: [ 4438.129928] Code: 8b 4c 24 18 4c 8b 4f 10 9c 58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 65 8b 04 25 a8 e3 00 00 48 98 49 8b b4 c1 f0 02 00 00 <48> 3b 7e 08 75 16 83 7e 10 00 78 10 8b 56 14 48 8b 06 48 89 44 
Feb 22 10:04:33 debian kernel: [ 4438.129953] RIP  [<ffffffff810e7008>] kfree+0x8e/0xcb
Feb 22 10:04:33 debian kernel: [ 4438.129956]  RSP <ffff880062cf5be8>
Feb 22 10:04:33 debian kernel: [ 4438.129958] CR2: 0000000000000007
Feb 22 10:04:33 debian kernel: [ 4438.129960] ---[ end trace f95bb4be205daf9c ]---

[ruehlchr]

Clint,

your shmall is 2meg only !

please try (256m):

268435456
for shmmax and shmall in your /etc/sysctrl.conf

kernel.shmall = 268435456
kernel.shmmax = 268435456

and call sysctrl -p /etc/sysctrl.conf