Raspberry pi3 and h264_mmal

[mastertheknife]

The difference should be only in zma. Make sure to use 8bit grayscale or 32bit colour, the optimized delta function isn't used with 24bit colour.
This is very similar to what i am seeing on my server. Thanks for sharing
Sometime in the future i'll probably also optimize AlarmedPixels, which currently is the major CPU consumer.

[cmisip]

I have been trying to max out this RPI3. I have found out that I was probably already using libturbojpeg. I was able to build a libturbojpeg 1.5.1 package and it resulted in the following debs:

Code: Select all

ii  libjpeg-dev                      1:1.5.1-12                                all          Development files for the JPEG library [dummy package]
ii  libjpeg-turbo-progs              1:1.5.1-12                                armhf        Programs for manipulating JPEG files
ii  libjpeg-turbo-progs-dbg          1:1.5.1-12                                armhf        Programs for manipulating JPEG files (debugging symbols)
ii  libjpeg62-turbo:armhf            1:1.5.1-12                                armhf        libjpeg-turbo JPEG runtime library
ii  libjpeg62-turbo-dbg:armhf        1:1.5.1-12                                armhf        Debugging symbols for the libjpeg-turbo JPEG library
ii  libjpeg62-turbo-dev:armhf        1:1.5.1-12                                armhf        Development files for the libjpeg-turbo JPEG library
ii  libopenjpeg5:armhf               1:1.5.2-3                                 armhf        JPEG 2000 image compression/decompression library - runtime
ii  libturbojpeg1:armhf              1:1.5.1-12                                armhf        TurboJPEG runtime library - SIMD optimized
ii  libturbojpeg1-dbg:armhf          1:1.5.1-12                                armhf        TurboJPEG runtime library - SIMD optimized (debugging symbols)
ii  libturbojpeg1-dev:armhf          1:1.5.1-12                                armhf        Development files for the TurboJPEG library

Notice the "dummy package". With dpkg -L libjpeg-dev you get:

Code: Select all

/usr
/usr/share
/usr/share/doc
/usr/share/doc/libjpeg-dev
/usr/share/doc/libjpeg-dev/README-turbo.txt.gz
/usr/share/doc/libjpeg-dev/changelog.Debian.gz
/usr/share/doc/libjpeg-dev/changelog.gz
/usr/share/doc/libjpeg-dev/README.gz
/usr/share/doc/libjpeg-dev/structure.txt.gz
/usr/share/doc/libjpeg-dev/copyright

The actual libjpeg libraries and include files are in libjpeg62-turbo-dev as shown by dpkg -L libjpeg62-turbo-dev

Code: Select all

/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/pkgconfig
/usr/lib/arm-linux-gnueabihf/pkgconfig/libjpeg.pc
/usr/lib/arm-linux-gnueabihf/pkgconfig/libturbojpeg.pc
/usr/lib/arm-linux-gnueabihf/libjpeg.a
/usr/lib/arm-linux-gnueabihf/libjpeg.so

So I guess zoneminder is probably already linking with libturbo-jpeg.

The next thing I tried was to tune zoneminder specifically to the RPI3 processor by adding the following CPP flags to debian/rules:

Code: Select all

export DEB_BUILD_MAINT_OPTIONS = hardening=+all -march=armv8-a -mfloat-abi=hard -mfpu=neon-fp-armv8 -funsafe-math-optimizations -ftree-vectorize -fPIC -O3

That did not result in any appreciable increase in performance as well. Just FYI for anyone traveling down that road.

Chris

[cmisip]

I have an idea why multiple zmc's can seemingly access a single decode pipeline concurrently. I think the zmc's are able to successfully open the codec but they dont have exclusive access to it. That's why it never falls over to software decoding regardless of having more than 1 zmc. It can however fail on decode if the port is not available because it is busy servicing one of the other zmc's. The following code will probably never get to execute the second line.

Code: Select all

// Try and get the codec from the codec context
  if ( (mCodec = avcodec_find_decoder_by_name("h264_mmal")) == NULL )
  if ( (mCodec = avcodec_find_decoder( mCodecContext->codec_id )) == NULL )
    Fatal( "Can't find codec for video stream from %s", mPath.c_str() );

The solution is to specify the use of the h264_mmal codec on monitor creation and edit.

I created a rough patch to allow for the functionality of selecting the hardware decoder at the monitor creation and edit screen to test the idea. By selecting Ffmpeghw, the zmc for that monitor will specifically use the h264_mmal codec. The user can still select Ffmpeg for software decoding which is default. I made changes to the web UI for this and also some db changes:

Code: Select all

ALTER TABLE Monitors modify column Type enum('Local','Remote','File','Ffmpeg','Ffmpeghw','Libvlc','cURL') NOT NULL default 'Local';
ALTER TABLE MonitorPresets modify column Type enum('Local','Remote','File','Ffmpeg','Ffmpeghw','Libvlc','cURL') NOT NULL default 'Local';
ALTER TABLE Controls modify column Type enum('Local','Remote','Ffmpeg','Ffmpeghw','Libvlc','cURL') NOT NULL default 'Local';

The RPI seems to max out at 1 h264 camera at 704x576 resolution plus another at 1280x720 for hw decoding. However, with the patch, additional monitors can now use the software decoder.

The following setup resulted in a load of 5.

Code: Select all

 7243 www-data  20   0  326940 198952 190580 R  93.2 22.5   3:38.14 zma                                                                                        
 7336 www-data  20   0  227344 101316  93640 S  52.2 11.5   1:01.26 zmc  704x576 S/W decoder                                                                                     
 6845 www-data  20   0  222672  94736  89780 S  40.3 10.7   2:23.14 zma                                                                                        
 7354 www-data  20   0  224256  95436  89692 R  39.3 10.8   0:44.99 zma                                                                                        
 7200 www-data  20   0  379968 218948 194928 S  30.7 24.8   1:26.79 zmc  1280x720 H/W decoder                                                                                    
 6836 www-data  20   0  264432 105628  93768 S  15.9 12.0   1:00.62 zmc  704x576 H/W decoder  

The following setup resulted in a load of 3.

Code: Select all

 8671 www-data  20   0  330544 200748 190524 R  93.6 22.7   3:21.86 zma   1280x720                                                                                     
 9001 www-data  20   0  190724  65712  59612 R  32.1  7.4   1:08.82 zmc   640x360 S/W decoder                                                                                    
 8662 www-data  20   0  380224 218724 194456 S  30.8 24.8   1:27.48 zmc   1280x720 H/W decoder                                                                                     
 8932 www-data  20   0  188152  60012  55480 R  23.9  6.8   0:52.09 zma   640x360                                                                                 
 9008 www-data  20   0  187264  59432  55680 S  23.6  6.7   0:47.76 zma   640x360                                                                                          
 8923 www-data  20   0  225528  67572  59632 S  10.8  7.7   0:22.26 zmc   640x360 H/W decoder

I am limited in my testing as I still have two cameras ( just was utilizing the sub stream of one of the cameras for the third monitor). The SV3C camera can be configured for 1920x1080 or 1280x720 for the main stream and 704x576 or 640x360 for the sub stream.

Here's the rough patch.

Code: Select all

diff -Naur ../src-versions/src-orig/zm_ffmpeg_camera.cpp src/zm_ffmpeg_camera.cpp
--- ../src-versions/src-orig/zm_ffmpeg_camera.cpp	2017-05-14 22:01:18.765897796 +0000
+++ src/zm_ffmpeg_camera.cpp	2017-05-14 22:02:26.205693726 +0000
@@ -33,11 +33,12 @@
 #include <pthread.h>
 #endif
 
-FfmpegCamera::FfmpegCamera( int p_id, const std::string &p_path, const std::string &p_method, const std::string &p_options, int p_width, int p_height, int p_colours, int p_brightness, int p_contrast, int p_hue, int p_colour, bool p_capture ) :
+FfmpegCamera::FfmpegCamera( int p_id, const std::string &p_path, const std::string &p_method, const std::string &p_options, int p_width, int p_height, int p_colours, int p_brightness, int p_contrast, int p_hue, int p_colour, bool p_capture, int codec_type ) :
   Camera( p_id, FFMPEG_SRC, p_width, p_height, p_colours, ZM_SUBPIX_ORDER_DEFAULT_FOR_COLOUR(p_colours), p_brightness, p_contrast, p_hue, p_colour, p_capture ),
   mPath( p_path ),
   mMethod( p_method ),
-  mOptions( p_options )
+  mOptions( p_options ),
+  h264_codec(codec_type)
 {
   if ( capture )
   {
@@ -312,9 +313,15 @@
   mCodecContext = mFormatContext->streams[mVideoStreamId]->codec;
 
   // Try and get the codec from the codec context
-  if ( (mCodec = avcodec_find_decoder_by_name("h264_mmal")) == NULL )
-  if ( (mCodec = avcodec_find_decoder( mCodecContext->codec_id )) == NULL )
-    Fatal( "Can't find codec for video stream from %s", mPath.c_str() );
+  if (h264_codec) {
+     if ( (mCodec = avcodec_find_decoder_by_name("h264_mmal")) == NULL )
+         Fatal( "Can't find h/w codec for video stream from %s", mPath.c_str() );
+  }
+
+  if (!h264_codec) {
+     if ( (mCodec = avcodec_find_decoder( mCodecContext->codec_id )) == NULL )
+         Fatal( "Can't find s/w codec for video stream from %s", mPath.c_str() );
+  }
 
   Debug ( 1, "Found decoder" );
 
diff -Naur ../src-versions/src-orig/zm_ffmpeg_camera.h src/zm_ffmpeg_camera.h
--- ../src-versions/src-orig/zm_ffmpeg_camera.h	2017-05-14 22:01:18.765897796 +0000
+++ src/zm_ffmpeg_camera.h	2017-05-14 22:02:26.205693726 +0000
@@ -57,6 +57,7 @@
   bool mCanCapture;
   int mOpenStart;
   pthread_t mReopenThread;
+  int h264_codec;
 #endif // HAVE_LIBAVFORMAT
 
 #if HAVE_LIBSWSCALE
@@ -64,7 +65,7 @@
 #endif
 
 public:
-  FfmpegCamera( int p_id, const std::string &path, const std::string &p_method, const std::string &p_options, int p_width, int p_height, int p_colours, int p_brightness, int p_contrast, int p_hue, int p_colour, bool p_capture );
+  FfmpegCamera( int p_id, const std::string &path, const std::string &p_method, const std::string &p_options, int p_width, int p_height, int p_colours, int p_brightness, int p_contrast, int p_hue, int p_colour, bool p_capture, int codec_type );
   ~FfmpegCamera();
 
   const std::string &Path() const { return( mPath ); }
diff -Naur ../src-versions/src-orig/zm_monitor.cpp src/zm_monitor.cpp
--- ../src-versions/src-orig/zm_monitor.cpp	2017-05-14 22:01:18.765897796 +0000
+++ src/zm_monitor.cpp	2017-05-14 22:02:26.285693484 +0000
@@ -2634,7 +2634,152 @@
       contrast,
       hue,
       colour,
-      purpose==CAPTURE
+      purpose==CAPTURE,
+      0
+    );
+
+    monitors[i] = new Monitor(
+      id,
+      name,
+      server_id,
+      function,
+      enabled,
+      linked_monitors,
+      camera,
+      orientation,
+      deinterlacing,
+      event_prefix,
+      label_format,
+      Coord( label_x, label_y ),
+      label_size,
+      image_buffer_count,
+      warmup_count,
+      pre_event_count,
+      post_event_count,
+      stream_replay_buffer,
+      alarm_frame_count,
+      section_length,
+      frame_skip,
+      motion_frame_skip,
+      analysis_fps,
+      analysis_update_delay,
+      capture_delay,
+      alarm_capture_delay,
+      fps_report_interval,
+      ref_blend_perc,
+      alarm_ref_blend_perc,
+      track_motion,
+      embed_exif,
+      RGB_WHITE,
+      purpose,
+      0,
+      0
+    );
+    Zone **zones = 0;
+    int n_zones = Zone::Load( monitors[i], zones );
+    monitors[i]->AddZones( n_zones, zones );
+    monitors[i]->AddPrivacyBitmask( zones );
+    Debug( 1, "Loaded monitor %d(%s), %d zones", id, name, n_zones );
+  }
+  if ( mysql_errno( &dbconn ) )
+  {
+    Error( "Can't fetch row: %s", mysql_error( &dbconn ) );
+    exit( mysql_errno( &dbconn ) );
+  }
+  // Yadda yadda
+  mysql_free_result( result );
+
+  return( n_monitors );
+}
+
+int Monitor::LoadFfmpeghwMonitors( const char *file, Monitor **&monitors, Purpose purpose )
+{
+    std::string sql = "select Id, Name, ServerId, Function+0, Enabled, LinkedMonitors, Path, Method, Options, Width, Height, Colours, Palette, Orientation+0, Deinterlacing, Brightness, Contrast, Hue, Colour, EventPrefix, LabelFormat, LabelX, LabelY, LabelSize, ImageBufferCount, WarmupCount, PreEventCount, PostEventCount, StreamReplayBuffer, AlarmFrameCount, SectionLength, FrameSkip, MotionFrameSkip, AnalysisFPS, AnalysisUpdateDelay, MaxFPS, AlarmMaxFPS, FPSReportInterval, RefBlendPerc, AlarmRefBlendPerc, TrackMotion, Exif from Monitors where Function != 'None' and Type = 'Ffmpeghw'";
+  if ( file[0] ) {
+    sql += " AND Path = '";
+    sql += file;
+    sql += "'";
+  }
+  if ( staticConfig.SERVER_ID ) {
+    sql += stringtf( " AND ServerId=%d", staticConfig.SERVER_ID );
+  }
+  Debug( 1, "Loading FFMPEG H/W Monitors with %s", sql.c_str() );
+  MYSQL_RES *result = zmDbFetch( sql.c_str() );
+  if ( ! result ) {
+    Error( "Cannot load FfmpegMonitors" );
+    exit( mysql_errno( &dbconn ) );
+  }
+
+  int n_monitors = mysql_num_rows( result );
+  Debug( 1, "Got %d monitors", n_monitors );
+  delete[] monitors;
+  monitors = new Monitor *[n_monitors];
+  for( int i = 0; MYSQL_ROW dbrow = mysql_fetch_row( result ); i++ )
+  {
+    int col = 0;
+
+    int id = atoi(dbrow[col]); col++;
+    const char *name = dbrow[col]; col++;
+    unsigned int server_id = dbrow[col] ? atoi(dbrow[col]) : 0; col++;
+    int function = atoi(dbrow[col]); col++;
+    int enabled = atoi(dbrow[col]); col++;
+    const char *linked_monitors = dbrow[col]; col++;
+
+    const char *path = dbrow[col]; col++;
+    const char *method = dbrow[col]; col++;
+    const char *options = dbrow[col]; col++;
+
+    int width = atoi(dbrow[col]); col++;
+    int height = atoi(dbrow[col]); col++;
+    int colours = atoi(dbrow[col]); col++;
+    /* int palette = atoi(dbrow[col]); */ col++;
+    Orientation orientation = (Orientation)atoi(dbrow[col]); col++;
+    unsigned int deinterlacing = atoi(dbrow[col]); col++;
+    int brightness = atoi(dbrow[col]); col++;
+    int contrast = atoi(dbrow[col]); col++;
+    int hue = atoi(dbrow[col]); col++;
+    int colour = atoi(dbrow[col]); col++;
+
+    const char *event_prefix = dbrow[col]; col++;
+    const char *label_format = dbrow[col]; col++;
+
+    int label_x = atoi(dbrow[col]); col++;
+    int label_y = atoi(dbrow[col]); col++;
+    int label_size = atoi(dbrow[col]); col++;
+
+    int image_buffer_count = atoi(dbrow[col]); col++;
+    int warmup_count = atoi(dbrow[col]); col++;
+    int pre_event_count = atoi(dbrow[col]); col++;
+    int post_event_count = atoi(dbrow[col]); col++;
+    int stream_replay_buffer = atoi(dbrow[col]); col++;
+    int alarm_frame_count = atoi(dbrow[col]); col++;
+    int section_length = atoi(dbrow[col]); col++;
+    int frame_skip = atoi(dbrow[col]); col++;
+    int motion_frame_skip = atoi(dbrow[col]); col++;
+    double analysis_fps = dbrow[col] ? strtod(dbrow[col], NULL) : 0; col++;
+    unsigned int analysis_update_delay = strtoul(dbrow[col++], NULL, 0);
+    int capture_delay = (dbrow[col]&&atof(dbrow[col])>0.0)?int(DT_PREC_3/atof(dbrow[col])):0; col++;
+    int alarm_capture_delay = (dbrow[col]&&atof(dbrow[col])>0.0)?int(DT_PREC_3/atof(dbrow[col])):0; col++;
+    int fps_report_interval = atoi(dbrow[col]); col++;
+    int ref_blend_perc = atoi(dbrow[col]); col++;
+    int alarm_ref_blend_perc = atoi(dbrow[col]); col++;
+    int track_motion = atoi(dbrow[col]); col++;
+    bool embed_exif = (*dbrow[col] != '0'); col++;
+
+    Camera *camera = new FfmpegCamera(
+      id,
+      path, // File
+      method,
+      options,
+      width,
+      height,
+      colours,
+      brightness,
+      contrast,
+      hue,
+      colour,
+      purpose==CAPTURE,
+      1
     );
 
     monitors[i] = new Monitor(
@@ -2692,6 +2837,8 @@
 }
 #endif // HAVE_LIBAVFORMAT
 
+
+
 Monitor *Monitor::Load( unsigned int p_id, bool load_zones, Purpose purpose )
 {
   std::string sql = stringtf( "select Id, Name, ServerId, Type, Function+0, Enabled, LinkedMonitors, Device, Channel, Format, V4LMultiBuffer, V4LCapturesPerFrame, Protocol, Method, Host, Port, Path, Options, User, Pass, Width, Height, Colours, Palette, Orientation+0, Deinterlacing, RTSPDescribe, Brightness, Contrast, Hue, Colour, EventPrefix, LabelFormat, LabelX, LabelY, LabelSize, ImageBufferCount, WarmupCount, PreEventCount, PostEventCount, StreamReplayBuffer, AlarmFrameCount, SectionLength, FrameSkip, MotionFrameSkip, AnalysisFPS, AnalysisUpdateDelay, MaxFPS, AlarmMaxFPS, FPSReportInterval, RefBlendPerc, AlarmRefBlendPerc, TrackMotion, SignalCheckColour, Exif from Monitors where Id = %d", p_id );
@@ -2896,12 +3043,36 @@
       contrast,
       hue,
       colour,
-      purpose==CAPTURE
+      purpose==CAPTURE,
+      0
+    );
+#else // HAVE_LIBAVFORMAT
+    Fatal( "You must have ffmpeg libraries installed to use ffmpeg cameras for monitor %d", id );
+#endif // HAVE_LIBAVFORMAT
+  }
+    else if ( type == "Ffmpeghw" )
+  {
+#if HAVE_LIBAVFORMAT
+    camera = new FfmpegCamera(
+      id,
+      path.c_str(),
+      method,
+      options,
+      width,
+      height,
+      colours,
+      brightness,
+      contrast,
+      hue,
+      colour,
+      purpose==CAPTURE,
+      1
     );
 #else // HAVE_LIBAVFORMAT
     Fatal( "You must have ffmpeg libraries installed to use ffmpeg cameras for monitor %d", id );
 #endif // HAVE_LIBAVFORMAT
   }
+  
   else if (type == "Libvlc")
   {
 #if HAVE_LIBVLC
diff -Naur ../src-versions/src-orig/zm_monitor.h src/zm_monitor.h
--- ../src-versions/src-orig/zm_monitor.h	2017-05-14 22:01:18.765897796 +0000
+++ src/zm_monitor.h	2017-05-14 22:02:26.305693423 +0000
@@ -429,6 +429,7 @@
   static int LoadFileMonitors( const char *file, Monitor **&monitors, Purpose purpose );
 #if HAVE_LIBAVFORMAT
   static int LoadFfmpegMonitors( const char *file, Monitor **&monitors, Purpose purpose );
+  static int LoadFfmpeghwMonitors( const char *file, Monitor **&monitors, Purpose purpose );
 #endif // HAVE_LIBAVFORMAT
   static Monitor *Load( unsigned int id, bool load_zones, Purpose purpose );
   //void writeStreamImage( Image *image, struct timeval *timestamp, int scale, int mag, int x, int y );

Chris

[iconnor]

That's really weird. 24bit vs 32bit I mean..

Thing is that most of these CPU instructions work best on 32bit or more.... obviously 24bit uses less ram... but everything else uses less cpu.
I'm wondering if you are low on ram so other effects are coming into play?

I find it interesting that 24bit makes zmc use more... because zmc shouldn't be doing any more processing either way... just allocating more RAM.
All the motion detection code (and hence the code we are talking about) gets used in zma..

So... some really interesting results... that are going to need more analysis.

[mastertheknife]

Regarding the 24bit vs 32bit debate, i have an AArch64 VPS from Scaleway (ARM64-2GB) running 1.30.4 with the neon stuff in it.
So i did some tests:

ZoneMinder Version: v1.30.4 with ARM Neon code merged in
OS: Ubuntu 16.04.2 LTS Server AArch64
CPU: 2 cores of Cavium ThunderX
RAM: 2GB, unknown speed
Monitor: CURL source type, 800x600, MJPEG stream: http://206.140.121.226/mjpg/video.mjpg
ZM_FAST_IMAGE_BLENDS: On


ZM_CPU_EXTENSIONS=off, 24bit color:

Code: Select all

PIDSTAT zmc : Reporting average of 12 readings at 5 seconds intervals

ZMC Process ID : 14302 ==> ( 5.40 + 5.60 + 5.60 + 5.40 + 5.20 + 5.80 + 5.40 + 5.00 + 5.40 + 5.00 + 5.80 + 6.20 ) / 12 
AVERAGE : 5.48

PIDSTAT zma : Reporting average of 12 readings at 5 seconds intervals

ZMA Process ID : 14305 ==> ( 6.59 + 6.20 + 6.40 + 6.40 + 6.20 + 6.60 + 6.20 + 6.00 + 6.40 + 6.20 + 6.00 + 6.40 ) / 12 
AVERAGE : 6.30

ZM_CPU_EXTENSIONS=off, 32bit color:

Code: Select all

PIDSTAT zmc : Reporting average of 12 readings at 5 seconds intervals

ZMC Process ID : 13593 ==> ( 5.00 + 4.80 + 5.00 + 5.20 + 5.00 + 3.40 + 4.80 + 6.20 + 6.60 + 6.20 + 6.40 + 6.00 ) / 12 
AVERAGE : 5.38

PIDSTAT zma : Reporting average of 12 readings at 5 seconds intervals

ZMA Process ID : 13596 ==> ( 7.20 + 7.00 + 7.80 + 7.20 + 7.20 + 4.80 + 7.20 + 7.80 + 8.40 + 8.00 + 8.60 + 7.40 ) / 12 
AVERAGE : 7.38

ZM_CPU_EXTENSIONS=on, 24bit color:

Code: Select all

PIDSTAT zmc : Reporting average of 12 readings at 5 seconds intervals

ZMC Process ID : 14614 ==> ( 5.40 + 5.40 + 5.80 + 6.20 + 6.00 + 5.80 + 5.60 + 5.20 + 5.20 + 4.80 + 5.40 + 5.60 ) / 12 
AVERAGE : 5.53

PIDSTAT zma : Reporting average of 12 readings at 5 seconds intervals

ZMA Process ID : 14617 ==> ( 3.80 + 4.20 + 4.20 + 4.20 + 4.00 + 4.40 + 4.20 + 4.00 + 3.80 + 4.00 + 4.00 + 4.00 ) / 12 
AVERAGE : 4.07

ZM_CPU_EXTENSIONS=on, 32bit color:

Code: Select all

PIDSTAT zmc : Reporting average of 12 readings at 5 seconds intervals

ZMC Process ID : 14898 ==> ( 5.80 + 5.80 + 5.80 + 6.00 + 6.40 + 6.00 + 6.00 + 5.80 + 5.19 + 5.60 + 5.60 + 4.20 ) / 12 
AVERAGE : 5.68

PIDSTAT zma : Reporting average of 12 readings at 5 seconds intervals

ZMA Process ID : 14901 ==> ( 2.20 + 2.60 + 2.80 + 2.40 + 2.40 + 2.80 + 2.80 + 2.20 + 2.40 + 2.40 + 2.60 + 1.80 ) / 12 
AVERAGE : 2.45

So with ZM_CPU_EXTENSIONS on, the ARM Neon version of fast blend is used for both 24bit and 32bit. However, the ARM Neon version for delta is only used with 32bit colour. So while 24bit is faster without Neon, because its smaller, 32bit seems to be faster when Neon is enabled.
But to be fair, although not evident with 32bit color, it seems that this processor (Cavium ThunderX) really benefits from Neon, as can be seen in the benchmarks i posted here:
https://github.com/ZoneMinder/ZoneMinder/pull/1881
The reason is probably because its a server CPU and the RAM there is probably DDR3, DDR4 or better, while the Raspberry Pi 3 uses LPDDR2.

By the way, the script didn't work right away. The CPU% usage column in pidstat is number 8 here, not 7, but then the average didn't work because it was cutting the wrong field. So i went for direct offsets.
While at it, i made it faster by sampling zmc and zma at the same time:
https://gist.github.com/mastertheknife/ ... 6a87dd399d


Kfir

[cmisip]

Regarding the script, I ran into those issues with bash sometimes. Different flavors of linux or updates change the fields. Thanks for fixing.

Has there been any work on using motion vectors to detect motion in video frames? As a consequence of h264 software decode, the macroblock motion vectors are available as side info. The proper parameter needs to be passed before opening the decoder. You get source and destination x and y values. The distance the macroblock traveled if crosses a minimal threshold would indicate motion and enough macroblocks moved, then you can say motion was detected in the frame. This might cut down the work of zma when zmc is doing software decode of h264.

The motion vectors unfortunately are not available on hardware h264 decode on the Pi. The decoded frames I think need to be passed to the hardware encoder to reveal the sideinfo. Don't know if ffmpeg is extracting this info when it is using the hardware encoder.

Chris

[cmisip]

I'm trying to grasp the inner workings of zm_zone checkalarm. It seems that each y line is checked from lo_x to hi_x within extents defined by the zone's polygon for that line. How does the program handle multiple lines of lo_x and hi_x when each y is intersected by the polygon more than once?

I saw bool isInside(const Coord &coord) const; in zm_poly.cpp. Is this the way to check if a certain pixel coordinate is within the bounds of the zone's polygon? I am trying to find an easy way of testing if a coordinate is within the bounds of the polygon for that zone.

Thanks
Chris

[cmisip]

I wrote a test code based on ffmpeg's extract_mvs example to see how the motion vectors could be extracted and processed from an h264 stream. Can't guarantee that it will work and its probably got tons of bugs. It should compile against the latest ffmpeg-3.x. In Ubuntu xenial, pkg-config does not link to the proper libraries so I needed to use the following options:

Code: Select all

-I/usr/include/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu/libavcodec.so -L /usr/lib/x86_64-linux-gnu/libavutil.so -lavcodec -lavutil -L /usr/lib/x86_64-linux-gnu/libavutil -lavformat `pkg-config --libs opencv` -L /usr/lib/x86_64-linux-gnu-libswscale.so -lswscale   -lpthread -lboost_thread -lboost_system  

The code is at

https://github.com/cmisip/h264_motion_v ... r/main.cpp

What is the possibility of including motion_vector analysis as an alternative to pixel analysis in zoneminder? Is there an idea to explore that direction?
I don't really know if from a performance standpoint it will be better. But the motion vectors are there already as a consequence of h264 software decode.

Thanks,
Chris

[cmisip]

Maybe you can give me a little hint.

How is information passed from Monitor::Capture to Monitor::Analyse?

I have motion vector information saved in a uint8_t * pointer and it is present in image_buffer[index] in Monitor::Capture. However it is not present in image_buffer[index] in Monitor::Analyse.

I had assumed image_buffer[] updates in Monitor::Capture would show up in Monitor::Analyze because they are referring to the same array.

Thanks,
Chris

[cmisip]

I figured it out. Needed to extend mem_ptr to allocate for the sidedata info. I can see the sideinfo in Monitor::Analyse now but I segfault when I create a vector from it.

Chris

[iconnor]

I love the efforts here. Good stuff.

I might suggest you start familiarising yourself with github so we can follow/help/collaborate easier.

[cmisip]

Thanks,
I will look into that. I am unraveling the code piece by piece. I still dont know why I get a segfault with vector creation in Monitor::Analyse. Do you know why? Hopefully, I might not need to create a std::vector. I assumed that it was because the vector was referring to memory addresses that existed in the memory space of zmc. So I dereferenced all the values in Capture and packed them into a uint8_t buffer and tucked that at the end of mem_ptr. I can read them in Monitor::Analyze, but just can't build a std::vector from them.

I'm now trying to build an overload of Zone::CheckAlarm that take the buffer as input. If I understand correctly, I really only need to set the values of class member score and alarm_centre. Since each zone has a polygon ( I assume from a list of Coord points created when the zone shape was created), I am using the polygon::isInside to test for inclusion in the polygon. I recognize the form of that function. DetectMotion has an extra parameter to tell it wether to work on images or motion vector buffers. I run DetectMotion on the motion vector buffer to get a score and then run it again with the image to see the difference. They seem to both be providing consistent results ( scores > 0 when there is motion). But every now and then the score from motion vector DetectMotion would spike up to 8000 plus with no motion and I wonder if it is because it may be from the type of frame that is the source of the vectors. I am going to have to figure that out and exclude that scenario by looking at the pict_type. Some extra work needed to figure out a fast filtering solution to remove isolated motion vectors because they probably dont describe real motion. I thought I had a solution with using std::partition, but I can't build a std::vector. And a weighting scheme need to be devised because macroblocks would come in different sizes. The bigger macroblocks 16x16 should count more than 4x4 blocks. I am thinking I would just pack more vectors into the buffer with the dst_x and dst_y adjusted to +4 in the x and y dimension (i.e. a 16x16 block would be 16 4x4 blocks contiguosly). So this dissolves all macroblocks into 4x4 size and the weighting would be a measure of their number. A proper algorithm for finding out the centre of the vectors need to be devised as well to get alarm_centre.

For now the memory for mv_buffer is preallocated with Image creation and is a new member of class Image. It probably shouldn't be this way as other video sources may not need it. However, at the risk of code complexity, it would probably be better to do it this way as the buffers are read from shared memory in one iterative line and therefore needs to be constant size. The size of the buffer probably needs to be dependent on the size of the video. For now, I am working on a single case scenario.

As a side effect or maybe its completely unrelated, streaming via the UI stopped working. Everything else works. I have to track that down. I have to recompile and reinstall the deb file each time I make a change and I am wondering if that is the cause of the issue.

I'm not really a programmer. I went through learncpp.com lessons. And I regularly post questions in cplusplus.com/forums when I hit a wall. So can't promise anything fruitful. But I will continue to try. At the very least, I am learning more and more.

Thanks,
Chris

[cmisip]

Making some headway. I got a working solution now. Will have to do some more testing and post it on git once I figure out how to do that. I dont see the spikes in motion vectors anymore. The spikes might be the cause of the std::vector creation segfaults in Monitor::Analyse. It was due to me miscalculating the size of the vector buffer. I was memcpying past the end of the buffer into the next one. The macroblocks are dissolved into 4x4 size. I dont have any filtering solution though. My algorithm depended on c++11 standard with use of lambda. There are not a lot of macroblocks even dissolved into 4x4 size. I thought I could just multiply the number of vectors x4 to get number of pixels. The scores are lower compared to alarmed_pixels. Alarm_centre is just the average of all the vectors coordinates. I have tested Active Zone with custom shaped polygon. But all the other zones should work. A put in a new type of Monitor Function, Mvdect. I also put in a new type of camera Ffmpeghw for the use of h264_mmal. The nph-zms problem was completely unrelated. I did some database changes to make it work as well as some web UI changes. There needs to be three additional config options for Mvdect, for now hardcoded in zm_zone.cpp.

Thanks,
Chris

[cmisip]

I am trying to run master. I successfully built an ubuntu 16.04 package but there are issues that prevent me from testing. Some I was able to workaround. I thought I'd post here.

Install problem

Code: Select all

Unpacking zoneminder (1.28.108-nmu2015100101) ...
Setting up zoneminder (1.28.108-nmu2015100101) ...
Use of uninitialized value $Config{"ZM_DIR_EVENTS"} in pattern match (m//) at /usr/bin/zmupdate.pl line 81.
Use of uninitialized value in concatenation (.) or string at /usr/bin/zmupdate.pl line 81.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/ZoneMinder/Logger.pm line 154.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/ZoneMinder/Logger.pm line 199.

Database already at version 1.31.1, update aborted.

Use of uninitialized value $Config{"ZM_DIR_EVENTS"} in pattern match (m//) at /usr/bin/zmupdate.pl line 81.
Use of uninitialized value in concatenation (.) or string at /usr/bin/zmupdate.pl line 81.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/ZoneMinder/Logger.pm line 154.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/ZoneMinder/Logger.pm line 199.

Freshening configuration in database
Loading config from DB 207 entries
Saving config to DB 207 entries

There was an issue with creating the sql command to insert a new monitor. This is despite setting

Code: Select all

set GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES';
set SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES';

The sql command that was generated by the web UI when adding a new monitor:

Code: Select all

INSERT INTO Monitors SET LinkedMonitors = NULL, Name = 'Monitor-1', ServerId = NULL, Type = 'Ffmpeg', Function = 'Monitor', Enabled = '1', RefBlendPerc = '6', AlarmRefBlendPerc = '6', AnalysisFPS = NULL, MaxFPS = NULL, AlarmMaxFPS = NULL, Device = '/dev/video0', Channel = '0', Format = '255', Palette = '0', V4LMultiBuffer = NULL, V4LCapturesPerFrame = '1', Protocol = 'rtsp', Host = NULL, Port = '80', SaveJPEGs = '3', VideoWriter = '0', EncoderParameters = '# Lines beginning with # are a comment \r\n# For changing quality, use the crf option\r\n# 1 is best, 51 is worst quality\r\n#crf=23', RecordAudio = '0', RTSPDescribe = '0', LabelFormat = '%N - %d/%m/%y %H:%M:%S', LabelX = '0', LabelY = '0', LabelSize = '1', ImageBufferCount = '50', WarmupCount = '25', PreEventCount = '25', PostEventCount = '25', StreamReplayBuffer = '1000', AlarmFrameCount = '1', EventPrefix = 'Event-', SectionLength = '600', FrameSkip = '0', MotionFrameSkip = '0', AnalysisUpdateDelay = '0', FPSReportInterval = '1000', DefaultView = 'Events', DefaultRate = '100', DefaultScale = '100', WebColour = 'red', Exif = '0', SignalCheckColour = '#0000c0', Path = 'rtsp://admin:admin@192.168.0.78/12', Method = 'rtpRtsp', Options = NULL, Colours = '3', Width = '704', Height = '480', Orientation = '0', Deinterlacing = '0', Sequence = '1';

I set Protocol to 'rtsp' and Options to "" because they cannot be NULL. Sequence had a missing '. Corrected sql command will run in mysql:

Code: Select all

INSERT INTO Monitors SET LinkedMonitors = NULL, Name = 'Monitor-1', ServerId = NULL, Type = 'Ffmpeg', Function = 'Monitor', Enabled = '1', RefBlendPerc = '6', AlarmRefBlendPerc = '6', AnalysisFPS = NULL, MaxFPS = NULL, AlarmMaxFPS = NULL, Device = '/dev/video0', Channel = '0', Format = '255', Palette = '0', V4LMultiBuffer = NULL, V4LCapturesPerFrame = '1', Protocol = 'rtsp', Host = NULL, Port = '80', SaveJPEGs = '3', VideoWriter = '0', EncoderParameters = '# Lines beginning with # are a comment \r\n# For changing quality, use the crf option\r\n# 1 is best, 51 is worst quality\r\n#crf=23', RecordAudio = '0', RTSPDescribe = '0', LabelFormat = '%N - %d/%m/%y %H:%M:%S', LabelX = '0', LabelY = '0', LabelSize = '1', ImageBufferCount = '50', WarmupCount = '25', PreEventCount = '25', PostEventCount = '25', StreamReplayBuffer = '1000', AlarmFrameCount = '1', EventPrefix = 'Event-', SectionLength = '600', FrameSkip = '0', MotionFrameSkip = '0', AnalysisUpdateDelay = '0', FPSReportInterval = '1000', DefaultView = 'Events', DefaultRate = '100', DefaultScale = '100', WebColour = 'red', Exif = '0', SignalCheckColour = '#0000c0', Path = 'rtsp://admin:admin@192.168.0.78/12', Method = 'rtpRtsp', Options = '', Colours = '3', Width = '704', Height = '480', Orientation = '0', Deinterlacing = '0', Sequence = '1';

I think this needs to be added to /etc/zm.conf

Code: Select all

#add this to /etc/zm at least in ubuntu
ZM_PATH_SOCKS=/tmp/zm

I also needed to do this:

Code: Select all

sudo chmod 740 /etc/zm/zm.conf
chown root:www-data /etc/zm/zm.conf

The following error I dont have a workaround. It seems, the mmap file is missing the path string.

Code: Select all

FAT [zmc_m1] [Can't open memory map file /zm.mmap.1, probably not enough space free: Permission denied]
Jun 24 09:59:41 myxps8100 zmdc[19416]: ERR ['zmc -m 1' exited abnormally, exit status 255]
Jun 24 09:59:51 myxps8100 zmwatch[19455]: ERR [Memory map file '/zm.mmap.1' does not exist.  zmc might not be running.]

I think, it should be :

Code: Select all

/dev/shm/zm.mmap.1

If I find a solution to this, I would be able to test the motion vector stuff with the latest master branch. I have it working on an older version of master.

Thanks,
Chris

[cmisip]

Found the solution. /etc/zm/zm.conf did not contain the config parameters required. I found that I had to put in:

Code: Select all

ZM_PATH_SOCKS=/tmp/zm
ZM_PATH_MAP=/dev/shm
ZM_DIR_EVENTS=/usr/share/zoneminder/events
ZM_DIR_IMAGES=/usr/share/zoneminder/images
ZM_PATH_ZMS=/zm/cgi-bin/nph-zms
ZM_PATH_SWAP=/tmp

I also had to :

Code: Select all

mkdir /usr/share/zoneminder/events
mkdir /usr/share/zoneminder/images
sudo chown -R www-data:www-data /usr/share/zoneminder/

Thanks,
Chris