How to identify dropframes/missing frames

[marvin]

Hello all,

I´ve been working with ImageMagick on a very basic level from time to time. Now I think I have come to a point where I would like to ask you guys out there for some advice:

I am trying to detect missing frames/images from a digital video camera which occasionally drops frames during recording. In other words there are some missing motion phases within my sequence of frames. The motion "jumps" in an unnatural way.

My initial thought was do do some image comparison with ImageMagick, but I can´t figure out how to determin a certain extent of "difference" as a normal motion portrayal and another "difference" for an abnormal "jump" of content when a frame is missing.

Needless to say that I have to deal with thousands of frames at a high resolution and speed of detection should be as fast as possible. And of course the "difference" between images in a scene with i.e. handheld camera can be pretty large without having any dropframes...

If anybody has an idea it would be greatly welcome!

THX, Marvin

[snibgo]

Code: Select all

compare -metric RMSE f200.png f201.png NULL:

will return a numerical measure of the difference between the two frames. Higher numbers are more different.

You could write a script to generate a list of such differences, then look for entries that are peaks, or are more than say 10% greater then the rolling average.

A different metric might work better.

[marvin]

Thanks snibgo, I wil give it a try asap.

Best, AMrvin

[snibgo]

It works quite well.

Code: Select all

#!/bin/bash

rm framediffs.lis

ffmpeg -i MPEG0004.AVI f%d.png

framenum=1
while [ $framenum -lt 50 ]
do
  file1=f$(( $framenum )).png
  file2=f$(( $framenum + 1 )).png
  compare -metric RMSE $file1 $file2 NULL: 2>diff.lis
  fdiff=`cat diff.lis |cut -d ' ' -f 1`
  echo $file1 $file2 $fdiff>>framediffs.lis
  framenum=$(( $framenum + 1 ))
done

cat framediffs.lis

# Simulate some dropped frames:
compare -metric RMSE f45.png f47.png NULL:
compare -metric RMSE f45.png f48.png NULL:
compare -metric RMSE f46.png f48.png NULL:
compare -metric RMSE f47.png f50.png NULL:

Output:

f1.png f2.png 3711.96
f2.png f3.png 3752.95
f3.png f4.png 7046.19
f4.png f5.png 4776.78
f5.png f6.png 5051.19
f6.png f7.png 5043.03
f7.png f8.png 5405.14
f8.png f9.png 5374.37
f9.png f10.png 5483.51
f10.png f11.png 5940.59
f11.png f12.png 6784.67
f12.png f13.png 7222.97
f13.png f14.png 7303.03
f14.png f15.png 7311.46
f15.png f16.png 7099.41
f16.png f17.png 10940.5
f17.png f18.png 435.007
f18.png f19.png 6755.05
f19.png f20.png 6525.19
f20.png f21.png 6042.01
f21.png f22.png 6397.25
f22.png f23.png 6700.4
f23.png f24.png 6750.85
f24.png f25.png 6995.79
f25.png f26.png 9769.39
f26.png f27.png 316.394
f27.png f28.png 5947.32
f28.png f29.png 6902.82
f29.png f30.png 6853.72
f30.png f31.png 6209.44
f31.png f32.png 5904.37
f32.png f33.png 6092.26
f33.png f34.png 9146.11
f34.png f35.png 363.379
f35.png f36.png 5954.34
f36.png f37.png 5821.97
f37.png f38.png 6192.74
f38.png f39.png 6218.78
f39.png f40.png 8910.85
f40.png f41.png 839.893
f41.png f42.png 6357.41
f42.png f43.png 6741.14
f43.png f44.png 6860.18
f44.png f45.png 6591.64
f45.png f46.png 6608.83
f46.png f47.png 6531.38
f47.png f48.png 9912.45
f48.png f49.png 0
f49.png f50.png 8646.73
9847.31 (0.15026)
13307.6 (0.20306)
11976.9 (0.182756)
13253.8 (0.202239)

It runs about 6 comparisons per second (on 640x480 frames). It shows that my ancient cheap stills camera (Lumicron LDC-825Z3) doesn't exactly drop frames, but does weird things at frames 17, 26, 34, 40 and 49. The simulated dropped frames produce distinctive numbers.

You could use more sophisticated image-matching techniques (see some recent posts in this forum, eg viewtopic.php?f=1&t=15641), but they would be much slower.

[marvin]

Hi snibgo,

thanks for the script. Unfortuately I am tied to Windows and will have to rewrite it...

Another nice sideeffect: It fairly detects duplicate frames as well: diff is simply 0.

Any idea how to catch the error which is produced when the images are too different? I tried to compare a black and white image but I get an error message: images too dissimilar...

Would be a nice scene detection or find single "wrong" images...

Best, Marvin

[snibgo]

I'm currently converting my brain from Windows.

The actual script isn't important, except as a proof of concept. You would have to determine how to process the output numbers, perhaps highlighting returned values that are either higher or lower than expected. As you say, 0 = no change. I reckon it shows the processor in my camera sometimes delays the capture of a frame (a large number from the previous frame, followed by a small number to the next frame).

In a production environment I would code it in MagickWand, which would save reading every image twice, and simplify calcs of rolling averages or whatever.

When images are too dissimilar, compare returns 1, instead of 0 = no error. Catch this in Unix with "$?"; in Windows with "ERRORLEVEL".

[marvin]

Hi snibgo,

just got a new Windows machine, so my brain will be windowed again... Thanks for your extensive help!

Marvin

[fmw42]

Any idea how to catch the error which is produced when the images are too different? I tried to compare a black and white image but I get an error message: images too dissimilar...

try adding -dissimilarity-threshold 1

that will allow it to work for all images and return a proper number for the rmse error rather than an error message saying they are too dissimilar.

see http://www.imagemagick.org/script/comma ... -threshold