Page 1 of 1

whiteboard unique keyframes

Posted: 2020-01-31T09:20:17-07:00
by bvsp
I want to identify unique keyframes from a video of a lecture (whiteboard).
Let's say I extract frames every 1 second and there are a total of 1000 frames.

Frames will be additive, meaning that:
- frame 1 will have just one word written in the whiteboard
- frame 4 will have two words
- frame 10 will have five words
- frame 100 will be the point of the most writing on the canvas, just before the erasing
- frame 105 will have one word
- frame 110 will have two words
- frame 250 will be the point that contains the most writing on the canvas, just before the erasing
etc.

The output should be frame 100 and frame 250 (keyframe 1 and 2 respectively).

How can the spatial information be considered?
I'm thinking of calculating the frame difference with compare ( http://www.imagemagick.org/Usage/compare/ ), but comparison should be made between frame 1-100 and 100-250 for example and not frame 5 with 105.

Re: whiteboard unique keyframes

Posted: 2020-01-31T10:40:55-07:00
by snibgo
What version of IM, on what platform?

You want the frames with the most writing, just before erasure. If your board really is white, then these are the darkest frames. As more words are written, the frames become darker. Then it suddenly becomes light again.

So your script takes the mean of each frame. This mean will decrease. When it suddenly increases, the previous frame is a keyframe.

Re: whiteboard unique keyframes

Posted: 2020-01-31T11:29:33-07:00
by bvsp
Linux, version 6.9.7

What about if there is occlusion by the writer? This would confuse the above method.
Also how do you suggest to get the darkest frame? histogram?

Re: whiteboard unique keyframes

Posted: 2020-01-31T11:42:15-07:00
by snibgo
To get the average darkness of a frame:

Code: Select all

convert frame.png -colorspace Gray -format %[fx:mean] info:
This gives a number between 0.0 (black) and 1.0 (maximum white).

If the writer occludes the whiteboard, there will be a major darkening of the frame, much more than writing a word, so readily detectable. But is the erasure done by a person standing there? Then you have problems.

If you post some sample frames (upload to anywhere on the web, and paste URLs here), that would help.

Re: whiteboard unique keyframes

Posted: 2020-01-31T12:12:26-07:00
by bvsp
Thanks snibgo!
Hm maybe I can check the gradual darkening of the picture and disregard the frame (the one with the person occlusion) if it happens to be not within some threshold.
The erasure will be done by the person standing there I guess.
I will try to prepare some sample frames.

Re: whiteboard unique keyframes

Posted: 2020-01-31T12:15:11-07:00
by bvsp
Also is there a way to crop the whiteboard region (large white rectangle)?
https://media.gettyimages.com/photos/sh ... d876659458

Re: whiteboard unique keyframes

Posted: 2020-01-31T13:07:04-07:00
by GeeMack
bvsp wrote:
2020-01-31T12:15:11-07:00
Also is there a way to crop the whiteboard region (large white rectangle)?
If the location of the whiteboard is constant, you can even stretch the entire whiteboard to fill the frame from corner to corner with an ImageMagick command like this...

Code: Select all

convert board.png -distort perspective "55,74 0,0 943,77 1024,0 36,514 0,683 962,512 1024,683" result.png
Edited to add: This command will perform the above stretching operation to an entire directory of PNG images...

Code: Select all

mogrify -distort perspective "55,74 0,0 943,77 1024,0 36,514 0,683 962,512 1024,683" *.png