The circumstances are when the large image has a totally flat (constant) color the size of the small image such as the upper right corner of the large image where it is perfectly flat white. The standard deviation for that subsection will be zero causing a divide by zero in the formulae.
In actuality, in my FFT NCC, the divide by zero is handle OK, but values within some threshold are the ones that are causing the issue. I might imagine that this will be the same for the compare -metric ncc.
ncc=(Small-MeanS)*subimage((Large-MeanL))/(Ns*StdS*subimage(StdL))
where subimage(StdL) is the standard deviation of the subimage in the large image of size equal to the small image.
See http://en.wikipedia.org/wiki/Cross-correlation for the full expression of the NCC.
Any way, I just fixed my FFT version by testing the standard deviation of the subsection to see if it is too close to zero (both positive and negative differences).
I used -fx "(abs(u)<0.002)?1:u", which worked fine.
I reimplemented this via a mask composition with a white image to avoid using the slower -fx. However, this might be a nice new function (a bit similar to -perceptible, except it would have a user specified value to set the result to in addition to the threshold value.
In any case, I just wanted to let you know about this. Here are my tests.
Large Image: wizardsq.png
Small Image (subsection 50x50+130+80) : wizard_sub_130_80.png
FFT NCC: normcrosscorr
time imh normcrosscorr wizard_sub_130_80.g wizardsq.png null:
Match Coords: (130,80) And Score In Range 0 to 1: (1.00003)
real 0m20.700s
user 0m17.173s
sys 0m4.012s
Compare NCC:
time compare -metric ncc -subimage-searchizardsq.png wizard_sub_130_80.png null:
0 @ 196,0
real 4m1.259s
user 4m0.775s
sys 0m0.435s
Compare RMSE:
time compare -metric rmse -subimage-search wizardsq.png wizard_sub_130_80.png null:
0 (0) @ 130,80
real 0m51.298s
user 0m50.964s
sys 0m0.326s