Page 3 of 3

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2013-12-18T15:25:40-07:00
by kaefert
Okey, So I've written myself two shell scripts:

one to do the tiling of an image that has already been converted into the mpc format:

Code: Select all

#!/bin/bash

#usage:
#./tile-mpc.sh wall2-right_merged_55697px.mpc 55697 871

mpcin=$1
picsize=$2
tilesize=$3

tiledir=$4
if [ -z ${tiledir+x} ]; then tiledir="tiles/"; fi

x=0
y=0

while [ $y -lt $picsize ]
do
  ya=$[$y/871]
  yl=$[$picsize-$y]
  yl=$(($yl<871?$yl:871))

  while [ $x -lt $picsize ]
  do
    xa=$[$x/871]
    xl=$[$picsize-$x]
    xl=$(($xl<871?$xl:871))
    
    convert -limit memory 32 -limit map 32 $mpcin -crop $xl"x"$yl+$x+$y +repage $tiledir$xa"_"$ya.jpg
    x=$[$x+871]
  done

  x=0
  y=$[$y+871]
done
And another one that will do the scaling necessary to get a deep-zoom pyramid - and call the above script repeatedly to tile each layer:

Code: Select all

#!/bin/bash

input=$1
tilesize=$2
if [ -z $tilesize ]; then tilesize=512; fi
skipfirsttile=$3

#dim=$(mediainfo --Output="Image;%Height%" $input)
dim=$(identify -format "%w" $input)
type=$(identify -format "%m" $input)

if [ $type = "MPC" ]
then
  echo "good, input is already an MPC, we don't need to convert it"
  mpc=$input
else
  echo "will convert input image to MPC format before tiling"

  mpc="${input%.*}"".mpc"
  convert -verbose -monitor -limit memory 2G -limit map 4G $input $mpc
fi

lvl=$(./salado-level-calc.sh $dim)
if [ $skipfirsttile -eq 1 ]
then
  echo "skipfirsttile parameter set, will continue with first resize operation now"
else
  echo "tiling original image (lvl $lvl) now"
  mkdir $lvl
  ./tile-mpc.sh $mpc $dim $tilesize "$lvl/"
fi

lvl=$[$lvl-1]
scale=2
mpcpre=$mpc
mpcl="${input%.*}""_lvl$lvl.mpc"
diml=$[$dim/$scale]

while [ $diml -gt $tilesize ]
do
  echo "will resize for lvl $lvl with scale $scale -> edge length $diml into $mpcl now"

  #for best quality, use $mpc as source, for faster processing, use $mpcpre
  convert -verbose -monitor -limit memory 2G -limit map 4G "$mpcpre" -resize $diml"x"$diml "$mpcl"

  echo "finished resize for lvl $lvl - will start tiling it now"
  mkdir $lvl
  ./tile-mpc.sh $mpcl $diml $tilesize "$lvl/"

  lvl=$[$lvl-1]
  scale=$[$scale*2]
  mpcpre=$mpcl
  mpcl="${input%.*}""_lvl$lvl.mpc"
  diml=$[$dim/$scale]
done

mkdir $lvl
convert -verbose -monitor -limit memory 2G -limit map 4G "$mpcpre" -resize $diml"x"$diml "$lvl/0_0.jpg"
update:
I've outsorced the functionality to find the right numbers for the pyramid-levels to an extra script, since I need floating point calculations for it:

Code: Select all

#!/bin/bash

# Default scale used by float functions.
float_scale=2

#####################################################################
# Evaluate a floating point number expression.

function float_eval()
{
    local stat=0
    local result=0.0
    if [[ $# -gt 0 ]]; then
        result=$(echo "scale=$float_scale; $*" | bc -q 2>/dev/null)
        stat=$?
        if [[ $stat -eq 0  &&  -z "$result" ]]; then stat=1; fi
    fi
    echo $result
    return $stat
}

#####################################################################
# Evaluate a floating point number conditional expression.

function float_cond()
{
    local cond=0
    if [[ $# -gt 0 ]]; then
        cond=$(echo "$*" | bc -q 2>/dev/null)
        if [[ -z "$cond" ]]; then cond=0; fi
        if [[ "$cond" != 0  &&  "$cond" != 1 ]]; then cond=0; fi
    fi
    local stat=$((cond == 0))
    return $stat
}

#####################################################################
# main script

dim=$1
scale=1
lvl=0
diml=$dim

while float_cond "$diml > 1"
do
  #echo "$diml - $lvl"
  scale=$[$scale*2]
  diml=$(float_eval "$dim / $scale")
  lvl=$[$lvl+1]
done

echo $lvl

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-29T07:24:14-07:00
by kaefert
so sadly this doesn't work anymore with new imagemagick versions. I've just converted a 2gb lzw compressed tiff file into an 93gb mpc (cache) file but upon trying to use this mpc file in another imagemagick call it copies it to the tmp directory instead of just reading it directly.

Version: ImageMagick 7.0.8-39 Q16 x86_64 2019-04-07

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-29T10:42:54-07:00
by fmw42
I do not think MPC formats are not the same in IM 6 and IM 7. So the same data will not work if create in IM 6 and processed in IM 7.

In IM 7, use magick, not convert. Also use magick identify, not identify.

See https://imagemagick.org/script/porting.php#cli

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-29T11:17:41-07:00
by kaefert
the mpc file has been created with IM7 and I'm using it with IM7. The problem is not that there is some incompatibility throwing an error, but that it's completely read and copied to a new temporary pixelcache file of the exact same size instead of it beeing just used directly like with whatever version I used in 2013. That ruins the performance of course since the associated cache file is 93gb in size and I was looking for the performance I got in 2013...

I just replaced all the "convert" calls with "magick" which gladly seems to have the same exact syntax, but the behavior is the same.

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-29T11:40:18-07:00
by magick
Can you distill the problem down to a one or two command lines? We tried these commands and the MPC file was memory-mapped and did not create the pixel cache on disk:

Code: Select all

convert -limit area 0 logo: -resize 28000x28000 logo.mpc
convert -debug cache -limit map 4G logo.mpc -crop 1000x1000+0+0 logo.miff

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-29T12:51:53-07:00
by kaefert
I've tried your example and unlike with my own mpc file it doesn't copy the cache when the second call runs, but it still reads the whole cache instead of just the part that it needs for the cropped area (and therefore takes minutes instead of seconds)

update: I've ran the second line a second time and timed it by putting `date +%H:%M:%S:%N` before and after the call in a bash script and it took nearly 5 minutes for the single crop run:
21:48:37:947840937
21:53:23:464553612
for the cache debug output see: https://pastebin.com/raw/VXaFejNQ

instead trying to create a tile of my own mpc fails because the default /tmp/ location only has 8gb space instead of the 93gb it's trying to copy there:

Code: Select all

$ bash playground.sh 
22:05:34:476933087
2019-04-29T22:05:34+02:00 0:00.001 0.000u 7.0.8 Cache convert[20106]: cache.c/DestroyPixelCache/1040/Cache
  destroy 
2019-04-29T22:05:34+02:00 0:00.002 0.000u 7.0.8 Cache convert[20106]: cache.c/DestroyPixelCache/1040/Cache
  destroy 
2019-04-29T22:05:34+02:00 0:00.002 0.000u 7.0.8 Cache convert[20106]: cache.c/PersistPixelCache/4024/Cache
  attach persistent cache
2019-04-29T22:05:34+02:00 0:00.002 0.000u 7.0.8 Cache convert[20106]: cache.c/SetPixelCacheExtent/3577/Cache
  extend x_f.mpc[0] (x_f.cache[4], disk, 99.6135GB)
2019-04-29T22:05:34+02:00 0:00.002 0.000u 7.0.8 Cache convert[20106]: cache.c/OpenPixelCache/3949/Cache
  open x_f.mpc[0] (x_f.cache[4], Disk, 78904x78904x4 99.6135GB)
2019-04-29T22:05:34+02:00 0:00.003 0.000u 7.0.8 Cache convert[20106]: cache.c/SetPixelCacheExtent/3577/Cache
  extend x_f.mpc[0] (/tmp/magick-20106Wrx8-9GuyZRA[5], disk, 99.6135GB)
2019-04-29T22:05:34+02:00 0:00.003 0.000u 7.0.8 Cache convert[20106]: cache.c/OpenPixelCache/3949/Cache
  open x_f.mpc[0] (/tmp/magick-20106Wrx8-9GuyZRA[5], Disk, 78904x78904x4 99.6135GB)
2019-04-29T22:07:09+02:00 1:35.242 10.030u 7.0.8 Cache convert[20106]: cache.c/DestroyPixelCache/1040/Cache
  destroy x_f.mpc[0]
2019-04-29T22:07:10+02:00 1:35.954 10.730u 7.0.8 Cache convert[20106]: cache.c/SetPixelCacheExtent/3577/Cache
  extend x_f.mpc[0] (/tmp/magick-20106I3g7CAqYyiVV[5], disk, 99.6135GB)
2019-04-29T22:07:10+02:00 1:35.954 10.730u 7.0.8 Cache convert[20106]: cache.c/OpenPixelCache/3949/Cache
  open x_f.mpc[0] (/tmp/magick-20106I3g7CAqYyiVV[5], Disk, 78904x78904x4 99.6135GB)
2019-04-29T22:08:45+02:00 3:11.283 20.830u 7.0.8 Cache convert[20106]: cache.c/DestroyPixelCache/1040/Cache
  destroy x_f.mpc[0]
2019-04-29T22:08:46+02:00 3:11.990 21.520u 7.0.8 Cache convert[20106]: cache.c/DestroyPixelCache/1040/Cache
  destroy x_f.mpc[0]
convert: unable to persist pixel cache `x_f.mpc' @ error/mpc.c/ReadMPCImage/995.
convert: no images defined `crop1.jpg' @ error/convert.c/ConvertImageCommand/3300.
22:08:46:480322164
playground.sh contains:

Code: Select all

date +%H:%M:%S:%N
convert -monitor -verbose -debug cache x_f.mpc -crop 617x617+0+0 crop1.jpg
date +%H:%M:%S:%N

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-30T04:22:55-07:00
by kaefert
I've now dug up an older laptop of mine which still runs on Linux Mint 18.3 Sylvia with
ImageMagick version 6.8.9-9 Q16 x86_64 2018-09-28

With this my above scripts work with a reasonable speed. And the mpc cache file is only 50gb instead of 100gb in size.

Re: Imagemagick deepzoom - tiling and resizing huge pictures

Posted: 2019-04-30T17:02:17-07:00
by magick
Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ http://www.imagemagick.org/download/beta/ by sometime tomorrow.