Linux: mmap()-ing instead of read()-ing input files

Questions and postings pertaining to the development of ImageMagick, feature enhancements, and ImageMagick internals. ImageMagick source code and algorithms are discussed here. Usage questions which are too arcane for the normal user list should also be posted here.
Post Reply
mtodorov3_69

Linux: mmap()-ing instead of read()-ing input files

Post by mtodorov3_69 » 2006-07-14T12:05:03-07:00

Hi!

I faced a situation with convert when it consumes 900+ MB quickly, when only 58 MB of input PNGs were present. I doubt PNGs have such compression rate.

In fact, a lot of mem could be saved by mmap() of input files instead of open()/malloc()/read()

From my experience, it is twice as fast for larger files. Kernel will pagein files and then pageout those pages which are probably uncompressed and we no longer need them. besides, ti is space efficient, as no memory is allocated by malloc(), kernel does this job for us through virtual memory subsystem.

Is it too high of a complexity to implement in IM 6.2.6+?

I have experimented with this Unix call, and of course there can and should always be fallback procedure for "classic" read() in case mmap() fails. I use it this way in my clustering tools, and it works.

But of course, all system libraries are mapped that way.

Thanks for considering this!

Best regards,
Marvin

Post Reply