Montage of lots of files. Lots.

[arizonagroovejet]

I'm trying to produce a montage of a lot of files. This many

Code: Select all

mike@continuity:~/Videos/bttfimages_smaller$ ls | wc -l
166887

Each image is a jpeg 160x88 and less than 15KB.

My first attempt failed:

Code: Select all

mike@continuity:~/Videos/bttfimages_smaller$ montage '*' -tile 409x409 ../bttfmontage.jpg
montage: MemoryAllocationFailed `????\??L*f' @ fatal/splay-tree.c/IterateOverSplayTree/1002.

So I set some memory limits which got rid of the MemoryAllocationFailed error but now I get this error and I can't figure out what it means or how to avoid it.

Code: Select all

mike@continuity:~/Videos/bttfimages_smaller$ montage '*' -limit memory 1000mb -limit map 1000mb -tile 409x409 ../bttfmontage.jpg
montage: UnableToAcquireString ??????*??(???' @ fatal/string.c/AcquireString/126.

Is it possible to montage that many images or am I being over ambitious?

Code: Select all

Version: ImageMagick 6.6.1-0 2010-07-29 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC
Features: OpenMP 

32bit Linux (openSUSE 11.3). 4GB ram. 11GB free space for /tmp, 50GB free space for home directory.

[brianw]

I don't know anything about your specific error messages and I probably won't help, but have you thought about dividing your files up and then montage-ing them?

You've got 16k files. That's a lot of files, bro. Can you try creating a separate montage for each 1k files, and then maybe appending the resulting 16 files together?

[magick]

Try this command:

• montage -monitor -limit area 0 -define registry:temporary-path=~ -define jpeg:size=160x160 \
  -geometry 160x160 '*[160x160]' montage.png

[arizonagroovejet]

Try this command:

• montage -monitor -limit area 0 -define registry:temporary-path=~ -define jpeg:size=160x160 \
  -geometry 160x160 '*[160x160]' montage.png

Thanks for the suggestion, but it got as far as file 123126 then output 'Memory allocation failed' and stopped doing anything. I hit ctrl-c and my harddisk is now thrashing clearing up ~10GB of files with names like magick-XXCnCZ9c. Ho hum. Maybe I should have also used memory and map limits?

You've got 16k files.

More than ten times that number in fact.

That's a lot of files, bro.

I know, mate

Can you try creating a separate montage for each 1k files, and then maybe appending the resulting 16 files together?

I'm going to try that. Doing it in a single command is neater and arguably cleverer, but it's no good when that single command chugs away for ours then spews and error. Multiple smaller operations seems a lot more attractive right now. I've previously been able to montage ~6k files in one go with no problems. I'll post back with how I get on.

[arizonagroovejet]

The few-at-a-time approach worked just fine. I split them up in to to batches of less than 10000, montaged them, then montaged some of the resulting montages together and then montaged some of the resulting montages together until I had one final montage. I could maybe have skipped a couple of steps but I didn't want to risk a repeat of waiting hours for a command to then fail.

One thing that did surprise me was the amount of disk space required. I have about 50GB free space on the /home partition and used that as the temporary space as suggested. It was just barely enough. Doing the final montage command used enough that I got a disk space warning pop up telling me I had less than 1GB free. It all got freed up again once ImageMagick cleaned up the temporary files when the montage was done, but given that the images I was montaging together were only a couple of hundred MB each and the resulting montage is ~700MB I'm surprised at the size of the temporary files.

Additionally, I'm currently running 'identify -verbose' on the ~700MB jpeg, it's been going a while now and there's an 18GB temporary file. Is this expected behaviour?

[magick]

See http://www.imagemagick.org/script/archi ... .php#cache. If you install ImageMagick @ Q8, the resource requirements are cut in half.