Setting preview cache images to a standard size

[NeilR]

I am assuming that the new "Preview Cache" replaces the IM3.6 "offline cache".

In order to control the size of the cache we can set a quality level, and also a size as a percentage of the original image.

I have 350K images in IM3.6, and I have carefully calculated that an offline cache image size of 1400 pixels works for me, in terms of the size cache I want. 

I don't think I can do this now.  I have embedded JPG images in my raw files anywhere from 800 pixels (approximate) on D2h raw files to 4300 pixels on my 12 mpx raw files.  With new sensors of 36mpx now, and surely larger in the future, I would think most of us will accumulate a large variety of image sizes in our iMath catalogs.

Is there a way for me to specify a standard image size of n pixels, as in IM3.6?  if not, you should consider that you had a very simple system that worked, and I think it is a huge mistake to abandon it.

Some users may like relative size, but I would think many are like me.  I think it will be critically important to allow either method of size determination.  Relative percentage will not work for me, and it will create even more problems when I upgrade my cameras to larger megapixel sensors.

I considered making my thumbnail 1400 pixels.  Since I currently offline cache all my images, it is not clear to me why I would not want to do that, EXCEPT for the problems of backing up the database.  My iMatch DB is now about 10GB (with 400 pixel thumbnails), and my offline cache is now 30GB.   That would expand my iMatch 5 DB to somewhere between 30-40GB, and at that point it becomes a problem with daily backups because merely opening the DB would force a 40GB copy in my various daily backups. 

Having the images, which are basically stagnant, in a cache scheme outside the DB does have its advantages.    And of course, the DB, and the cache, only grows with time.  It never shrinks  .  What is 40GB today will be 100GB some years from now int eh not too distant future.  Plus, there may be issues with an IM database of 100GB or wherever it is headed in the future.

Just thinking out loud here.

[Richard]

I have embedded JPG images in my raw files anywhere from 800 pixels (approximate) on D2h raw files to 4300 pixels on my 12 mpx raw files.

Don't Cache JPEG Files
If you work mostly with files in JPEG format and you keep your files on-line all the time (e.g., on your hard disk) you can set this option to conserve disk space. If this option is on, IMatch does not create cache images from JPEG files anymore. Instead it accesses the original image when the image is displayed somewhere in IMatch.

Since IMatch cache images are by default 100% copies of the original image in JPEG format, it often makes no sense to produce cache images for JPEG files.

[NeilR]

Hi Richard,

In fact I do NOT have all my images online all the time.   My images are on a file server and I work from a laptop.  I would not think that an unusual setup.

My issue has nothing to do with JPG image files because we have an option to control that.  My issue has to do with setting the size of the cached preview files, regardless of image file format, in the case where a laptop or similar device is not always connected to the server.

As far as I am concerned, the old offline cache is one of the most important features of the old iMatch, hence my question.

[Richard]

In fact I do NOT have all my images online all the time.   My images are on a file server and I work from a laptop.  I would not think that an unusual setup.

I doubt that yours is an unusual setup. In reading the Cache topic in Help I saw nothing about limiting maximum image size. So your feature request makes sense. I only questioned JPEGs in a cache unless the originals were off-line. Which yours are.

[NeilR]

So your feature request makes sense. I only questioned JPEGs in a cache unless the originals were off-line. Which yours are.

It doesn't just "make sense"... it is imperative.

Some numbers...

My beta test image folder contains 7200 images but only 1400 of them are NEFs and cached (I have not flipped the switch to cache JPGs yet).  Those images consume 20GB of disk space.  The cache of only 19.4% of my images (the NEF's) consumes 1.00GB.  In real life 80% or more of the images in my cataloged folders are NEF, so in real life the cache would be more like 4GB, or about 20% of the size of the images.

My real life image collection is about 4TB, so, with those initial settings we are talking about a preview cache on the order of 800GB- that is simply not an acceptable situation on a laptop!   That is 20x the size of my current offline cache.

My cache is using the default 65 quality and 100% image size.

It is obvious to me that in real life, in a large or even moderate size image collection, that the defaults build a cache way too big.

I can limit the cache to a certain total size but then the result would be that only 5 or 10% of my images are cached, and it is not clear to me what that would accomplish.  And certainly for my purposes (an offline cache) I need 100% of the images cached, with no total limit on cache size.  I have to size the cache by tailoring the individual image sizes, not by limiting the total cache size.

I can reduce the image quality setting but that will only tinker around the margins unless I set it so low that the image quality is useless.

By process of elimination, that leaves me with setting the cache image size to some fixed percentage of the original.  Let's see how that will work...

My current cameras deliver images of 4256 pixels wide.  If I want to approximate my current 1400 pixel image I have to set the image size to 1400/4256 or 33%.  That takes care of my D300 and D700 images.

I have a very significant collection (50K images) of D2h images.  That was a peculiar camera (by modern standards) because its raw files  contain a preview JPG only about 800 pixels wide.  If I reduce those by 33% I will end up with cached images 264 pixels wide, smaller than the thumbnails embedded in the app!  That is certainly not acceptable.

Actually, those 800 pixel embedded jpg images are not acceptable either, and I solved that with many of my images by either forcing a re-edit in CaptureNX to deliver the correct 2464 x 1632 full size images, or using iMatch to re-render the images from the raw file (which was basically unworkable due to issues in the rendering routines).

So even in the case where I have the full size 2464 pixel wide embedded JPG, my preview image will be only 813 pixels wide, which is not acceptable.

You may think I have a "unique problem" with my 4mpx D2h but a lot of those cameras were sold, as well as other models with low resolution such 6 mpx sensors.  And all those owners that use iMatch will presumably be cataloging their old images taken with those cameras and they will not be happy with the result.  And most if not all those shooters have since upgraded to 12-36mpx cameras so they all have the same problem of dealing with these very disparate image sizes.

Even if I started with a 12 mpx camera, I am probably upgrading to a 24 or 36 mpx camera.  If I optimize the percentage for a 36mpx D800 image (7,360 x 4,912 pixels) I must use a percentage of 1400/4926 = 28%.

That 28% number will create previews for the 12 mpx camera only 1200 pixels wide- probably barely acceptable for most people.

I've gone into all this math to demonstrate that the current configuration options simply will not work for most users.  For me it is totally unacceptable, and it will be just as unacceptable for anyone with older 4-6mpx sensor images in their catalog.

Mario had it right 10 years ago with his initial condiguration scheme.  I hope he now understands that 

[Mario]

One of the design principles for IMatch 5 was make things easier. Less options.
In my experience from the past years, I can tell that most users use 100% cache files (off-line cache for IMatch). And they just want the largest possible size in IMatch 5 as well, because this allows them to use cached images also for features like the Batch Processor, Scripting, etc.

Hard disks are getting larger by the month, and disk space is cheap. When I designed the off-line cache system for IMatch 3, it was 2003. A large hard disk had maybe 50 GB. Now we have hard disks with multiple Terrabytes so donating 5%-20% of the total size of the images indexed in the database for a fast access cache is usually no problem at all.

These are the reasons why you can dial in only percentages and I strongly recommend to use 100% cache files all the time (except for JPEG files, if your files are always on-line). I've even pondered to remove the size setting completely, always using the largest possible preview.

[NeilR]

Mario,

If I were using iMatch on a desktop machine then I would agree with you, and simply add a terabyte of hard disk space.

But I am on a laptop, and the numbers I illustrate above should make it clear that your plan will NOT work on a laptop.  I would need something on the order of 800GB just for my existing cache, and a future upgrade to a 24 or 36 mpx camera would put those numbers through the roof in just a few years.

Your thinking here is not realistic, and at best I would have to spend a couple thousand on a high end laptop just so I could add a 2nd hard drive (that I cannot do with my perfectly acceptable laptop now) to support the cache you think I should be using .  My existing working laptop has a 750GB 7200rpm drive, and that drive is essentially full, that with a 30GB offline cache.  With a new high density sensor, and a couple years of shooting,  I could easily fill even a largest possible 1TB laptop drive.

Disk is not always cheap.  It is only cheap if you are willing to chain yourself to a desk, which I am not, nor are most people these days.  I find it difficult to believe you would even think of enforcing a 100% image size.

[Mario]

Neil,

350.000 files managed on a laptop is exceptional. I doubt that many users will ever have this amount of photos, especially not on a laptop.
And you can control the size of your cache files to limit the amount of disk space needed, just not with pixel-precision. Which is what you need and what makes sense for your specific workflow and environment. I understand that.

But this is the feature request forum and the right place to post such requests. And I will consider your request for a future release, especially if a number of other users support it here.

I guess that it will cost about one or two working days for me and a few hours for the testers and translators to implement such a change. But my to-do list has about 200 features now already...and features useful/helpful/important for larger user groups are implemented first.

[NeilR]

The fact that I have 350,000 image files is not exceptional.  You even brag about your app handling "even 300,000 images" in your beta guide.

The fact that I use a laptop is not exceptional.  It is now more the norm.

I have a good friend that sits on a number of high tech boards- companies everyone has heard of.   As such he understands the "vision" that the high tech companies are "forcing on us" (my interpretation of the state of things).

He believes that within 10 years time, if you want a desk top computer you will have to pick used or NOS stock out of some Nebraska barn via ebay.  That they will be fully obsolete.

I spent 2 entire days arguing the point with him, mainly because my world is imaging, where people need more than a tablet or smart phone.  Just to say I don't agree with him (more accurately I just hope he's wrong, at least in his timing estimate, because he understands the world in a way I can not).

Without getting into the merits of the arguments of his assertions, I think it is safe to say that the world is changing.  The world at large is moving to smaller, more mobile machines.  We photographers may be the last people left on earth doing anything on other than an iPhone, but the fact is that we are all demanding more mobility.

And the world is changing fast.  The idea of designing your app without a high priority to its being run on a laptop, even with my volumes,  is a huge mistake.

I was doing just fine with iMatch 3, despite some issues.  And the more I think about it, the more I realize you have probably designed people like me out of your new app.

Consider iMatch 3.6 ....

I now have a choice...  I can use the offline cache, even if my image library is online, thus improving performance at the cost of resolution and image quality.  I can turn the cache on and off, depending on my needs.  If I need speed, I turn the cache on (always use offline cache, even if images are online).  If I need image quality, at the cost of performance, then I have to get connected with a gigabit wired lan to my image file server.  Not great choices, but at least I have a choice.

Now consider iMatch 5.  If I understand things correctly, then I can NOT turn off the use of the preview cache, even if I am online with my image library.

That means that unless I generate 100% size previews, then I will NEVER see high res images in iMatch.  So my plan for using 1400 pixel images can't work.

I understand well the performance reasons why you did what you did.  I've suffered through it for years, particularly when paging through images with zoom turned on, which was particularly slow.

But your solution forces me to add a terabyte of storage to my laptop, and I don't really have a choice.  The only viable option is to not use iMatch 5 at all.  Forcing me to chain myself (and my primary working machine) to a desktop is not an "upgrade" or "enhancement", it is a huge downgrade in flexibility.  Alternatively, forcing me into a huge hardware upgrade expense (new laptop with multiple hard drives) is similarly unacceptable.

I think you made a seriously bad decision here, in terms of your desire to minimize user options.  I don't think it is reasonable to say that iMatch needs a terabyte of scratch space on the client machine, even with size image collection.

You badly need an additional option, to turn off the cache.  And without that option, and the ability to generate fixed size previews or offline cache files, I can't be a customer of this product.  It simply will not work for me, where I could at least limo along with iMatch 3.6.

You basically got everything right with iMatch 3.6, except the performance issues.  At least in terms of the mechanics needed to work with the performance issues you had, and likely would still have.  You seem to have thrown all thatout for a far more imperfect solution, at least for laptop users.

For me, this is not just another feature request.  This is life and death, as far as this new version is concerned.

[David_H]

You badly need an additional option, to turn off the cache.  And without that option, and the ability to generate fixed size previews or offline cache files, I can't be a customer of this product.  It simply will not work for me, where I could at least limo along with iMatch 3.6.

That'd be the Cache->Off/Default/On Demand, option, wouldn't it?

However, I have to say, I agree with the request to have an absolute size for the cached images - I can't think of why I'd want one greater than my screen resolution.

[NeilR]

That'd be the Cache->Off/Default/On Demand, option, wouldn't it?

My understanding, reading the help, is that the preview cache image would not be created until an image is accessed.  That might be a sensible choice if your image library is always online.  But for a situation where the image library is NOT always online, which is the thrust of my thread, then only images previously viewed would be available when working offline.  IOW it would be hit and miss, on a rather random basis.  An image never previously viewed would not be available at all (except for the database thumbnail, typically - and by default - only 400 pixels in size).

The only way to solve my problem is have an additional option, something like "Don't use preview cache if image is online", more or less like what happens now with the 3.6 offline cache, except the context is somewhat reversed because the core app philosophy has been effectively reversed. 

And like the current option, it would need to be easily and quickly changed back and forth on the fly.  I often flip that several times a day when doing intensive iMatch sessions, depending on what I'm doing at the moment.

I can't think of why I'd want one greater than my screen resolution.

The first time you try to zoom into an image, maybe to do some culling where you are trying to pick the best shot, and you will learn why you need access to an image larger than your screen.  And remember, if I am correct then if you do set the preview size smaller than 100% you are effectively "stuck" with that image size for apparently most or all viewing within the app.

[David_H]

I can't think of why I'd want one greater than my screen resolution.

The first time you try to zoom into an image, maybe to do some culling where you are trying to pick the best shot, and you will learn why you need access to an image larger than your screen.  And remember, if I am correct then if you do set the preview size smaller than 100% you are effectively "stuck" with that image size for apparently most or all viewing within the app.

I review my images prior to ingestion - so I'm still not seeing the need for IMatch to have a bigger than screen cached image.
For the ones I have processed, the visual proxy version as I understand it takes over (which may or may not be cached itself) - if it is cached, again, there's no need for a 'bigger' than screen size cache, so long as all operations done to it are on the proxy rather than the cached version.

[picolo]

Mario,

I review my images prior to ingestion - so I'm still not seeing the need for IMatch to have a bigger than screen cached image.

Sorry, but I have to agree with David. I too don't have a need for such huge previews (esp. with a D800)

I prefer to set the size of my previews in pixels, so they match my screen resolution. This way space and time can be saved...
Ok, harddrive space is cheap - but only for standard harddrives, and not for SSD's.

see also Mantis 781

[Mario]

However, I have to say, I agree with the request to have an absolute size for the cached images - I can't think of why I'd want one greater than my screen resolution.

Consider the fundamental change between caching in IMatch 3 and IMatch 5. IMatch loads all files through the cache. If you look at a file in the Viewer, Slide Show, Quick View panel etc. it will be retrieved from the cache. If there is no cache file, it will be created on-demand and then either stored permanently or temporarily. But the size you dial in for the cache controls how large you can see a file in the viewer. No 100% cache, no 100% view in the viewer or other modules. If you don't need that, a pixel-exact version of cache files might be a solution.

[David_H]

However, I have to say, I agree with the request to have an absolute size for the cached images - I can't think of why I'd want one greater than my screen resolution.

Consider the fundamental change between caching in IMatch 3 and IMatch 5. IMatch loads all files through the cache. If you look at a file in the Viewer, Slide Show, Quick View panel etc. it will be retrieved from the cache. If there is no cache file, it will be created on-demand and then either stored permanently or temporarily. But the size you dial in for the cache controls how large you can see a file in the viewer. No 100% cache, no 100% view in the viewer or other modules. If you don't need that, a pixel-exact version of cache files might be a solution.

Thinking it through a bit more, I think there's one more option needed - use original JPG if online (this isn't the same as don't cache JPG). Thus :

1. All images get a pixel-exact cached version created as usual (or otherwise depending on setting).
2. If opening a non-JPG file without a visual proxy, use the cache (pixel-exact) version.
3a. If opening a JPG file that is ONLINE, use the original (full sized) version.
3b. If opening a JPG file that is OFFLINE, use the cache (pixel-exact) version.

I would also suggest that the preferred behavior is to use the cached visual proxy if the original isn't available (this might already happen, and I'm probably confusing myself):
TEST.CR2 has a visual proxy of TEST.JPG; if TEST.JPG is offline, then the cached version of TEST.JPG is used when viewing TEST.CR2

[gunda]

Disk space might be cheap, but it isn't free and completely limitless.  I've created a database based on a duplicate copy of my entire image collection (77,000 indexed files, 750Gb) and the cache is 40Gb, which is not small.  This is at default settings, so "Don't Cache JPG Files" is on, and so I assume that this doesn't include JPGs, which are about 40% of the collection. 

I don't generally use Imatch for 1:1 examinations and so I'd find a pixel size limit useful as well.  After all, there's already a % size limit option.

[ChrisMatch]

IMatch loads all files through the cache. If you look at a file in the Viewer, Slide Show, Quick View panel etc. it will be retrieved from the cache. ... No 100% cache, no 100% view in the viewer or other modules. ...

Does this mean if I would set the cache to 50% and look at an image file with the viewer that is 'online' I would not see the full image (only the reduced one)?

[Mario]

Exactly. The cache is a very important and core feature of IMatch 5.

It simplifies internal processing, provides previews for off-line images, allows IMatch to display all file types at the same blazing fast speeds. All display modules use the same cached file, IMatch can streamline many internal operations because it all goes through the cache. A very elegant and effective design.
As I outline in the help and also here, you should use non 100% cache files only under very specific conditions. There are no options to view files by-passing the cache.

[ChrisMatch]

There is also a qualitiy setting for the cache:


Does this mean that if I set the quality to 75% and view an image file
-> I never see the real pixels of that file but only the 75% lower quality image?

[attachment deleted by admin]

[Mario]

Yes.

Basically you see the same as if IMatch would load the 'real' file each and every time, process it for on-screen display (ICC profile, resampling to the required size etc.). The only difference by using the cache is that this is done only once (at the time the cache is created) and a potential minimal quality reduction by a one time only storage to JPEG format.

The resulting quality is more than sufficient for a document management system like IMatch. You can see out-of focus issues, cropped highlights or shadows and everything else that is required to assess quality and perform image management tasks. Of course when you force a lower quality or a smaller cache image size, you limit IMatch severely.

You can disable the cache completely and force IMatch to load the original image every time. If you are concerned about the potential quality reduction of using the cache. You can only cache selected folders. You can leave out JPEG files (Caching JPEG files at 100% means IMatch just copies the original file into the cache folder).

You have a lot of options. From disabling the cache to using the tried-and-true default values.

[ChrisMatch]

The resulting quality is more than sufficient for a document management system like IMatch.

I agree!

This whole topic is important and even after reading the help section about the cache I was not aware what that really means.

One last question: If I set the cache to 'Don't cache JPEG files' this loss in quality does only apply to RAW files because iMatch will generate a JPEG cache image 'on the fly' but will not save it (so no quality loss here) - correct?

[Mario]

Don't cache JPEG means just that: IMatch does not produce cache files for JPEG files. It always uses the original JPEG file for display purposes.

This setting can save lots of space in the disk cache if you use many of JPEG files and always keep them on-line. A cache image for a JPEG file would be basically a copy of the original, sans the metadata. Makes no sense if you keep the original JPEG files on-line anyway. But for users who work with off-line media or only temporary available image storage, the option to produce cache images for JPEG files exists.

[gunda]

Of course when you force a lower quality or a smaller cache image size, you limit IMatch severely.

I understand this.  But there is already an option in preferences to limit the cache image size.  Given this, all we are suggesting is an additional way of doing it.  One that will give us the smallest full-screen cache image.

[Mario]

I have added options to specify the maximum cache image size by giving a width/height in pixels for build 5.0.104.
I still recommend 100% size, but if you need to conserve disk space and you never need to display files larger than screen size, you can dial that in now.

Tip: You can re-create cache files for selected files at any time. Just select the files in a file window and then choose Rescan from the content menu while holding down <Ctrl>. This opens a dialog with advanced options, from which you can choose Update Cache only. This command re-creates the cache images for all selected files using the current cache settings.