Image size limit for viewer display?

[mking]

I have a 16bit Tif (RGB) that's 14660 x 7349 and Imatch5 won't display in viewer.
If I reduce it to 14500 x 7269 it displays fine.

To save me testing all combinations - what is the limit - total pixel size? one dimension? and can it please be raised.
That's only a 3 image stitch with a £2000 camera these days ! I would think 30K x 30K is a minimum, 50K x 50K would be ok, but ideally just go for 100K x 100K and that should then be good for a few years

Many Txs,
Mike

[Mario]

IMatch 5 is a digital asset management system with a strong focus on images. Exceptionally large files as produced by panorama applications require very specialized image processing capabilities. IMatch currently supports files up to a size which fits into the largest consecutive area IMatch can allocate in its address space, usually about 2 to 2.5 GB. If there is no consecutive memory space of that size, IMatch cannot load the file.

Pano applications are designed to handle files that large, a DAM like IMatch is not necessarily so. To overcome this limitation I will need to buy and implement a different set of image libraries plus migrate IMatch completely to 64-Bit. Which also requires me to upgrade the scripting engine to 64-Bit, which is a purchase of several thousand dollars up-front, plus several thousand dollars per year for the distribution license.

I will think about that when the initial release of IMatch 5 is out and I can see if people like it and buy it. If I make some money, I'll be happy to re-invest it into buying new image libraries and scripting licenses.

Tip: Make a smaller JPEG version of the panorama and put it into the same folder. Then set up a file relation which makes the JPEG a version of the pano file, and make that version the visual proxy. This allows you to see the pano in IMatch.

[exb53]

Hi, as iMatch now seems to be close to release (I am waiting desperately for that) I again looked into the beta. As I have a lot of large files (stitched or lots of layers.) I found this thread about size limitations. I understand the reasoning behind that, but there are 2 questions:

I use the Ardfry codec to display .psb files in Windows explorer: works fine! Is there any way to use this technology to display PSBs in iMatch ?

In order to create JPG buddy files, is there a search function in iMatch for images that failed to render a preview (TIFs and/or PSBs)? Unfortunately the size of a failed file is not always the same.

Thanks

Eckart

[Mario]

With "codec" you mean WIC codec?
Did you test one of your files with the WIC diagnosis in IMatch to see if IMatch can use the codec?

There is no search function for files which failed to import. Maybe you can just set a filter on image width / height or sort by that attribute to find all files without a proper size. Add a file format filter as well so you get all TIFF/PSB files with a width/height of zero. Just a few clicks to set this up. Check out the help topics on Filters and Sort Presets for details.

[exb53]

Hi,

This is the WIC Diagnostic for a PSB file:

List of installed codecs:
   Codec 'BMP Decoder' for extensions .bmp,.dib,.rle
   Codec 'GIF Decoder' for extensions .gif
   Codec 'ICO Decoder' for extensions .ico,.icon
   Codec 'JPEG Decoder' for extensions .jpeg,.jpe,.jpg,.jfif,.exif
   Codec 'PNG Decoder' for extensions .png
   Codec 'TIFF Decoder' for extensions .tiff,.tif
   Codec 'WMPhoto Decoder' for extensions .wdp,.jxr
   Codec 'DDS Decoder' for extensions .dds
   Codec 'PSD Decoder' for extensions .psd, .psb, .pdd
   Codec 'Microsoft Camera Raw Decoder' for extensions .ARW,.CR2,.CRW,.ERF,.KDC,.MRW,.NEF,.NRW,.ORF,.PEF,.RAF,.RAW,.RW2,.RWL,.SR2,.SRW
   Codec 'Nikon .NEF Raw File Decoder' for extensions .nef


Testing file 'E:\ASRock MyPics 01\10 - Japan 2013\7 - Panos\Japan 2013 Panos Done\2013-10-06-Tokyo-1272-O1.psb'
   Thumbnail: Codec 'PSD Decoder'
      () 160x120 pixel in 32 ms.
   Preview: Codec 'PSD Decoder'
      () 160x120 pixel in 0 ms.
   Full resolution: Codec 'PSD Decoder'
      () 18189x13696 pixel in 0 ms.

Whether Ardfry is a WIC or something else, I dont know. But I have disabled your Adobe Photoshop Codec (Options, File Formats) and there is a PSD Decoder in the protocol. Here is the link to Ardfry: http://www.ardfry.com/psd-codec/.

For the search: Maybe there is a change to search for Files with an error Flag set. All failing TIFs have a flag with the comment: There was an error reading metadata ...

Regards
Eckart

[Mario]

You have a WIC codec installed ('PSD Decoder') and it will be used by IMatch automatically.

But I have disabled your Adobe Photoshop Codec (Options, File Formats)

I don't understand. PSB/PSD are built-in formats, you should not change that or try to override them with a user format.
This will not cause IMatch to use an installed WIC codec for .PSB/.PSD files.

It requires a change to a configuration file. To save some time (and me installing the codec and somehow creating such a large panorama file) I suggest you try the following and let us know how it works out:

Close IMatch.
Open the IMatch 5 program folder on your system in Windows Explorer:

%PROGRAMFILES%\photools.com\IMatch5

Open the "plugin" sub-folder.

Make a backup copy of the file named ptpiconfig.xml in that folder.
Open this file in Windows Notepad.
Search for .PSD

Change the section from

<plugin type="ptpipip"/>
<detect mode="direct"/>
<extensions>
   <ext>.PSD</ext>
   <ext>.PDD</ext>
   <ext>.PSB</ext>
</extensions>


to
<plugin type="ptpipip"/>
<detect mode="direct"/>
<wic>enabled</wic>
<extensions>
   <ext>.PSD</ext>
   <ext>.PDD</ext>
   <ext>.PSB</ext>
</extensions>


(Add the line <wic>enabled</wic>)

Save the file and restart IMatch.
Select some of the failed files in a file window and press <Shift>+<Ctrl>+<F5>. In the dialog, choose "Force Update".

IMatch should now use the WIC codec for your files. Let us know how it turns out.

[exb53]

Hi Mario,

Thank you, but it does not help. Some PSB (very few) are displayed, the rest is not.

I have been working with the buddy file model for a while. (PSBs as a buddy to JPGS,  a Photoshop action generates JPGs from PSBs with a big text "PSB" on it. So they can be easily identified.) So I can live with that. Especially there seem to be no DAM capable handling PSBs. (Exception seem to be Thumbsplus 9, which works fine with large TIFs, too. But that is a bad product, with low support.). Bridge does, too, but is not really a DAM.

Anyway, the bigger worries are the TIFs. The Buddy model could work fine here, but there should be a way to reliably find the failing files.

So I will play around a bit more, maybe I find a way...

Regards

Eckart

[Mario]

Did you try my filter trick? What dimensions does IMatch report for the failed files?
You only need to find some metadata or attribute which identifies the files and then you can filter them out easily, or use a data-driven category.

I would need a sample file which fails (please upload somewhere or contact me for an FTP upload link).
And I would need the WIC codec you are using? Is this a commercial product or a free one?

[exb53]

Hi,

The Codec is not free, but has a 15 day eval period (http://www.ardfry.com/psd-codec/).

I tried the trick with zero dimensions, which seems to work. I will check that tomorrow, as I have currently a large import running, which will need a few more hours. I will let you know.

I could download a relative small TIF file, but that is still 1,3 GB and my internet is slow. That would take 7 hours plus ... Or go to an internet café .... But in the old thread you mentioned that as a 32 bit program, imatch and its components have a limited address space and will fail to handle files going close to 4 GB. Photoshop (and others) allow TIFs up to 4 GB. So there is a limitation, which cannot be changed.

I could live with the TIF buddy thing, and your little trick seems to give me all failing images, so I can generate the buddy JPGs.

I'll let you know how it works ...

Thanks


Eckart

[Mario]

The WIC system is a bit fuzzy but as far as I know, IMatch uses 64-Bit WIC codecs on 64-Bit machines. So the codec loading the file has no address space problem, it can use Exabytes of memory (and Photoshop will require that amount of memory some day, no doubt).

A 18189x13696 pixel file requires 18189x13696*4 bytes in memory (about 1 GB) if loaded at 8 bits per channel, and about 2 GB of RAM if loaded at 16 bits per channel. IMatch can use a bit more than 3 GB of RAM. But to load a 2 GB chunk of image data, the memory has to allocated in one consecutive block, and memory fragmentation in Windows can make this impossible. IMatch then fails to load the file.

As said above, handling such large image files is the scope of specialized applications, and not a typical DAM feature.

If IMatch 5 sells sufficiently well I will definitely look into a 64 Bit port. If I make some money from it I'm willing to spend it to buy and license 64-Bit versions of the third party components. The most expensive will be the 64-Bit engine which costs initially what I pay now in fiver years for royalties - plus annual royalties.

[exb53]

Hi,

I again did some testing. Finding Tif files that failed to load, works for the moment. I could use the selection of zero dimensions to generate smaller JPGs as buddy's, which then are displayed as a "placeholder". I put a watermark on them so that I know there is a big TIF (or PSB) hidden.

Unfortunately this is time consuming and difficult to work with, for example to start the file out of iMatch in Photoshop and save it as a new variant. Also, as the memory usage changes depending on other apps running or the sequence of tasks started, it is not predictable what happens on a rescan. Suddenly some that did not display, are displayed, or, worse, are suddenly not displayed anymore.

I understand that it would be expensive to integrate full 64 bit support, but just a few layers in Photoshop bring TIFs into the "critical" area. And that are not just pano's, even a D800 or D600 or similar cameras these day easily generate TIFs larger than 2 Gig. Or PSBs.

I have no idea of the cost and your business plan etc., but 64 Bit could be a differentiator, too! At least for the moment!

Thank you

Eckart

[Mario]

Suddenly some that did not display, are displayed, or, worse, are suddenly not displayed anymore.

That's because another application is fragmenting the memory and IMatch, despite being able to use more than 3 GB of RAM, is unable to allocate a consecutive memory block to hold your large files.

even a D800 or D600 or similar cameras these day easily generate TIFs larger than 2 Gig. Or PSBs.

I'm not sure about this. I work with D800 and even source files from digital backs often. I do retouche in PS on many layers, and save them as PSD. IMatch handles these files with between 500 to 1 GB on disk quite well.