COMING: 64-bit versions of IMatch 2017 and IMatch Anywhere 2017

[Mario]

Porting IMatch and IMatch Anywhere to 64-bit...

This was hard. Much harder than I'd have thought - and I thought it would be hard.

For example, I've spent several frustrating days (!) trying to figure out the reason for a 'under stress' crash when using SSL in IMatch WebServices 64-bit. As soon as my automated test suite (a set of apps which simulates IMWS users) 'upped the stress a bit, IMWS was crashing when SSL (HTTPS) was used. An absolute no-go for production use of course. Damn.
I finally went the looooong way (meaning: many changes) and upgraded to the 1.1 of the SSL library. Problem gone. Yeah!. Sigh.

And tons of more 'experiences' like that with the multitude of 3rd party libraries and components, open source tools I use for IMatch and IMatch Anywhere. Porting my own code (all of IMatch) was a breeze compared to that. I've planned ahead over the past years, you know

But, after successful internal tests and a short but intensive Beta with the local tester group (using IMatch 2017.7.8 ) I will ship the IMatch 2017.7.10 release (or the 2017.8.2 if I'm lazy) in both 32- and 64-bit editions. MILESTONE RELEASE!

All users with a license for IMatch 2017 or IMatch Anywhere 2017 get twice as many bits for free. If this isn't good news, I dunno what is.

What's this 64-bit & 32-bit all about?

In short: IMatch 2016 64-bit can use all the memory available on your computer!
The 32-bit edition is limited to about 3.5 GB, and this only if Windows is able to provide this memory in consecutive, non-fragmented fashion.
This could cause problems for users with large databases, users with many categories, users working with very large images (panoramas!) etc....

But no more! IMatch 64-bit can use Gigabytes of memory. Panorama images with 32,000 pixels? Hah! Easy. High-end cameras with 80MP images? U-huh. Databases with 500,000 or more files. Yep, no problem. Modern A.I. algorithms which require Gigabytes of memory. No sweat.

What does this change for you?

Not much. If you have a 64-bit Windows version (Most likely) you will just install the 64-bit version of IMatch or IMatch Anywhere in the future (if the Beta test is over). Everything works as before, just better.

For users with low-end hardware, the 32-bit version of IMatch and IMatch Anywhere will be around for a while.

[thrinn]

Congratulations!

[mastodon]

Fantastic work! I always wandering when do you do programming etc., because you give so much "customer service". 

[DigPeter]

I am sure this is significant and a great achievement.   Not knowing much about technicalities, what is "low end hardare"?

[pmbvw]

Congratulation also from me 

[Mario]

(...) Not knowing much about technicalities, what is "low end hardare"?

Small notebooks, tablets etc. which usually have only 2 or 4 GB RAM and use a 32-bit version of Windows. Or maybe an old PC running Windows 7.

[cthomas]

Congratulation Mario. Only somebody with some programing  will appreciate this for it takes a lot of hard work!

[BanjoTom]

This is great news! I assume you will let us all know when the 64-bit version can be trusted as a regular (not BETA) release?  And thank you, Mario. for your never-ending efforts to improve an already wonderful DAM product.  I recommend it to Windows users all the time!!

[Darius1968]

How would an older Dual Core Desktop PC, with 8GB Ram, 64-bit Windows 10 fare with the 64-bit version of IMatch? 

[Aubrey]

Can the same database be opened using the 64 or 32 bit executable?

[Mario]

This is great news! I assume you will let us all know when the 64-bit version can be trusted as a regular (not BETA) release?

I use the 64-bit version for my own databases exclusively for the past two weeks. I only run the 32-bit version for testing purposes 

[Mario]

How would an older Dual Core Desktop PC, with 8GB Ram, 64-bit Windows 10 fare with the 64-bit version of IMatch?

Better I assume. IMatch 2017 needs a bit more memory (maybe 50 MB) but can utilize all 8 GB available.

[Mario]

Can the same database be opened using the 64 or 32 bit executable?

Yep.

[Menace]

Absolute great. Can't wait to install the 64-Version to feed my ryzen. 

[DigPeter]

......Or maybe an old PC running Windows 7.

That's me!

[cartersfarm]

Great News! So on my main dual Xeon 24 core workstation with 48GB physical ram will use as much as needed - but does this mean that iMatch will naively use up to 48GB for cache to speed up access from slower disks?  Also, will this make any difference in the number of threads it will use on the processors?  I currently see only a handful of threads in use ever for iMatch...  Looking forward to no more "out of memory..." messages on large category sets and stacks! Ed

[ianrr]

Awesome !  Thanks heaps Mario.

[Mario]

but does this mean that iMatch will naively use up to 48GB for cache to speed up access from slower disks?

No. Windows speeds up disks by using file system caches. IMatch does not do its own disk caching. The database system used by IMatch is cached out with 1 or 2 MB RAM - more RAM does not help.

A web or database server serving hundreds of users can utilize 48 GB RAM. Or maybe Photoshop or Lr - these are well-known massive memory hogs.

Also, will this make any difference in the number of threads it will use on the processors?

No. IMatch utilizes multiple processors for certain tasks, e.g. ingesting files, calculating categories, running filters. For most things IMatch uses only one core because it makes not much sense to break everything down into tiny steps which can be run in parallel. This increases the complexity and potential for bugs by a factor of 10.

A computer with dual Xenon processor and 48GB RAM s designed to be used as a server - serving web sites or databases to dozens or hundreds of users. This is not a typical desktop machine designed to run Windows for one user. Maybe when you run DAW software or 3D rendering applications which can be written to truly utilize all CPUs you will get a benefit from all your CPUS. Or when you always run many applications at the same time.

No disk can deliver as much data as your cores can consume, so the bottleneck for applications like IMatch will always be the disk. Or the database system. Or how fast ExifTool can extract metadata from files (again, I/O limited).

Writing an image filter which divides an image into <CPU times> segments and calculates the new pixels in parallel is easy. No I/O. A 3D-renderer which uses one CPU per image or multiple CPUs per image is easy. No I/O. A web server or database server which handles connections from dozens of users and spreads them over all CPUs, that's how to utilize these processors.

IMatch WebServices does that to handle parallel connections. And as long IMWS has cached data available it can work really fast on systems with multiple cores. When IMWS needs to access the disk to load data, things will slow down.
And if it needs to do something outrageous like checking whether or not a file exists in the file system, IMWS may need to wait for a few hundred ms to several seconds before it receives a reply from the Windows file system or a remote NAS box. Even when you have 48 CPU cores 

Looking forward to no more "out of memory..." messages on large category sets and stacks!

You should not get these with the current version of IMatch. Please show me a log file from a session where you did get such a message. What is a large stack? How many files do you manage in your database?

[cartersfarm]

Hi Mario!

I kept seeing Cache Manager in the error messages when I get an Out of Memory. error so I assumed it was iMatch internal caching (see below)...

I understand the previous discussion awhile back about multi-threading, but now I see multiple (12) cores running on one CPU when ingesting files, but not on the second CPU...  Just wondering. This workstation is my primary rendering workstation for video, FX, and audio all of which utilize the cores and RAM. I'm just hoping that my DAM software will also benefit from the workstation capacity when possible.  The references in the OP that you will be using "All RAM" is what prompted me to ask 

I've attached a clip (showing 2 Out Of Memory. errors & one crash) of the debug level log because it is too big to upload (I also have the memory dump from when it crashes thereafter - but it is too big to attach, please let me know if you'd like it sent via another method).  It looks like it is a problem with the Cache Manager and it is using in excess of 2825MB as a peak and then crashing...  This particular one occurred right after updating keywords on about 7,000 files and then viewing approximately 31,119 and scrolling down the screen.  This is easily reproducible on my machine and db if you need me to test a fix...

Note: this workstation is a HP Z800 running 64-bit Windows 10 Professional (Version 1703 OS Build 15063.483) w/ Dual X5650 & 48.0 GB ram.

07.30 16:51:57+114032 [1950] 00  E> CException(CException): 'Out of memory.'  'IMRelationManager.cpp(3110)'
07.30 16:52:59+61281 [BB2C] 01  W> 'database table is locked' [6]  'IMSQLite.cpp(2400)'
07.30 16:52:59+    0 [BB2C] 01  W> SELECT tag_oid,tdata,rdata,lang,flags,count(*) AS cnt FROM md_tag_data WHERE tag_oid = ?1 AND IFNULL(lang,'') = ?2 GROUP BY tdata ORDER BY cnt DESC  LIMIT ?3  'IMSQLite.cpp(2401)'
07.30 16:52:59+    0 [BB2C] 01  W> 'database table is locked' [6]  'IMSQLite.cpp(2406)'
07.30 16:52:59+    0 [BB2C] 01  W> SELECT tag_oid,tdata,rdata,lang,flags,count(*) AS cnt FROM md_tag_data WHERE tag_oid = ?1 AND IFNULL(lang,'') = ?2 GROUP BY tdata ORDER BY cnt DESC  LIMIT ?3  'IMSQLite.cpp(2407)'
07.30 16:54:24+85328 [B980] 01  W> 'database table is locked' [6]  'IMSQLite.cpp(2400)'
07.30 16:54:24+    0 [B980] 01  W> SELECT tag_oid,tdata,rdata,lang,flags,count(*) AS cnt FROM md_tag_data WHERE tag_oid = ?1 AND IFNULL(lang,'') = ?2 GROUP BY tdata ORDER BY cnt DESC  LIMIT ?3  'IMSQLite.cpp(2401)'
07.30 16:54:24+    0 [B980] 01  W> 'database table is locked' [6]  'IMSQLite.cpp(2406)'
07.30 16:54:24+    0 [B980] 01  W> SELECT tag_oid,tdata,rdata,lang,flags,count(*) AS cnt FROM md_tag_data WHERE tag_oid = ?1 AND IFNULL(lang,'') = ?2 GROUP BY tdata ORDER BY cnt DESC  LIMIT ?3  'IMSQLite.cpp(2407)'
07.30 16:55:35+70750 [1950] 00  E> CException(CException): 'Out of memory.'  'IMRelationManager.cpp(3110)'
07.30 16:55:54+19750 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:54+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:54+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:55:54+   78 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:54+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:54+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:55:57+ 2266 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:57+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:57+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:55:57+   78 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:57+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:57+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:55:59+ 2453 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:59+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:59+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:55:59+   94 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:55:59+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:55:59+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:56:00+  875 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:56:00+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:56:00+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:56:00+   78 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:56:00+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:56:00+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:56:13+12984 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:56:13+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:56:13+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:56:14+  391 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:56:14+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:56:14+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:56:25+11625 [B2F0] 00  E> CreateBitmapFromWicBitmap (1) failed with 8007000E  'IMDXCacheManager.cpp(478)'
07.30 16:56:25+    0 [B2F0] 00  E> Exception [8007000e]  'IMDXCacheManager.cpp(837)'
07.30 16:56:25+    0 [B2F0] 00  E> D2D CacheManager: [8007000e] Not enough storage is available to complete this operation. on line 854 file IMDXCacheManager.cpp  'IMDXCacheManager.cpp(1564)'
07.30 16:58:09+103422 [1950] 00  E> CException(CException): 'Out of memory.'  'IMRelationManager.cpp(3110)'
07.30 16:58:51+42687 [1950] 03  W> # Process Memory Info: WSC: 2825MB, WSP: 2831MB (NEW PEAK), PF: 1217985841
07.30 16:58:51+    0 [1950] 00  I> # Loglevel changed to 50.
[Full Log File Attached]

[cartersfarm]

I couple of things I left off...

22.52 GB File Size
9,942 Folders
797,389 Files
998 Categories
4.85 TB Total Size (1 GB dedicated pipe NAS)

iMatch Version: 2017.7.6 (32-bit) Licensed (installed & caches on C: - HyperX 4 channel PCI raw SSD)
ExifTool Version: 10.59

3 Monitors, 2 HP 30" ZR30w (Wide Gamut 2560 x 1600) & 1 27" Viewsonic 1920 x 1080 & 2 Nvidia linked Quadro 5000 (w/ CUDA enabled) GPUs

[sinus]

I couple of things I left off...

22.52 GB File Size
9,942 Folders
797,389 Files
998 Categories
4.85 TB Total Size (1 GB dedicated pipe NAS)

iMatch Version: 2017.7.6 (32-bit) Licensed (installed & caches on C: - HyperX 4 channel PCI raw SSD)
ExifTool Version: 10.59

3 Monitors, 2 HP 30" ZR30w (Wide Gamut 2560 x 1600) & 1 27" Viewsonic 1920 x 1080 & 2 Nvidia linked Quadro 5000 (w/ CUDA enabled) GPUs

Phew, what a amount of files and size.
And, generally, does IMatch handle this all well?
And it seems to me, that also IMatch5 did handle it good?

The next level of IMatch (64bit) can help also to make such huge DB more comfortable.
IMatch is simply a great piece of software. 

[Mario]

I couple of things I left off...

22.52 GB File Size
9,942 Folders
797,389 Files
998 Categories
4.85 TB Total Size (1 GB dedicated pipe NAS)

You are using IMatch way beyond spec. Managing 800,000 files in a 100 US$ software is quite adventurous, frankly.
If you would ask companies like Canto, FotoWare, Extensis, AssetBank or Widen what it takes to manage 800,000 files, they will send you a pre-sales consultant or talk to you earnestly about things like "dedicated server farm"...

The sheer work IMatch has to do to recalculate even one data-driven category for 800,000 files  is massive. Same for calculating relations or stacks or Collections etc.

Most of the error messages in your log file come from the WIC / DirectX subsystem. The graphic card runs into out-of-memory errors when IMatch tries to load mages. Usually you see this large number of errors only on low-end graphic cards which share memory with the system (e.g. Intel graphic cards). Modern 100$ graphic cards all have 2 or 4 GB RAM and show such errors only rarely. IMatch can deal with the occasional out-of-memory condition by freeing GPU memory and loading the images again. But your GPU produces these errors in the dozend...

Try to reduce the number of images IMatch pre-loads for the Viewer under Edit > Preferences > Application. This should reduce the stress on your graphic card.

There are many other errors where IMatch runs out of RAM - these should be solved with the 64-Bit edition.

BUT I see also errors caused by database locks, which is unusual. This basically means that IMatch runs into timeouts when performing certain database operations, because the database cannot deliver the data fast enough. Not unexpected for 800,000 files. IMatch can handle this amount of files, but all the advanced features like Relations, Collections, data-driven categories, stacking etc. slow down the more files there are in the database. Twice as many files means double + X execution time.

A typical IMatch user has 50,000 to maybe 200,000 files. 300,000 to 400,000 files OK. But 800,000 files is way beyond what a single database should hold. There are many stock photo agencies out there which have less files to manage...

[cartersfarm]

Thank you for your analysis of my log, there is a lot of stuff you mention that explains some of the problems I've been having. On the Video Card front though, I think there may be a problem with the code since these are linked Quadro 5000 cards with "352 CUDA parallel processing cores clocked at 513MHz and 2.5GB of GDDR5 frame buffer memory. In addition to running at 750MHz over a 320-bit interface for a total bandwidth of 120GBps, the memory also supports error-correcting code" - each was $2,200...  If I'm running out of GPU memory with 5Gb of ECC GDDR5 memory, do you think using 64-bit addressing will help?

As to the size of the DB, that is exactly why I've been using your creation for years - I don't want proprietary solutions locking me into their niche products, iMatch has served me well 😁. I'll look forward to giving the 64-bit version a series of tests when it is released and I suspect it will do better. Keep up the awesome work you do, and I hope you continue to enjoy a loyal customer base who push your product to new heights with every major release! Ed

[Mario]

Yeah, let's try the 64-bit version first. It works wonders for some users already.

As for the GPU error messages: I just log them. There is no "how much memory left?" call in DirectX. You cache files until the graphic card tells you to stop. If the graphic card issues errors outside that box, it's usually some driver problem or an Intel card on a Notebook which has to share 2 GB RAM with Windows or something.

This may also be affected by the 64-bit edition of IMatch because it is no longer prone to Windows memory fragmentation and also uses the 64-bit versions of WIC and DirectX.

[cartersfarm]

Hi Mario, I guess I've reached the end of the road with my dual Xenon HP X800 workstation with twin Nividia RTX2070FE video cards I use for video and 1M+ photo libraries. The lack of AVX support across my 24 cores and 48GB of ram with 40TB of storage is not going to work with your software any more. I saw other threads where you asked what kind of computer doesn't have AVX support in 2020 - this is the kind of machine...  It has been a good ride.  If I upgrade hardware in the future I'll return and upgrade at that time - thank you for your years of support for me and the community! Peace

[Mario]

I get from your post that you are complaining about the fact that IMatch requires a vector math technology that has been standard since 2011 in pretty much all processors manufactured by Intel and AMD.

I have no idea why Intel did not include at least a minimum of vector math extensions to their expensive Xeon server processors.
Usually they disabled AVX only on low-end Notebook processors to shave of a few dollars off the price point. Why they did so on the early Xeon lines, I have no idea.