Sometimes IMatch hangs

[Arthur]

Hi, I had similar situations before, where IMatch was not resposive for 2 minutes or so. I have updated a single JPEG file from DxO, and executed "Rescan now". The relevant JPEG preview never updated completely. It was shown blury with the blue updating icon in the preview. Later I deleted a file.

Only after closing and reopening IMatch the preview could be recovered. Automatic metadata write back is always disabled. Background indexing is on.

Now I have a log for this. What can be seen arround 02:21 and 02:29? I have only windows defender running. Nothing special.

[Mario]

Please always ZIP and attach the complete log file.
There is important info at the beginning which I need. For example, the IMatch version. If your database has 20,000 or 500,000 files, ...

I see many very long database locks, This means that IMatch cannot perform correctly because the database is too slow.
Did you exclude the database folder from your virus checker?

I also see reports that ExifTool is handing (not responding) and IMatch has to shut it down hard after waiting for a minute.
This either means that ExifTool has encountered a unknown metadata format which breaks it, that the metadata in the files is badly corrupted or that some other application is blocking ExifTool or the file. If this happens for all ExifTool instance IMatch is running, IMatch may end up dead in the water...

Please close other applications while you run IMatch and see if this solves the problem.
Some applications keep files locked, preventing other applications from accessing them.

Also

[Arthur]

I have the newest 2017.11.4 with around 400 files.

I haven't excluded anything from windows defender. Will exclude the folder tonight. The data base is on my system SSD the managed files on a Hard Drive.

There were only DxO -> new JPEG file -> IMatch (and maybe the virus scanner) involved. With "Rescan Now" and "Delete" fired inbetween.
Because the JPEG is a new file, the Metadata generator can only be DxO or IMatch itself. I have seen metadata writebacks, like this:

11.21 02:27:13+    0 [1204] 10  M>                  > 17 PTMetabase::ImportFiles  'PTMetabase.cpp(2493)'
11.21 02:27:13+    0 [1204] 10  M>                   > 18 PTMetabase::ImportFileValues  'PTMetabase.cpp(2134)'
11.21 02:27:13+    0 [1204] 50  M>                    > 19 CIMRelationManager::Propagate  'IMRelationManager.cpp(3178)'
11.21 02:27:13+    0 [1204] 10  M>                     > 20 PTMetabase::WriteBack  'PTMetabase.cpp(7330)'
11.21 02:27:13+    0 [1204] 10  I>                         1 files to write-back to
...
11.21 02:29:41+    0 [1204] 10  I>                        PTMetabase::WriteBack failed for file [606] 'E:\RAWS\2016\2016-07\2016-07-30\IMG_1127.jpg'

Why is this writeback triggered, although I have disabled the auto write back functionality?

The last times when IMatch was non responsive there was always metadata handling involved. The first time this happened was on the initial file import into the catalog. There the dialog which shows
disc activity in a graph just stopped to do anything. Another time there was a progress bar dialog on the screen which reported metadata writeback, this dialog stopped to respond also.

[Mario]

This looks like a write-back triggered by metadata propagation. When you propagate metadata IMatch first writes-back data to the master, then copies the data from the master to the versions (which also causes write-backs, naturally).

ExifTool was clearly locked according to the lock.
If this happens for multiple files at the same time (IMatch always runs ET multiple times to better utilize processor cores) this can bring down IMatch to a crawl. Or even cause database locks.

But this is super-rare. One ET instance may crash or even block when it encounters a badly damaged file.
But multiple instances hanging at the same time, multiple times is usually caused by something external, like a virus checker or another application keeping the files open and locked.

[Arthur]

This looks like a write-back triggered by metadata propagation. When you propagate metadata IMatch first writes-back data to the master, then copies the data from the master to the versions (which also causes write-backs, naturally).

Hm, we should separate things:

1) There is a CR2 and a XMP on disc. These are not changed externally. So no need to update metadata for master.
2) The relation is CR2 -> JPEG and >>>auto write back to file is off<<<

So if I update the JPEG on disc, I would expect that IMatch updates the JPEG metadata from CR2 metadata only in the data base (even this is not needed because the master metadata was not changed). It should not change the JPEG file on disc as a result of metadata propagation, until I trigger the Write Back Metadata Command explicitely on that file. This is how Lightroom does it. It does not propagate metadata, but it displays an icon on a managed file that informs the user that the metadata in the catalog is different from the metadata in the file and the user can selectively commit the catalog metadata as the new file metadata at the right point in time.

Additionaly an application should not deadlock itself, because an external application uses the same resources. Is it not possible to precache (clone) or aquire a lock to the external file stream before passing it to the exiftool? If precaching is not possible, because the file is locked by somebody, just abort the processing of that file and display a could not access icon or something. The user could press F5 some seconds later after he ensures that all locks are freed.

Currently I must ensure that IMatch and the external development tool run exclusively, so that IMatch does not kill itself while waiting for the other tool to free the handles.

[Mario]

Do you have background relation updates enabled?
I have no idea why this locks on your system. I use IMatch all the time along with other applications like Photoshop, Lr or Affinity software.
No other reports about similar issues. Very likely that this specific issue only appears on your system, with your files, relation and other settings and application suite.

[Arthur]

Any DxO+IMatch users out there with the following setup?:

- CR2 raws as masters
- JPEG as versions
- Buddy files JPEG, XMP, DOP
- Related CR2, JPEG, DOP, XMP are stored in the same directory on disc, no subdirectories for JPEGs
- JPEG is set as version stack proxy, not as version proxy
- Metadata auto propagation is turned on
- Metadata auto write back is turned off
- Categories are propagated from masters to versions
- Edit in DxO is started via Applications Panel, so that an extrenal selection is created in DxO.

This setup is enough currently to let IMatch hang here sometimes, anybody with the same experiences? Anybody with a different setup, where writing metadata to disc causes the tool to stop responding?

[herman]

Using DxO with IMatch, but a completely different setup......
My OOC files are a mix of .CR2 and .DNG raw files..
DxO outputs JPG files which I store in a subdirectory of the OOC files.
DxO created JPG files are handled as versions in my setup, not versions AND buddy files simultaneously (when I understand you correctly the latter is what you do?).
DxO .dop files are buddy files in my setup, stored in the same directory as the OOC.
I am not using proxies here, I want to see the raw file on top of the version stack (or the camera jpg thumbnail rendition of it)
I almost never write metadata to files, the rare occasion when I do is to repair metadata screwed up by some other application   
On all other occasions I work with categories exclusively which are propagated from OOC to versions.

Works like a charm here, but that was not exactly your question......

[Arthur]

Using DxO with IMatch, but a completely different setup......
My OOC files are a mix of .CR2 and .DNG raw files..
DxO outputs JPG files which I store in a subdirectory of the OOC files.
DxO created JPG files are handled as versions in my setup, not versions AND buddy files simultaneously (when I understand you correctly the latter is what you do?).
DxO .dop files are buddy files in my setup, stored in the same directory as the OOC.
I am not using proxies here, I want to see the raw file on top of the version stack (or the camera jpg thumbnail rendition of it)
I almost never write metadata to files, the rare occasion when I do is to repair metadata screwed up by some other application   
On all other occasions I work with categories exclusively which are propagated from OOC to versions.

Works like a charm here, but that was not exactly your question......

Thanks for replying,

1) Are versions handled in the same way as buddy files when it comes to moving and renaming? If not, then it is not applicable for me. Yes I have JPEG as version and as buddy at the same time.
2) I use stack proxys because I do not want to see the original raw after development. As as stupid Lightroom user I create exactly one JPEG from a RAW in 98% of all cases and I want to see exactly that in the DAm tool.
3) What is the advantage of putting the JPEGa in a subfolder from the workflow side? Maybe you have no problems because your export directory is different from your import directory. Maybe IMatch updates less in that case.
4) The only reason I write metadata, is to store my keywords, so that I can search better outside of Imatch.

[sinus]

Using DxO with IMatch, but a completely different setup......
My OOC files are a mix of .CR2 and .DNG raw files..
DxO outputs JPG files which I store in a subdirectory of the OOC files.
DxO created JPG files are handled as versions in my setup, not versions AND buddy files simultaneously (when I understand you correctly the latter is what you do?).
DxO .dop files are buddy files in my setup, stored in the same directory as the OOC.
I am not using proxies here, I want to see the raw file on top of the version stack (or the camera jpg thumbnail rendition of it)
I almost never write metadata to files, the rare occasion when I do is to repair metadata screwed up by some other application   
On all other occasions I work with categories exclusively which are propagated from OOC to versions.

Works like a charm here, but that was not exactly your question......

Thanks for replying,

1) Are versions handled in the same way as buddy files when it comes to moving and renaming? If not, then it is not applicable for me. Yes I have JPEG as version and as buddy at the same time.

I think No. Versions and masters are independent, only "linked" with each other (viewable with the icon for example.

[sinus]

2) I use stack proxys because I do not want to see the original raw after development. As as stupid Lightroom user I create exactly one JPEG from a RAW in 98% of all cases and I want to see exactly that in the DAm tool.

I do the same.

[herman]

Are versions handled in the same way as buddy files when it comes to moving and renaming?

Definitely not.
When you have a master-version relation you can rename / move / delete / ..... the master without touching the version, it remains as it is. Buddies travel with the master though, are renamed / moved / deleted together with their master.

What is the advantage of putting the JPEGa in a subfolder from the workflow side?

It mirrors essentially what I did 50 years ago in the chemical darkroom (yes, I am that old......  )
The YYYY-MM-DD named master folder contains the equivalent of my roll of film, the out-of-camera original files (together with the inevitable sidecars thrown in by post processing software).
Subfolders of that roll are aptly named after their analog darkroom counterparts: developed, retouched, output - print / mail / web
The advantage is that I know where to find a specific file even in Windows Explorer.
I also find it very helpful in defining formula-driven categories, so that for example I can easily find all e-mailed images based on the folder name.
There are more ways to skin that cat in IMatch though.....

[Arthur]

OK, thanks.

I only need one version to look at on a PC/smartphone/TV screen synchronized through google photos and to send it to a discounter for printout.
I put the version near the raw and dop so that the files are handled together as a package. I don't get it, what breaks IMatch in that very simple setup.

[sinus]

The advantage is that I know where to find a specific file even in Windows Explorer.

But this point it not specialy an advantage from a subfolder.
If you have a proper filenaming-system, you can find a specific file in the Windows Explorer, it they are in a Subfolder or not.

I think, subfolders or not does not really matters.
I like to have no subfolders for the jpgs, because I like to have all files together. I can see in the Windows Explorer for example very easy, what files (e.g. from an Event) has versions, what not and I can see the versions.

[herman]

I put the version near the raw and dop so that the files are handled together as a package. I don't get it, what breaks IMatch in that very simple setup.

I am not sure about this, it is a shot in the dark.....
I vaguely remember (perhaps from beta testing days of IMatch, again, I am not sure) that a file can be either a buddy file or a version.
You have both relations defined for one derivative of a raw file.
Have you considered to temporarily deactivate one of the relations and then repeat your tests?

[Mario]

Version relations and buddy relations are independent.
You can configure a version to be a buddy file, but you don't have to.

[herman]

If you have a proper filenaming-system, you can find a specific file in the Windows Explorer, it they are in a Subfolder or not.

Actually I have both in place.
From a filename I can tell if a file is an original or a derivative, what software(s) created the derivative and for what purpose.
Still I like the subfolders as it seems so logical to me.
As in the wet darkroom days long ago where I kept the negatives and prints in separate binders too 

[Mario]

Still I like the subfolders as it seems so logical to me.

And tidier, too.
I also use sub-folders to keep edits, versions for different targets, auxiliary files, documentation etc.
IMatch handles this without issues.

[sinus]

As in the wet darkroom days long ago where I kept the negatives and prints in separate binders too 

True.
But I think, it is interesting comparing the good old stuff from ages gone, but it makes finally not really sense.

A digital workflow is different.
It is different take pictures with  a film or with a digital camera, but handling digital and analog pictures is really another thing.

But of course, we do sometimes think back, very normal. 

[David_H]

Any DxO+IMatch users out there with the following setup?:

- CR2 raws as masters
- JPEG as versions
- Buddy files JPEG, XMP, DOP

This setup is enough currently to let IMatch hang here sometimes, anybody with the same experiences? Anybody with a different setup, where writing metadata to disc causes the tool to stop responding?

Careful; IMG_1234.CR2 and IMG_1234.JPG in the same folder will have issues as you when you get IMG_1234.XMP you have a master and a version sharing the same xmp which is unlikely to be what you want, nor is it likely to be a good idea.

[Arthur]

I thought the XMP belongs to the CR2 only and that a JPEG stores its metadata internally. Is this a wrong assumption?

[sinus]

I thought the XMP belongs to the CR2 only and that a JPEG stores its metadata internally. Is this a wrong assumption?

no, this is correct. xmp is only for raws. Almost ever.

[Mario]

By definition an XMP file belongs to all all files with the same name in the same folder.
In case of the JPEG images merges the XMP metadata embedded in the JPEG with the XMP sidecar file.

[jarraun]

In case of the JPEG images merges the XMP metadata embedded in the JPEG with the XMP sidecar file.

Hi,

Sorry Mario , can´t follow, please could you explain this concept a little.

Best regards

Javier

[Mario]

An XMP sidecar file belongs to all images with the same name as the XMP file.
If you keep a RAW and a JPG/TIFF/PSD in the same folder and with the same name, the XMP metadata in the sidecar file is used for all files.
IMatch prefers the embedded XMP data (e.g. in the JPEG) but also merges the XMP file. This ensures the most complete metadata for all files.
Usually you don't need to care for this, it's automatic.

See also the extensive explanation about all things metadata in the IMatch help. Just search for metadata.

[jarraun]

So kind Mario,

So assuming my NEF - JPEG - XMP live together in the same folder, NEF´s are masters and JPEG´s are versions and I propagate metadata from masters to versions, I end with the same metadata in NEF - JPEG - XMP files, no inconsistency between these files. Is that right.

[Mario]

Yep, if everything is going according to plan.