Propagation accross folders

[ERBRO]

Hello,

Will propagation work accross folders ?  I'm using capture one session and the DNG are in one folder and the output in JPG or TIF are in another one.

Best regards
Eric Bromey

[Mario]

Sure.

It is irrelevant for IMatch where the versions of a master image are, as long as it can find them. And when it has found them, it can propagate metadata.

Having the master image in one folder, and the versions in a sub-folder or even on a different drive is a pretty common situation.

[ERBRO]

HEllo,

I'm still fighting to get the propagation to work.

I have for each event, a sub-folder with RAW and a second one with the created JPEG.

Example :

C:/pictures/Christmas 2024
C:/pictures/Christmas 2024/capture (contains the RAW) and to which I have assign keywords and faces
C:/pictures/Christmas 2024/output  (contains the JPG resulting of the editing of the RAW)

I can not figure out how to set the parameters in the file relations version windows in the "where to search" block as

1) "entire database" gives a pop up that it will be very slow  and is not recommended (what I understand)
2) "Master Folder" give the option Up, Down or Up and down
3) "Folder List"  I could put here "C:/pictures/Christmas 2024"  but that woulf mean I need to add each time I create and event  the folder here what is combersome.

Can iMatch cope with my file structure without having to choose the option "entire database" ?

Best Regards
Eric

[thrinn]

C:/pictures/Christmas 2024
C:/pictures/Christmas 2024/capture (contains the RAW) and to which I have assign keywords and faces
C:/pictures/Christmas 2024/output  (contains the JPG resulting of the editing of the RAW)
...
3) "Folder List"  I could put here "C:/pictures/Christmas 2024"  but that woulf mean I need to add each time I create and event  the folder here what is combersome.

Can iMatch cope with my file structure without having to choose the option "entire database" ?

Yes, IMatch can cope with this requirement.
The solution is to use the "Folder List" option, combined with Folder Patterns.
Following your example, your master file is in "C:\pictures\Christmas 2024\capture\". The so-called "path segment token" {p0} would resolve to this path. {p1} would resolve to the folder one level up, that is "C:\pictures\Christmas 2024\". You want to search for versions in a subfolder called "output", so you let IMatch search in "{p1}\output". Folder patterns are very helpful to access parts of a given path.

• Set "Where to search" to "List of folders".
• Set "Direction" to "Specified folders only".
• Use the Plus sign to the right with "Add pattern..." and enter "{p1}\output".

[ERBRO]

Hello,

Thank you for the advice. I tested it on a small DB but can not get it work. the keywords are not propagate. I suppose I still miss something but can not figure out what.

I set the relation as in attached printscreen.

Would the extension be case sensitive ?

Best regards
Eric

[thrinn]

Please show us a screenshot of the "Versioning" tab. The "Detection" tab makes sure that IMatch can identify versions of a given master. But the "Versioning" tab specifies what exactly  should be propagated. For example, to propagate keywords, you might want to set the checkmark for "XMP keywords" (or "XMP Keywords (Merge)". It depends on your workflow requirements which of the other options you might want to set or unset.

The "*" behind the XMP Keywords option is also important: It means that you have to perform a Metadata Writeback to trigger the propagation.



Please note that File Relations are already an advanced topic. There are a lot of specifics and options to cover different use cases. I understand that you are just evaluating IMatch, correct? Configuring File Relations in the IMatch help has a lot to say about File Relations. I use IMatch since around 2003, I believe, and I am still using the excellent IMatch help on a regular basis to lookup certain aspects.

But don't hesitate to ask in this community if you run into troubles or can't figure out something. Chances are very good that somebody can and will help you.

[Tveloso]

You should also note that Propagation happens when you write back the Master, so the JPEG will not contain the Metadata you have added to the Master, until after the Writeback takes place.

Also, after setting up the Version Relation as thrinn has advised (and having refreshed the relations - F4, R - as IMatch suggests, following changes to relation configs), you should ensure that IMatch has actually identified your files as Masters, having one or more Versions.  Your Link Expression may not be doing what you expect, and is failing to create the Version Set.

The Help Topic thrinn has linked to provides all the details.  Once IMatch has established a Version Set, the Thumbnail for the Master will contain this icon:

   

...and the Version(s) will contain this one:

   

If you don't see those Icons, either the rule is not working, of you have not refreshed the Relations.

[Tveloso]

...and I am still using the excellent IMatch help on a regular basis to lookup certain aspects.

Here here.  It will definitely be worth your while to read over the Help topics thrinn has linked to.

[Mario]

Did you check that the versions are detected?
They will show a version icon (see Versions in the File Window) and when you open the Version Panel via (View menu > Panels > Versions) and you select a master, the versions must show.

File extensions are not case-sensitive.

If your versions are properly detected, make sure to enable what you want to propagate on the Versions tab, as explained by @thrinn.

He also mentioned that versioning is an advanced feature. Usually this is not something you work while still on your first couple of days with the Trial version of IMatch.

Make sure you read the explanations in the corresponding help topic completely. Press <F1> in the Version dialog.
Don't just copy metadata between files. A lot of metadata is not really designed to be copied, or must be copied in entire groups.

XMP metadata like hierarchicalSubject, Description, date created, create date, XMP GPS data etc. are usually safe to copy.
Be selective.
Don't just copy all XMP data, unless you know what this means. Don't copy native EXIF or GPS data, stick to XMP.

[ERBRO]

Hello,

With the support above and the onlinr help, I figure out the issue.  I had a prefix (like _DSC in the online help) that was not taken in consideration.

Thank uou for the quick support.

Best regards
Eric

[Mario]

Very good!

Can you help me out? 

I know you're already working with advanced concepts like versioning. But you have recently started using IMatch, right?
What were your initial impressions? Where did you struggle first? Do you like the help system?

I'm always trying to make onboarding for new users easier. To make IMatch work right out-of-the-box for new users. Which is not easy, due to the wide variety of workflows, previous experiences and expectations.

If you have some comments or tips to share, let me know.

[ERBRO]

Hello Mario,

the major area where I did struggled  is around performance, background processes etc. 

- dashboard freezing each time a background task run.
- Difficulty to close imatch (for instance if I cancel the metadata write back, imatch will be unresponsive).
- I can not find a place to view all pending back ground tasks

The online help is fine but the (quick) support of this forum to interpret the online help is invaluable especially to explain quite technical topics to non IT people like me.

The video showing a few concepts is also very welcome.

Best regards
Eric Bromey

[Mario]

dashboard freezing each time a background task run.

Are you sure?

Because, when background tasks run, the Dashboard dims, shows a spinner icon and stops updating panels all the time - because it would be fruitless because the information displayed changes several times a second. You see the activity panels at the top, informing you about what IMatch is currently doing. Trying to keep the dashboard updated would only waste processing cycles.

Unless you make IMatch process tens of thousand of files, this should never be an issue.
What where you doing at the time?

I can not find a place to view all pending back ground tasks

The Info & Activity Panel

If your system struggles with processing demanding background tasks, reduce the number of parallel threads IMatch runs.
See Process Control (Advanced Setting)

Especially if your images are on a slow NAS or disk and your computer has many processor cores, your system may not be able to keep up.

Difficulty to close imatch (for instance if I cancel the metadata write back, imatch will be unresponsive).

Cancelling 20, 30 or more background tasks all waiting e.g. for Windows to return data or ExifTool to finish reading or writing can take a while. IMatch cannot prevent this, at some point it has to wait for the operating system or external helpers to finish processing. Especially if your system is maxed out or the files you are processing are very large.

If you experience such issues, the FIRST THING to do is to secure the IMatch log file (log file) after IMatch has closed by copying it to somewhere. IMatch logs what it is doing, how long things take, etc. Very helpful when diagnosing such issues and providing tips to the user.

I don't know if you were processing 100 JPG files at the time you experienced this, or IMatch was churning through tens of thousands of RAW and video files stored on a slow NAS. Big difference. The more details you provide, the better I can understand.

Typical "IMatch is slow" reasons are:

- virus checker scanning the database constantly
- too many large files processed and the system cannot copy
- video files being processed
- ExifTool running into a bad batch of files with tons of metadata errors, slowing it down from 5 files per 2 seconds to 1 file per 30 seconds
- System overloaded

If you have a log file, I can tell you more.

[ERBRO]

Hello Mario,

Thank you for the detailled explanation.

The process that frozen iMatch (for 27 hours) was the "Mata Data write-back"-"All pending files".  It as about writing-back about 40 000 files meta data to the NAS.  I suspect you will tell me it is because I write to the NAS and/or my NAS is slow and/or my network is slow.  That is all fine as long as this long process would not refrain me of using iMatch in between what is not the case.

Concerning the dashboard, as the above process just end up, I when to the dash boardboard and it is waiting with the spinner icon. (see print screen).

Why would iMatch again and again "read metadata" as I did not add any files to my NAS ? In addition it looks to imply a kind of loop process. It is reading the metadata... and then indicated they need to be written back.

I also join the log file. 

Concerning the background processes, I set up the number of threads as in attached print screen but it do not imporve things as far as I can see.

I notice that there are always about 8 exiftools runing in parallel (see print screen)

I hope we can solve this performance issue as in actual state it is not usable at all. It would be a pitty s the rest of the iMatch functionlities are very impressive.

Best regards
Eric  imatch_log20241229.txt

[Mario]

Why would iMatch again and again "read metadata" as I did not add any files to my NAS

Writing back produces new metadata, e.g. timestamps, digests (checksums) and more.
IMatch always reloads metadata after write-back, to ingest the actual data produced by ExifTool and by IMatch mapping from XMP to EXIF, legacy IPTC, GPS and other metadata.

It as about writing-back about 40 000 files meta data to the NAS.

Writing back 40,000 files in one go and to a NAS, which is the slowest option, will take a long time.
ExifTool creates a copy of the image, updates the metadata and when everything works fine, deletes the original and replaces it with the updated copy. On a SSD, this is irrelevant. On a NAS, this will be quite slow.

I notice that there are always about 8 exiftools runing in parallel (see print screen)

IMatch uses multiple ExifTool instances all the time, that's normal. The actual number depends on what IMatch is doing, how the process control settings are etc. I usually run 48 ExifTool instances in IMatch 2025 when indexing files on my machine and it works perfectly.

Your system reports 8 processors, which is 4 physical processors and hyper-threading. IMatch users use all kinds of hardware, from computers with only 4 cores to workstations with 32 or even 48 processor cores.

IMatch 2025 uses a much improved background processing system.
But that cannot change the fact that writing back to a large RAW on a SSD takes one second (+ 0.5 seconds for reloading the metadata) and that the same process can take 10, 20 or more seconds.

The log file you have attached is from a database with only 4000 files. Then there is a database with 100,000 files. I assume this is the database used for write-back?

I see quite a number of warnings from ExifTool about bad IIC profiles (color profiles) in your images. Some warnings about invalid timestamps. You can search the log file for W> to find the names of the problem files, and then process them.
Each file with metadata problems will slow down ExifTool a bit.

The log file is not in debug logging mode (see log file) and Help menu > Support. It contains only minimal information and now information or timing data for write-backs.

Please repeat writing back after enabling debug logging. Write back a few dozen files of your typical type on your NAS.
Then copy a folder from the NAS to your local SSD, add the folder to your your database. Make some metadata changes (rating, label, keywords) and write back.

Then use Help menu > Copy Logfile to copy the log. ZIP and attach.

This will show us how long each write-back takes on the NAS compared to the local SSD (to rule out non-NAS related issues) and what probably slows down IMatch.

[ERBRO]

Hello,

Please find attached the new log file with debug active.

I'm still concerned that the pop up coming when we do the write back stay open during all the process of write back. As a result as far as I understand, I can not use imatch at all. If I hit cancel the process stop (see print screen)

On the other side, when reading meta data, there is a "dismiss" button that allows to close the windows (and the process continue in background.  For the write back this looks not to be possible. Is this correct or I still miss something ?

Best regards
Eric 

 

[Mario]

I'm still concerned that the pop up coming when we do the write back stay open during all the process of write back. As a result as far as I understand, I can not use imatch at all. If I hit cancel the process stop (see print screen)

Writing back metadata is very demanding task. Even more so when propagation comes into play and writing a master may mean to write 2, 3 or even 10 versions.

When you write back more than a few dozen files, IMatch shows this progress dialog to inform you about this, and sends the user interface to sleep while utilizing all available processor cores to write files in parallel.
This is the most efficient way to do this.

When you press cancel, IMatch no longer writes-back new files and waits for all current write-back threads to finish.

The typical number of files users write-back (according to IMatch telemetry) is between 50 and 1000 files. Users let IMatch write-back when they don't need it, which is one of the reasons of the entire write-back mechanism.

Writing back 40,000 files in one go is quite the exception.

Looking at your log file, there are 177 warnings, most of them about bad ICC profiles embedded in files. You may want to look at these files and maybe re-save them in a good-quality image editor.

There are also some warnings about bad maker notes. This can happen if a file with EXIF maker notes is saved by an application not aware of them, moving the maker notes around and breaking the data pointers.

A few XCF files fail to load from Z:\2021\09-septembre\Arde... with an "unknown file format error".

There are also 205 "slow" entries. A "slow" entry is logged when an operation takes an unexpected long time.

I see a large number of slow entries for UpdateFileAttributes. This method is used by IMatch to read file attributes from the fiel system, e.g. file size or last modified date.
This is a file system operation and thus "slow" by nature, but the typical time is a few milliseconds, at most. Zero time on a SSD and maybe 0.5s to 1s on slow remote network shares.
On your system, IMatch has to wait 5 to 8 seconds (!) for the file system to return the information about a file. This would indicate that the SAMBA implementation on your server is very slow.

Now looking at the write-back times.

Write backs to the NAS take 20, 30.. 50 or more seconds per DNG file and 2 and 8 seconds per JPG file.
For DNG files on the C:\ disk, write back takes 1 to 2 seconds for DNG files and 0.3 seconds for JPG files.

So the NAS takes about 30, even 50 times as long to write a DNG file than your local disk.

From what I can tell, the problem is the NAS, which is not performing well. Many consumer-grade NAS systems are designed for streaming data, e.g. streaming movies or making backups.

The write pattern IMatch / ExifTool uses during write back (create copy of original image, apply metadata changes, check the resulting file and if successful delete original image and replace it with the updated copy) causes issues for some SAMBA implementations on some NAS systems - they don't handle it well and performance drops to the bottom. Especially if IMatch is using multiple threads to process multiple files at the same time. See also IMatch and NAS systems in the IMatch Knowledge base.

I would suggest you either wait until this is all finished, or your reduce the number of write-back threads to 1 (Edit > Preferences > Application: Process Control) to see if your NAS box agrees more to only one file being processed at a time.
Or, move a folder to your SSD, write back there, and when the files are finished, move them to your long-term archival NAS.

[Tveloso]

Or, move a folder to your SSD, write back there, and when the files are finished, move them to your long-term archival NAS.

This is the approach I started taking quite some time ago, based upon advise given here.

I have quite a large number of files that have yet to be properly updated in IMatch, and when I work on these, the first step is to move them to my local SSD.  Writing back there is quite a few orders of magnitude faster than doing it directly on the NAS.

Many consumer-grade NAS systems are designed for streaming data, e.g. streaming movies or making backups.

The write pattern IMatch / ExifTool uses...causes issues for some SAMBA implementations on some NAS systems - they don't handle it well and performance drops to the bottom.

I can attest to this.  When I get to the point where I'm ready to store a batch of completed files on the NAS, even that is a bit slow...(I've heard that many RAID configurations are fast for reading, but a bit slower for writing). 

In fact, I recently posted about experiencing a significant slowdown in storing files to the NAS (for which I use the Renamer), that I thought could possibly be related to something in IMatch (I noticed the "extra slowness" with my first use of the Renamer, following a new Release of IMatch).  But sometime later, I decided to try re-booting the NAS to see if that could help...and it did.  Afterwards, storing a batch of a few hundred files was back to being relatively quick again (down to a minute or two, compared to the several tens of minutes I was seeing previously). 

But a week or so later, the slowness returned.  I checked the RAID Status and "System Health" on the NAS, and it reports that everything is good, but it seems that there's something squirrely going on there.

[Mario]

Maybe your NAS has a memory leak, and after a week of operation, it struggles because the memory is low?

NAS systems are great for long-term storage, and for keeping your video and music collection. But there is always a chain of technology involved that is not used when you work with local files.

network => SAMBA => Linux => Linux file system

SAMBA is a software that runs on the Linux box and simulates a Windows server. This allows Windows PC's to easily access files stored on the NAS. But there is always overhead.

It also depends on how fast the NAS is (processor cores, spinning disks or SSD storage) etc.

[ERBRO]

Hello,

Just to tell the NAS files are now OK. 

Thank again for yoour help and explanations.

Best regards
Eric

[Mario]

Just to tell the NAS files are now OK. 

What did you change?
A lot of time was spent in discussing your problem and log file analysis. If you let us know what has changed, maybe other users can benefit from it.

[ERBRO]

Hello,

I just wait and wait :-). 

I suspect that the initial write back collapse in the middle, (stopping imatch by the hard way) and leaving the status in Imatch of these 40 000 files as pending for update. I restart twice and at the second run finally all files where processed.

Probabely linked to the speed of the NAS and/or the 1GB connection to it.

Just for my information, is it correct that the manual write back do not remove the files from the files pending for write back ?




Best regards
Eric

[Tveloso]

It's not uncommon for some files to require a second write-back...(especially if they were processed by another software, that didn't do a good job with synching all the metadata).

If you hover your mouse cursor over the write-back pen icon, a tool tip will be displayed showing which tags are still pending (usually it's keywords).  A second writeback should take care of the remaining issues.

If not, and some files continue to return to a writeback pending state, you can run the The Metadata Analyst App on them to get more details on the issue.

[Mario]

Just for my information, is it correct that the manual write back do not remove the files from the files pending for write back ?

No.