Progress bar for pre-write-back preparations

[janb83]

As reported in this topic and discussed here during write-back there are two phases before IMatch actually starts modifying files. The actual file modification already has a progress bar.

When you run write-back for a large amount of files, there is a significantly long waiting phase until the actual modification bar shows any progress (1...n/n instead of just 0/n). The runtime highly depends on your usage of face annotations, but it's significant (~4-8 minutes per 10k files).

This feature request asks for two additional progress bars before the existing one is shown. Currently, pre-write seems to have two phases before file modification starts.

1) A phase before the existing progress bar dialog even opens. This can already take several minutes
2) A phase during which the existing progress bar dialog is opened, but shows 0 for a long time because no files are yet being written.

The proposal, especially for phase 2), is to add a progress bar that shows progress for these pre-write-back phases per file (as it seems to process files file by file). This prevents user confusion and lets users estimate how long it will probably take.

If not handled by a separate bugfix, this would also solve the issue that the existing progress bar cannot be canceled during phase 2) described here.

Additionally, speed improvements to this process would be nice. Looking at procmon, it seems these pre-write-back steps are not utilizing parallelization. As they process individual files apparently, it should be possible to do this in parallel.

[Mario]

Regarding 1) and 2)
There was indeed a check for user cancellation missing. I considered this to be a bug and added it yesterday.
I have also added a separate "Preparing metadata..." message to the write-back dialog, which is displayed with a progress bar while IMatch is preparing metadata for the write-back. You won't see this unless you write-back thousands of files, but...

I've also pondered ways to improve the performance in this section.
For each write-back file, IMatch checks if the file contains face annotations. If this is the case, it propagates the face annotation geometry and tag into one or more XMP face regions.
If the file has no face annotations but the metadata contains face regions in XMP, it deletes the XMP face regions.

These steps take few milliseconds, but it sums up when users write-back thousands of files. Or, as in your case, 100,000 files in one go.

I could improve this considerably by not performing the database operations for each file while it is being processed, but instead use a purpose-build routine which figures out the required information separately and as close to the database as possible.

For a test set of 1,200 files, this improved the total runtime from about 3 seconds to less than 80ms! Which is awesome.

According to telemetry, most users write back a few dozen to a few hundred files. They won't notice much of a difference since the most time of write-back is spent "in" ExifTool reading and writing data. But for users who have to write-back thousands of files regularly, this will be a very noticeable improvement.

[Carlo Didier]

I have also added a separate "Preparing metadata..." message to the write-back dialog, which is displayed with a progress bar while IMatch is preparing metadata for the write-back. You won't see this unless you write-back thousands of files, but...

Great idea!

[janb83]

Thank you, Mario, these sound like definite improvements I'm happy to hear we agree on the definition of a bug in this case as well!

[Mario]

As I said, I considered the too irregular checks for user cancellation during the first phase of the write back as a bug. You only notice this when you write-back hundreds of thousands of files in a go and actually wait for it.
The idea of write-back is to start and let it run when you don't need IMatch.