Metadata writeback

[Aubrey]

When I go through my images and rate them with metadata writeback set to "Write-back changes to metadata immediately", then it takes a lot of time (2-3) seconds to move to next image.

Is it normal for there to be a delay with metadata writeback when this is set to "immediately" , I had thought that this would be a background task?

Currently I have "Write-back changes to metadata immediately" untoggled and then periodically apply Commands |metadata writeback | all pending files. This works fine in my workflow.

Thanks,
Aubrey.

[Mario]

Please always include a log file. I cannot tell anything without it.

then it takes a lot of time (2-3) seconds to move to next image.

Sounds OK.
You write back one file. IMatch does not enqueue this because this is so fast. But depending on your options, writing back a file will re-import the metadata, update collections, set data-driven categories to pending (which may cause an immediate rebuild if you are working in the Category View, or you use a filter which depends on data-driven categories or collections).

Please give details about what exactly you do, where you do it, which files you process. Attach the log file so we can see exactly what takes how long, which operations IMatch has to perform (depends on your workspace and a dozen other things).

By default background write-back is off for good reasons.

[Aubrey]

Mario,
Thank you for the explanation.The re-importation, update of collections (over 3000 categories in my database etc.), filters that are currently activated... now it makes sense!

I had not realized that by default background write-back was set to off.

There is probably no need to attach log file; your explanation is sufficient and my current workflow works fine.

Thank you again,

Aubrey.

[Mario]

The general problem is that the simple task of writing back metadata can change so many things in IMatch...

Writing back also causes a re-read of the metadata. Because, depending on the chosen options, writing back one tag may cause ExifTool to update several other tags, in order to ensure proper data migration between standards, fulfill MWG rules about digest data etc.

And a re-read of metadata for one file can cause things like:

- Running Metadata Templates
- Re-run File Relation Rules
- Propagate and write-back metadata
- Invalidating all data-driven categories
- Invalidate @Keywords
- Invalidate Collections
- Update the timeline
- Run scripts
- Reload all panels
- Re-evaluate filters
- Reload 'expensive' panels like the Map Panel, Filter Panel (Value filters) Category Panel
- Reload Keyword and Metadata Panels
- Reload History Panel
- ...

There is a lot of stuff going on, even when you do only an apparently simple thing like, say, changing a rating or a keyword.
There are good reasons that other DAM software is much simpler and does a lot less than IMatch 5 

[Panther]

I've been experiencing a number of crashes/problems associated with metadata writing with 5.2.6 and 5.2.8 - as far as I can recall they all seem to have occurred when I've had it set to do the write back "immediately".  Just doing some (what seem like) simple activities such as  adding an author and date created to the metadata of 4 or 5 files at a time (which were scanned in and don't have any metadata otherwise) has caused iMatch to choke from time to time.  Sometimes it will seem to do several small batches like that just fine, and then choke.

Since the problems seemed to be related to the "immediately" setting, I've switched over to doing them in batches of 20-40 or so files at a time, and so far so good.  As an experiment I tried it with a group of about 720 files, and while it took about 30 minutes it got through it just fine.  I'm not really sure why it takes so long, but perhaps it has something to do with all those checks and re-reads iMatch seems to be doing per Mario's post above.  As far as I know, I haven't done and don't have anything complicated in my database dealing with metadata (I only have about 2200 images in my database so far, and almost all my files so far have been images I've scanned in which don't seem to have any metadata in them except for the simple couple of entries I'm trying to make, and I haven't created any special scripts or anything that acts on or depends upon any metadata). 

I'm wondering if there couldn't perhaps be some sort of optional, simpler write process that would just act on the specific metadata being changed without having to go through all those other steps Mario outlined and thus might be a lot faster, for those folks (like me) who don't have databases that use any fancy metadata-related features?  I'm not sure what most of those steps are that are listed in Mario's post above, but perhaps iMatch could check and see whether the database has those sorts of complex metadata-related functions and, if not, could just use the simple write process?  Just a thought.

[Mario]

I would like to see a log file from a write-back of only 700 files which takes 30 minutes. So far I cannot even say which file format you use (JPEG? RAW? 200 MB TIFF files?) or if your files are on a local disk or a slow NAS box...

Write-back for JPEG files is actually pretty quick. For RAW files it is XMP only, except your RAW files contain IPTC/EXIF/GPS data which needs to be updated to keep it in synch with XMP. This can be a serious slow-down.

If your database is simple, IMatch keeps the process simple. If you have no metadata templates defined to be run during ingest, IMatch skips this step. If you don't have data-driven categories, IMatch does not keep them up-to-date. If you don't have 15 panels open which need to keep updates all the time while IMatch is importing files - again, speed improvement.

No file relations / versions: Also, speed improvement.

All the extras cost time and resources - just normal.

Show me a log and I can see why this is so slow. Writing back data to 50 MB each RAW files of course takes a lot longer than writing back data to JPEG files.

As for crashes and hang-ups: I cannot fix them when you don't report them. I've solved maybe 10 potential reasons for hang-ups and locks for the 5.2.6 - just because users reported problems and supplied me with log files and DUMP files.

[Panther]

Mario - the issues I referred to are the same ones we've been corresponding about for the last couple of weeks.  I know you get swamped with such emails and reports so I apologize for not making that clearer.

Since the process with those 700 files eventually completed and everything seemed OK I didn't think about grabbing any log file or reporting a bug, and probably wouldn't have mentioned it until I saw this thread and it seemed like it might be related.

I'm not sure if it's too late to find a log file that would show anything about that process at this point, but if it happens again I'll try to remember to do that.

Another weird thing that may be related happened when I opened up iMatch a few minutes ago.  Nothing had changed with my files since last closing down iMatch yesterday (after writing all those metadata changes to about 850 or so files total and renaming maybe 100 or so using iMatch's renaming function), and the database closing operation appeared to proceed and conclude just as normal.  However, when I started iMatch today it first popped up a window saying it was "importing metadata" from some 850-ish files, and then when that finished and went away the status bar on the bottom right kept saying it was "adding and updating files" for about 8 or 10 minutes.  Since iMatch itself had written the metadata and renamed the files yesterday, I couldn't figure out why it would need to import any metadata or add and update any files today, so that seemed odd to me.  I then closed iMatch (which seemed to close normally) and popped over here to mention it.

BTW - my files are all .tif files, generally ranging from 20-50 MB in size (a few a little smaller and a few a little bigger), and are stored on a local removable drive (USB 3) plugged into my PC.  As to the 700 files issue, maybe that's just how long it takes to do file operations on that many files that large - it wouldn't surprise me too much, which is why (along with all the other things you listed as were going on with metadata writes) I didn't think of that so much as a bug as perhaps just a matter of some possible optimization.  I don't have any metadata templates or data-driven categories or file versioning or anything like that, as far as I know - just some simple categories I've created, some attributes (title and description) left over from version 3.6, and now trying to use three of the standard metadata fields (author, date created and description).  As far as open iMatch windows on screen, I have the list of categories on the left, the preview/thumbnail pictures in the middle, and the metadata and category assignment windows stacked on the right.

At any rate, I've attached the iMatch log file from today, in case it's helpful.  I tried to find some sort of exif tool-related log file but couldn't find anything like that by searching my hard drive (I saw you mention something about it in another thread somewhere but haven't been able to find that thread again yet).



[attachment deleted by admin]

[Mario]

This log file shows just a startup and close of IMatch. No folders have been scanned, no files added or anything.

1. If IMatch is not finished with something like reading or writing data, it will stop when you close IMatch, and continue the processing when you start it again. You can always check the Info & Activity panel to see if operations are pending.

2. IMatch automatically rescans folders with a newer "last modified" file system date than recorded in the database. If you run software which updates folders (including e.g. Windows Explorer, which may update the thumbs.db files or similar) IMatch has to rescan folders.

[Panther]

If you run software which updates folders (including e.g. Windows Explorer, which may update the thumbs.db files or similar) IMatch has to rescan folders.

Wow - that probably explains it.  I opened that folder with Windows Explorer to see how much stuff was in there because I was thinking about making a backup copy of the image files to another drive, and then decided to finish adding some more metadata through iMatch first before making the backup copies, so I then opened iMatch and that's when it started updating.  Thanks for the explanation.

[ubacher]

An aside:
I use the manual update only option of IM. This way I see how often IM sees a folder as having changed.
Even if IM itself did the change - it shows the folder as needing update. Quite often a nuisance.
All in all a useful learning exercise - although my reason for the manual option was to not have things go on
in the background without me knowing it.