How long should it take to.....

[Joe Austin]

import 1000 CR2 files, some xmp sidecars and a few jpgs) ?

I am currently importing  a folder with those files.  There has been an "Importing Metadata" window on the screen for 4 hours now with blue bars and green and red lines that advertise disk and system usage.  The window still says (0 of 1108).   

There is steady disk activity, but I am beginning to suspect a problem.  I don't recall 3.6 taking anywhere near this long to import this quantity of raw images.

[Mario]

This should take only a few minutes, even if cache images are generated.

The log file (see How to report bugs) would be required to analyze this in any way.

[Joe Austin]

How can I safely shut this process down?     

Can I dismiss the import window and close Imatch?

[Joe Austin]

I copied the log while the import process was running and then shut Imatch down.

The log has been uploaded to the ftp site with this topic number  - 2531.0

[Mario]

Unfortunately your log file was a bit garbled. Most likely a wrong setting in your FTP tool.
Can you ZIP the file the next time before you upload ti? This not only reduces the size of the text file from 300 MB (!) to 10 MB, but also prevents accidental changes introduced by the FTP software. Thanks.

By the looks of it, IMatch is happily indexing files, writing back modified metadata, reading files in, writing back metadata etc.
It looks like some sort of infinite  loop created by metadata being "created" or changed when IMatch imports the files. You have background write-back enabled so IMatch writes back the file immediately after importing it, and then it imports it again.

1. Disable background write-back under Edit > Preferences > Background Processing.
IMatch will then just mark the files as pending after import.

2. Check which data is modified during the import. Point the mouse cursor at a yellow pen in the file window to see what's shown in the tooltip. My guess would be keywords.

[scope]

Just as a reference; it took me roughly 1.5hours to convert the 3.6 database in v5 - it contained 27.000 photos..

I have still not been able to open that said database file though, as iMatch keeps telling me another process has a lock on the file (after hanging for a while).

[Mario]

Just as a reference; it took me roughly 1.5hours to convert the 3.6 database in v5 - it contained 27.000 photos..

Without seeing the log and protocol files (see Database Converter help for details) I cannot say if this is good or bad. Usually the slow parts are the re-ingest of all metadata using ExifTool (needs to be done once) and thumbnail conversion.


Did you close the converter before opening the database in IMatch 5?
IMatch 5 by default opens files in exclusive mode, and this is only possible if no other application has the file open.
Check task manager for processes named "IMDBConverter.exe" or "imatch5.exe".

[Joe Austin]

Unfortunately your log file was a bit garbled. Most likely a wrong setting in your FTP tool.
Can you ZIP the file the next time before you upload ti? This not only reduces the size of the text file from 300 MB (!) to 10 MB, but also prevents accidental changes introduced by the FTP software. Thanks.

By the looks of it, IMatch is happily indexing files, writing back modified metadata, reading files in, writing back metadata etc.
It looks like some sort of infinite  loop created by metadata being "created" or changed when IMatch imports the files. You have background write-back enabled so IMatch writes back the file immediately after importing it, and then it imports it again.

1. Disable background write-back under Edit > Preferences > Background Processing.
IMatch will then just mark the files as pending after import.

2. Check which data is modified during the import. Point the mouse cursor at a yellow pen in the file window to see what's shown in the tooltip. My guess would be keywords.

To get control of the database I had deleted the folder of images I described above.   I tried again with a different folder containing the same number and types of images, and had the same result.   The import window displayed "Adding images" for about 6 minutes with updates on the number of images added, then switched to "Importing Metadata" where the loop apparently begins.   I just closed the database.

I read your post and opened the database again.   The process of reading metadata began again so I turned off background write-back and closed and reopened Imatch.   Upon reopening, I saw one more cycle of "Reading Metadata" in the status bar at the bottom of the screen and then it switched to "Adding and updating files".   This ran for about 5 minutes and then all images seemed to be imported.   There are no yellow pen icons to follow your tooltip instruction, and the "Files with pending Metadata writeback" filter showed no files.

As I scrolled through the files they looked normal, and seem to have the correct metadata, but I would occasionally see a file that had the blue "updating" icon, and I would see periodic "Adding an updating" files in the status bar.   This would usually disappear a few seconds after I scrolled the file into the file window.

There are also a few images (raw files) with a 90 degree left rotation that they should not have.

I have re-uploaded the original log file (zipped) and also uploaded the log from this latest session (part 2).

Are images not meant to be imported while using immediate metadata writeback?

[Mario]

There are also a few images (raw files) with a 90 degree left rotation that they should not have.

IMatch uses the EXIF orientation flag embedded in the file to rotate images. Check this metadata field (Metadata Panel, Browser mode) to ensure that what is recored in the file matches the real orientation of the image in the file.

but I would occasionally see a file that had the blue "updating" icon, and I would see periodic "Adding an updating" files in the status bar.

When you look at the Info & Activity panel (View > Panels > Info & Activity) you can see if IMatch is still processing data. Also check the information in the status bar (progress bar visible? Flashing busy icon).

Are images not meant to be imported while using immediate metadata writeback?

IMatch always performs two steps when importing files:

1. Reading and importing Metadata
(because the metadata may impact how the files need to be processed)

2. Extracting thumbnail, visual query data, check sum, creating the cache image etc.

When there are files with pending metadata in the processing queue and background write-back is enabled, IMatch will pause import tasks and first perform the pending write-backs.

Usually this is all automatic and you won't notice this (except for IMatch being busy).

In your case, I assume that

a) IMatch created new hierarchical keywords while importing your files (because your files have no hierarchical keywords yet or the ones produced by IMatch as per your settings differ from what's in the file)

b) This marks the file as pending. If background write-back is on, IMatch will thus update the metadata in the file

c) IMatch now re-imports the file again. This is usually where this process ends, because the metadata in the database matches what's in the file.

But on your system and with your files, IMatch does not stop at c) because apparently again new metadata is created. Sometimes this can happen if a user uses keyword import settings, which together with the users thesaurus and the keywords already in the file, create "new" keywords on every import. It's hard to explain generally because this always depends on the settings and files used.

To analyze your problem I would need more info.

1. What are your Metadata and Metadata 2 settings?

2. You say that no yellow pen is shown for the files in your second test. This means that IMatch has not found data to write-back and thus the process is now performing as normal.  How did your 2nd test differ? Other files only or also some changed settings?

[Joe Austin]

For the rotation issue,  I have no idea why, but some, not all,  of my images (again from a 7D) have an EXIF orientation="Rotate 270 CW".   The raw files display normally in thumbs, viewer, and in PS.   Jpegs created from the versions with this EXIF rotation however, get this rotation which was displaying on the CR2 because of my visual proxy.  Jpegs created from CR2s without the EXIF rotation display normally.    Is it correct that Imatch only uses this EXIF rotation on versions and not on the CR2s?

Attached are images of my metadata and metadata2 settings   I see that I left 'default' unchecked for the CR2 file extension in metatdat2, but the values are the default values.

In the first test I started the import, then when it would not finish I dismissed the import window and deleted the files.

In the second test I started the import, saw that is was doing the same thing when it reached "Importing Metadata",  ao I shut down Imatch.   Later read your post about turning off write-back, started Imatch which immediately began doing "Reading Metadata" according to the status bar.   I turned off write-backs,  but it kept reading metadata so I shut down Imatch, and restarted.   Upon restart I saw "Reading Metadata" in the status bar for a few seconds, then it switched to "Adding and updating files".    This lasted for a couple of minutes and then stopped.   After that the images appeared properly imported.

It appears that the automatic write-backs were causing the loop (?)

[attachment deleted by admin]

[Mario]

It appears that the automatic write-backs were causing the loop (?)

Yes, we had assessed that in the first posts. The question was which data has to be written back. Or more than once?
If IMatch imports files and ads new metadata, the file will be pending. Since you have the background write-back off, the files should show a yellow pen. If not, there is something strange going on.

Do you always test with the same files? IMatch may have updated and thus synchronized them already so the behavior will not show again.


You wrote:

There are also a few images (raw files) with a 90 degree left rotation that they should not have (...)
but some, not all,  of my images (again from a 7D) have an EXIF orientation="Rotate 270 CW". (...)
The raw files display normally in thumbs, viewer, and in PS.

And I assume IMatch this displays them rotated CCW by 90°? If so, IMatch is correct. If this is a RAW file, your WIC codec also play a part. And if the RAW file contains an embedded preview, the preview may have been rotated wrong but the RAW not. Without a sample to look at I cannot be sure. So far IMatch was always correct and the rotation issue was caused by the image file or the embedded preview.

[Joe Austin]

I have done several small imports with up to 100 files without any problem.

I have done two imports of folders with 1000+ files that I described above.   These are the ones that have had the problem.
In the first test I dismissed the import window and the files remained greyed out and 'updating' until I deleted them.  No yellow pen.
In the second test the I dismissed the import window, had greyed out files with no yellow pen.  Read your post, started Imatch, cancelled the write-back, restarted, and that's when Imatch started "Adding and updating" following which I got properly imported files.

I'll try a third test with write-back cancelled from the start and see if I ever get a yellow pen to review.

It may not have any bearing on this problem, but these are images that were cataloged in 3.6.   I have copied over folders from my production location to my Imatch 5 testing area to run these import tests.


I am learning more about the rotation issue, it does not seem to reproduce consistently.

[Joe Austin]

I ran a third test on a folder of images similar to the first two tests.   This time write-backs were turned off from the start.   The import window went through "Importing Metadata" and "Adding and updating files" in about 10 minutes.

The file window loaded and this time I had the yellow pen icon on all of the images.

After looking at a great many images, all of the images were pending these tags:  (these were the only tags indicated for CR2 files)

XMP::Lightroom/HierarchicalSubject
XMP::dc/Rights
XMP::Photoshop/Credit
XMP::xmpRights/marked
XMP::xmpRights/Webstatement

Some of the jpeg versions also were pending these:

Composite/Keywords
IPTC::Applicationrecord/Keywords

Some of the jpeg versions were pending one or both of these:
XMP::dc/Subject
XMP::xmp/Rating

Should I try another test of a large folder with write-backs turned on again?

[Joe Austin]

Mario, I decided to try the same folder I had imported in the original test (whose log you reviewed), but this time with immediate write-back turned off, to see if the set of tags that would be pending write back would be different than above.

I sampled about 150 images in a set of 1000.   

The results were the same as above with the addition of the XMP::dc/Subject tag for a group of CR2 files.

[Mario]

That looks as if IMatch knows that it is doing.

Most of the write-backs are caused by IMatch importing IPTC keywords from your files and adding them to XMP.
I would need to see one of those files and your settings for IPTC Import and Export used for this test (if different from the screen shot posted above somewhere).

I wonder about the credit / marked / webStatement because I cannot recall from which IPTC or EXIF fields these are filled. We're not talking about CR2 files here which have embedded IPTC/EXIF and embedded XMP and a sidecar file as well and you let IMatch update only the XMP sidecar file?

[Joe Austin]

I would need to see one of those files and your settings for IPTC Import and Export used for this test (if different from the screen shot posted above somewhere).

I'm not sure what files you want to see,  CR2, jpegs (with which pending tags?), both?   My settings have not changed for IPTC import/export.

I wonder about the credit / marked / webStatement because I cannot recall from which IPTC or EXIF fields these are filled. We're not talking about CR2 files here which have embedded IPTC/EXIF and embedded XMP and a sidecar file as well and you let IMatch update only the XMP sidecar file?

I don't understand the question.   The folders being imported have CR2 files with embedded IPTC, not sure about embedded XMP,  they also have sidecars which are used presumably by Imatch 3.6 and Adobe Camera raw.  I am not allowing embedded IPTC updates to CR2's in Imatch 5.

[Mario]

I had a look at the log files.

IMatch reads metadata from your CR2 files at a rate about 0.2 - 0.4 seconds per file (20 files in 4 to 8 seconds).
The load times for the CR2 files vary between 0.3 and 0.6 seconds per file. Creating thumbnail and full size cache images (3456 x 5184 pixel) takes a total of about 2 seconds per file.

To speed up the processing switch the cache to on-demand. This will speed up the ingest considerably but cause a short delay when IMatch first needs to access the cache file (Viewer / Quick Preview).

[Joe Austin]

Mario,   I think the title of my post (written when I did not have an understanding of what was happening) clouds the real issue.   

The time to read each file is not the problem,  2 seconds to ingest and cache each image is fine.   The problem is that the images were always pending write-backs no matter how many times I gave the command to process all pending writebacks. 

I have not had this problem since discovering the cause of the problem I described in this issue:

https://www.photools.com/community/index.php?topic=2309.0
(see my posts of June 21-22)

This was causing endless cycling of writing-back and reading metadata whenever automatic metadata write-back was enabled.

The lingering question on that issue is why those keywords, unintended thought they may be, created a write-back situation that could not be written and therefore was always pending.

[Mario]

I just went through "open" log files / downloads / samples provided by FTP and I thought the initial question "How long..." was not answered. I can now delete the log files provided for this post.