@Keywords not updating or reflecting changes in Thesaurus despite refreshing

[Damit]

I have noticed that most, if not all of the new keywords I have added recently to the thesaurus are not being reflected in the @Keywords category. I have assigned those keywords to photos, and the keywords still do not appear in the category either. I have gone to the keywords in the keyword panel and clicked "Go to Category" in the context menu and nothing happens. The category is set to show empty categories and auto refresh is also enabled. As far as I understand, this is not normal behavior for iMatch.

I also noticed the @Keywords category folder constantly shows the blue circular arrow, signifying it needs to be refreshed. I have attached a zip of the log where the last thing I did was refresh all data-driven categories. I am not versed in reading the log files but I do see entries showing the category was refresh ("02.14 08:06:16+    0 [93D4] 10  I> RefreshGroup-Completed: '@Keywords' in 15 ms."). I have refreshed many times, sometimes just the pending categories other times everything.  I have restarted iMatch. I have optimized the database. I have searched this forum and read the help section and I cannot figure out why this is happening. Any help would be appreciated.

[Mario]

Quite a large number of warnings and even errors about unsupported file formats in that log.
I see many RefreshGroup-Skipped entries, where one or more categories skipped updating because IMatch was busy ingesting files. Since each new file invalidates most categories, it makes no sense to update them during phase of high activity. These logs are also near the end of the log file, which indicates that IMatch was still busy.

The database has over 250,000 files, and depending on what options you have configured, how many "expensive" categories there are, if they are expanded (and thus visible) etc. IMatch may keep the system very busy.

Let the system complete the background processing and the categories will update at the next opportunity.

[Damit]

How do I see if there is background processing going on other than the info & activity panel, because I see nothing going on there.  I am looking at the info & activity panel and it says DB idle Time 5 minutes and Idle Time 2 minutes.  I do not see it processing anything and some of these entries are over 48 hours old.  I would think iMatch would be done processing seeing that I had not touched it for 10 hours prior to trying these refreshes.

In addition, should not a manual refresh do it immediately? "Under certain conditions, the @Keywords category needs to be manually refreshed, e.g., when IMatch thinks that updating it now in the background would interfere too much with your workflow." That statement seems to imply that doing a manual refresh would force iMatch to do it immediately.

I tried to refresh again, and I see "refreshGroup-Completed: '@Keywords' in 15 ms." Does this not mean that the category was refreshed?

Yes, I see the errors and warning.  I hope to one day be able to understand how to address some of them.

[Mario]

The last entry in your log file related to @Keywords is

RefreshGroup-Skipped (Queue Busy): '@Keywords'.

which indicates that background processing is busy.  I cannot see your system or the activity indicators in the Activity Panel or the Dashboard. All the info I have is from your log file.

Close IMatch. Restart your PC, just in case.
IMatch updates non-manual categories after loading a database, before background processing starts.

[rolandgifford]

How do I see if there is background processing going on other than the info & activity panel, because I see nothing going on there.

I use Windows Task Manager
 

[Damit]

Ok, I will try what you suggested, but I searched that log and the last two entries with @keywords are this:

02.14 08:58:46+    0 [93D4] 10  I> RefreshGroup-Skipped (Queue Busy): '@Keywords'.

02.14 08:58:46+    0 [93D4] 10  I> RefreshGroup-Completed: '@Keywords' in 16 ms.

Am I not getting something? Doesn't completed mean it was completed? Should this not result in the keywords being reflected in @Keywords?

The missing keywords (and there are many) were added prior to this entry.

[Mario]

RefreshGroup-Completed is always emitted, even if the operation was skipped.
The minimal runtime of 16ms also indicates that no work was done.
Otherwise I would have commented differently.

The missing keywords (and there are many) were added prior to this entry.

So the problem is no more?
I'm confused.

Note that refresh group may complete without being skipped, it might be as fast as 16ms when there is no work do to, a skipped entry my follow immediately when the refresh was triggered quickly afterwards but then the background processing queue is busy etc. All these operations, and many of them, run at the same time. This is non-deterministic.

I get that the problem has been fixed automatically!?

[Damit]

Thanks for clarifying about the Complete message.  That was confusing.

No, unfortunately, the problem is not fixed.  The missing keywords are still missing in the @Keywords category. I reset the computer and restarted iMatch and the Keywords are still not reflected in @Keywords. I really believe there is an issue.

I did find that I have 195,151 unassigned files. Just in case this gives a clue to what is going on.

[Mario]

@Keywords is dynamically built from the hierarchical keywords in your files. It is refreshed very often.
When you force a refresh of the category, IMatch recreates it by fetching all keywords for the files in your database from the database.

No other user reported a similar problem, no open bug reports etc.
Something similar was never reported, as far as I know.
Data-driven categories are dynamic and always rebuilt from the actual metadata cached for your files in the database.
Not much that can go wrong, especially not for "simple" categories like @Keywords.

In your initial post you wrote:

(...) not all of the new keywords I have added recently to the thesaurus are not being reflected in the @Keywords category

This is normal. The thesaurus is independent from @Keywords. Adding keywords to the thesaurus does nothing, unless you actively assign the keywords to files in the Keywords Panel.

Or, when you modify or re-arrange keywords in the Thesaurus you use the option to apply the changes to existing keywords in your database.

Try that:

Restart IMatch.
Select a file.
In the Keywords Panel, add the keyword ABCD to that file and commit your changes.
Switch to the Category View and watch @Keywords. It should refresh after a short while and show ABCD on the top-level, with one file assigned.

that I have 195,151 unassigned file

If you refer to the default Unassigned Files category with that, as the description explains:

This category contains all files not directly assigned to a category.

This excludes files assigned to data-driven categories or files in formula-based categories.
Directly assigned means you assigning a file directly to a category.

[Damit]

The thesaurus is independent from @Keywords. Adding keywords to the thesaurus does nothing, unless you actively assign the keywords to files in the Keywords Panel.

As I said in my original post

I have assigned those keywords to photos, and the keywords still do not appear in the category

So, these new keywords have been applied to photos and they have been saved. Then all the pending files had their metadata written to it. And it has been days, for some of these keywords, since I have done this. I have also tried to refresh several times, as you know.

No other user reported a similar problem, no open bug reports etc.
Something similar was never reported, as far as I know.

This is not the first time you have said that you have never seen something I have reported. One such instance was last year, when I had 2 @Keywords and you said that was impossible to do that and you could not replicate creating a secondary @Keywords. Maybe that issue is still unresolved, even though I followed your instructions to resolved the problem. I only have one @Keywords now. But maybe the problem was not fully resolved?

Restart IMatch.
Select a file.
In the Keywords Panel, add the keyword ABCD to that file and commit your changes.
Switch to the Category View and watch @Keywords. It should refresh after a short while and show ABCD on the top-level, with one file assigned.

I did this shortly after you posted and the result is the same.  If I click on the file I see ABC in the panel.  When I click "Go to Category" a small window appears for a millisecond and then nothing happens.  I do not get sent to the Category like I with other keywords that have existed in this database for relatively longer. But if I click "Find" in the context menu, iMatch finds the file and displays it in the Media & Folders view under a panel named "Keywords." It shows just that one photo, but again, it is not reflected in the @Keywords category.

I restarted and did this again with the keyword DEF and the result is the same.

[Mario]

So when you add a keyword in the Keywords Panel, the keyword is not showing in @Keywords after the category has been updated?

[Damit]

You are correct, it is not showing after updating.  I just checked again. It is applied and read from the file I applied it to, but it is not showing up in @Keywords. If I go to the Keyword in the Keyword Panel and I click "Find" in the context menu, iMatch will display the file in the File and Media Window, so it can find the file and the keyword has been applied, and yet, if I click "Go to Category" nothing happens because the category has not been created.  As far as I know, this is not how iMatch is supposed to work.

[Mario]

Strange. Run a database diagnosis and see if this reports any problem.
Upload your database to your cloud space and send a link to support email address.
When I can reproduce the issue here, I can advice or fix it.

[Damit]

So I kept trying to work on this while you are looking at my database and I wanted to report some findings:

I made another database and imported one folder.  That database seems to work fine.  I added ABC and it came up immediately in the categories.  Also, stubborn Dublin Core and IPTC application data that would not delete, even using exiftool, now are not present. Which is wierd, because if I open the old database (the one I am hoping to rescue), the values still exist! It is as if iMatch was not letting me change XMP data and all changes I made were only reflected in the database and not on the actual file, but only in iMatch. Well at least partially.  It would show in the keyword panel and when I would as iMatch to find the file in the context menu of the keyword panel, it would find the file, but it was not reflected in the categories as if somehow it was not writting the information in the file or perhaps in the proper place of the file. As, while using other programs I could see the added keywords, but I could still not remove the stubborn Dublin and Application data. That still remained. 

So maybe there is a setting, perhaps in the metadata preferences that would result in this behavior of protecting the legacy data?  I don't know, but that might explain my I have so many warnings about keyword mismatches from the metadata mechanic. I was thinking of just reimporting all the files, but I think it is imperative that I figure out what is going on, so it does not rear its ugly head again.  I have everything customized in this set up including some complicated file relations (which I would have to study my notes for a long time to re-figure out), so I would prefer to keep things as they are.

Update: So I loaded the old database, and all of a sudden a bunch of keywords started appearing in the @keywords category.  It seems all of the missing keywords are now present, BUT the problem with the legacy Dublin & IPTC Application Keywords remain with this database. As soon as I writeback data, the old keywords reappear even though I deleted them in the keyword panel and clicked on the green checkmark. Once I do that, I see the problematic keywords disappear but when I writeback, they reappear in the metadata panel, but NOT in the keyword panel.  This is extremely bizarre!

I then listed and copied the exiftool output for the file in the new database and then a copy of the exiftool output of the file in the old database after rescanning and writing the information without changing anything.  I just hit a rescan and metadata write-back and you can see the keywords change and the Who|In Pic|Alex....  rather than the Who|People|In Pic|Alex.... reappears in the Dublin Core & IPTC Application record.  I am not sure why the old database is so insistent on keeping these values, but at least it is now recording the old values.

[Damit]

I have attached a copy of the exiftool log for the same file in the old and new database in case it may help determine what is going on. I also loaded my People from the old database to the new database and that had no effect on the keywords.

[Mario]

As I wrote in my initial email reply, the first analysis of the database reveled that the number of files shown in @Keywords (40,699) matches the number of keywords in the database, and adding new keywords to files updates @Keywords within a few seconds. Looks perfect.

Also, stubborn Dublin Core and IPTC application data that would not delete, even using exiftool, now are not present.

If not even ExifTool can delete the data, how comes IMatch into play?

because if I open the old database (the one I am hoping to rescue), the values still exist! I

Yes. Why not?
When ExifTool has imported XMP metadata, it keeps it in the database. Unless you delete the contents of tags, e.g. in the MD Panel.

Did you write back changes to files in the "new" database before opening the files in the "old" database?

When you modify metadata of files managed in IMatch, IMatch re-imports the data into the database again. Unless you have enabled metadata protection under Edit > Preferences > Metadata 2, IMatch will not retain any metadata in the database when it ingests files. Metadata protection can protect metadata modified in IMatch and in the external file during re-ingest.

that would result in this behavior of protecting the legacy data?  I

Check metadata protection. Also note that Dublin Core and IPTC Core are not legacy metadata. They are actually modern metadata. IPTCCore and IPTCExt is what has replaced legacy IIM IPTC metadata, for example.

Update: So I loaded the old database, and all of a sudden a bunch of keywords started appearing in the @keywords category.  It seems all of the missing keywords are now present,

That's exactly what I see here and what I've reported to you on Thursday (15.).

As soon as I writeback data, the old keywords reappear even though I deleted them in the keyword pan

This usually indicates that the file has out-of-sync keywords or (for RAW files) an embedded XMP record and an XMP sidecar file.

Please run  the Metadata Analyst for the file and use the GREEN button to copy and paste the results.

See also Metadata Problems and Pitfalls for typical causes for this problem and solutions.

Upload a sample image that causes this problem and post a link.

Note: I've noticed that some keywords you have created end with a blank (space). That's something that often causes issues, because blanks at the beginning and end of keywords may be stripped during processing, causing a mismatch between keywords that still have the blanks and others that have been cleaned.

I'm currently looking into the diagnosis issue about "missing person keywords" that might be caused by this issue.

[Damit]

As I wrote in my initial email reply, the first analysis of the database reveled that the number of files shown in @Keywords (40,699) matches the number of keywords in the database, and adding new keywords to files updates @Keywords within a few seconds. Looks perfect.

Yes, as I reported, the same thing happened with mine after I experimented with the new testing database.  In the new testing database, all the values showed up immediately, and then they showed up in the old database when I opened it up afterwards. That was very strange.  The only thing I think I may have changed, though I am quite sure I did not, was changing the Background indexing option, which I had set to off while trying perform another task. Would having the background indexing off stop the @keywords to stop populating even if you ask iMatch to manual scan those files and also asking iMatch to refresh the Categories?  If so, that may have been the reason the new keywords were not populating.

Also, stubborn Dublin Core and IPTC application data that would not delete, even using exiftool, now are not present.

If not even ExifTool can delete the data, how comes IMatch into play?

Because iMatch may be "protecting" these values and either not deleting them or repopulating them after exiftool deletes them (See below).

Yes. Why not?
When ExifTool has imported XMP metadata, it keeps it in the database. Unless you delete the contents of tags, e.g. in the MD Panel.

The Problem is that iMatch has a lock on these values and I cannot delete them.  I wish I could.

Did you write back changes to files in the "new" database before opening the files in the "old" database?

I went back and wrote the changes to the Dublin Core and Application Record on the new testing database.  I then went back and opened my old database and rescanned the file. I then ran the exiftool to list the metadata. It reported the files containing:

[XMP-dc]Subject: Who|People|In Pic|Alex M.....
[IPTC] Keywords: Who|People|In Pic|Alex M.......

But the metadata panel on the old database lists both the Dublin Core XMP, and the IPTC ApplicationRecord Keywords as
Keywords: Who|In Pic|Alex M...... without its hierarchical parent "People" preceding "In Pic"

Again, this was all in the old database after writing the data to the file in the new database, with the appropriate keywords, as reported in the exiftool. I also made sure that the metadatachanges writen in the old database actual were written as the pencil did not reappear on the file. 

I then went back to the new testing database, rescanned the file and the new database listed everything as it should with the correct keywords including the "People" hierarchical parent keyword both in the exiftool and the  Metadata panel.

So, it seems that, for some reason, the old database wants to hold on to those old values. To confirm this, I wrote the metadata changes in the old database and opened up the file with the exiftool in the new testing database and it confirmed that the old database had converted the Dublin Core and the Application Record back to Who|InPic|Alex M....
BUT
This was only the listing in from the exiftool. If I looked in the metadata panel, the listings still retained the "People" hierarchical parent keyword and when I wrote the changes, these values remained.
So, after this experimentation it seems that there must be a setting protecting the values already scanned.

When you modify metadata of files managed in IMatch, IMatch re-imports the data into the database again. Unless you have enabled metadata protection under Edit > Preferences > Metadata 2, IMatch will not retain any metadata in the database when it ingests files. Metadata protection can protect metadata modified in IMatch and in the external file during re-ingest.

And this is the setting that I think may be causing my "problems." I think settings "Protect existing XMP" and "Keep existing XMP" probably are very beneficial, especially the latter, in most cases, but I think they may be causing the unwanted metadata to "stick." However, after turning both those options to off and trying to write the corrected metadata (by first changing it in the new testing database and updating it in the old database), the old values still remain.  And vice versa, when the old database changes the values, as confirmed by exiftool in the new database, when I try to write those "wrong" values, the new database overwrites them and keeps the "correct" keywords.  Like I said, something is preserving those values and it seems to be something other than those 2 settings.

quote author=Mario link=msg=98648 date=1708156355]

As soon as I writeback data, the old keywords reappear even though I deleted them in the keyword pan

This usually indicates that the file has out-of-sync keywords or (for RAW files) an embedded XMP record and an XMP sidecar file.

Please run  the Metadata Analyst for the file and use the GREEN button to copy and paste the results.

See also Metadata Problems and Pitfalls for typical causes for this problem and solutions.

Upload a sample image that causes this problem and post a link.

Here is the metadata analyst output for the file in the new database:
Metadata Analyst Results. Version 2023.7.2. 2/20/2024 6:09:24 PM
File analyzed: N:\[] Personal\2008-04-25-Wedding and Reception pictures\337_277502_IMG_1337.JPG
Errors: 0
Warnings: 15

Warning: [Detailed Validation] ExifIFD tag 0x9010 OffsetTime requires ExifVersion 0231 or higher
Warning: [Detailed Validation] ExifIFD tag 0x9012 OffsetTimeDigitized requires ExifVersion 0231 or higher
Warning: [Detailed Validation] ExifIFD tag 0x9011 OffsetTimeOriginal requires ExifVersion 0231 or higher
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa002 ExifImageWidth
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa003 ExifImageHeight
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x011a XResolution
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x011b YResolution
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0128 ResolutionUnit
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x011a XResolution
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x011b YResolution
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x0128 ResolutionUnit

And here is the metadata Analyst results from the old database after trying to write the "correct" Keywords with the parent hierarchical keyword "People"
Metadata Analyst Results. Version 2023.7.2. 2/20/2024 6:13:32 PM
File analyzed: N:\[] Personal\2008-04-25-Wedding and Reception pictures\337_277502_IMG_1337.JPG
Errors: 2
Warnings: 15

Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match IPTC keywords.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match XMP keywords.
Warning: [Detailed Validation] ExifIFD tag 0x9010 OffsetTime requires ExifVersion 0231 or higher
Warning: [Detailed Validation] ExifIFD tag 0x9012 OffsetTimeDigitized requires ExifVersion 0231 or higher
Warning: [Detailed Validation] ExifIFD tag 0x9011 OffsetTimeOriginal requires ExifVersion 0231 or higher
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa002 ExifImageWidth
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa003 ExifImageHeight
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x011a XResolution
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x011b YResolution
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0128 ResolutionUnit
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x011a XResolution
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x011b YResolution
Warning: [Detailed Validation] Missing required JPEG IFD1 tag 0x0128 ResolutionUnit

As you can see the old database displays the error you cited.  I have noticed this error on A LOT of my files. It is curious that the values that cause the file to be out of sync are the old values that will not allow themselves to be deleted.
Here is a copy of the file:
Problem file

Note: I've noticed that some keywords you have created end with a blank (space). That's something that often causes issues, because blanks at the beginning and end of keywords may be stripped during processing, causing a mismatch between keywords that still have the blanks and others that have been cleaned.

I'm currently looking into the diagnosis issue about "missing person keywords" that might be caused by this issue.

This is worrisome.  Is there a way to detect and eliminate trailing spaces from keywords?  I work with a music metadata manager that will allow you to search for and delete such problem files.  Hopefully iMatch can help me in a similar way.

You also mentioned to me in the email that there was a person with a space at the end of their name, can you let me know which one?

I have noticed that alot of my problems on individual folders are resolved when I reimport them into iMatch forcing an update of all files.  Do you think reimporting all folders would be a good step to take in resolving these problems? Can this approach be problematic? If so, how so?

[Mario]

But the metadata panel on the old database lists both the Dublin Core XMP, and the IPTC ApplicationRecord Keywords as
Keywords: Who|In Pic|Alex M...... without its hierarchical parent "People" preceding "In Pic"

Check your thesaurus and make sure "People" is set as a group level keyword.

Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match IPTC keywords.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match XMP keywords.

Check your thesaurus for group levels.
Check your keyword flattening settings under Edit menu > Preferences > Indexing. See Write hierarchical keywords

This warning typically means that IMatch cannot fix out-of-syn issues between hierarchical and flat keywords (legacy IPTC keywords usually) by itself, because your thesaurus and flatten keyword options don't allow it.

See Keyword Problems in Metadata Problems and Pitfalls for typical situations and how to fix them manually.

I think settings "Protect existing XMP" and "Keep existing XMP"

Protect existing XMP ensures that metadata you have modified in IMatch but not written yet is not overwritten when you also change the file in another application. See Protect existing XMP
See Keep existing XMP
You want these settings to be enabled. They are unrelated to your problem.

Is there a way to detect and eliminate trailing spaces from keywords? 

Sure. Search for them using a regular expression that looks for spaces at the end.
See Regular Expressions

I have noticed that alot of my problems on individual folders are resolved when I reimport them into iMatch forcing an update of all files.

Then just write-back all files and then perform a forced rescan by selecting a folder and then pressing Shift+Ctrl+F5 or by selecting the files in a File Window and press Shift+Ctrl+F5 > Force Rescan.
Not sure if this can fix any problems or which. IMatch already did that, else you would not see the files. Unless you use an external software that modifies files but does not reset the "last modified" timestamp.
Or you have enabled the option for ExifTool to not update the last modified timestamp (keep original file date and time) which is well hidden but available for expert users who need to fix expert-level problems.

[Damit]

I think settings "Protect existing XMP" and "Keep existing XMP"

Protect existing XMP ensures that metadata you have modified in IMatch but not written yet is not overwritten when you also change the file in another application. See Protect existing XMP
See Keep existing XMP
You want these settings to be enabled. They are unrelated to your problem.

Thank you! I was wondering what I should do, and you confirmed that my inclination was correct.

Check your keyword flattening settings under Edit menu > Preferences > Indexing. See Write hierarchical keywords

I did not find this in Indexing, but found it under the Metadata tab and I had/have it set to "Write hierarchical keywords."

But the metadata panel on the old database lists both the Dublin Core XMP, and the IPTC ApplicationRecord Keywords as
Keywords: Who|In Pic|Alex M...... without its hierarchical parent "People" preceding "In Pic"

Check your thesaurus and make sure "People" is set as a group level keyword.

No, People is not a Group Level keyword.  I do not use group level keywords because I do not like how, with iMatch, the @Keywords will not list group level keywords which screws up how things are organized visually and inhibits my ability to swiftly edit keywords using the @keyword category. I really wish there was an option to use them as such, but I understand the logic that since they are not technically a keyword, so they are not listed as such in @Keywords, but they are groups and could be listed as such in @keywords. But I digress.

Either way, I do not understand how if People was a group level keyword, why it would be listed in one database one way and in another another way BUT when you asked me to check I did see that I set the "Exclude in flat keywords" to "Yes" and when I switched that to "No," everything started showing the proper hierarchical keywords.

So, why was that happening? In your manual, it states: "If this is enabled, IMatch will not include this keyword when mapping from hierarchical keywords to flat keywords." But Application record and Dublin Core are not flattened keywords, AFAIK. All the other components are shown in these hierarchical keywords and, as such, these are obviously not flattened keywords, so, why are they not listing "People" if it is not set as a Group Keyword? I know it is not supposed to be shown in flattened keywords, but it should be showing here.

This has been a huge headache for me. It was the reason for my confusion and this option seems to have created duplicate people keyword categories in Who|In Pic when the files contained therein should have been place in their counterparts in Who|People|In Pic|. Furthermore, it seems as it was doing this with some of my files and not others, mainly to the files I initially imported.

I suspect that I may have changed this "Exclude" setting after assigning some files to "People," and this is why it could be doing it to the earlier files? Maybe iMatch does not retroactively change the keywords to reflect this change in the thesaurus? Maybe that is why when I reimport these files, and perhaps in doing so affect this change, some issues seem to go away? I would think not, because once I changed the option to NOT exclude the keyword in flat keywords, all the files now showed the correct structure, with "People" included, and I was not asked to apply changes to the database, so I assume this is not something that has to been written to the file and is handled internally through the database, becuase, if not, why did they change automatically in some of my files?

I assume that not including a keyword that is not a parent but has the "Exlude in flat keywords" option is not the way things are supposed to work. I assume it is supposed to list it in hierarchical keywords, so is this a bug?

I really like to use the "Exclude in flat keywords" option because it at least allows me to remove these words from flattened keywords, but I can still use them with @Keywords to visually organize my files, as I would like to, but cannot do, with groups for aforementioned reasons.

Also, could you please answer if having the background indexing off stop the @keywords to would populating even if you ask iMatch to manual scan those files and also asking iMatch to refresh the Categories?

Sure. Search for them using a regular expression that looks for spaces at the end.
See Regular Expressions

How? I can figure our the expression but where do I use it?  Which app do I use?

Or you have enabled the option for ExifTool to not update the last modified timestamp (keep original file date and time) which is well hidden but available for expert users who need to fix expert-level problems.

I have not used or enabled this as I do not even know where it is, but this would have been a good option for when I was trying to correct erroneous and missing face detection in iMatch by replacing existing files with backup files, recreating Face regions and annotations with XMP data and then saving the existing metadata (that had been changed since the backup) from the database without the files being first rescanned by the system.

[Mario]

BUT when you asked me to check I did see that I set the "Exclude in flat keywords" to "Yes" and when I switched that to "No," everything started showing the proper hierarchical keywords.

So, why was that happening? In your manual, it states: "If this is enabled, IMatch will not include this keyword when mapping from hierarchical keywords to flat keywords." But Application record and Dublin Core are not flattened keywords,

This setting is off by default for all keywords.

XMP keyword and legacy IPTC keywords are flat keywords. They don't handle hierarchies.

I assume that not including a keyword that is not a parent but has the "Exlude in flat keywords" option is not the way things are supposed to work.

This is the purpose of this option. To exclude keywords when flattening hierarchical keywords. This is a feature that deals with some very specific problems some IMatch user types have to deal with when exchanging files with clients and agencies. 99% of all users never need to change this setting or even consider it.

I really like to use the "Exclude in flat keywords" option becaus

Please don't. This is for expert users only.
It also requires a very strikt thesaurus management, else IMatch will be unable to re-map the flat keywords to hierarchical keywords during re-import - because unless it's in the thesaurus, there will be no way to map flat keywords with missing parts (levels) back the hierarchy. Resulting in keywords on wrong levels, duplicated keywords on different levels. This can get real messy real quick and then a lot of manual repair has to be done.

How? I can figure our the expression but where do I use it?  Which app do I use?

You can do that in the File Window search bar, for example. In a value filter in the Filter Panel. Formula-driven categories.

See the Search Engine and Finding Files: The Search Bar

Make yourself acquainted, try the examples, maybe use the RegExp Tester app in IMatch to make some experiments. Time well spent.

To find data that ends with some specific character, just apply the corresponding example given in the help topic I linked. I'm sure there is one in there. In general: <text or character>$
Note that | has a special meaning in regular expressions. It's all explained in the IMatch help.
Knowing a bit about regular expressions will help you solve tricky search questions in the future.

[Damit]

XMP keyword and legacy IPTC keywords are flat keywords. They don't handle hierarchies.

So to understand better, why are they listed as such in the metadata panel.  I don't mean explicitly, but in the panel both the Application Record IPTC and the Dublin Core XMP have values such as Who|People|In Pic|Abillio Ramirez; Who|People|In Pic|Dairys Ramirez; Who|People|In Pic|Luca Salazar etc.  Are these not hierarchical keywords? Does not the inclusion of the "|" make it a hierarchical keyword? I thought I had at least learned correctly that flattened keywords would only have the final value in their names, like the XMP Adobe Lightroom weighted flat subject. I am confused!

This is the purpose of this option. To exclude keywords when flattening hierarchical keywords. This is a feature that deals with some very specific problems some IMatch user types have to deal with when exchanging files with clients and agencies. 99% of all users never need to change this setting or even consider it.

OK, but may I suggest you include some information/warning on that in the manual or help text pop ups?  I had no idea what its intended purpose was, only what I could use it for. I do manage my thesaurus strictly, but I will try to avoid using this option, for now. Again, if @Keywords could represent group level keywords (I am not the only one who wants this, I have read others stating similar sentiments) then this would not be needed. I just find it very confusing to have one representation in the Thesaurus and, if you will, the keyword panel, and another in the Category Panel.

Thanks for the tips in finding the problematic keywords with spaces at the end.  I will check your examples and hopefully will be able to figure it out.  Yes, I know a bit about regular expressions, and every bit I learn I seem to forget, but I can get reacclimated!

Lastly, 2 unaddressed questions:
So, am I correct that using this "exclude in flat keywords" caused the problem with the two people keyword categories one with "People" in it and one without? I had two sets and most people where duplicated, some under Who|In Pic and some under Who|People|In Pic.

Also, can you please answer if having the background indexing off stop the @keywords to would populating even if you ask iMatch to manual scan those files and also asking iMatch to refresh the Categories?

I am still getting those 400+ errors of missing keywords that keep repeating despite the diagnostics says they are fixed, I am still waiting on you to resolve that, correct?

Lastly, I keep getting files that need metadata written to them even though I have not touched the program for long periods of time.  If I do not make changes to the files I thought this would not happen, but it seems iMatch is constantly updating the files even though I am not making changes.

[Mario]

I don't mean explicitly, but in the panel both the Application Record IPTC and the Dublin Core XMP have values such as Who|People|In Pic|Abillio Ramirez; Who|People|In Pic|Dairys

Sigh. It's just text. When you set IMatch (as you said) to write hierarchies to flat keywords, it will exactly do that.
Edit only hierarchical keywords in IMatch in the Keywords Panel. Leave the other keyword tags alone.

Thanks for the tips in finding the problematic keywords with spaces at the end.

I gave you the regexp to use. When you don't know how to search in keywords, revisit The Search Engine and Searching

So, am I correct that using this "exclude in flat keywords" caused the problem with the two people keyword categories one with "People" in it and one without?

Could be. Probably. Messing with the exclude settings can have very different effects, depending on what keywords where in the file before writer-back, your thesaurus and whatnot. I don't have the time to analyze all this for you.

Cleanup the keyword mess your thesaurus settings have created and all this will work for you as it does for all other users.
Again, see Metadata Problems and Pitfalls for pointers.

Sorry, but so much time spent already on this one problem of one user. So much text to read. Non-standard thesaurus settings used causing harm etc. There are many other users who also need my support.

I am still getting those 400+ errors of missing keywords that keep repeating despite the diagnostics says they are fixed, I am still waiting on you to resolve that, correct?

Just ignore. No harm done. The diagnosis is wrong, not the keywords.
As I said maybe a Kilometer above in this thread, I will look into it. Problems encountered by one user only and only with special circumstances like keywords ending in a blank have priority 3. I will mention it in the release notes when I have made some changes regarding this particular issue.

[Damit]

Well, unfortunately, on more than one person and on many occasions, MY iMatch is not behaving. I removed all "Exclude" options (I only had one other one with this option enabled) from the thesaurus. They are all set to default except one "Group"Keyword that has nothing to do with these keywodrs, hierarchically.

I had a bunch of files that had Mona Lisa as a Person.  I moved her to a Keyword under Who|Art Subject and added that Keyword to her "Person."  I did this about 60 minutes ago.  The keywords are present, but the category in not there.  I have refreshed many times. I even did a reload with a force update and that did not populate the keyword. Please see the screen shot:

https://imgur.com/3OY8H4f

You can see that the image has the keyword, the face is confirmed, but the Category is not there. This is after many refreshes, and 2 reloads. This is just one example of many files with different Persons that is doing this. I looked into the thesaurus and Mona Lisa has the default settings. Indexing is set to on. The faces are confirmed.

[Damit]

And here is another example of the opposite. The @Keywords are reporting that the photo has keywords that the keywording panel is not even listing:
https://imgur.com/a/lxc3Ij4
These keywords are not listed in the exiftool output either:
[ExifTool]      ExifTool Version Number        : 12.76
[System]File Name: img106.tif
[System]Diectory: N:/[] Mario P.....
[System]File Size: 85 MB
[XMP-lr] Hierarchical Subject: When|Estimated|Estimated Day & Month, Where|Regions, Where|Regions|North America, Where|Regions|North America|United States, Where|Regions|North America|United States|Florida|Coral Gables|1105 Almeria
[XMP-lr] Weighted Flat Subject: United States, North America, Regions, Where, 1105 Almeria, Coral Gables, Florida, Estimated Day & Month, Estimated, When

Photoshop Lightroom Classic 13.1 (Windows)
[IPTC]Keywords: When|Estimated|Estimated Day & Month, Where|Regions, Where|Regions|North America, Where|Regions|North America|United States, Where|Regions|North America|United States|Florida|Coral Gables|1105 Almeria

I have checked the thesaurus, and all these keywords have default settings.  Every level of the hierarchy is a normal keyword.  No Groups or "Excluded" options used. I have compacted and optimized the database and run the diagnostics twice. I have refreshed all data-driven categories several times. Everything remains the same.

There also seems to be a mismatch between the number of keywords and the number of @keywords because the last diagnostics I ran reported over 64K entity keywords.  You said there was a match between the amount of @keywords and keywords, but I cannot find the place that lists the number of keywords (unless it is the amount of entity keywords, which I was able to find, obviously).

[Damit]

Are you still looking into this issue?  As I posted, the issue is still present, and the pictures document the issue. Though I understand you could not replicate it, it is happening, and represents a serious bug in my system.  This is widespread.  About half or more of the keywords are not populating to the category @keywords.  I believe they are the more recent keywords I have created in the last 2-3 months. This severely handicaps my ability to work on my files and I really need to use this time I have to tag my pictures. I would appreciate a solution that resolves this issue.

[thrinn]

And here is another example of the opposite. The @Keywords are reporting that the photo has keywords that the keywording panel is not even listing:

As long as the @Keywords category is in the "must be refreshed" state (the blue arrows) it does not make much sense to compare it to other panels.

If Exiftool does not report a given keyword, it is not in the file. The first step would be to write-back.

[Damit]

Thank you thrinn for trying to help. I realize what you say is true, but I have written back and instructed iMatch to force a write back at least 20 times. I have restarted both iMatch and the computer at least 6 times. Some of the keywords not populating to @keywords have been in existence and assigned to a file for over 2 weeks. So, the question is, why is iMatch not populating them and performing the reload/refresh? It is a basic function of the program and why I find this issue very disconcerting.

[Mario]

As I said in my email reply, the @Keywords category in your database updates just fine and matches exactly the keywords and number of keywords stored in the database. I did a manual comparison!

Did you make sure that you have not set @Keywords to manual update (open the properties panel in the Category View) and check if auto update is on (default):



That's the only explanation I can come up with.
No other user ever reported this problem.
I cannot reproduce it, not even with your very own database.

[Damit]

Thank you Mario. I checked and it is indeed set to automatic.  In fact, iMatch will not let me change it.  The drop down does not exist. Is that normal?

I understand that you cannot replicate the issue, which is unfortunate, and that I may be the only user reporting it, but that does not mean the problem is not occurring and should not be addressed. I would be happy to prove that the issue exists (though I think the last screen shots do so), and provide any information you need, or take problem-solving steps, or you could remote in for a minute to see for yourself. I assume you want to know if this bug exists in your program and how to resolve it.

Again, I am not using any "non-standard" setting. But I will again remind you that I did have a secondary @Keywords category that you said was impossible to create (remember, you could not do it yourself; Deja Vu! But that did not mean the problem did not exist), that you asked me to delete.  Here is a link to help remind you:  https://www.photools.com/community/index.php/topic,13084.msg92574.html#msg92574 Could that have anything to do with the issue?

If there was a way to open another database with all the same settings, I could attempt to reimport some of the images and see if that resolves the problem. When I opened the new database last time everything worked fine and it also, inexplicably, got the old database to update the @Keywords as well, but then it stopped working and the only thing I changed since it worked was re-enabling the background processing and changing the XMP protections in the Metadata2 tab to yes, as you suggested, but I assume that would not lead to this problem? I will point out that "Protect existing XMP" set to off in default, maybe I should change it back?

You also wrote in Protect existing XMP
"In rare cases, e.g. when the XMP data contained in your files has been written by an old or defective application, you can set this setting to 'No' to force IMatch to ignore existing XMP data and to produce a new XMP record from scratch. Usually you set this option to 'No', import the files (or run a Metadata Reload manually) and then switch this setting back to 'Yes'."
Should I try this to see if this helps?  Could iMatch be protecting the XMP record from itself? I really don't understand how this would affect the @Keyword Category,as it is a category not a keyword, and the keywords themselves are attached to the image and reported by the Keyword Panel but since are the only two settings changed before it stopped working, I am just positing. It does seem the issue mainly lies with the mechanism that creates, drives and refreshes the @Keyword category within iMatch.
UPDATE: So, I tried this and I guess I stumbled onto the solution, or rather, the cause of the error. I set the Protect existing XMP setting to off, and, lo and behold, all the @Keywords started updating!!! But, unfortunately, it brought back a lot of erroneously tagged or tags moved by iMatch to other areas of my hierarchy. It took a day or two to correct all this stuff and I have done this twice as I had to do it again after I set this option to no when I made the new testing database and the old database.  This was the reason everything was occurring. It was due to this setting or what this setting was preventing. It is like I had posited. It was as if iMatch was protecting itself from data changed within itself, because I created and moved Mona Lisa to Art Subjects within iMatch. And I did this almost a week ago, but iMatch would not populate it to @keywords despite several attempts to refresh the data category until I removed this protection. Why would iMatch think another application made this change if I did this in iMatch? I will say that after I update and write changes in iMatch, I do go to lightroom and update the folder, but that should not trigger this protection.

I don't know what to do. I don't feel like I can trust iMatch to work properly, which is a real bummer because I love what this program can do, if I could only tame it. Can you please let me know how to circumvent or fix this so I will have a stable, reliable platform? I assume, since you set it default to off on Protext existing XMP, and after reading what it does, that it is not a good idea to leave this off. Also, I basically did what you instructed in the note quoted above the lst time I tried to resolve this, as I turned both these options to no when I created the new testing database, which is why everything started working on the old database. It was because I turned this option to "No." But when I turned it back to "Yes," everything stopped working.  I assume that should not be the case.

Please advise. Reimporting everything would take a day, but I really need the @Keywords to work. They are an essential part of iMatch. Unfortunately, it seems that only some of the preference windows and tabs have save options and there does not seem to be a global save setting that I could use to reimport the settings to a new database.  However, I do suspect that this issue may have to do with the settings. Still, it may be a good problem-solving step. I think the main settings that I need are the versions I have set up, which are quite convoluted.

I am sorry this a difficult issue to resolve, but look at the bright side, at least I am a good beta-tester. Some pay for them, I am doing this for free!

[Mario]

Some pay for them, I am doing this for free!

Fiddling with settings without knowing what they cause is not beta testing. Like, for example, you deliberately skipping keyword levels in the thesaurus. Doing things like this creates problems for you and costs me time and money.

Thank you Mario. I checked and it is indeed set to automatic.  In fact, iMatch will not let me change it.  The drop down does not exist. Is that normal?

Sorry, my bad. This is not an option for @Keywords. It works for other data-driven categories.

Protect existing XMP setting to off,

This option is enabled by default.

If a file is rescanned and it already has XMP data in the database, IMatch does not re-import IPTC/EXIF/GPS data from the original image into the XMP record. This protects XMP data if the metadata in the image and in the XMP record are not synchronized (because the data was not written by IMatch or you do not allow IMatch to update the original image file).

This only plays a role if you modify XMP metadata for files managed in IMatch in other applications. It does nothing specific with keywords.

If you disable this and modify a file externally, IMatch ignores existing XMP data and produces a fresh XMP record from existing EXIF/GPS/IPTC and XMP data in the file. The same happens when you to a Shift+Ctrl+F5 > Reload Metadata or Shift+Ctrl+F5 > Force Update.

The database you have provided does not show any problem with @Keywords, and it has protect XMP data enabled.
That's what I see in @Keywords for your database:



and this matches the keywords in the database. I've just opened the database and @Keywords and other data-driven categories refresh within a couple of seconds. All working just fine.

Maybe you have changed other settings from their default values? I don't see why @Keywords is not working for this one database and only on your PC. No reports from other users, not reproducible with the very same database on my PC and laptop...

[Damit]

Protect existing XMP setting to off,

This option is enabled by default.

Yes, obviously I know this, for I stated as much in the post above:

I assume, since you set it default to off on Protext existing XMP

Are you asserting that setting this to off is like selecting the file and the hitting Shift+Ctrl+F5 > Reload Metadata or Shift+Ctrl+F5 > Force Update? I do not think so, because I tried both those and it did not resolve the problem. There has to be a difference in what the program is doing when this setting is set to off or else this would have fixed the problem when I did the reload as I stated in post 26 and 22.

I even did a reload with a force update and that did not populate the keyword.

So what is the difference between these actions?

This only plays a role if you modify XMP metadata for files managed in IMatch in other applications. It does nothing specific with keywords.

I don't think this is the case, because on two separate occasions the same thing happened: the @Keywords would not update until I took this setting off. Then they immediately started updating.  I saw it myself.  As soon as I set this option to off and closed the window Imatch started doing a lot of processing and the keywords started populating to @Keywords. So, it must be doing something to the keywords so that they actually get propagated.

Fiddling with settings without knowing what they cause is not beta testing. Like, for example, you deliberately skipping keyword levels in the thesaurus. Doing things like this creates problems for you and costs me time and money.

Sorry, but I have to push back on this. I knew what it could do, which is why I started using it.  I just did not know that Dublin Core or the IPTC Application Record Subject were flat keywords. Please don't mischaracterize what I am doing. It seems you are taking the position that I am a bull in a China shop when I am just using the options you have written into your program. And I am doing so because you do not have an option to list group keywords in the @Keyword category, which I and at least two other users (from my reading in your forum) would like to be able to do so we can have visual similarity between the thesaurus. But I digress. I did not deliberately do anything to screw with how this program functions and I did not skip levels in the thesaurus because the keyword still was present in the thesaurus. If you consider this an advanced setting that can cause problems, you should take my advice and place a warning in your manual about it. You cannot expect your users to be clairvoyant or prescient.

And I still don't understand why this setting would cause problems. You stated:

Please don't. This is for expert users only.
It also requires a very strikt thesaurus management, else IMatch will be unable to re-map the flat keywords to hierarchical keywords during re-import - because unless it's in the thesaurus, there will be no way to map flat keywords with missing parts (levels) back the hierarchy. Resulting in keywords on wrong levels, duplicated keywords on different levels. This can get real messy real quick and then a lot of manual repair has to be done.

With the new feature where one can apply changes in the Thesaurus to the whole database, how could this happen? If you do any changes to the excluded keyword in the thesaurus, such as deleting this keyword, would those changes not be applied to the whole database as long as you tell iMatch to do so? SHould this also not remove the keyword from the flattened ones? How can one remove this leaved keyword imbedded in a hierarchy from the files without doing so in the thesaurus?

No problems other than a visual incompatibility between my Keywords and the IPTC Application Record and 
XMP Dublin Core were caused by using this option. If I would have known that the were flat keywords(and I admit it is a newbie problem), I would have never had an issue, because I read the information and I knew it would not be included. So, how can "Exclude in flat keywords" cause problems? Can anyone please provide an example, so I can understand, because I have been thinking about it for a few days and I cannot understand why it would.

The database you have provided does not show any problem with @Keywords, and it has protect XMP data enabled.

Yes, because the one I sent you was from the day before, when you asked me to send it to you.  I then started problem solving myself while I was waiting  for your diagnosis.  That is when I turned the protection to off, and, as I reported, that is when everything started updating.  I then turned it back on, after you advised me to do so.  I did not realize the correlation at the time, but, in hindsight, it was when I turned it back to on that then problems rematerialized. Then, as I just reported in my last post, I turned it back off and everything started updating.  There is no doubt in my mind that this setting is directly related to the problems I am having in this database. I am not saying it is the cause, but it is related, and I think provides a clue.

and this matches the keywords in the database

I asked this before, where can you find the number of keywords in the database.  I cannot find where it is listed to see if there is discord if or when keywords and @keywords misalign again. Please let me know.

I've just opened the database and @Keywords and other data-driven categories refresh within a couple of seconds. All working just fine.

But you are not really replicating my situation. First, this is a fresh database for you, not one that has replaced other values several times, and even if that is inconsequential, I am working with both Lightroom and iMatch, constantly updating and refreshing appropriately when I do changes in the other.

Whenever I change something in lightroom I rescan the files in Imatch, wrte any metadata changes found and then work with the files in IMatch. After doing so, I go back to Lightroom (which does, in fact, have an icon with an upwards pointing arrow which indicates the files have been changed in another program) and read the changed metadata from the file, so its database is updated.  Maybe this procedure is confusing IMatch? I don't know.  All I know is the problem exists, whether or not you can replicate it. Again, when this pops up again, and I am pretty sure it will if I set the Protect XMP data to yes again, I invite you to do a remote session so that you can see for yourself.

[Damit]

Well, I must confess that after some experimentation and problem solving, it does seem that the "Exclude" option will cause a mismatch of flattened keywords as you stated, and that stops some of my files from saving metadata. It is unfortunate, because I was going to use them in the place of groups but misappropriation never leads to good things.

[rolandgifford]

No other user ever reported this problem.

I have had the reverse of this problem which has been mentioned in a thread here but I can't remember which one.

I have had instances when correcting Keywords by dragging images from an incorrect Keyword to a correct one that the old keyword retains a count and linked images which don't contain that keyword. The linked images all contain the new corrected keyword. I have tried multiple writebacks etc and have always been able to 'fix' the problem by simply deleting the old/incorrect keyword as that is what I intended to do anyway if nothing else worked.

I remember there being some issue with embedded and xmp metadata in raw files the last time this happened but that may have been something discovered when investigating rather than a linked cause.

I have certainly had occasions where the total against a keyword doesn't match the data in the keywords panel for images linked to that keyword.

[Mario]

If you experience such issues, first thing is to retain all evidence.
The IMatch log file from that session. Switch to debug logging before trying to reproduce the problem for a more detailed logfile.

• The files that were processed, before the write back and after.
• Maybe metadata propagation came into play and the results were unexpected?
• Your thesaurus. Metadata 1 and Metadata 2 settings.
• Which operations you performed in which order.
• Where did you drag keywords? In the @Keywords category? Your thesaurus?
• Was IMatch maybe indexing files at the time or processing metadata?
• If this may be database-dependent, provide a copy of the database for me to analyze.

Since these are "one person only" or "One person only on one computer only" or "I remember that I..." or "Maybe this was actually caused by competing keywords embedded in the RAW (XMP and/or legacy IPTC?) and keywords in the XMP sidecar file (in combination with my thesaurus)" and maybe "I have changed the default settings for keywords in my thesaurus and then this started happening" and similar hard to reproduce situations, the more data we get, the better.

This is surely no general problem, else the bug report board would be full of it. Most IMatch users work with keywords every day, after all.

I have never experienced this, but then I work differently with IMatch than you do.

I have downloaded the 12 GB database of the OP and the number of keywords listed in @Keywords matches exactly the number of keyword tag values in the database. Adding, removing, replacing keywords always caused the @Keywords category to update. No luck finding any problem, unfortunately. Meanwhile, I get 30 - 40 emails per day plus other users requesting support via the community...

Of course, when the "exclude in flat keywords" option was used in the thesaurus and IMatch was forced to reprocess the files after external changes or changes to keywords done by the user, the keywords may not necessarily resolve into @Keywords as a user might expect. But that again depends on the thesaurus and the keywords already in the image and/or sidecar file!

All this has to be taken into account.
Analysis based on specific files, databases, user workflows and custom user settings for keywords and thesaurus makes this very complicated, time consuming and expensive for me. And it one one or a handful of users are occasionally and under some conditions affected by this, it may not be worth spending lifetime (aka money) on this.

If I can somehow reproduce it a problem reported by a user, I will fix it.
But I can easily waste 20, 30, 40 or more hours or lifetime with no result. Sometimes an issue happens only for one user and on one PC. And then it sometimes make sense to deem it a "no fix". I cannot invest an endless amount of time. There are many IMatch users requiring support of sorts, and only one Mario.

[Damit]

Mario, did you get the same warnings I got in the diagnostic I sent you when testing my database, the ones reporting missing keywords for entity? You said you investigated them, and determined that they were diagnostic errors, so I assume you had to reproduce them to make that determination.

[Mario]

This has been already marked as fixed for the next release. #2290 in https://www.photools.com/release-notes/.

[Damit]

Ok, but you were able to reproduce them, correct?

[Mario]

Yes. I said so above, somewhere in this mile-long thread. And also in the email I wrote to you a week or two ago.

[Damit]

Yes, you said you addressed them, but I did not know if it was just from the diagnostic I sent you or if you had gotten them when you ran your own. Good.  I am glad you did, because I think you are incorrect. They were not errors! There are problems with those photos. There are missing keywords. They were indicative of the problem I am facing, not diagnostic errors. Try to resolve those errors on those files. You will see they will not resolve. I have searched out the files and corrected the metadata and it keeps returning. For instance, in file 6470 one face, Gary Chister keeps being erased no matter what I do. I have taken all protections off. I have redrawn the face region. I have deleted and then reintroduced the linked keywords. I have reconfirmed the face.  Each time everything shows as fixed for about 5 minutes and then it reverts back, deleting the face. See if you can add Gary's face and that it will retain after a write-back and restart.  You will see that it won't.

I really believe there is a problem with the way IMatch protects data. I have run into it many times in the past, especially with file rotation. I started a new database and old exif rotation information was not retained for several .jpegs even though the picture shows the correct orientation in the preview (file window).  If you open the same file in explorer or lightroom or in the viewer, the photo shows the original orientation of the photo, the one that remains as the correction done in the previous IMatch database was not retained, for some reason-probably because it was never written because of the way IMatch protects files. Yet, interesting, it still has some sort of way to know that the file was rotated because it is showing it in the file window, even though it is not really rotated.

But I am not asking you to look into the rotation problem.  I am asking you to resolve this problem with the missing keywords.

[Mario]

I have tested the diagnosis fix with your database and it works just fine. Keywords in files are as they should be.

IMatch produces the thumbnail from JPG files using a 3rd party image library.
IMatch uses WIC, DirectX and the GPU to render files in the Viewer and Quick View panel.
Sometimes there is an embedded thumbnail large enough for the thumbnail. And sometimes not. Sometimes the thumbnail orientation does not match the metadata EXIF orientation, but the JPG data does. Many reasons. All unrelated to metadata or keywords or metadata protection. Because neither Windows WIC nor the 3rd party image library I use accesses the metadata stored in your database. They access the EXIF data in the image itself.

Re-saving the JPG in an image editor fixes this issue in almost all cases.

The same happens sometimes with RAW files when the camera vendor did not rotate the preview to neutral as required and also not setting the EXIF orientation correctly. Sometime WIC gets the orientation wrong, sometimes LibRaw gets the orientation wrong. It is how it is.

[Damit]

I have tested the diagnosis fix with your database and it works just fine. Keywords in files are as they should be.

But could that not be self-fulfilling, especially since you could only test it on my system with the assumption that the keywords are not missing?  It is odd that they are pointing to files that I specifically found problems with.  Could you add the face to the photo? Can you at least see if what I am reporting replicates in the database in your computer? Go to that file and try it. Maybe then you will be able to replicate my issue.

And the issue still exists!  I took the pains to start over.  I have been doing it folder by folder, making sure things are working. Everything was working fine and then, this morning, once again my @keywords are not reflecting changes on a brand-new database with less than 2100 files! I emailed you the new debug log and diagnostic now.

The minute I set Keep Existing XMP to no, the keywords updated. What is going on?  Why would that happen? Again, there seems to be an issue with the way IMatch is protecting XMP data.

All unrelated to metadata or keywords or metadata protection.

Yes, but why was the rotation resetting over and over, such as it did when I encountered it on the new database when I know I rotated it on the old one and saved the changes on the old database and the photo was not showing up in the write-back collection?

[Damit]

Another note.  When I took the "Keep Existing XMP" off, one drawback was that the pictures I had rotated and saved the change in the metadata revered back to their original orientation even though the change in orientation is supposed to be written into the file.  I am dealing with jpegs. I am not sure why this is occurring, but it may explain why these same images and lost the orientation changes done on the old database when I imported them into this fresh database.

[Mario]

You have rotated how and saved how?

[Damit]

I rotated using the Exif Orientation pull-down on the metadata panel. After setting the appropriate rotation value, I then click the green check mark and then the pencil.  I then make sure everything is written by monitoring the pending write-back collection.   

[Mario]

I don't see this command.
There is a legacy EXIF Orientation command in Commands menu > Image > Orientation which directly changes the main EXIF orientation tag in JPG files. This is done directly in the file by the 3rd party image library IMatch uses.
This was used in the olden days, when cameras did not always got their orientation right. It works in most cases, unless there are different orientation for thumbnail and main JPEG data and similar special cases.
IMatch reloads the metadata of the image afterwards automatically.