Can someone explain how IMatch deals with changes to metadata by 3rd party apps?

[eyemack]

If someone could clarify this for me, I would appreciate it.

1) So, just to demonstrate the issue simply, I created a jpg file in Affinity Photo with keywords of 'Holidays', 'Holidays|Cyprus'.  Saved it, and created a database in IM with only that file for simplicity.

2) IM read the keywords OK.  All good.

3) I edited the file again in Affinity Photo, adding the keyword 'Test'

4) Back to IMatch, which recognised there was a change and the pen icon appeared.  Before I did a writeback, I ran ExifTool and the keywords shown in ap.jpg attachment showed.

5) Then I did a writeback in IMatch, which overwrote the file with the original keywords, i.e. 'Holidays', 'Holidays|Cyprus', and the ExifTool result after the writeback is shown in attachment ap2.jpg. It shows that the added keyword, 'Test', has been ignored and overwritten by IMatch which is confirmed by loading the file into Affinity Photo which only shows the original keywords with no 'Test'.

From what I am doing, IMatch ignores any changes I make to the files in another app and just overwrites what it sees in its database.

I'm guessing there is something incorrect about my understanding of how this works, and I would appreciate if someone should show me the way here.

[Mario]

Edit > Preferences > Metadata 2. Which protection settings are enabled?
Did you write back pending changes in IMatch before you changed the metadata in the other application?
IMatch by default protects unwritten changes to prevent other applications from wiping out metadata updates you have made in IMatch but not written back yet.  See Protect unwritten metadata for details.

[PandDLong]

4) Back to IMatch, which recognised there was a change and the pen icon appeared.  Before I did a writeback, I ran ExifTool and the keywords shown in ap.jpg attachment showed.

In addition to Mario's questions and suggestions.

At Step 4 - what was iMatch showing for keywords?  Had it put the 'test' keyword into the iMatch database?  And if so - for which specific metadata tags? exiftool tells you what was in the file on disk at that moment but you need to know what was in the iMatch database before write-back.

I notice that in your first exiftool output, the 'test' keyword is in 2 of the 3 keyword metadata tags but not in the Lightroom tag. 
This can be a factor as discrepancies between these tags need to be resolved - I am not sure of the order of precedence in keywords but I had a similar issue with some geo-location data which has duplicate tag locations and is rationalized as part of the write-back/re-read process with exiftool - experimentation and internet research (primarily the exiftool site) explained how those differences were rationalized.

I hope this is helpful.

Michael

[Mario]

IMatch always updates hierarchical keywords and mirrors them into flat keywords (XMP::subject and legacy IPTC::keywords when the file still has legacy IPTC data). It seems Affinity Photo only writes flat keywords?

IMatch will merge flat keywords into hierarchical keywords on ingest, though. This step depends on the Edit > Preferences > Metadata settings and your thesaurus.

[eyemack]

Sorry, sleep etc. got in the way!
So, in Preferences > Metadata 2, there is no protection set.
In response to other questions above,
1) yes, I wrote back before making changes in Photo.
2) no, IMatch had not put in 'Test' to its database. It never actually shows at all in IMatch.
3) yes, I know ExifTool tells what was in the file at that moment, that's why I ran it before and after writeback.
Basically, IMatch is correctly taking in the keywords on first import to library, but ignores any subsequent changes by the external app and reverts the file back to what was in the library with respect to keywords after a writeback.
I'm clearly doing something wrong here.

[Mario]

Since your other application updated only flat keywords and ignored the existing hierarchical keywords in the file, make sure to disable the Don't replace existing hierarchical keywords option. And then do a <Shift>+<Ctrl>+<F5> to rescan the metadata for affected files using the new settings. Images coming into the database in the future will use the new settings automatically.

This option is enabled by default to protect the (usually) richer and properly mapped hierarchical keywords with potentially less rich / correct flat keywords some applications out there write.

In general, it is not a good idea to use multiple applications to modify metadata when not all of these applications stick to the same standards / mapping rules. If your other software only updates some of the keyword  tags (and probably also only some of the native / XMP EXIF tags), there will be trouble. Also, legacy IPTC data has been retired 20 years ago and should not be written anymore. XMP exists for 20 years and is a much better alternative to the old IPTC data from the 90's.

You have looked at keywords so far, but who knows what other metadata IMatch has written is ignored / changed / wiped by your other software, causing problems down the road. XMP and native EXIF and native IPTC and native GPS are interlinked and should always kept in sync. IMatch does that with the help of ExifTool. If you bring in software which does not do that into the mix, results are unpredictable.

[eyemack]

Thanks a lot, Mario. Changing that option has resolved the issue!
Quick question, what does the 'Today at 00:00' in the attachment mean?  Looking at the metadata panel, it suggests that is 'Date/Time Original', but I created it an hour or so ago. Is this because it is a generated jpg and not from a camera? Does that piece of data normally come from a camera?

[Mario]

This depends on the File Window layout you have selected. Check which timestamp is shown in the layout and then compare to the corresponding metadata tag. They date seems to be from today, but the time is 00:00:00, so check that.
The "Default" File Window layout uses the Date and Time Original.

See How IMatch uses Date and Time Information for detailed information about how IMatch uses metadata and file system timestamps.

[eyemack]

Yes, it's Date Time Original, but the info is incorrect in the images I've been working with today.  See attached. The Date/Time Original is before the Create Date!

[Mario]

What is not correct? The DateTime Original shows a timestamp of 00:00:00 and that's what IMatch is showing in your file window layout.

The data you see here was delivered by ExifTool and extracted from the metadata in your file.

[eyemack]

Sorry, what I mean is the info is wrong.  It is impossible for it to be 00:00!  It's before the creation date, and certainly wasn't when I was asleep.
What I was asking is where does the Date/Time Original data originally come from, i.e. embedded onto the file.

[Mario]

What I was asking is where does the Date/Time Original data originally come from, i.e. embedded onto the file.

Yes. As I said, this data was produced by ExifTool.
Check your file in the The ExifTool Command Processor or run the Metadata Analyst to check the metadata.
Or, attach the file so we can have a look.

[eyemack]

Sorry, I'm not explaining myself well here. surely the info is just reported by ExifTool. I'm asking where does the info originate?
File attached.
Thanks.

[Mario]

A quick look with the IMatch ExifTool Command Processor reveals:

[ExifIFD]      Date/Time Original              : 2024:05:01 00:00:00
[ExifIFD]      Offset Time                    : +01:00
[ExifIFD]      Offset Time Original            : +01:00
[ExifIFD]      Offset Time Digitized          : +01:00
[XMP-exif]      Date/Time Original              : 2024:05:01 00:00:00
[XMP-photomech] Time Created                    : 00:00:00+01:00
[IPTC]          Time Created                    : 00:00:00+01:00
[IPTC]          Digital Creation Time          : 11:45:54+01:00


The metadata was created / updated by IMatch. I have no original file so I cannot tell where these timestamps come from. But legacy IPTC, EXIF, XMP EXIF and the Photo Mechanic (?) XMP timestamp agree. Since IMatch does not write or modify proprietary XMP data like XMP-photomech, I believe the 00:00:00 timestamp was already in the file when IMatch ingested it.

[eyemack]

Thanks again, Mario. I have no idea where photomech data is coming from.  That file was created in Affinity Photo and subsequently by IMatch. Nothing else.

[Mario]

IMatch does not create this namespace.
Maybe create a new file and check it out in the ECP before writing back. This will show you the metadata as written by AP.

[eyemack]

I know we're not really in IMatch territory here now, but it's interesting to see what is occurring.
Attachments show -
p1.jpg - after initial creation by AP
p2.jpg - after writeback by IM
p3.jpg - after 2nd edit by AP
p4.jpg - after another writeback by IM.
Looks like the 2nd AP edit produces the photomech item. Why on earth would that be?

[Mario]

Ask Serif?

[JohnZeman]

Just to add my 2 cents worth here a few months ago I changed to using Affinity Photo for processing my raw and JPG photos.  I loved PhotoLab but their refusal to support the newer iPhones is a deal breaker for me so I switched.

I have never had a metadata problem with Affinity Photo.  But then again I never touch the metadata with Affinity Photo only IMatch takes care of the metadata.

My workflow is import the raw photo into IMatch, add metadata, categories, etc and then I use IMatch to send the raw image to Affinity Photo 2 for processing.  Affinity Photo exports 2 JPG versions back to IMatch, one color version, one black and white version.

XMP:XMP-photomech

Does not exist in my raw or JPG metadata.

[Jingo]

Not sure why it would be in your file - but XMP-photomech appears to be a shorthand from exiftool for the PhotoMechanic namespace: https://exiftool.org/TagNames/PhotoMechanic.html 

[eyemack]

Thanks guys. It only appears when the jpg has been created in AP. If the file is an image from a camera, it doesn't appear.
Anyway, I've asked the question on Affinity Forums, so we'll see.

[jch2103]

@JohnZeman has good advice: Only process metadata in one program even when you use other programs for image processing. Mixing platforms can lead to unexpected metadata results.

[Tveloso]

Just another observation:  Not only has Affinity Photo added that XMP-photomech tag, but also IPTC Date Created, without including the corresponding IPTC TimeCreated (as shown in p3.jpg).

So when IMatch then writes to the file again (p4.jpg), it adds the missing IPTC TimeCreated, and sets it to zero (which corresponds to its "input value").  And this appears to be what causes XMP DateTimeOriginal to then change from 16:11 to 00:00.

[eyemack]

Thanks guys. So, are we saying that AP is not handling metadata correctly?

[eyemack]

Looking into this further, can I ask for a bit more clarification on what is going on here?

Are we saying IMatch uses [EXIF:ExifFD DateTimeOriginal] for the display in the 'Default' File Window for 'Date and Time Original'?

If so, then it is correct in p2.jpg in post #16 above. It is also correct in p3.jpg after a further edit in AP.

But then IM changes it afer a further writeback (p4.jpg) maybe for the reason above, but why would it do that if it already has the data?

The 'photomech', 'IPTC:DigitalCreationTime' still hold the correct info.

I'm confused by the process here.

Thanks.

[Mario]

See How IMatch uses Date and Time Information which explains which tags in which sequence IMatch uses to create the global {File.DateTime} and timestamps derived from that.Keep in mind that as soon you mix in human-made metadata like IPTC or XMP, it gets priority over machine-made metadata like EXIF.

[eyemack]

Thanks Mario.

1) So, looking at the help section you linked to, it would seem that IMatch takes the Create Date ( and Time) from the XMP::photoshop/DateCreated tag.  In my first attachment (p1.jpg) in post #16, that shows the correct info.

2) Then it looks like – after writeback  - IMatch retains that, see attachment (p2.jpg).

3) Then, after a 2nd edit, AP splits the date and time into XMP::photoshop with the date, and XMP::photomech with the time (p3.jpg).

4) Finally, after a writeback of the change, IMatch doesn't take the info from the 'photomech' entry and puts in 00:00.  However, the correct info is still in XMP::CreateDate which the help file suggests is where it looks next. (p4.jpg)

It seems to me that IMatch is taking the non-existent info for time and putting 00:00 in the metadata, rather than looking for it in XMP::CreateDate where the correct info is.

[eyemack]

Following exchanges with the Affinity devs on this issue, it looks like Photo is not handling certain metadata tags correctly.

They have promised a fix.

[Mario]

As always, it would be very helpful to have the actual images you work with here in my lab.

The ExifTool output you produce is irrelevant for diagnosis since you use different parameters than IMatch, you don't apply EXIF->XMP, IPTC->XMP, GPS->XMP and MWG rules. What IMatch sees for your files will be quite different than what you see for your arguments in the ECP.

IMatch ignores the phtomech tag group. It does not care for proprietary XMP data, it just retains it unmodified.
It looks at legacy IPTC data, XMP metadata and native EXIF data. In XMP, it cares for XMP::photoshop\DateCreated\DateCreated tag (Date Subject Created in the Metadata Panel) and XMP::xmp\CreateDate\CreateDate (Create Date). If these don't exist or are invalid, IMatch produces time stamps from legacy IPTC data or EXIF data or one of the other tags listed in the help topic.

For the particular mess of tags in your file 3, IMatch may fall back to legacy IPTC, but since the IPTC is not complete and missing the mandatory time field, IMatch sets the time to 00:00:00 to complete the missing time portion of the IPTC timestamp.

Please understand that I don't plan for or support all combinations of half-written metadata tags (like a missing time portion for the IPTC timestamp) and that IMatch does not copy timestamp data between XMP or IPTC to "fill" in missing information it may encounter. I have never seen a file with this kind of half-broken metadata so I cannot tell what actually happens. Not without having image 3 here.

I wonder if it would be a god use of my time to add yet another exception or code to deal with this rather particular set of metadata mess produced by your version of Affinity Photo. This problem not been reported before, so either nobody here is using Affinity Photo or this is something new or something linked to your particular product version.

Provide the file with the second update from AP and I will have a look when I have a free time slot.

[eyemack]

I just seem to get permanently told off here, but... I'm getting used to it! 

I am just a simple user who is seeing behaviour that I have not come across before, and come here to find a solution. 

In this case, I have identified behaviour that the Affinity devs have said has shown a bug in how AP handles metadata. That seemed like progress to me.

I have learned a lot from your very quick responses, but I am still learning. I have only been using the combination of IMatch and Photo for about a month.

I am using the latest version of AP, i.e. 2.4.2, and I don't consciously do anything with metadata.  I just take files, edit them in AP and return to IMatch.  All the background stuff is done without my intervention.

I have attached a simple file created in AP and edited twice.  It shows the issue.

As I said, the Affinity devs have said that AP is not handling the metadata output by IMatch correctly and they are going to fix it.  The first edit of the file in AP is always OK because IMatch has not yet done a writeback on it.

I hope this helps.

[eyemack]

Looking again at your last post, I think that is one of the points I made above, the XMP::xmp\CreateDate tag is there and is complete, so I am wondering why it is not being used by IMatch.

[Mario]

I just seem to get permanently told off here, but... I'm getting used to it! 

What makes you think that?

You present ExifTool outputs with invalid metadata, missing tags / half-written tags produced by another software.
Other users and I try to help you figure out what the problem is. Users and I spent time looking at your metadata and posts, explaining what the problem is and that the problem is not ExifTool or IMatch.

What else do you expect?
Frankly, telling off other users who tried to help you, spending their time to read your posts and writing answers, is probably not a good idea. This is a friendly and helpful community, and the moderators keep it that way.

the XMP::xmp\CreateDate tag is there and is complete,

Yes. And it is only one of the tags IMatch uses to produce metadata. There are an explicit tag->tag mappings between IPTC, EXIF and XMP as specified by the IPTC and the MWG. I've explained in the help topic.
IMatch does not populate XMP data across these implicit source tags, just because one of the other tags has a time stamp but the actual tag defined by the IPTC for mapping has not. This could cause chaos.

The time stamps may specify two very different points in time (e.g., image from 1950 but digitized in 2024) and mixing up date and time to fill missing mandatory tags would not be a good idea. IMatch does the right thing by assuming that a missing time stamp in IPTC is 00:00:00.

Make your other software to not omit the mandatory time portion of the legacy IPTC tag, and I'm sure everything will work fine. Or, do as I suggested, and remove the half-baked legacy IPTC data (who needs this 20 years after it was officially discontinued?) and things will also work. Did you try that?

Mapping time between EXIF, legacy IPTC, XMP and XMP IPTC is already complex. Trying to anticipate issues caused by other software omitting mandatory time or date portions in legacy IPTC and coming up with automatic schemes to deal with it is even more complicated. In your case, IMatch apparently opted to assume that the missing time portion of the legacy IPTC timestamp is 00:00:00, which sounds reasonable. If the other software would have written both XMP timestamps that map to legacy IPTC, IMatch would have used these. But the other software did not, and also omitted the time portion of the legacy IPTC timestamp...

Thanks for providing the output image 3 (from the 2nd write of AP). I will look into it when there is a free time slot. Next week, most likely.

[PandDLong]

Looking again at your last post, I think that is one of the points I made above, the XMP::xmp\CreateDate tag is there and is complete, so I am wondering why it is not being used by IMatch.

The xmp\CreateDate represents a different date-time than photoshop\DateCreated so they shouldn't populate each other unless one is totally missing - IMHO.

I would expect that since DateCreated has a valid date (and 00:00:00 is a valid time of day), iMatch wouldn't look further.  Doing so could be dangerous in terms of messing up valid metadata.

If it is creating a problem for you before Affinity issues a fix, I would recommend using a metadata template that would build date-time stamps the way they should be in your particular situation.

Michael 

[eyemack]

Thanks again, Mario.

I think maybe you misunderstand my humour, but there we go.
 
Having broken metadata messing things up from AP is clearly the answer to the problem here. And, I have followed this up with Affinity devs as I have said, so hopefully a resolution to this is in hand.
 
As a major contributor on several forums myself trying to help others, I really do appreciate the time that people have spent here on this issue.

[eyemack]

Looking again at your last post, I think that is one of the points I made above, the XMP::xmp\CreateDate tag is there and is complete, so I am wondering why it is not being used by IMatch.

The xmp\CreateDate represents a different date-time than photoshop\DateCreated so they shouldn't populate each other unless one is totally missing - IMHO.

I would expect that since DateCreated has a valid date (and 00:00:00 is a valid time of day), iMatch wouldn't look further.  Doing so could be dangerous in terms of messing up valid metadata.

If it is creating a problem for you before Affinity issues a fix, I would recommend using a metadata template that would build date-time stamps the way they should be in your particular situation.

Michael 

Thanks Michael.

I understand what you are saying. I just thought that the missing time in 'photoshop\DateCreated' would make IMatch go to its next port of call, i.e. 'xmp/CreateDate' for the info rather than assume that a blank field means 00:00.  But, a wrong assumption!

Anyway, I guess we've taken this far enough until we see a fix from Affinity.

[Mario]

Looking at the "test.jpg" file, relevant tags are:

Code Select Expand

[XMP-photoshop] Date Created                  : 2024:05:03
[XMP-xmp]      Create Date                    : 2024:05:03 17:52:18+01:00
[IPTC]          Date Created                  : 2024:05:03
[XMP-photomech] Time Created                  : 17:52:18+01:00
The IPTC timestamp is missing the time portion.

The XMP::photoshop is missing the time portion (which is legal as per the XMP standard but rather uncommon). If the time portion is omitted, 00:00:00 is assumed.
These timestamp is the source for "Date Subject Created", which thus shows with a 00:00:00 time in IMatch. This is the expected result. IMatch does not populate missing time from other metadata tags.

The XMP-xmp Create Date is used as "Date Created". It has a fully-qualified date and time.

Removing the incomplete legacy IPTC data from the file and then forcing a reload of the metadata with Shift+Ctrl+F5 changes Date Subject Created to

2024:05:03 17:52:18+02:00

which matches the EXIF Date/Time Original. Since no time zone offset exists in the metadata, IMatch uses my local time zone (according to the File.DateTime Mapping Mode (User-defined Time Zone Offsets) I use.

IPTC as human-made metadata overrides EXIF (machine-made) and thus the time portion missing in IPTC causes IMatch to assume 00:00:00 as the date and time, which is in accordance with the legacy IPTC spec, when I recall correctly. This is all very old stuff.

The incomplete legacy IPTC data in the file causes the problem. IMatch handles IPTC time stamps with missing time values by assuming 00:00:00 for the time. IMatch does not try to substitute the missing time with any other time that may exist in the metadata. This is a can of worms I won't open for such rare cases where a software does  still write legacy IPTC date but skips the time portion for some reason.

Not even IMatch can anticipate and deal with all of such rare special cases.  This would lead to code bloat, more potential bugs and increase maintenance depth on my part. Nope, sorry.

You can easily fix files affected by this by updating the entire timestamp or just the time portion with IMatch's Time Wiz and when IMatch writes back the files, it will automatically fill in the missing IPTC time portions from the fully-qualified XMP timestamps.

[eyemack]

Thanks again, Mario for your detailed response.
It is clear to me now what is happening after a long time of investigating this (plus the contributions here and on the Affinity forum).
1) The original files don't have any ITPC tags
2) once the ITPC tag is there - and there is debate about where they come from, but it's a moot point -  Affinity Photo does not process it correctly after editing. AP thinks it's from photomech, and splits the  (previously correct) XMP:photoshop info into photoshop (with the creation date) and photomech (with the creation time)
3) then when IMatch looks at it, it ignores photomech and therefore does see any creation time, hence the 00:00.
It's clearly a bug in AP which the devs have told me will be fixed.
I will look into 'Time Wiz', so thanks for that.

[eyemack]

Mario, just out of curiosoty, why does IMatch use the XMP:photoshop data rather than the XMP:xmp for date/time created?  Is it more reliable, in general?

[eyemack]

Hopefully, the last question here!
I'm trying to use Time Wiz to change the time, but I'm struggling.
I can see how to 'Set Absolute Date and Time' to fix the erroneous 'Date Subject Created' time by manually inputting the data, but obviously that is a pretty awkward and time consuming way of dealing with the issue.
I can also see that 'Shift Date and Time' is not appropriate here, so I assume that 'Set From Variable' is the way forward in this case.
I have tried various variables, but can't get this to work. Any hints on what I should be doing?
Thanks.

[Mario]

Mario, just out of curiosoty, why does IMatch use the XMP:photoshop data rather than the XMP:xmp for date/time created?  Is it more reliable, in general?

Because these mappings are defined in the XMP standard / IPTC Standard and my the MWG mapping rules. And this is also how companies like Adobe do it, which makes this an industry standard. the photoshop and xmp time stamps have very different meanings. I recommend you read the XMP standard and the IPTC standard if you want to learn more.

I assume that 'Set From Variable' is the way forward i

Yes.
Use the date from whatever time stamp you want and combine it with the time stamp from whatever tag you want. Make sure you use the correct format. A quote from the help:

You have to use the format variable function to produce the YYYY:MM:DD HH:MM:SS format IMatch expects, for example: {File.MD.XMP::xmp\CreateDate\CreateDate|format:YYYY:MM:DD hh:mm:ss}

In your case you would use YYYY:MM:DD for the first variable, then a blank and then a second variable with the hh:mm:ss format to combine date from one variable with time from another variable.

[eyemack]

Thanks again, Mario. That exact format and variable you quoted above was exactly what was required!
I can't see any XMP references in the Variable Selector within TimeWiz, so does this mean that the above command has to be recreated manually each time I want to use it?

[Mario]

Thanks again, Mario. That exact format and variable you quoted above was exactly what was required!

Yes. This is why this is explicitly mentioned and demonstrated in the TimeWiz help: Set from Variable
I recommend reading the help when you use some feature for the first time. This will save you a lot of time and grief.

I can't see any XMP references in

Not sure what you mean by reference, but to create variables for tags, click on the "Tag Selector" button.
See The Variables Selector Dialog for details.

command has to be recreated manually each time I want to use it?

If you switch between different variables in the TimeWiz, I suggest you keep frequently used variables in the IMatch Notepad for quick access and retrieval. I and I'm sure many users keep frequently used variables with some notes there.

The need to use variables with the TimeWiz is rather rare, and so there is no "history" or "recently used" feature implemented in TimeWiz. It just remembers the last settings used.

[eyemack]

Thanks again for that. The Notepad looks really useful.

I can't see any XMP references in

Not sure what you mean by reference..

What I meant was that in the list shown in the drop down 'Variable Selector', I could see no mention of XMP anywhere, so it wasn't clear what variable I should use and that it had to be entered manually.

[Mario]

Click on Tag Selector to open the regular tag selector. Select the tag you want. The variables selector then produces the corresponding variable. See help:

The Variables Selector Dialog
ShortCodes
Metadata Tag Selector...

[eyemack]

Thank you. All good now, or at least it will be when Affinity do their fix!

[eyemack]

Just to say that Affinity have fixed the issue with how AP manages the IPTC data and missing photoshop time issue. I have tested in the new public beta, and it looks good.