Face Annotations in HEIC Files and IMatch File Formats config

[Tveloso]

I recently began adding/updating Face Annotations, and am finding that the data related to my changes doesn't "stick" in (and is actually removed from) HEIC Files, during the writeback.  It appears to be related to the fact that HEIC Files are configured to use XMP sidecar files (and the XMP sidecar doesn't exist yet on the first write-back).

When working with HEIC Files which already contain XMP Face Regions (from which IMatch has created Face Annotations), I found that Labels added to those existing Face Annotations are not retained after the write-back (and neither are the Hierarchical Keywords resulting from those annotations).

For example, a newly indexed HEIC File shows the following tags as pending WriteBack (the Location Data and the Keywords were added at ingest, by virtue if an IMatch Location):

   

After adding a label to a Face Annotation on that file, the pending tags now looked like this (the MWG::Regions\ReginoInfo item is now included in the list):

   

After the WriteBack, the file continues to show a pending writeback, and the tags are now (the keywords only):

   

The Hierarchical Keywords previously shown for the file (before the write-back) in the Metadata Panel, included a Keyword Path created from the Location Data, and one created from a Face Annotation.  The "location path" was retained, but the "face path" was removed (which is why the file showed as pending write-back again – since the keywords had changed)...and the label on the annotation was also removed (also, if the annotation's location and size had been changed slightly, it was reverted to its original dimensions, per the XMP regions data embedded in the file).

So when creating the XMP Sidecar File, the "location keywords" are written there, but the "who keywords" (from the face annotation) are not...it's not until the annotation is added again, and a second writeback is done, that the "who keywords" are then written (presumably because the XMP SideCar now exists).

Mario, is it possible for the user to configure the HEIC format to embed the XMP data in the file, rather than use an XMP sidecar file?...I'm guessing that you might need to make a change to provide that facility (i.e. that the HEIF format must be updated to have the "x:rw(e)" designation as discussed in the Help?):

   

Currently the available options in the File Format Metadata Options dialog for the HEIC format:

   

...are far fewer than those available for JPEG, so I'm guessing that an IMatch update may be needed to allow the user to elect to embed XMP data.

I'll send the Debug Log and the ExifTool output to the support email address (and will link back to this post), in case the "vanishing face data issue" might be a bug...but having HEIC Files behave the same as JPEGs would make that moot, as the issue doesn't manifest with JPEGs (presumably because, since they use embedded XMP data, there is no "contradictory" information out there after the write-back?).

Mario, thank you so much for your support, and for the incredible software.

[Tveloso]

I actually had a sixth ScreenShot to include in my previous post, but the max per post if five, so I thought I'd include it here.

I was asking Mario if the File Format Metadata Options dialog for the HEIC format could provide all of the same options as it does for the JPEG format:

   

...and if it could be delivered that way (to embed the XMP data in the file, rather than use a sidecar file)...or if a user could configure that.

I also wanted to mention something I noticed that strikes me as odd...the XMP Face Regions written by Apple Devices appear to be relative to dimensions:

Code Select Expand


[XMP-mwg-rs]    Region Applied To Dimensions W  : 4224
[XMP-mwg-rs]    Region Applied To Dimensions H  : 3168


...that exceed the size of the image itself:

Code Select Expand


[ExifIFD]       Exif Image Width                : 4032
[ExifIFD]       Exif Image Height               : 3024


To my limited (very limited) understanding of this, that just doesn't make sense...but IMatch still creates properly placed Face Annotations from these files.  When an updated Face Annotation is written back (to a JPEG File), those "bad" Applied To Dimensions are replaced with the actual image dimensions...so I suppose this does represent another instance of "metadata mess"...

[Mario]

Provide a sample image file which exhibits this behavior.
I have never heard about such an issue. I guess its as usual some incompatible metadata already in your files which causes this.

[Tveloso]

Mario:

A sample image has been sent to the support email.

This is not a big issue, as I can just preform a Write-Back on HEIC Files before updating their Face Annotations, but just wanted to mention it in case this could be a bug.

I would be more interested though, in the ability to configure IMatch to embed the XMP record, rather than using a SideCar.  Is that something that a user can configure under the current version of IMatch?

[Mario]

As per standard, XMP is embedded only in PSD, GIF, TIFF, PNG, JPG formats. All RAW files use XMP data in sidecar files. IMatch also uses sidecar files for which ExifTool reports that it cannot write the format.

HEIC/HEIF is an awful file format with so many patents attached that even touching the file with a long stick may cause the software vendor to face patent infringement charges or requirements to pay money to the MPEG, Nokia, Apple and others.  Even Microsoft requires you to download a separate WIC plug-in to support it in Windows - it does not come pre-installed. Because Microsoft has to play royalties for every installation and this saves them a lot of money.

There are rumors that Apple only introduced this format so they can increase profit by fitting their devices with smaller built-in memory. Which I think is believable. No other platform has adopted HEIC/HEIF yet and hopefully it goes the same way as JPEG2000 and other patent-infested file formats.

[Tveloso]

Thank you Mario.

I was afraid of that...you have said before that HEIC/HEIF is a patent-infested format...

I do like that it reduces the size of each image file quite a bit...but maybe I'll re-think using it.

[Mario]

I do like that it reduces the size of each image file quite a bit.

Well, disk space is cheap. 4 TB for ~ 70 bucks gets you a long way...

[Jingo]

I do like that it reduces the size of each image file quite a bit.

Well, disk space is cheap. 4 TB for ~ 70 bucks gets you a long way...

cheap in all places.. expect Apple products!    Unfortunately, since HEIC is turned on by default... most users have no clue they are using that format!

[Mario]

most users have no clue they are using that format!

That's how Apple locks them in.

[Tveloso]

True!...and when that format was first introduced, it seemed like Apple was "forcing" its use, and there didn't appear to be any means to view those files in a Windows environment (perhaps Windows didn't offer the codecs right away?), if I'm remembering correctly.

I actually thought, "well, I might have to move to an Android Phone".  It was my son who "indoctrinated" my wife and me into the Apple world (he's a big Apple fan).  As Mario has said before, if you stay in that world, this will work ok...but I'm still a Windows user (and only use the iOS devices), so I've had to deal with some "not getting along" issues from time to time...

[Mario]

I'm a big fan of open protocols, file formats, open standards. I don't like software patents.
You should be able to use your data and files on any platform and operating system.

If you look at the 'allowed' file formats for archival (libraries, universities, ...) you won't find any RAW formats, patent-affected file formats etc. for good reasons.

[Mario]

Back to the original topic:

The supplied HEIC file contains embedded XMP data.
But IMatch uses XMP in sidecar files for HEIC files - partly because ExifTool just recently added support for writing HEIC files, partly because Phil seems not to be sure to have all variants covered and party I could not find anything definitive that specifies whether or not HEIC files should use embedded XMP data. And, because the format is not used often outside the Apple golden cage.

In your case this produces two XMP records - one embedded record and one in the sidecar file IMatch creates.
As far as I know, Adobe software also uses XMP sidecar files for HEIC. I may be wrong. Comments anyone?

I shall ask a question in the ExifTool forum to learn what the common way to deal with this is.
Actually, I did already ask and discuss this in Match. See http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,9999.msg52007.html#msg52007

I'm still not sure how to deal with this.
Appel does their own thing, as usual. But what is the standard. What does other software do?
I don't want to switch IMatch to update XMP in HEIC and then learn that Apple software/cameras, Photoshop or other industry standard tools no longer can open the files...

[Tveloso]

Mario:

While you work out whether it would be appropriate for IMatch to embed the XMP in HEIC files, or to continue using a SideCar, I had a go at using an ECP preset to embed the XMP. 

This seems to work ok in my test Database...but I'm a little apprehensive about actually doing it in my main Database.  I'm worried that in doing it (outside of the protections of IMatch), I might be creating problems for myself down the road.  In particular, it does seem to create "a slight hiccup" when a file that has been processed that way, is then re-loaded via a Forced Update.

So, after I've made all of my Metadata updates, and done a Write-Back, I then run this ECP Preset:



...and delete the XMP file.

The resulting HEIC file can be opened fine in the Windows Photos application, and in the Viewer in IMatch (and the face annotations are visible there, and in the Quick View Panel).

The only thing is that if I do a Forced Update of that file, IMatch now indicates that it is Pending Write-Back...but when I hover my cursor over the Pen Icon, no Tooltip displays (and clicking the Pen Icon results in no change).  If I then run a Database Diagnostics, the Pending Write-Back is fixed for the file:

Checking Metabase:
      Warning: 1 files marked as pending for write-back which have no data to write-back. Fixed.
Completed.

I wonder if that "false Pending Write-Back" condition is actually a concern?...or whether there might be some other parameters that I need to add to that ECP Preset to prevent it?...

[Mario]

IMatch 2020 writes XMP to HEIC files. I hope this does not violate any patents from Apple or Nokia. But then Phil from ExifTool will be the first one Apple sues, and I can vanish somewhere int he jungle.

When you use the ECP to modify files, you take full responsibility. The pending write-back may be caused by many factors. Not sure how to remote-diagnose this, of if it is even worth the time (isolated problem, one user only, use has used the ECP to modify files, probably now an embedded XMP and sidecar XMP record, ...?)

[Tveloso]

Thank you so much Mario.  I'm very much looking forward to IMatch 2020.

Haha...I sure hope that there won't be any legal issues that drive you into the jungle!

Although there might possibly be some appeal in getting some of that back-to-nature solitude (and not having to constantly deal with us users...and probably some great photography opportunities too), we users would absolutely miss you!...and would be lost without you!!

Thank again.

[Mario]

Haha...I sure hope that there won't be any legal issues that drive you into the jungle!

If Apple, the MPEG or Nokia see money in suing people over using HEIC/HEIF files or modifying metadata, the will sue.

[vsammy]

Hi Mario, hello all

IMatch 2020 writes XMP to HEIC files. I hope this does not violate any patents from Apple or Nokia. But then Phil from ExifTool will be the first one Apple sues, and I can vanish somewhere int he jungle.

I am very interessted to write the metadata to HEIC files. Right now all the Metadata were written by iMatch 2020 in a sidecarfile .xmp.
How / Where can I change that behaviour? And when the data is written - I probably have to delete to .xmp File manually?

Thanks in advance for help,
VSammy

[Mario]

IMatch writes metadata via ExifTool and leaves HEIC/HEIF files alone.
I'm not sure which metadata formats HEIC/HEIF supports and I cannot find anything in the XMP standard that claims that XMP metadata must be embedded in HEIC/HEIF files.
Do you have more info? Can you show me the standard document?

[vsammy]

Thanks for replying, Mario.

I am completely new in this topic, I know the Lock-In of some companies with heic/heif could be problematic.... Still, I do have tenthousands of heics - yes there is a way to covert them to jpg - but I don't want to loose quality by converting.

I found this:
- it seems that Exiftool suports writing the Data into Heic
https://exiftool.org/#supported

- and there is a way how to write the data into the heics (??)
https://exiftool.org/forum/index.php?topic=11721.0

Greetings, vsammy

[Mario]

I shall then consider this, check for corresponding additions in the XMP standard etc.
I have no idea if adding/embedding XMP in HEIC/HEIF files maybe breaks a ton of applications out there.
HEIC is a format mostly used by Apple, and they have their reasons: platform lock-in, patent-protected file format, ability to sue everybody using it without proper permission and royalty payments etc.
I wonder why users allow their devices to store their data in such a patent-infested and legal horror format. I have not much experience with this format, because I don't use Apple hardware.

[Jingo]

I wonder why users allow their devices to store their data in such a patent-infested and legal horror format. I have not much experience with this format, because I don't use Apple hardware.

Marketing.. and the majority of users are not technically savvy.... they hear over and over that apple has this new format that SAVES SPACE without losing quality, it is turned on by default and voila.... millions and millions of photos in this new format! 

I had a client send me three thousand HEIC files from their phone for a book project... the person had no idea was HEIC was.  Good ole Apple!

[vsammy]

True! Same with me. Heic has many advatages (16bit color, smaller size, etc.) and it is set as standard in iphones and some android phones. After storing thousands of (good-quality-)photos and starting to sort them, I got touches with iMatch. After reading about meta-tags and fileformats, more and more is getting clear. But as a standard-just-use-the-camera-user, I guess many don't know what the disadvantages are.
Thanks Mario!
VSammy

[Mario]

Apple/Nokia inveted HEIC/HEIF to save disk space so Apple could get away witch selling smaller disks at much higher prices.
At least as far as I can tell, HEIC is not really used outside the Apple world. And has hundreds (!) of software patents attached to it.

Actually, HEIC has over 7000 patents affecting it.

I will look into this, maybe making an option to let ExifTool write XMP data to HEIC. Not sure what other metadata HEIC may contain, I have only 20 sample files or so.
This will of course open up another problem field: Adding a feature to IMatch to copy existing XMP data from sidecar files into HEIC, and then deleting the sidecar file (only one source of truth).

Unfortunately, I don't have the money to pay a specialized lawyer to determine if and how many of Apple/Others patents writing metadata into a HEIC file via ExifTool would violate. Or my own patents to trade with the patent holders. Or buy an insurance to protect me from the Apple law department, which has the size of a small city. And Apple is suing all the time, from small independent developers to entire governments.

All our technology, the stuff we use everyday, is more and more owned by the big companies, their legal departments and tens of thousands of patents filed each year for every minute piece of tech or algorithm. Some day one of these will find a way to patent breathing or air or iAir  and then we're doomed. But, shiny new iPhone me...

I fear that the times of small independent software vendors will be over in a decade or so. Then it's either unpaid open source or Adobe/Microsoft/SAP/Oracle ware.

[vsammy]

Will switch back to jpg / raw for future photos from now on.... Sounds like horror how they could come because of patents.
Absolutely - small devellpoers ar so important, especially becaue they can react so quick and implent features without having endless discussions with many thoughts in a bigger company.

Thanks a lot for supporting, that great!

If I can help you with sending HEIC Pictures, it would be a pleasure... I could send you a link with pictures made with Apple iPhone 7, iPhone XS, iPhone SE2 and iPhone 11Pro...

Best regards,
vsammy

[Mario]

I have 30 or so HEIC/HEIF sample files in my test suite.
But a few more is always better. You can send me a link to my email address, see support email address

[Mario]

But as a standard-just-use-the-camera-user, I guess many don't know what the disadvantages are.

Most users don't care and Apple does not tell them. They want you to use HEIC because this is an advantage to them. Apple wants to get away with the smallest internal storage, for cost-cutting reasons.

On the same principle, Adobe and Microsoft Office now uses their clouds as the standard storage location for your files (and nag you about storing your personal files in their clouds, because, you know, cloud and safety).
Storing files on your local computer now actually requires some extra steps and clicks.

If you spend some time to read the 60+ pages of legalese you agree to when you store anything in their clouds, you learn that they can basically do anything they want with your files, to "improve their service". Just think about what Adobe/Microsoft/Apple/Google/... can learn about you by just letting their AIs analyze the content of what you upload...
The company who has your data has more control over you.

And people slowly learn that their account can be temporarily or permanently disabled, because of something the company's AI think you did that violates one of the 400 rules in their service agreement. And then good luck with reaching an actual support person at Google or Adobe or Microsoft or...

I've just read today that another Apple user is suing Apple for damages - because they disabled his account without telling him why. Just like that. No way to appeal.
All his data and Apple mails: inaccessible. All his Apple Store purchases: inaccessible. All bought movies: inaccessible.
You really don't own anything anymore when you buy digital goods from Apple. They close your account because of a computer hickup and that's that. Pay for a lawyer and court to get back the stuff you have bought.

I don't like patent-infested or proprietary file formats. I don't like proprietary metadata schemas. I still recall the mess the GIF patent caused a decade ago. And that was only one patent, not 7,000 like in the case of HEIC/HEIF.

Not even Windows supports HEIC out of the box. You must download and install the HEIC codec yourself. Because Microsoft has to pay royalties for every user.
Which is why the video HEIC codec costs extra money.
I don't touch HEIC when I don't have to. HEIC is also not one of the file formats permitted for long-term image storage in libraries and archives. Guess why.

[Tveloso]

Hi Mario, hello all

IMatch 2020 writes XMP to HEIC files. I hope this does not violate any patents from Apple or Nokia. But then Phil from ExifTool will be the first one Apple sues, and I can vanish somewhere int he jungle.

I am very interessted to write the metadata to HEIC files. Right now all the Metadata were written by iMatch 2020 in a sidecarfile .xmp.
How / Where can I change that behaviour? And when the data is written - I probably have to delete to .xmp File manually?

Thanks in advance for help,
VSammy

IMatch 2020 does embed the XMP in HEIC Files by default - that's how it's behaving for me.  Might it be that you might no longer have the Default set up for that FileType in your IMatch Installation?:
       

...and you at some point selected the force or favor Sidecar option for them?...(so IMatch 2020 continues to use Sidecars in your case)?

Or am I misunderstanding, and this actually relates to wanting to "move" the XMP into the HEICs for the older files that were processed when IMatch still used XMP Sidecars by default?...

I did use the ECP approach mentioned in my previous post for a little while in IMatch 2019, to forcefully embed the XMP...but I hesitate to recommend that, now that IMatch embeds the XMP (and of course does so properly - instead of via the brute force approach that the ECP Preset I mentioned used).

Mario, would re-scanning the HEIC files that have a Sidecar (with a Force Update) cause the data to now be embedded?...(and the Sidecar Files can then be safely deleted?)...

On the subject of using HEIC Files in the first place...I do become a little apprehensive about continuing to use them, every time I see these discussions of their "patent infestment".  The majority of the files that I'm adding to IMatch these days are in fact HEICs...

[vsammy]

Hello Tveloso,

I checked - I steill have the default settings with heic-Files as you described.

I have just imported some new photos out of my iPhone (heic). After analyzing and adding some keywords, faces, etc, the metadata is written as a sicar-file. (Same photos in jpgs - metadata is within the jpg-File).

Regards,
vsammy

[Tveloso]

Hello vsammy,

I'm not sure then why IMatch is still using a sidecar in your case.  I also double-checked my IMatch installation to ensure that it is in fact embedding the XMP, and found that it is (checked in the ECP, and verified that no XMP Files exist in my ingest folder - where I preform the Writebacks).

Although you shouldn't have to do it, you could uncheck the Use default settings CheckBox, and then explicitly set the XMP sidecar fie parameter to Embed XMP in file:

       

[vsammy]

Hello Tony,

wow... that works!!! 

I deleted the sidecar xmp files and set some Tags and keywords. After unchecking the option standard in settings and explicitly setting the option to embed in xmp file the face data is now inside my heic-files!

Thanks a lot for your help!

But hierarchical data isn't inside the file.... Do you miss these  data too?

Regards, vsammy

[frlindla]

Hello there.
I have about 3000 heic files in my database. Is this the way to go then, to delete the xmp and embed into the file? Did you search for all xmps connected to these files and then delete? Did you delete from windows explorer?

[Tveloso]

wow... that works!!! 

I deleted the sidecar xmp files and set some Tags and keywords. After unchecking the option standard in settings and explicitly setting the option to embed in xmp file the face data is now inside my heic-files!

Hi vsammy,

I'm glad it's working with the option to explicitly embed the XMP...(but it's still curious that you had to do that, since the Default is also to embed for HEIC Files).

But hierarchical data isn't inside the file.... Do you miss these  data too?

For me, the Hierarchical Keywords are also included.

I'm guessing that you have already identified the issue with that though, and the absence of the Keywords in your case is related to the discussion in your other post:

        https://www.photools.com/community/index.php?topic=11327.msg80881#msg80881

...where you found that Excire and/or the ExifToolGIU was the cause.

In fact, I wonder if that might also be responsible for the apparent continued use of Sidecars by IMatch...(could it be that even with the Default setting, IMatch is in fact ebmedding for you also?...and it's actually Excire that's creating the Sidecar?).  Perhaps, if you repeated your test with the Default settings for the HEIC FileType, it would still behave the same way as in your first test?...

Hello there.
I have about 3000 heic files in my database. Is this the way to go then, to delete the xmp and embed into the file? Did you search for all xmps connected to these files and then delete? Did you delete from windows explorer?

Hi frlindla,

Since the Default in IMatch 2020 is to embed the XMP for HEICs, I'm thinking that's the behavior you have now for new files.  So the question is really how to go about "moving" the data from the Sidecars into the Files themselves for your older files.

The Technique I mentioned in my other post above (of using an ECP Preset to embed the data):

        https://www.photools.com/community/index.php?topic=9355.msg67209#msg67209

...I used only for a short period of time in IMatch 2019 (when the default was still to use a Sidecar).  I had relatively few XMP Sidecars, and I actually held off on doing Writebacks, and waited for IMatch 2020, where the XMP would be embedded "naturally".

I did delete the XMP Sidecars in Windows Explorer for the few older files that had them (once had completed the "manual embedding").  But I worry that doing that in your case, when you have so many older files that have Sidecars might lead to problems.  I wonder if there might be a more elegant solution, using the now "native embedding behavior" in IMatch 2020...

[Mario]

So the question is really how to go about "moving" the data from the Sidecars into the Files themselves for your older files.

IMatch 2021 has a special command for that. See https://www.photools.com/community/index.php?topic=11377.0