Error: Can't write multi-segment EXIF with external pointers

[ubacher]

When I create a jpg from a NIKON Z6 NEF file (using ACR V13) and then write out metadata I get sometimes the error
Error: Can't write multi-segment EXIF with external pointers. The file shows a yellow error triangle.
If I the rescan the file the error triangle disappears.


I gathered from the exiftool blogs that this has to do with the exif data being larger than a block
(64k?). I thus had a careful look at the data to make sure there is no spurious data taking up space. No luck.
To find the culprit data I would like to know the size of the exif data in a file  - How?

I

[Mario]

This sounds more like an ExifTool question? They have their own forum.
This sounds as if a security mechanism kicks in because the EXIF data in your file uses pointers to other areas in the file and updating the EXIF data could break these pointers.

Gee, I'd wish camera vendors would finally stop using this 30 year old EXIF nonsense and switch to XMP metadata.
EXIF makes so many problems, with all the proprietary maker notes, half-assed implementations, ever-changing structures and all the other crap camera vendors come up with.
This all could be so easy if cameras would just put all this info into XMP.

[ubacher]

....but: It is the metadata that Imatch creates which creates this error when Imatch via exiftool writes this out.
The original RAW file shows no problem. (Well... it could be photoshop (ACR) creates wrong or excessive metadata
which Imatch then reads in.)  I have to find out what data causes the error and who creates this.

[Mario]

Unless you explicitly change an EXIF tag value, IMatch updates Exif only indirectly, by letting ExifTool map XMP back into EXIF.
IMatch has no control over how ExifTool does it, it uses the XMP2Exif args file to do the mapping.
Maybe the metadata in your file is broken or there  is some other issue. I have never experienced this nor seen this ExifTool error message before.

Maybe disable EXIF synching for your file format until (Edit > Preferences > Metadata: File Formats) you have figured out what's causing the problem?
This may cause other side-effects, but unless you force an update, the XMP will stay OK.

I cannot help you with this issue.
You are using an undocumented and proprietary image format from Nikon as the source.
You used an Adobe product to produce the JPEG file with the unusual EXIF data.

[ubacher]

Harvey says this:
https://exiftool.org/forum/index.php?topic=9512.msg49283#msg49283

I will investigate further. It is probably that the size of the metadata plus the size of the thumbnail
exceeds 64k. If this is the problem then I will have to see which of the two to reduce in size.

Is there a way to inspect the file to see the size of the metadata block? Years ago I remember looking
at jpg files with a program called hexdump. Is that still available and a viable way?

[Mario]

Use the "List Metadata" or "List EXIF Metadata" in the ExifTool Command Processor.
It shows the length of embedded preview images, like

[IFD0]          Preview Image                   : (Binary data 621883 bytes, use -b option to extract)
[IFD1]          Thumbnail Image                 : (Binary data 18062 bytes, use -b option to extract)

You can also add the -v parameter to make ExifTool list all segments with their respective length.

[ubacher]

Now the question came up as to the meaning of the error message. With reference to Harvey:
https://exiftool.org/forum/index.php?topic=11302.msg60576#msg60576
copied here:
BTW, I also added a NoMultiExif API option to allow you to avoid writing multi-segment EXIF if you want.  Just add -api nomultiexif to your command.  With this option you will get an error if the EXIF doesn't fit in a single JPEG segment.

I assume that exiftool writes multi-segments and the error message just informs me that other applications may have trouble reading this.
Or does Imatch use the nomultiexif option and thus not write out the exif block (or write it truncated)? The error message mentioned
by Harvey would indicate that this is the case.

[Mario]

IMatch does not use this option.
From the ET forum I understand that probably only Adobe applications produce this kind of multi-segmented EXIF records and that ExifTool can handle them?
I won't make any changes to how IMatch makes ExifTool write EXIF data until absolutely required. The way it works now works very well and I don't want to brake anything for 99% of all users just because one user has a specific issue with some files produced by some Adobe software.

[ubacher]

I did not want any changes - just to know if I can ignore the error (exiftool wrote it ok) and
just rescan the file to remove the warning
or if I need to fiddle with the metadata to avoid this error.

[ubacher]

I re-read the discussion in the exiftool blog:
https://exiftool.org/forum/index.php?topic=11302.0
and I interpret it thus:
When the warning appears it means a mult-segment EXIF was read or written (--> some programs may be unable to handle this)

So until I encounter problems with some other programs I need not worry about it.
If it becomes a problem I will try to just delete the thumbnail as was suggested.

[Mario]

If you can see the metadata in IMatch and in Photoshop/Lr/Affinity etc. it should be OK.
Not sure what NIKON/Adobe place in the EXIF record to make it that big. The EXIF thumbnail is usually 160x120 pixel in JPEG format and hence does not need much space...

[ubacher]

The NEF file is OK. The jpg generated by ACR adds additional info - lots about ICC profile.

I tried using the batch processor to strip all Metadata but essential EXIF (and some IPTC fields).
This ICC stuff seems to sneak into this output. I attach a EXIFTOOL listing (from exiftoolGUI)
so you can have a look yourself. May be worth revising?

ADMIN: Removed the attachment because it contains your email address.

[Mario]

The ECP output tells me not much. You did not use the -v ExifTool option to include the lengths of the segments in bytes. This would be useful to know.
An embedded ICC profile is kept by default, unless you explicitly convert to another profile or you strip it in the Metadata output. Color profiles are not that huge.

[ubacher]

An embedded ICC profile is kept by default,

I specified to copy only essential EXIF fields in the batch processor. This is why I am surprised to find
the ICC data.  (in other words: how can I strip the ICC profile with the batch processor).

And it is Photoshop which adds this ICC Profile information even if one selected standard sRGB.

[Mario]

EXIF is not ICC.
sRGB can also use a ICC profile. This is preferable over just setting the hint field in the EXIF record, which may or may not be interpreted by other applications. Or may be interpreted wrong.
You can ask an Adobe technician about the details when you have a support or maintenance contract.

I've typed in ExifTool remove ICC profile and the first hit was https://exiftool.org/forum/index.php?topic=3979.0
If you know the correct parameters and are sure about the potential consequences of stripping the ICC profile from your image, you should be able to apply this via the Metadata panel in the BP or the ECP.

[MattMattMattMatt]

I know this is a bit old, but I'm having the same problem with the error message " Error: Can't write multi-segment EXIF with external pointers." It happens for me when I edit files from an OM System OM-1 camera. The camera seems to save an insane amount of metadata, including every auto-focus point and if it was in or out of focus at the time of the shot, plus large amounts of data regarding color profiles.

I have developed a work around by using the command 'exiftool -overwrite_original -makernotes:all= "2022-11-24 140807.jpg"'to remove the makernotes from each of the files.

Just wanted to let you know to add a voice as a second user with this issue. No support needed right now, but an option in the future to help fix this instead of needing a command line would be cool.

PS: of some annoyance, when I use ExifTool to remove the makernotes from a single file iMatch insist on rescan every file in the folder, which does drive me bonkers.

[Mario]

All these are ExifTool issues and I'm sure ExifTool is right.
If you think otherwise, the ExifTool forum (https://exiftool.org/forum/) is the best place to ask.

Provide Phil with sample images so he can analyze what makes your files special or why the maker notes have external pointers (very unusual). Camera vendors sometimes do strange things, cameras have buggy firmware which messes up EXIF or maker notes and all that. Happens often, and camera vendors intentionally don't document their proprietary maker notes for "customer retention" purposes...

PS: of some annoyance, when I use ExifTool to remove the makernotes from a single file iMatch insist on rescan every file in the folder, which does drive me bonkers.

When you change files in a folder, Windows will send one or multiple file system modified / folder modified / file modified messages.

IMatch reacts on these messages by placing the folder into the rescan queue (it does not know at that time what has changed in the folder).

If Windows reports a modified file, IMatch rescans the file in the background (unless more than 100 (?) new/modified files were found, in which case IMatch displays the load progress dialog).

When IMatch processes modified folders, it quickly checks for new files and compares the timestamps of existing files with the timestamps recorded in the database. This is very fast (hundreds of files per second). Only when a timestamp has changed, IMatch rescans the file.

What is the problem with this that makes this annoying? Usually you won't even notice this, because it is so fast (unless new or modified files are found and processed).

You can see in the log file which files IMatch has scanned, if modified files were found etc.  Search for AddOrUpdateFile in the log file. Unless files have actually been modified, IMatch reports "File is current" for each checked file.

If a folder scan takes very long, either many new/updated files have been found or the virus checker is somehow interfering with IMatch, slowing it down badly (if you have a virus checker other than Windows Defender).

[ubacher]

I am constantly plagued by this error message. Since it clears when one rescans the file it is actually
just a warning. Maybe there would be a better way to inform the user of this rather than the yellow triangle
requiring a rescan to clear.
Associated - after some change to such a file later on the error message appears again and the same rescan
is required. Because of this I remove the Makernotes which, because large (62k bytes in my case)  cause this error.
It is Photoshop which copies these to the jpg file from the raw file. I do not want to remove the Makernotes
from the raw files though.

Because of this having to run exiftool I thought of requesting that we can run an exiftool preset via a favorite.
Would that be doable?

For those of you who want to see the sizes of the various parts of metadata here is the
exiftool preset I use:

Code Select Expand

-G1
-a
-s
-exif
-xmp
-iptc
-JPEGImageLength
-Warnings
-Adobe
-U
-Makernotes
{Files}
A sample output:

Code Select Expand

[EXIF]          EXIF                            : (Binary data 78392 bytes, use -b option to extract)
[XMP]           XMP                             : (Binary data 16402 bytes, use -b option to extract)
[IPTC]          IPTC                            : (Binary data 126 bytes, use -b option to extract)
[File]          JPEGImageLength                 : 4358810
[Adobe]         Adobe                           : (Binary data 12 bytes, use -b option to extract)
[ExifIFD]       MakerNoteNikon                  : (Binary data 62000 bytes, use -b option to extract)

[Mario]

Please send me one of these files (original RAW and Ps). support email address with a link back to this thread.
Thanks.

Ps usually does not copy maker notes, because Adobe considers this potentially dangerous.
Maker notes often contain offsets (pointers) to data blocks in the original file, and copying them to another file will break these pointers - they will point at the wrong offsets or even outside the file.
When ExifTool copies maker notes, it also copies the data the pointers point to and updates the pointers in the maker notes to match the new file layout. It does all that only when it is sure that it understands the maker notes and knows how to process them. This is very complex but allows to retain the often informative data in the maker notes.

(Which could all be stored in the XMP record, avoiding all the proprietary shit and problems caused by undocumented maker notes. But camera vendors want to keep them proprietary, as another attempt at customer retention.)

If Ps copies the maker notes from the RAW but does not update the offsets/pointers, they will point everywhere, maybe even "outside" the JPG file. Which sounds like the error ExifTool reports, right? The maker notes are invalid and damaged beyond repair.

Please contact Adobe support and make them aware of the problem their software causes. If you have a current license. Older versions are no longer supported.

IMatch displays the yellow warning icon when ExifTool reports an Error, not when it reports warnings. Warnings are pretty common and IMatch needs them to log them. But the error icon should only show up when there is a real error. Like corrupted maker notes.

Check the file in the Metadata Analyst in IMatch to learn more about the warnings and errors reported by ExifTool.
They will also be logged to the log file.

A forced rescan or a reload metadata for such a file should produce exactly the same result. ExifTool should report the same error.

Deleting maker notes can be done, and ExifTool only does it when it is sure that it will not damage the file.
But you should try to fix the root of the problem, aka the Adobe software or whatever you use and which produces invalid maker notes.

[ubacher]

I was wrong about PS being the culprit. Somehow I had IM propagation set to copy Makernotes.
Why I did this and why I did not discover this earlier I can't say. SORRY.

(Racking my brain I think there must have been some info from the Makernotes that I wanted to have
available in the JPG file - which made me set this propagation. And then I forgot about this setting.)

[Mario]

ExifTool usually only copies maker notes when it is sure about their structure and it is safe to copy them to another file.
It's not recommended doing this, though. Not unless you really, really badly need some of the info. EXIF and especially maker notes are just designed to be copied between files. We have XMP for that.

When I really want to preserve some maker note data, I use a MD template to copy the info into a safe XMP tags.