Metadata truncation - how to prevent it?

[Pawel]

I would appreciate some help regarding IMatch and Lightroom.

I am migrating my image management from Lightroom to IMatch. I have a lot of old JPGs for which I added some metadata fields (Transmission Reference, Location) that are of type String in XMP (so the length of a value is not a problem) which however are related to IPTC fields (like Original Transmission Reference) having 32 character limit.
It used to be no problem in Lightroom, now however I see that for many images (not all) the XMP metadata is repeatedly truncated to 32 character limit whenever I let Photoshop or Lightroom write metadata to images and rescan the files in IMatch.
I need to do some more investigation however I think I noticed a following sequence:

• Write "long" metadata in IMatch.
• Save file in Photoshop. Checking with Exiftool, still the XMP value is "long", the IPTC field is truncated.
• Rescan the folder in IMatch. The XMP value is truncated.

Now I've read that sometimes it can happen that IPTC value will overwrite XMP field on import if the metadata is not synchronized. I've read also that removing IPTC data from files can be a solution.
What should I do to prevent truncating XMP data? I think all of my "Metadata 2" settings are default as set by IMatch install.

Is there a setting in IMatch that can be used to make IPTC "write-only" for images that are already managed in IMatch? I am not sure if removing IPTC from images is really the best solution as the process will take many, many hours and there is no guarantee the problem will not reoccur should any other application (Lightroom, Photoshop, Windows Photos etc.) write IPTC records to files.

[Mario]

Why do you still work with legacy IPTC data? This has been deprecated 10 years (!) ago by the IPTC. New files should never contain legacy IPTC data, and IMatch does not create it.
To avoid this kind of problem, I recommend to strip the legacy IPTC data from your files (there is a preset for that in the IMatch ExifTool Command Processor) and only work with XMP data.

When IMatch detects that a file has changed, it re-reads the metadata in the file, applying the XMP mapping from IPTC/EXIF/GPS into XMP.
If the IPTC data in the file has been truncated to 32 characters (which is correct and has to be done) (the OTR is usually only a 6 digit code) ExifTool will map the truncated value into XMP.

If you are combining legacy IPTC data and you modify your metadata in multiple applications, you will be affected by such problems. I suggest you only manage your metadata in IMatch. IMatch will truncate the IPTC data when writing, but will retain the full XMP data.

I also suggest that you stick to the character length limits specified for legacy IPTC when you will in the XMP fields which are mapped. Especially if you continue modifying metadata in multiple applications and also want to keep the legacy IPTC data. For me, this would way too much trouble. Remove the legacy IPTC data and use only XMP. Problem solved, application interoperability restored.

[Pawel]

Thank you for your reply. I am finishing deleting IPTC data from images and hope the problem won't come back. :-)

[Pawel]

Well, the problem does come back, sort of.
I deleted all the IPTC data from images and now the images metadata contain only the following tags (content deleted):

Code Select Expand


[ExifTool]      ExifTool Version Number         :
[System]        File Name                       :
[System]        Directory                       :
[System]        File Size                       :
[System]        File Modification Date/Time     :
[System]        File Access Date/Time           :
[System]        File Creation Date/Time         :
[System]        File Permissions                :
[File]          File Type                       :
[File]          File Type Extension             :
[File]          MIME Type                       :
[JFIF]          JFIF Version                    :
[JFIF]          Resolution Unit                 :
[JFIF]          X Resolution                    :
[JFIF]          Y Resolution                    :
[File]          Exif Byte Order                 :
[IFD0]          Make                            :
[IFD0]          Camera Model Name               :
[IFD0]          Orientation                     :
[IFD0]          Page Name                       :
[IFD0]          Modify Date                     :
[ExifIFD]       Exif Version                    :
[ExifIFD]       Date/Time Original              :
[ExifIFD]       Create Date                     :
[XMP-x]         XMP Toolkit                     :
[XMP-dc]        Format                          :
[XMP-dc]        Subject                         :
[XMP-exif]      Date/Time Original              :
[XMP-exif]      Exif Version                    :
[XMP-lr]        Hierarchical Subject            :
[XMP-mwg-rs]    Region Applied To Dimensions H  :
[XMP-mwg-rs]    Region Applied To Dimensions Unit:
[XMP-mwg-rs]    Region Applied To Dimensions W  :
[XMP-mwg-rs]    Region Area H                   :
[XMP-mwg-rs]    Region Area W                   :
[XMP-mwg-rs]    Region Area X                   :
[XMP-mwg-rs]    Region Area Y                   :
[XMP-mwg-rs]    Region Rotation                 :
[XMP-mwg-rs]    Region Type                     :
[XMP-photoshop] Date Created                    :
[XMP-photoshop] Transmission Reference          :
[XMP-tiff]      Make                            :
[XMP-tiff]      Camera Model Name               :
[XMP-tiff]      Orientation                     :
[XMP-tiff]      Software                        :
[XMP-xmp]       Create Date                     :
[XMP-xmp]       Creator Tool                    :
[XMP-xmp]       Metadata Date                   :
[XMP-xmp]       Modify Date                     :
[XMP-xmpMM]     Document ID                     :
[XMP-xmpMM]     History Action                  :
[XMP-xmpMM]     History Changed                 :
[XMP-xmpMM]     History Instance ID             :
[XMP-xmpMM]     History Software Agent          :
[XMP-xmpMM]     History When                    :
[XMP-xmpMM]     Instance ID                     :
[XMP-xmpMM]     Original Document ID            :
[XMP-crs]       Raw File Name                   :
[Photoshop]     IPTC Digest                     :
[ICC-header]    Profile CMM Type                :
[ICC-header]    Profile Version                 :
[ICC-header]    Profile Class                   :
[ICC-header]    Color Space Data                :
[ICC-header]    Profile Connection Space        :
[ICC-header]    Profile Date Time               :
[ICC-header]    Profile File Signature          :
[ICC-header]    Primary Platform                :
[ICC-header]    CMM Flags                       :
[ICC-header]    Device Manufacturer             :
[ICC-header]    Device Model                    :
[ICC-header]    Device Attributes               :
[ICC-header]    Rendering Intent                :
[ICC-header]    Connection Space Illuminant     :
[ICC-header]    Profile Creator                 :
[ICC-header]    Profile ID                      :
[ICC_Profile]   Profile Copyright               :
[ICC_Profile]   Profile Description             :
[ICC_Profile]   Media White Point               :
[ICC_Profile]   Media Black Point               :
[ICC_Profile]   Gray Tone Reproduction Curve    :
[File]          Image Width                     :
[File]          Image Height                    :
[File]          Encoding Process                :
[File]          Bits Per Sample                 :
[File]          Color Components                :
[Composite]     Image Size                      :
[Composite]     Megapixels                      :


I update the Transmission Reference field, write-back metadata and there is an IPTC record again:

Code Select Expand


[ExifTool]      ExifTool Version Number         :
[System]        File Name                       :
[System]        Directory                       :
[System]        File Size                       :
[System]        File Modification Date/Time     :
[System]        File Access Date/Time           :
[System]        File Creation Date/Time         :
[System]        File Permissions                :
[File]          File Type                       :
[File]          File Type Extension             :
[File]          MIME Type                       :
[JFIF]          JFIF Version                    :
[JFIF]          Resolution Unit                 :
[JFIF]          X Resolution                    :
[JFIF]          Y Resolution                    :
[File]          Exif Byte Order                 :
[IFD0]          Make                            :
[IFD0]          Camera Model Name               :
[IFD0]          Orientation                     :
[IFD0]          Page Name                       :
[IFD0]          Modify Date                     :
[ExifIFD]       Exif Version                    :
[ExifIFD]       Date/Time Original              :
[ExifIFD]       Create Date                     :
[File]          Current IPTC Digest             :
[IPTC]          Keywords                        :
[IPTC]          Date Created                    :
[IPTC]          Time Created                    :
[IPTC]          Digital Creation Date           :
[IPTC]          Digital Creation Time           :
[IPTC]          Original Transmission Reference :
[IPTC]          Application Record Version      :
[Photoshop]     IPTC Digest                     :
[XMP-x]         XMP Toolkit                     :
[XMP-dc]        Format                          :
[XMP-dc]        Subject                         :
[XMP-exif]      Date/Time Original              :
[XMP-exif]      Exif Version                    :
[XMP-lr]        Hierarchical Subject            :
[XMP-mwg-rs]    Region Applied To Dimensions H  :
[XMP-mwg-rs]    Region Applied To Dimensions Unit:
[XMP-mwg-rs]    Region Applied To Dimensions W  :
[XMP-mwg-rs]    Region Area H                   :
[XMP-mwg-rs]    Region Area W                   :
[XMP-mwg-rs]    Region Area X                   :
[XMP-mwg-rs]    Region Area Y                   :
[XMP-mwg-rs]    Region Rotation                 :
[XMP-mwg-rs]    Region Type                     :
[XMP-photoshop] Date Created                    :
[XMP-photoshop] Transmission Reference          :
[XMP-tiff]      Make                            :
[XMP-tiff]      Camera Model Name               :
[XMP-tiff]      Orientation                     :
[XMP-tiff]      Software                        :
[XMP-xmp]       Create Date                     :
[XMP-xmp]       Creator Tool                    :
[XMP-xmp]       Metadata Date                   :
[XMP-xmp]       Modify Date                     :
[XMP-xmpMM]     Document ID                     :
[XMP-xmpMM]     History Action                  :
[XMP-xmpMM]     History Changed                 :
[XMP-xmpMM]     History Instance ID             :
[XMP-xmpMM]     History Software Agent          :
[XMP-xmpMM]     History When                    :
[XMP-xmpMM]     Instance ID                     :
[XMP-xmpMM]     Original Document ID            :
[XMP-crs]       Raw File Name                   :
[ICC-header]    Profile CMM Type                :
[ICC-header]    Profile Version                 :
[ICC-header]    Profile Class                   :
[ICC-header]    Color Space Data                :
[ICC-header]    Profile Connection Space        :
[ICC-header]    Profile Date Time               :
[ICC-header]    Profile File Signature          :
[ICC-header]    Primary Platform                :
[ICC-header]    CMM Flags                       :
[ICC-header]    Device Manufacturer             :
[ICC-header]    Device Model                    :
[ICC-header]    Device Attributes               :
[ICC-header]    Rendering Intent                :
[ICC-header]    Connection Space Illuminant     :
[ICC-header]    Profile Creator                 :
[ICC-header]    Profile ID                      :
[ICC_Profile]   Profile Copyright               :
[ICC_Profile]   Profile Description             :
[ICC_Profile]   Media White Point               :
[ICC_Profile]   Media Black Point               :
[ICC_Profile]   Gray Tone Reproduction Curve    :
[File]          Image Width                     :
[File]          Image Height                    :
[File]          Encoding Process                :
[File]          Bits Per Sample                 :
[File]          Color Components                :
[Composite]     Date/Time Created               :
[Composite]     Digital Creation Date/Time      :
[Composite]     Image Size                      :
[Composite]     Megapixels                      :


Exiftool output is as follows:

Code Select Expand


-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-tagsfromfile
E:\images\KonicaMinolta DiMAGE Scan 5400.1\n199x\n199x-001-24.jpg
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\exif2xmp.args
--Exif:rating
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\iptc2xmp.args
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\gps2xmp.args
-sep


-XMP-photoshop:TransmissionReference=1992-04-06 Wyjazd na targi w Poznaniu
-XMP:CreatorTool=photools.com IMatch 5.7.0.2 (Windows)

-xmp:InstanceID=xmp.iid:892e9126-01f4-4dea-9604-c12916ff4726

-XMP:MetadataDate=now
-XMP:ModifyDate=now
E:\images\KonicaMinolta DiMAGE Scan 5400.1\n199x\n199x-001-24.jpg
-execute
-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-EXIF:ImageDescription=
-EXIF:Software=
-EXIF:Copyright=
-EXIF:Artist=
-EXIF:UserComment=
-IFD0:ModifyDate=
-IFD0:Rating=
-IFD0:ImageDescription=
-IFD0:Software=
-IFD0:Copyright=
-IFD0:Artist=
-IFD0:XPTitle=
-IFD0:XPComment=
-IFD0:XPAuthor=
-IFD0:XPKeywords=
-IFD0:XPSubject=
-IPTC:By-line=
-IPTC:Caption-Abstract=
-IPTC:CopyrightNotice=
-IPTC:Keywords=
-IPTC:ObjectName=
-IPTC:By-lineTitle=
-IPTC:Writer-Editor=
-IPTC:Category=
-IPTC:City=
-IPTC:Country-PrimaryLocationName=
-IPTC:Credit=
-IPTC:DateCreated=
-IPTC:TimeCreated=
-IPTC:Headline=
-IPTC:SpecialInstructions=
-IPTC:Source=
-IPTC:Province-State=
-IPTC:SupplementalCategories=
-IPTC:OriginalTransmissionReference=
-IPTC:Urgency=
-IPTC:Country-PrimaryLocationCode=
-IPTC:Sub-location=
-IPTC:DigitalCreationDate=
-IPTC:DigitalCreationTime=
-tagsfromfile
E:\images\KonicaMinolta DiMAGE Scan 5400.1\n199x\n199x-001-24.jpg
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\xmp2exif.args
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\xmp2iptc.args

E:\images\KonicaMinolta DiMAGE Scan 5400.1\n199x\n199x-001-24.jpg
-execute9999

    1 image files updated

    1 image files updated

----- Runtime: 0,5 s.
Warning: Error converting value for ExifIFD:SubSecTimeOriginal (ValueConvInv) - E:\images\KonicaMinolta DiMAGE Scan 5400.1\n199x\n199x-001-24.jpg


My settings for metadata are the default ones:

• MWG Compliance = Yes
• All file formats, including JPG and JPEG file format: Default settings (Write IPTC = Yes, Allow create IPTC/EXIF/GPS = No).

I was sure the IPTC data shouldn't be created - what am I doing wrong?

[Pawel]

Oh, and by the way: I just found that just saving image (without IPTC data) in Photoshop also recreates the basic IPTC data, in it's correctly truncated form - so it is probably just a matter of time when the IPTC data will be imported by IMatch and replace the corresponding IPTC fields. So now I have two problems: 

• IMatch recreating IPTC data upon metadata write-back
• Photoshop recreating IPTC data upon Save

I think I'd like to switch off IPTC data import in IMatch - is it possible? Or is there any other solution?

[Mario]

IMatch only writes legacy IPTC data if it detects an IPTC record in your file.

Did you do a forced rescan after stripping the data?
Because by default IMatch protects unwritten metadata in your database (to protect if from being overwritten when you change a file in another application).This may have caused it to re-create the IPTC data.

I made a quick test.

I imported a file with a legacy IPTC record into IMatch.
I then removed all IPTC data from the file via the ECP preset.
Now I did a forced rescan of the file to flush out all pending write-backs.
I set the Job-Id (XMP name for the old transmission reference) to JOB-123.
Write-back.

Only the XMP data is written.
The ECP shows no legacy IPTC data in the file.
Works as designed.

[Pawel]

After I removed IPTC data by ECP preset I performed "Rescan drive" to rescan all the directories. However I am sure the "pending write-back" collection was empty before I started to work with images again.

[Mario]

Select the file, do a <Shift>+<Ctrl>+<F5> to force a rescan. This will also replace all cached metadata in the database for the file.

That's what I did and it removed IPTC data just fine. Updating the XMP (!) transaction reference did not re-create the IPTC data in the file.
Of course you need to make sure that Lr/PS do not interfere by flushing out IPTC data to the file again.

[Pawel]

OK, that's what I do now (forced full rescan of all files).

Of course you need to make sure that Lr/PS do not interfere by flushing out IPTC data to the file again.

And that's the second part of the problem. I just checked that Photoshop writes certain XMP tags to IPTC tags in truncated form - and probably several other applications do that, too. I can prune IPTC data periodically however I'd rather not do that on a daily basis.

What can I do to make IMatch ignore IPTC data in images whatsoever? Shoud I edit the Exiftool arg-file (iptc2xmp) or is there a setting for that somewhere in Preferences?

[Mario]

According to the recommendations of the Metadata Working Group (of which Adobe is a major part) new files should not receive legacy IPTC data anymore.

What can I do to make IMatch ignore IPTC data in images whatsoever? Shoud I edit the Exiftool arg-file (iptc2xmp) or is there a setting for that somewhere in Preferences?

Can't. Not without disabling MWG compliance and configuring everything manually, NOT recommended.

Some remarks:

1. If you work with multiple applications on the same set of metadata, and this causes problems, stop doing it. In such cases it is better to use only one application to edit all your metadata - and most users decide that IMatch is the best choice.

2. Solve your problem by not entering more data into the Job-Id field than allowed by IPTC.

3. This is the root of your problem. The OTR field is limited to 32 character - if you need to keep it in sync with legacy IPTC data.
Consider using shorter references or to move the data to a tag that allows for more text (description, title, ...).
Most workflows in the old days used this tag only to store short bits of info, or plain job number. Barely more than 10 chars or so.

I'm not sure why PS is insisting in writing legacy IPTC data in your case.

[Pawel]

1. If you work with multiple applications on the same set of metadata, and this causes problems, stop doing it. In such cases it is better to use only one application to edit all your metadata - and most users decide that IMatch is the best choice.

I've already decided to work only in IMatch - but I have no control (or I don't know how to impose it) over Photoshop creating this IPTC data.

2. Solve your problem by not entering more data into the Job-Id field than allowed by IPTC.

I'll look into available tags to enter job description however the problem is not only with Transmission Reference/Job ID tag. The Location tag gets truncated, too - and I don't know which tag I can replace it with. 

Is editing iptc2xmp arg-file really such a bad idea?

[Mario]

Is editing iptc2xmp arg-file really such a bad idea?

Oh, yes.
Firstly, this file is replaced with every IMatch update.
Secondly, this file has been very carefully created by Phil Harvey (Author of ExifTool) to work across all applications. I would not dare to fiddle with it.


Photoshop seems to always write legacy IPTC, for whatever reason.
Photoshop limits all IPTC values correctly to the maximum length specified in the standard.
This means that it writes the full text into transmission reference and location, but crops them when writing them into IPTC.
This causes a problem in your case, when you insist on entering more than the allowed 32 characters.

[Pawel]

OK, I understand.

I'll refrain from modifying the arg-file and look into reworking all the locations and job descriptions to fit into the 32 character limits. It is a pity though that one can not take advantage of the new, more flexible XMP standard because of how the thing work now.

[Mario]

You're good when you only work in IMatch - because it does not create legacy IPTC data.
You are also good as long as you don't modify files in other applications and force IMatch to re-read metadata.
Your good as long as you let IMatch write legacy IPTC data. IMatch does not crop (unless ExifTool requires it).
You're good if you stick to the maximum limits as long as you have a workflow that creates/requires legacy IPTC data. IMatch does not.
IPTC sub-location and OTR are both 32 characters.

[Pawel]

Waiting for a full rescan of my images  I looked into MWG Guidelines for Handling Image Metadata (v. 2.0) to learn how the IPTC, EXIF and XMP should interact with each other and so I am coming back to this topic again. If I understand the guidelines correctly, there is a notion there, that if an image has both IPTC-IIM block and XMP block, the checksum of the blocks should be compared and if it doesn't match, the XMP should "win".

That IMatch behaves differently - is it by design, or are there any other circumstances that led to the situation where IPTC data "wins" over XMP while doing rescan?

[Mario]

IMatch does not care for the digests. They are not reliably anyway. Many applications don't update them.
ExifTool creates and updates them as needed.
The metadata implementation in IMatch is super-duper extremely complicated. It's in place now for almost 3 years. A proven implementation. I have no intention to change anything in the way IMatch handles metadata, unless changes in the standards require it.

If you need any non-standard functionality, disable MWG, ignore mapping, ignore IPTC. IMatch gives users so many possibilities - if none of the over 120 metadata options allow you to do what you need, you may look into other ways or applications.

[Pawel]

Please don't take it personally that I ask many questions - I am setting my new workflow and IMatch is pretty complicated because of all the options and possibilities it offers. I realize that the level of support you offer here is absolutely above-standard so I try to not waste your time and try as much as possible to gather knowledge by reading the help file, or by searching the forum. I ask questions here only when I can't find an explanation elsewhere or if it seems to me that the behaviour of the program differs from the standard or the description, or the information provided in the help file is insufficient - and I greatly appreciate your responses.

For a software offering so much it is probably not possible to meet every user expectation - and I don't expect you will change the software you build for years only because I ask if something shouldn't work some other way - but it is important for me to know whether I achieve such and no other effect because of my error (and I have to correct my ways), program error (then I can expect it will be fixed) - or simply it was designed that way (and then I have to find a workaround). 

[Mario]

If you need mapping between legacy IPTC (which has been discontinued 10 years ago for good reasons) and XMP, you have to make sure that your input is within the limits of legacy IPTC. As you said yourself, Photoshop cuts off the data you enter when writing IPTC because the data you enter is too long. If you stick to the limits, you have no problem.