How to write XMP data?

[RobiWan]

I think I'm too stupid for the software or the software is too stupid for me.

What I will is write Metadata - create XMP files but this don't work for me correctly. See my screenshots
1. There is no *.xmp file in directory
2. right panel shows some keywords AND copyright and the checkbox copyrighted is set

After I tell IMatch "write metadata for selected files" Ctrl+Alt+S Copyright has disappeared and Keywords shows different as previously

[Mario]

This looks like your MRW file contains embedded XMP data with copyright info and probably other tags as well.
This override the metadata IMatch is writing to the XMP sidecar file.
Some cameras write a rating or copyright data now. This fcks up the standard XMP metadata handling for RAW files implemented by IMatch.

Have a look at one of your MRW files in the ExifTool Command Processor. Use the "List Metadata" preset.
This will show us the metadata contained in your MRW files.
If the file contains XMP, we know what the problem is.

[RobiWan]

No the RAW file has no any infos about Copyright and other tags

Code Select Expand

[ExifTool]      ExifTool Version Number         : 10.68
[System]        File Name                       : pict0001.mrw
[System]        Directory                       : C:/DSLR/DSLR/ExtHDD 2501/Dynax7D/RAW/2006/03/13
[System]        File Size                       : 8.8 MB
[System]        File Modification Date/Time     : 2006:03:13 16:05:46+01:00
[System]        File Access Date/Time           : 2017:05:20 17:57:44+02:00
[System]        File Creation Date/Time         : 2017:05:20 17:57:44+02:00
[System]        File Permissions                : rw-rw-rw-
[File]          File Type                       : MRW
[File]          File Type Extension             : mrw
[File]          MIME Type                       : image/x-minolta-mrw
[MinoltaRaw]    Firmware ID                     : 21810002
[MinoltaRaw]    Sensor Height                   : 2008
[MinoltaRaw]    Sensor Width                    : 3016
[MinoltaRaw]    Image Height                    : 2000
[MinoltaRaw]    Image Width                     : 3008
[MinoltaRaw]    Raw Depth                       : 12
[MinoltaRaw]    Bit Depth                       : 12
[MinoltaRaw]    Storage Method                  : Linear
[MinoltaRaw]    Bayer Pattern                   : RGGB
[MinoltaRaw]    WB Scale                        : 2 2 2 2
[MinoltaRaw]    WB RGGB Levels                  : 417 256 256 441
[MinoltaRaw]    Saturation                      : 0
[MinoltaRaw]    Contrast                        : 0
[MinoltaRaw]    Sharpness                       : 0
[MinoltaRaw]    WB Mode                         : Auto (0)
[MinoltaRaw]    Program Mode                    : None
[MinoltaRaw]    ISO Setting                     : 100
[MinoltaRaw]    Color Mode                      : Embed Adobe RGB
[MinoltaRaw]    Color Filter                    : 0
[MinoltaRaw]    BW Filter                       : 0
[MinoltaRaw]    Zone Matching                   : ISO Setting Used
[MinoltaRaw]    Hue                             : 0
[MinoltaRaw]    Color Temperature               : 0
[File]          Exif Byte Order                 : Big-endian (Motorola, MM)
[IFD0]          Image Width                     : 3008
[IFD0]          Image Height                    : 2000
[IFD0]          Compression                     : Uncompressed
[IFD0]          Image Description               : KONICA MINOLTA DIGITAL CAMERA
[IFD0]          Make                            : KONICA MINOLTA
[IFD0]          Camera Model Name               : DYNAX 7D
[IFD0]          Orientation                     : Horizontal (normal)
[IFD0]          X Resolution                    : 72
[IFD0]          Y Resolution                    : 72
[IFD0]          Resolution Unit                 : inches
[IFD0]          Software                        : DYNAX 7D v1.10
[IFD0]          Modify Date                     : 2006:03:13 16:05:46
[ExifIFD]       Exposure Time                   : 1/1600
[ExifIFD]       F Number                        : 4.5
[ExifIFD]       Exposure Program                : Aperture-priority AE
[ExifIFD]       ISO                             : 100
[ExifIFD]       Exif Version                    : 0221
[ExifIFD]       Date/Time Original              : 2006:03:13 16:05:46
[ExifIFD]       Create Date                     : 2006:03:13 16:05:46
[ExifIFD]       Components Configuration        : Y, Cb, Cr, -
[ExifIFD]       Brightness Value                : 10
[ExifIFD]       Exposure Compensation           : 0
[ExifIFD]       Max Aperture Value              : 4.0
[ExifIFD]       Metering Mode                   : Spot
[ExifIFD]       Light Source                    : Unknown
[ExifIFD]       Flash                           : Off, Did not fire
[ExifIFD]       Focal Length                    : 300.0 mm
[ExifIFD]       Subject Area                    : 1504 1000 256 304
[Minolta]       Maker Note Version              : MLT0
[Minolta]       Exposure Mode                   : Aperture Priority
[Minolta]       Minolta Image Size              : Large
[Minolta]       Minolta Quality                 : RAW
[Minolta]       White Balance                   : Auto
[Minolta]       Focus Mode                      : AF-S
[Minolta]       AF Points                       : Center
[Minolta]       Flash                           : Off
[Minolta]       Flash Mode                      : Normal
[Minolta]       ISO Setting                     : 100
[Minolta]       Exposure Compensation           : 0
[Minolta]       Color Space                     : Adobe RGB
[Minolta]       Sharpness                       : 0
[Minolta]       Contrast                        : 0
[Minolta]       Saturation                      : 0
[Minolta]       Free Memory Card Images         : 215
[Minolta]       Color Temperature               : 0
[Minolta]       Hue Adjustment                  : 0
[Minolta]       Rotation                        : Horizontal (normal)
[Minolta]       F Number                        : 4.6
[Minolta]       Exposure Time                   : 1/1579
[Minolta]       Free Memory Card Images         : 215
[Minolta]       Image Number                    : 6
[Minolta]       Noise Reduction                 : Unknown (2)
[Minolta]       Image Number 2                  : 1
[Minolta]       Image Stabilization             : On
[Minolta]       Zone Matching On                : Off
[Minolta]       Compressed Image Size           : 0
[Minolta]       Preview Image Start             : 65399
[Minolta]       Preview Image Length            : 30531
[Minolta]       Scene Mode                      : Standard
[Minolta]       Color Mode                      : Adobe RGB
[Minolta]       Minolta Quality                 : Raw
[Minolta]       Flash Exposure Compensation     : 0
[Minolta]       Teleconverter                   : None
[Minolta]       Image Stabilization             : On
[Minolta]       Zone Matching                   : ISO Setting Used
[Minolta]       Color Temperature               : 0
[Minolta]       Lens Type                       : Minolta AF 300mm F4 HS-APO G
[ExifIFD]       User Comment                    :
[ExifIFD]       Flashpix Version                : 0100
[ExifIFD]       Color Space                     : Uncalibrated
[ExifIFD]       Exif Image Width                : 3008
[ExifIFD]       Exif Image Height               : 2000
[InteropIFD]    Interoperability Index          : R98 - DCF basic file (sRGB)
[InteropIFD]    Interoperability Version        : 0100
[ExifIFD]       Custom Rendered                 : Normal
[ExifIFD]       Exposure Mode                   : Auto
[ExifIFD]       White Balance                   : Auto
[ExifIFD]       Digital Zoom Ratio              : 0
[ExifIFD]       Focal Length In 35mm Format     : 450 mm
[ExifIFD]       Scene Capture Type              : Standard
[ExifIFD]       Gain Control                    : None
[ExifIFD]       Contrast                        : Normal
[ExifIFD]       Saturation                      : Normal
[ExifIFD]       Sharpness                       : Normal
[PrintIM]       PrintIM Version                 : 0300
[Composite]     Aperture                        : 4.5
[Composite]     Blue Balance                    : 1.722656
[Composite]     Image Size                      : 3008x2000
[Composite]     Lens ID                         : Minolta AF 300mm F4 HS-APO G
[Composite]     Megapixels                      : 6.0
[Minolta]       Preview Image                   : (Binary data 30531 bytes, use -b option to extract)
[Composite]     Red Balance                     : 1.628906
[Composite]     Scale Factor To 35 mm Equivalent: 1.5
[Composite]     Shutter Speed                   : 1/1600
[Composite]     Circle Of Confusion             : 0.020 mm
[Composite]     Field Of View                   : 4.6 deg
[Composite]     Focal Length                    : 300.0 mm (35 mm equivalent: 450.0 mm)
[Composite]     Hyperfocal Distance             : 998.46 m
[Composite]     Light Value                     : 15.0

[RobiWan]

After first write metadata is Copyright owner and copyrighted field empty (my screenshot above).
I can now set both again - and now will IMatch write it into xmp files. BUT - if I now delete the xmp file and tell IMatch write metadata again - all metadata will be deleted in the database. What is that behavior again?

[Mario]

Best upload a sample file somewhere (cloud space) or send it to my support email address.
Such obscure cases are easier analyzed with a sample file I can look at. I have about two dozen MRW files in my test suite and metadata updates for these files work without problems.

[RobiWan]

Any answer or Idea what happens?

[Mario]

Did you provide a sample file as requested?

[RobiWan]

Of course - sample with video link (18.12)

[Mario]

Where? How? Did you send me an email? I see no unanswered emails from yesterday in my system...

[RobiWan]

Not yesterday - 2 days ago.
I forwarded the mail again

Robert

[Mario]

I replied to that email already, explaining how IMatch/ExifTool produce and write XMP metadata, how the XMP is produced from the EXIF/IPTC/GPS data contained in the original image file.
And why it is a bad idea to delete the XMP sidecar file by hand. Deleting the XMP file manually is similar to wiping out your XMP metadata.

I've just made a test:

1. New folder, one RAW file. No XMP data. No legacy IPTC, just GPS and EXIF in the RAW.
2. Add the folder to IMatch. Add a rating, title, description.
3. Write-back. IMatch creates an XMP sidecar file with a full data-set (5KB data).

4. I delete the XMP file in Windows Explorer. Windows Explorer sends a "file system modified event to IMatch".
5. IMatch rescans the folder and finds that the XMP file for the RAW no longer exists. IMatch now re-imports the RAW, re-creating an XMP record in the database from scratch.
This is the intended behavior.
6. A new XMP file will be created when you write-back.

[RobiWan]

This is the intended behavior.
6. A new XMP file will be created when you write-back.

Maybe you thought so just it just does not work that way.
In settings you can disable background update (what I did)

After deleting the XMP file and rewriting, Imatch had to write the entire file with the contents from the database. And it just does not do that - and this is not a feature.

[Mario]

IMatch does not "have to write the entire file". IMatch does not know that there once was an XMP file and now you have deleted it.
When IMatch imports a file, it sets all tags in the database as "current". Nothing to write-back.
When you change tag data, these tags are marked as "updated". During write-back, IMatch instructs ExifTool to write these modified tags. Then to migrate between EXIF/IPTC/GPS (MWG rules).
IMatch never writes out all tags that are in the database.

When you now delete the XMP file in Windows Explorer, IMatch neither knows that an XMP file existed once, or that you once had made changes to the XMP in the database. When it has to re-import the file again, for what ever reason, it will start over with the fresh XMP data created by ExifTool from the original data in the image file.

Just don't delete the XMP sidecar file.

[RobiWan]

Nothing to write-back.

But you can say "write XMP Data for selected file" - and this does not work.
Do not delete is not a solution. If you absolutely need both the database and XMP data, the database is superfluous

I do not backup XMP files because the database is there so that you can always restore them if, for example, the disk fails. If Disk crashes with the database, then you can restore the database from the XMP files. Keeping both is nonsense

[Mario]

can say "write XMP Data for selected file" - and this does not work.

You can do this where?

If Disk crashes with the database, then you can restore the database from the XMP files. Keeping both is nonsense

XMP lives inside the image file, or the sidecar file. The IMatch database only serves as a cache, it is not a backup solution for your metadata.

[RobiWan]

You can do this where?

it is not a backup solution for your metadata.

I thought IMatch should be a DAM system

[Mario]

This is the normal write-back which writes all modified metadata tags to the image and/or the sidecar file, then performs the MWG mapping.

IMatch here does what I have described above. This cannot recover an XMP file you have deleted.
If you have deleted the XMP metadata, this will just re-create a fresh XMP record from the existing metadata in your file, then re-import the fresh metadata in your file.
This is not a "Create new EXIF/IPTC/GPS/XMP metadata from the metadata cached in the database". IMatch has no function to do that.

If you insist on deleting the XMP metadata for your files manually, you probably need to find another software. This actively works against how the metadata caching and update mechanism in IMatch is designed to work.

[RobiWan]

If you insist on deleting the XMP metadata for your files manually, you probably need to find another software.

Yes OK. Was a bad buy 
Is not software that I want to give my data


By the way - how I can remove completely IMatch from my computers (and I mean completely - not the most data!)

[Mario]

You can remove IMatch from the Windows Control Panel. Also remove C:\ProgramData\photools.com\IMatch6 and your database.

[RobiWan]

Has IMatch no entries in the registry?

[Mario]

Only a few:

Computer\HKEY_CURRENT_USER\Software\photools\IMatch\6

And you may clean your TEMP folder, because EXIF Tool and IMatch create files there.

If you want me to remove your customer record as well, send me an email to support email address.


Or, you just never write back your data. Then IMatch remembers all your changes in the database and keeps them in the pending status.

[Arthur]

Interesting discussion. Did also not realize that if I disable auto sync data base (cache) metadata from disc after initial import, that there is no way to write a XMP file with all available cached data once it was written back one time. So if someone eats my XMP file on disc, I cannot recreate (cached parts of) it from the data base. Here an on disc XMP backup gets really important.

The main "problem" seems to be, that the diff between cached XMP and on disc XMP is stored in the database, instead of being evaluated by comparing the two records on write back. This makes the XMPs to some sort of a "database sidecar files" instead of an alternative representation of the cached metadata.

Wouldn't it be cool when there would be a "Mark all metadata as pending" command for a selection or the whole data base, so as if the user has clicked on all yellow pens at once?

This would allow to keep things as they are, when this command is not used. At the same time it would enable the possibility to work with IMatch in a data base centric way, as it is the case with Lightroom. Lightroom never auto syncs metadata from disc after import. At the same time the user can write out everything what is cached, when he wants it. The command would be basically something like:

foreach file in selection/database
    foreach metadatum in metadata[file.id]
        metadatum.Touch()

Just an idea, which does not seem to cost much, and which would met the expectations of some. Then you could write something like: "Look, Lightroom can only write everything at once. IMatch can this also, but it gives you the additional possibility to write only the changed metadata, which improves performance and preserves the uncached metadata in existing files as they are." Something like that. 

[Mario]

The initial XMP record in the database is created from the XMP data in the sidecar file. Or a fresh XMP record is created out of thin air, by importing EXIF/IPTC/GPS and whatnot from the original image file (ExifTool does this). When you change metadata, IMatch only records the tags that were changed. And writes these back. At this time, the XMP record on disk and in the database is identical.
To recreate it after you have deleted/lost it, ExifTool needs to create the XMP record again from the original data (IMatch does not know how to do this - there is severe ExifTool magic at work in that phase). Now IMatch would have to keep track of all tags you have ever changed for that file (persistent even over a force reload) and then write all these tags out again via ExifTool.

This is doable but may be challenging and I really don't see a need.

You can always mark the tags you care for in the Metadata Panel as "updated" (click on the pen in front of the tag value). Works for all selected files.
Then write back.
This will either update the existing XMP record with the (marked as) modified tags or create a new XMP file from scratch by importing everything and then updating the result.

I consider the XMP data record stored in the image or the sidecar file as important as the image file itself. You should too, and include it in your backup.

[Arthur]

Ah, I see, you mean that when the user writes back, the changes since the last write back are lost. Like a cleaned redo stack after saving a text file to disc.
And even if the metadata tags are cached in the data base, you would need to create a diff to the initial record to update these values only.

[Mario]

Yes. After writing all modified tags, IMatch re-imports the XMP into the database. This is required because even writing just one tag may update several other tags, IPTC and XMP digest data blocks, timestamps etc. IMatch then considers the XMP dat as 'current'. It keeps no history all all changes done between write-backs or anything.

[Nordlicht]

If the XMP file gets lost, I can not create a new one from within the content of the database with iMatch?

Sorry, that is strange.This is bad and all competitors can do that.They consider the XMP file as a backup of the metadata that can be created from the program.

What do you do if you want to move with the pictures in another DAM at some point? Start all over?

[Mario]

IMatch of course will re-create missing XMP files from the original data embedded in your image.
But IMatch considers the XMP file (or the XMP record embedded in the image file itself) together with the EXIF, IPTC, GPS data as the core source for metadata. The IMatch database merely caches this data.
This means that if you delete the XMP data intentionally you may loose some of the edits you did if IMatch did write the data and hence considers the XMP record in the database to be an 1:1 copy of the metadata contained in your image and/or the sidecar file.

If you are in the habit of losing XMP sidecar files I can only highly recommend that you include them in your daily backup.

What do you do if you want to move with the pictures in another DAM at some point? Start all over?

Moving an image with 'another DAM' (why use several DAM's?) should not delete the XMP data record in the image or the XMP data record in the sidecar file. This is precious metadata and should not be deleted by accident or some sort of feature in your other DAM.

If you want a feature in IMatch which produces a 'fresh' XMP file not only from the metadata contained in your images but also in all/some of the XMP metadata contained in your IMatch database, feel free to add a feature request in the feature request board,

Please include details about how you would like to control what is written (because the data cached in the in-database XMP may differ from the data contained in the actual image and thus you would need to decide what to retain and what to overwrite). The largest part of the XMP data record cached by IMatch is produced from the EXIF/IPTC/GPS in the original image anyway (and thus is always re-created when the XMP goes missing. You only need to consider metadata that is exclusive for XMP.

If this is a popular feature I wll consider implementing it for a version next year.

[Nordlicht]

Perhaps there is a mistake on my side.

I've imported 2 RAW-Files. Then I set Geo-Tagging, label and rating. I write the metadata. A XMP-File was created.

In the next step I delete the XMP-Files and rewrite them from iMatch. In iMatch the Geo-Tag, label and rating are gone. Not XMP-Files was created.

[Mario]

Why do you deliberately delete the precious XMP file?

How do you rewrite them then?
Unless you modify XMP data, IMatch does not create XMP data in your image in that situation.
You are trying to trick IMatch and IMatch does not have features to prevent you from doing that.
Do a forced rescan and a write-back to re-create the XMP file you have deleted. The data you had previous added to XMP will of course be missing.

[Nordlicht]

I'm thinking about working with iMatch in the future. After reading the problems of RobiWan, I try to understand them. I want to see for myself if I could run into problems if the XMP files get lost.

When I work with iMatch, I do not want to block the way to another DAM in a few years. So it's important to me that I can export all the metadata in case of a case.

About the ExifTool command processor, the creation of XMP files should always be possible in any case or?

[Mario]

IMatch will create an XMP file if there is none (or an embedded XMP record) when you write-back.
Just click the pen in the metadata tags for the file in the Metadata Panel to mark the unchanged values you want to include. Then write-back.

You should really consider the XMP data in the sidecar file as important as the image itself. Always back it up with the image.
If you switch the DAM. If you lose your IMatch database. If you switch platforms. You may need it.

So far there was never a need for a "recover XMP files I have deleted from the database" or "I have used the wrong software and my XMP data is gone"...
If this happens often to you and you want such a feature in IMatch, please add a feature request in the FR board. We then see how many users would like to see this and then I can make a plan if and how to implement it.

[Nordlicht]

Thanks Mario, I've made serveral tests. It seems to work for me.

[Nordlicht]

Today I have done some tests and can understand and understand the problem of Robert. iMatch takes a different approach than competitors. The competitors see the XMP file as a backup of the metadata and store it completely in the database. And write this completely or as a synchronization back into the XMP file. Or create a new one with all the metadata from the database.

Even if you should be able to back up the XMP files with a backup, the handling of iMatch can cause problems.

The XMP files of e.g. 1000 pictures are broken or lost. I do not realize that immediately. Updated in the background, iMatch notes that there is no XMP file and reads the existing data in the image file into the database.

As a result, then all the added metadata are gone. That would not happen with  the other DAMs; tested with Daminion, Photo Supreme, Media Pro and (LR).

At least you have to be aware of how iMatch works and realize that the database is "not a safe place" to store the metadata. Not if accidentally changed or missing XMP files change the database.

If you are aware of it, you can adapt to it. Personally, I do not think that's ideal. Although probably the XMP files are not lost. Mam needs to know that you have to take care of her. You do not have to do that with other tools. I've already deleted the XMP files generated by LR more often. I can always produce them again.

[Erik]

I think there is something lost in a lot of the discussion going on here, although Mario has mentioned it...

Backup, Backup, Backup:

1. Backup your images
2. If your images have external XMP files (back those up -- should be included with the images)
a. side note that with external XMP files, backups are actually quick and easy as you will never touch the images, just the xmp files that are small.

3. Back up your IMatch database.

Aside from that, you need to get around the philosophy of a program, and of course recognize if it isn't going to work for you.  IMatch is designed to make it easy for users to change to other DAM software with a lot of flexibility.  How is that? because it functions where the XMP record is the primary record and the database is essentially a backup of the metadata.  Note that is what makes it a DAM.  So, if you want to change software, you can move your images and their metadata, and another software will have access to all the XMP records. 

Aside from that, part of managing this data and using a DAM without problems is to not do management outside the software.  When you delete an XMP file, you've deleted the XMP record.  If any software were to re-scan a file with a deleted XMP record it would replace the XMP record with a blank one.  However, without a rescan, the software will not no the XMP record is missing; this even goes for IMatch.  Note that no DAM software keeps a perfect copy of the XMP record within its database.  You might think it does because it keeps track of the data that is important to you, but that's rarely the case.  Most DAM programs are simple and don't necessarily keep track of proprietary XMP information.  I never even began using LR for DAM because it lacked any flexibility with regard to XMP data and the fields it kept were limited and didn't allow me to easily use other XMP fields.

As for IMatch.  If your goal is to operate in an environment similar to the other DAMs mentioned, I think it is as easy as not writing all pending metadata changes so that data never gets written and thus re-read from the files. You also want to watch for rescans IF you have deleted XMP files.  But, you probably shouldn't delete XMP files.

Finally, if you have made your backups and you've ever had a problem, you can restore all the XMP files, force a rescan in IMatch, and IMatch will pick up the XMP data that has now been restored.  Of course if you haven't written the XMP information, then that won't necessarily work.  But, if you have backed up your database, you can also recover an older version of that to get that data back.

In reality, you need to do that anyway.  You are in the same boat if you are relying on the database file in any other DAM.  If you ever lost the file or it went corrupt there goes all your metadata.  Thank goodness for hard drives and backups. 

[RobiWan]

You are in the same boat if you are relying on the database file in any other DAM.  If you ever lost the file or it went corrupt there goes all your metadata.  Thank goodness for hard drives and backups.

Wrong. With DAM Software I can always restore my XMP files from database or if needed my database from existing XMP files. Its never necessary to have a backup of XMP AND database. That is a big difference.
You may like to have a different view on it, but please tell nobody that it is the only one and right way..

[Mario]

And write this completely or as a synchronization back into the XMP file. Or create a new one with all the metadata from the database.

IMatch does not necessarily import all XMP data in your XMP file (or the embedded XMP record in your image). This is controlled by the Tag Manager.
Many XMP records these days contain proprietary name spaces of applications, often with massive amounts of data. One example are Adobe applications which store hundreds or sometimes thousands of metadata tags in the XMP record (processing instructions, history, sometimes even every brush stroke!).

IMatch by default does not import this metadata, It is of no use for the user and it would increase the database size considerably - and reduce performance for all operations dealing with metadata. IMatch currently manages between 200 and 500 tags (!) for each file. And many users have now 100,000 or 200,000 or even 500,000 files in their databases.

IMatch hence cannot recreate the entire XMP record if you intentionally delete the XMP file or you lose it without having a backup. In such disaster scenarios IMatch can IMatch can re-create an XMP record from the existing metadata in the image. And it also can write again the data cached in the database (controlled by what you mark as 'to write').

If losing XMP files or deleting them is a habit for you, I recommend you either improve your backup method or you switch to an application which always caches the entire XMP record and can write it back. So far I have no plans to add such a feature to IMatch.

There is no feature request yet and hence I don't know how many users have this problem.
I could implement such a feature but it would require some changes deep in the core. For example, IMatch would always have to keep a copy of the entire XMP record around. Including all the data it usually does not import. This is doable but would increase the database size considerably - think about databases managing 200,000 or 500,000 files...

Unless you did already, please add a feature request so we can see if and for how many users this is a problem. Then I can think about it.

[RobiWan]

It would be appropriate to answer the mails I had sent to support.

[Mario]

It would be appropriate to answer the mails I had sent to support.

When did you send me emails? I have currently about 40 emails to answer in my inbox, If you did sent me emails yesterday or today, please allow for 24 hours at least.

[Mario]

I have just replied to your email question "Why does IMatch update RAW files". Please see the IMatch help topic I mention there, which explains it all.

[sinus]

You are in the same boat if you are relying on the database file in any other DAM.  If you ever lost the file or it went corrupt there goes all your metadata.  Thank goodness for hard drives and backups.

Wrong. With DAM Software I can always restore my XMP files from database or if needed my database from existing XMP files. Its never necessary to have a backup of XMP AND database. That is a big difference.
You may like to have a different view on it, but please tell nobody that it is the only one and right way..

Surely this statement of RobiWan is wrong. Please do not tell for others, that this is the only one and right way!
There might be other DAM-software with this behaviour, but for sure not all.

[RobiWan]

Please do not tell for others, that this is the only one and right way!

I do not do that. Some people always try to make it clear to me that backup, backup, backup of XMP files is just as important as pictures. That may be the case with IMatch. I have accepted that the developer has gone this way here. But if I write that it is not correct in my opinion and just in other programs it is implemented differently, then the same people have to accept it too. No more.

When did you send me emails? I have currently about 40 emails to answer in my inbox, If you did sent me emails yesterday or today, please allow for 24 hours at least.

Mario, after I created this thread I have no any answer to my Mails.  I sent you an email today. But on my mail from 23.12 I have also received no answer.
We can have different opinions, but answers to questions are a good thing.

[Mario]

We can have different opinions, but answers to questions are a good thing.

I manage between 30 and 50 emails per day and I'm quite good at it, I can assure you. Remember, this community is a friendly place.

I've looked again at your email from the 23. and it does not ask a direct question.
Just asks me to change my ways and how IMatch works. I did not see a reason to reply to this. All I had to say I said in the reply to your community post.

I've explained why IMatch works the way it works, for the past 4 years.
I've explained that you are free to post a feature request. This will tell us quickly if you are trying to solve a problem that only affects you, or if this addition would be useful for many users.
Can't do more at this time, I'm afraid.
The only solution for your problem would require IMatch to make backups of your XMP files. And there is backup software out there which does this efficiently.

As I always say: Whatever works for you is good.

When you prefer how other software deals with XMP files you have deleted, use it, by all means.
Nobody forces you to work with IMatch. There are many DAM products out there to choose from. I'm sure you'll find one that does everything you want in exactly the way you want it.

[Erik]

You are in the same boat if you are relying on the database file in any other DAM.  If you ever lost the file or it went corrupt there goes all your metadata.  Thank goodness for hard drives and backups.

Wrong. With DAM Software I can always restore my XMP files from database or if needed my database from existing XMP files. Its never necessary to have a backup of XMP AND database. That is a big difference.
You may like to have a different view on it, but please tell nobody that it is the only one and right way..

Except you can't restore your XMP files, not exactly as they originally were.  No DAM (that I am aware of) stores ALL the XMP data.  XMP has been important, and I won't say that I haven't looked into replacing IMatch in the past (there was a long gap between IM3.x and IM5).  For instance, I know LR doesn't come close.  Adobe has little regard for all the original data. Convert a raw file to a DNG and use ExifTool to compare the XMP records from the original raw file and the derived XMP file.  They won't be similar, and that is pretty much how LR works with regard to metadata.  LR is probably at the low end of the DAM capabilities.

As far as backups go, everyone should really backup everything, especially databases.  It isn't a matter of XMP, it is just the fact that databases can become corrupt.  All it takes is a computer crash, power outage, other accident, etc. to corrupt a database.  Even with DAM software, they store more than just XMP data; they store other data, settings, etc. that may be only in a database.  If you lose that database you will lose all the data inside, not just the XMP data.   

Obviously it is your own computer and files, you CAN do what you want. That is the beauty of choice.  You can choose to use different software.  You can even choose to write your own as Mario did.  But you do need to make sure before you pick any software that it does what you need it to do, otherwise you try something else out.  I've also learned that no software is perfect, so you find ones that are close to it and you adjust your own workflow to fit the software's limitations.  And, as users we have to be aware (even if we don't agree with it) of what typical workflows look like and where we may differ from the "norm".

[RobiWan]

@Erik,

I agree 100% with you with one exception.

When I take a picture and import it into my computer - there is no any XMP file. Sure Metadata exist in my RAW Image but this is not the point.
If I also import a new Image into DAM system like IMatch, then first I create some new metadata that have ben never exist before (Email, Phone, Keywording and so on). All this data is stored in DAM database because I can create all this even if my image is offline.


Of course I have backup from my database. It was just that I expect that a DAM system the data I have entered into the database can always be restored.If the developer does not want it, it's like that. And if the users agree with, everything is OK. But one may have a different opinion, especially since, in my understanding, a DAM system is the central point and not just any extension. I also do not delete XMP files. In the 15 years I use digital cameras I already had many programs that write broke XMP data. And such programs will always exist. Mistakes can easily happen. Sometimes you notice them immediately, but sometimes you do not notice them either. And then maybe it's too late, because you do not even know when the mistake happened. No backup helps you.

Maybe my tone was a bit too hard. But I do not always have to hear that I'm better search for backup software instead of looking for a missing feature in IMatch.

[Mario]

Of course I have backup from my database. It was just that I expect that a DAM system the data I have entered into the database can always be restored

There is no need to 'restore' anything. As long as your database exists your data is accessible. You are mixing different concepts here, jumping from XMP files to metadata contained in your database etc.
As I explained in detail above, IMatch does not import all XMP data that may be contained in your files - it does not need to. IMatch also manages tons of data that cannot be written to XMP. And normally users don't manually delete their XMP files. They consider them as important as the image file.

As I said, whatever works for you is good. Look at other DAM software and choose whatever suits you.

[Nordlicht]

When I take a picture and import it into my computer - there is no any XMP file. Sure Metadata exist in my RAW Image but this is not the point.
If I also import a new Image into DAM system like IMatch, then first I create some new metadata that have ben never exist before (Email, Phone, Keywording and so on). All this data is stored in DAM database because I can create all this even if my image is offline.

I think that's the sticking point. If iMatch were able to store the metadata collected directly in the database and write it back to the XMP as needed, then everything would be fine.

Most of the time you do not really notice right away that something is going wrong. And when iMatch writes the metadata from scratch to the database, the data collected in the database are gone.

[Mario]

I think that's the sticking point. If iMatch were able to store the metadata collected directly in the database and write it back to the XMP as needed, then everything would be fine.
Most of the time you do not really notice right away that something is going wrong. And when iMatch writes the metadata from scratch to the database, the data collected in the database are gone.

No. And no.
IMatch stores the metadata collected directly in the database. It just does not import the entire XMP data or other metadata which may be contained in your file ("Tag Manager"). And it has no way to re-create an XMP file created by Lr other other RAW software - because it does not cache all proprietary data that may be in your XMP file. Or in the maker notes.

IMatch produces XMP on import and when you write back it updates the existing XMP data (or creates it from scratch). Then it only manages the "differences" (the changes you make to your data) and writes that. This is a proven workflow that apparently worked flawlessly since 2014.

Data in the database cannot be "gone". Only when you manually delete the XMP file (or you lose it without backup) and then IMatch is forced to re-import the file, the metadata will again represent the current state and metadata of the file.

It all boils down to this: Don't lose your images or XMP sidecar files. Always make backups and keep them for months (in case you recognize a problem several weeks later). IMatch is not designed to import all metadata other applications may write. It can, but does not for performance and disk space reasons. If you lose a XMP file written by Lr, only Lr can re-create it (probably). But IMatch can not, because it does not import all proprietary Lr settings. The same is true for other RAW processors which create their custom XMP files.

IMatch caches metadata in the database. it maintains metadata in your files, where it belongs.
It is not a backup tool.

Feel free to post a feature request for a metadata backup facility.

[RobiWan]

IMatch is not designed to import all metadata other applications may write. It can, but does not for performance and disk space reasons. If you lose a XMP file written by Lr, only Lr can re-create it (probably). But IMatch can not, because it does not import all proprietary Lr settings.

Why are you talking all the time about IMatch not being able to recreate XMP which it did not have created?
That's not the point. The point is that when I put a keyword "Hugo" in IMatch and because I am so stupid or other software is so bad and has deleted this keyword or file, I'm able to just write it again. I don't want that IMatch read/ understand/ write data created by others.

And leave please it with the hint that I should use other software. I have paid for IMatch and if such things are not documented that is not ok. Even if I do not agree with the procedure, I try to understand what motivation there was and look for a workable solution for me.
It would help us both if you also tried to understand my reasoning.

[Mario]

Just make keywords as modified then and write-back. IMatch will write all keywords to the XMP.

[RobiWan]

Just make keywords as modified

Is it possible to mark all data entered in the IMatch database (or all data that IMatch database store for a image) as modified?