Informations in a file, what is quick and safe?

[sinus]

Hi,
I tried to search, but I have not really found some good informations.

If we have a lot of images, say, 200.000, and want see some informations, what are slow and what are fast? And safe?

1) What about "Data driven categories" versus "formula cats"? Say, like it is written in the help-file, I want create some categories, with parts of informations, what are in the filename, like clients, format and so on.
In relation to the speed, what categories are quicker, "Data-driven" or "formulas"?

2) If I want write some "Infos" in a file (if necessary with a sidecar, in my case Nef), I have the impression, that Stars and Labels are not that safe as the collections of IMatch (Categories, Dots, Pins, Flags, Bookmarks), because Stars and Labels could be a problem with other programs (like Lightroom). Is my impression correct or wrong?

3) Attributes in IM5 are very quickly and quite safe, because they have nothing to do with other programs and will not be written into a file. Correct?

I ask this, because it makes sense for me, to know such things before deciding a final workflow.
Maybe somewhere knows this or has experiences.
Thanks for comments.

[Mario]

1)    Data-driven categories are calculated once and then remain unchanged until you add or update or remove files.
Formula-based categories are re-calculated every time you make a change to your database which may cause the result to change. Assigning images, renaming files, working with metadata, collections, categories etc. all invalidates formula-based categories. If you have many of them (several dozens) you can slow down IMatch.

2)    Rating (Starts) and Label are stored in the IMatch database and written to the XMP record int he image file or sidecar file. This allows you to share rating and label (and all other metadata) with other applications. Metadata changed in IMatch is visible in LR and Photoshop, and vice versa, ...

This is usually not a problem when you take care what you change were. And you have to keep in mind that XMP and other metadata (including rating and label) is only in the IMatch database until you have written it back to the file. You can do that manually when it fits, or let IMatch automatically write-back metadata by enabling this feature under Edit > Preferences > Background Processing.

Please note that Adobe applications write to the XMP metadata all the time and are not really co-operative with respect to non-Adobe products.

If you add a file to your IMatch database, IMatch reads all the metadata from the file into the database. When you change the rating of the file to 3 in IMatch, this data is first stored in the database. The XMP metadata in the file (or sidecar file) is not updated immediately. When you look at the file in LR or Photoshop or Bridge, you will not see that this file has been rated 3 in IMatch yet.

When you click on the Pen icon (or use the Process all pending write-backs) IMatch updates the XMP rating in the file or sidecar. Your Adobe application will (at some point in time) recognize that the XMP metadata has changed, and re-import the metadata. Then you will see the 3 star rating there as well.

When you change XMP data in the Adobe application, it will (at some point in time) write the modified data back to the XMP record in the file, maybe also update the IPTC, EXIF and GPS metadata. IMatch recognizes that the file or sidecar has changed and re-import the file and metadata. The changes done in the Adobe application will be visible in IMatch.

Now the problem case: You modify XMP data in IMatch but you don't write it back.
You open the same file in your Adobe application and work with it. Adobe updates the XMP record in the file. Since you did not write your changes back to the file in IMatch, the Adobe application will be unaware of the changes you did in IMatch.

You switch back to IMatch. IMatch has to re-import the XMP data in order to see the changes you did in your Adobe application. But IMatch has unwritten metadata changes. Re-importing the XMP data will delete these changes!

Only you user can handle this situation, and avoid it. You can set the options in IMatch to protect unwritten metadata (Edit > Preferences > Metadata 2). You have to take care that all pending metadata is written in IMatch before you open your image files in an Adobe application. Only then you'll have a guarantee that changes made in either application are visible in all applications.

If you regularly work on the same files in multiple applications, enable background write back in IMatch. This ensures that changes you make to your metadata is written as soon as possible to the files. This will slow down things a bit because writing XMP can be slow, especially when you work with large files.

Which is the reason why IMatch uses the database caching and delayed write-back mode of operation. Setting a rating and a headline in the database takes 0.1 seconds. Writing that data to an image file, doing the migration between different metadata standards etc. can take several seconds. More when your files are on a remote server or NAS.

See also the corresponding "How IMatch writes Metadata" in the help. Search in the index for write-back.

3)    Yes. IMatch stores all Attributes inside your database. No other application can reach this data (unless you export it). If you don't need (or want) to store information inside the image file itself (for privacy reasons or to protect the data from accidental damage) using IMatch Attributes is a good alternative. When you use Attributes you don't have to take care for all the metadata mess and the problems which may arise from using multiple software which modifies the same metadata.

[Ferdinand]

Now the problem case: You modify XMP data in IMatch but you don't write it back.
You open the same file in your Adobe application and work with it. Adobe updates the XMP record in the file. Since you did not write your changes back to the file in IMatch, the Adobe application will be unaware of the changes you did in IMatch.

You switch back to IMatch. IMatch has to re-import the XMP data in order to see the changes you did in your Adobe application. But IMatch has unwritten metadata changes. Re-importing the XMP data will delete these changes!

Only you user can handle this situation, and avoid it. You can set the options in IMatch to protect unwritten metadata (Edit > Preferences > Metadata 2).

Which is what I was warning you about Markus in my earlier posts about sidecars and workflow.  IMHO the best way to avoid the problem Mario describes is to have metdata writeback enabled in IMatch, and to configure Lightroom catalog options to always write settings to XMP.  This way there is never unwritten settings or metadata.  And also to use only one program at a time.

[Mario]

I know, Ferdinand.

I sometimes write up stuff just for helping the search engine, when I think that topics will be found often later, when more users use the community. Never a bad idea to repeat important info in different words.

As a side effect, I often get help feedback emails like 'The way you explained in this community post is better than what you did in the help. Please incorporate your post somehow in the help'. This is actually a very welcome side effect. When I wrote the initial version of the help, I did it into the blue sky, assuming what a user would need to know about a specific feature. As expected (hoped), the questions here on the community and the feedback emails I get help me to shape the help, to make it better and to incorporate questions asked in form of new examples or ! bars.

As you say, when you work with other XMP supporting applications (Adobe or other RAW processors etc.) it is important to setup a consistent workflow and to not cache metadata in either application. Only the metadata in the file can be seen and processed by IMatch 5 and the other applications.

@All Testers: I'm collecting configuration tips for typical applications. There is a separate section in the Metadata help which explains which settings work best for which apps. Send me feedback or post here which applications you are using and what your settings are. I will incorporate this into the help.

Or maybe post a FAQ tp this board where you explain which application you use and what your working settings are. Others can learn from you.

[sinus]

Thanks, Mario, for your detailed answer. Of course you are right, searching in this or earlier forums have been helped me a lot.

Your answer is very helpful, as usual. I do not think, that my workflow will be a danger for metadata, because I am very strictly in this. That I do write or change Metadata in IM5, not write it and then open this file in LR, is very unlikely. If I do: my fault.

But the speed and security is a issue for me. I have never had troubles with properties in IM3, hence I will trust the new attributes in IM5.

I will also trust categories, and the IM5-collections.

But I will be carefully with xmp-stuff, like stars and labeling.

I know from IM3, that formula-cats can be slow, hence your answer about data-driven cats are very interesting for me. I guess, I will go ahead with about 10-20 data-driven cats, and a dozen of formula-based cats. The rest are "real cats".

[sinus]

Which is what I was warning you about Markus in my earlier posts about sidecars and workflow.  IMHO the best way to avoid the problem Mario describes is to have metdata writeback enabled in IMatch, and to configure Lightroom catalog options to always write settings to XMP.  This way there is never unwritten settings or metadata.  And also to use only one program at a time.

Thanks, Ferdinand,
I will do so. But maybe I will not enable "writeback",  because it makes my workflow slow.
But I will let writing all the metadata, BEFORE I transfer images into LR or PS.
Because I have a strict workflow, I see no danger in doing this.

BTW: I had always a strict workflow, that is why I for example work in the categories with some numbers: first 1, then 2 and so on and all numbers MUST be edited or choosed. This reflects also the images, they MUST have 5 colour-coded cats under each thumb. On this way, I can see in a very short time, if something is wrong with an image, if a colour under a thumb is missing.
It "jumps" into my eyes. That is why I have also for example not only the word NEF under/above a thumb, but also a colour-coded cat.
Ahhhh, that brings me to an old feature request: I wanted two lines of colour-coded cats. The coloured label for example is VERY thick. This could be smaller. I would wish a second line for cats ABOVE each thumb (or as a second line under each thumb).
This way we could use one line for say things, like format, client and the second line for the progress of workflow.
Why? I think, more than 5-6 cats in one line is too much for remember and distinguish all the cats. A second line would be a good solution.
Hmmm, I will go ahead and put this in the feature-request-thread. 

[JohnZeman]

I will do so. But maybe I will not enable "writeback",  because it makes my workflow slow.
But I will let writing all the metadata, BEFORE I transfer images into LR or PS.
Because I have a strict workflow, I see no danger in doing this.

Hi Markus,

This is basically what I'm doing now.  Only I write the metadata back at the end of each IMatch session so I can leave it alone until the updating is done.  Before I settled on this routine I had automatic writeback enabled but I found the writing back speed (all my images are JPGs) can be anything from a few images per second, to over a minute per image.  Last night it took build 114 90 minutes to writeback the metadata to jpg 86 images (sorry Mario, my mind was on other matters at the time so I didn't think to save the log file). 

[sinus]

Hi Markus,

This is basically what I'm doing now.  Only I write the metadata back at the end of each IMatch session so I can leave it alone until the updating is done.  Before I settled on this routine I had automatic writeback enabled but I found the writing back speed (all my images are JPGs) can be anything from a few images per second, to over a minute per image.  Last night it took build 114 90 min utes to writeback the metadata to jpg 86 images (sorry Mario, my mind was on other matters at the time so I didn't think to save the log file). 

Well, great, John, so we are at least two! 

Seriously, I think, it is a good thing to do so. It is a kind of habituation to do so ... for example: after YOU work, let IM5 work and you enjoy a cup of coffee or tea ... hm, or do other interesting things! 

[JohnZeman]

m, or do other interesting things! 

That's what I keep suggesting to my wife but all she wants me to do then is to go fly a kite. lol 

[Mammut]

"so we are at least two"

Three.

I always turn off the automatic writeback in every program, it really slows down the programs.
After I'm done I write the data, usually at night while I'm sleeping, then I don't care how much time it takes.

[Kucera]

I I often get help feedback emails like 'The way you explained in this community post is better than what you did in the help. Please incorporate your post somehow in the help'.

@All Testers: I'm collecting configuration tips for typical applications. There is a separate section in the Metadata help which explains which settings work best for which apps.

"Applause!" Both the above is most welcome and of help to me in my struggles. I always felt that I am not utilizing  ver 3.5 to its potential, and now with ver 5 I am often overwhelmed, mainly because other pursuits prevent me to give it enough attention and because  I get some weird results on my main (xp) machine, whereas a win7 laptop works fine. Thus more of this kind of help is great! (AND I hope to get a new desktop soon for some proper testing). Please do keep up the good work, everybody!

[sinus]

m, or do other interesting things! 

That's what I keep suggesting to my wife but all she wants me to do then is to go fly a kite. lol 

   

[sinus]

After I'm done I write the data, usually at night while I'm sleeping, then I don't care how much time it takes.

Yep, exactly!

[Ferdinand]

This is basically what I'm doing now.  Only I write the metadata back at the end of each IMatch session so I can leave it alone until the updating is done.  Before I settled on this routine I had automatic writeback enabled but I found the writing back speed (all my images are JPGs) can be anything from a few images per second, to over a minute per image.

I understand.  In fact this is the main reason why I use Geosetter to input metadata rather than V3.6.  But the problem with flushing changes when you're finished is that you have to remember to do it.  Every time.  And for an Adobe workflow, this applies to both V5 and LR.  And you know what our memory is like at our ages .... 

[sinus]

This is basically what I'm doing now.  Only I write the metadata back at the end of each IMatch session so I can leave it alone until the updating is done.  Before I settled on this routine I had automatic writeback enabled but I found the writing back speed (all my images are JPGs) can be anything from a few images per second, to over a minute per image.

I understand.  In fact this is the main reason why I use Geosetter to input metadata rather than V3.6.  But the problem with flushing changes when you're finished is that you have to remember to do it.  Every time.  And for an Adobe workflow, this applies to both V5 and LR.  And you know what our memory is like at our ages .... 

Yes, Ferdinand, you are right: we have to remember. But you have also to remember to rating images, or to put some Metadata in it, you should remember make a copy, backup and so on. I think, such things should be in the responsibility of each user.

BUT: But MAYBE IM5 could add some new properties, for example:

x at end of IMatch (when you choose to end IM) display every time a warning window, want you this or not?

if yes, the warning could add some points to remember:
- want you writing all pending metadata?
- want you let run a backup of the database?
- want you let run a database diagnosis?

And so on. I think, IF IM5 would offer such a "remember-window", I would use this window. Hmmm, could be almost a feature request! 

[Ferdinand]

But you have also to remember to rating images, or to put some Metadata in it, you should remember make a copy, backup and so on. I think, such things should be in the responsibility of each user.

There's a difference between forgetting to do something which you can just as easily remember to do later, and forgetting to do something which could lead to data loss.

[Mario]

IMatch at least remembers which files have pending write-backs. So even if you forget to write-back, you can do it the next time you start IMatch.
The Pending Write-back collection always keeps track of which files have to be written.

IMatch cannot remind you about doing backups and stuff, though 

[sinus]

But you have also to remember to rating images, or to put some Metadata in it, you should remember make a copy, backup and so on. I think, such things should be in the responsibility of each user.

There's a difference between forgetting to do something which you can just as easily remember to do later, and forgetting to do something which could lead to data loss.

Yep, you are right of course. But anyway: I will not forget to writing pending Metadata, before I edit them in another program. If I touch them not in another program, IMatch is aware anyway of all pending Metadata.

I have, as I wrote somewhere, a really strict workflow with some kind of "steps to do". But we have to be careful, of course, your point is right.

[sinus]

IMatch cannot remind you about doing backups and stuff, though 

IMatch could - IMatch can almost everything (you remember my button for a coffee?)  ... but I understand you fully, if you want not add such things. At least IMatch is a DAM, isn't it!?

[Mario]

Maybe add a feature request like "IMatch should display a reminder when I close it and there are files with unwritten metadata". This will not be hard to implement and maybe some other users would want such a feature as well. I cannot tell.

[sinus]

Maybe add a feature request like "IMatch should display a reminder when I close it and there are files with unwritten metadata". This will not be hard to implement and maybe some other users would want such a feature as well. I cannot tell.

Yep, thanks for that, I have already written such a feature request.