LR and PS can not open images

[HansEverts]

Almost all the images in the same folder can no longer be opened by LR and PS. Other folders are fine. In LR they have a black exclamation icon with "LR has encountered problems reading this photo". The few images without exclamation mark can be opened normally. They are all NEF with an XMP sidecar and I can't see the difference.
I removed an image from the LR database and tried reimporting it but got a message "The file is not recognized by the raw format support in LR". On the LR import screen the thumbnails of those images can not be displayed.

I have only worked with those files in IMatch, so I figure I have done something to make them unrecognizable for LR, but I have no clue what it could be.

Does anyone have suggestions?

[Mario]

Did you try to delete the XMP file and then synch the LR catalog again?

Adobe regrettably only uses binary error messages: "Can't read that file, bla-bla". I would be more helpful if they could be bothered to add something like "I can't understand the image format" or "I can't figure out the metadata".

I have only worked with those files in IMatch,

And you did what, exactly? Looked at them? Changed metadata? If so, what did you change?
This is important. Since IMatch does not touch the image data itself, it can only be the metadata which confuses LR.
Since I work with NEF, IMatch 5, LR and CS every day, I know that there are no obvious problems.

I have one open issue (#774 I think) which causes LR to reject files after a user has applied what he calls "reverse versioning" - namely copying back all (!) metadata from TIF or JPEG to the RAW file (CR2) and this breaks LR. The problem here is that just copying all metadata from a TIFF or JPEG to a RAW was never part of the design of RAW files, ExifTool or IMatch. I can reproduce this problem, but only when all metadata is copied. Copying only safe metadata like XMP does not cause this problem. Copying EXIF data between files (as I point out so often) is potentially harmful. It's slightly bad when doing to from RAW to JPEG/TIFF etc. But it's definitely playing with fire when copying EXIF data from JPEG/TIFF files to CR2, NEF or other RAW formats. There may be reasons for it (which is why IMatch allows it) but if done wrong, things will fall apart.

[HansEverts]

That is called 'learning the hard way'.
Deleting the XMP files did not help.
Fortunately I had a backup of that folder dated 20 October. I copied the whole folder and all images  taken till that date are recognized again by LR and CS.

What I did in IMatch lately? Well quite a bit, but basically it seems to be an issue of workflow.
I shoot NEF, export the JPG to IMatch, change metadata in IM and copy them into the NEF. So the JPG is the master and the NEF a version. My version setting included propagation of all categories, hierarchical keywords and metadata.

I guess I should reconsider the workflow.

In the mean time, do you have a suggestion how to recuperate the images I took since 20 October?

Other lesson learned: make regular backups, with short intervals, but also with a longer interval, because sometimes you discover a problem only after a while, when it has propagated itself to all your backups.

[Mario]

Good to hear that you followed the BIG Beta Test advice in the Beta Tester Guide in the Help which explains that you should only ever work on copies of your images, and never work on original files with IMatch 5 Beta. So no harm can really be done if something goes wrong or you run into a bug.

I shoot NEF, export the JPG to IMatch, change metadata in IM and copy them into the NEF.

This is a very problematic workflow indeed.

Just see what you are doing:

You export your RAW with development settings in LR into a JPEG file.
Depending on your settings, LR creates a fresh JPEG with a subset of IPTC data, a mix of the EXIF data extracted from the NEF and new EXIF data created to match the JPEG file, and a stripped-down XMP record. All this makes up th metadata in the JPEG. It does not contain any Nikon maker notes, and the EXIF and XMP will differ in many parts from what LR maintains for the original NEF file.

When I understood you correctly, at some part of your workflow you copy all metadata (including EXIF, IPTC and XMP) from the JPEG to the RAW, replacing what's there. If this is correct, you will replace the important original EXIF data in the RAW with the EXIF data from the JPEG file, which of course differs in many ways, from the color profile to the critical maker notes, orientation info, image dimension info and more. Basically you are stuffing an invalid EXIF record into the RAW file.

I can understand that an application which depends in many ways on the EXIF data like LR will be confused by that and reject the file.

Maybe even the XMP data is dangerous or at least harmful to copy back. LK keeps its development settings in the XMP record it maintains for the RAW. These are not copied to the exported JPEG as far as I can tell (Adobe does not document this, at least I could not find any info on this). So maybe copying back the XMP record from the JPEG to the RAW will damage things or at least wipe out all development settings LR has stored in the XMP record for the NEF.

It's usually safe to copy IPTC data from a RAW to a JPEG and vice-versa. IPTC data is not involved in image generation.
Copying EXIF from a RAW to a JPEG is usually also safe (in IMatch) because ExifTool takes care to relocate all things as needed. Still, you may run into problems when the orientation of the image data in the JPEG does not match the RAW, or the JPEG uses another color profile and by copying the EXIF from the RAW to the JPEG you replace that info. As I often say, EXIF was never intended to be copied between files!

Copying XMP should be safe. But, LR and other products store proprietary (aka undocumented) data inside the XMP record. This data often belongs to one specific image (e.g. the RAW). Copying that XMP data to a JPEG may cause the JPEG to look "wrong" because the proprietary data, when interpreted by an application which understands it, will lead to wrong results. E.g. the crop record, all kinds of adjustments and so on.

I honestly never tested copying EXIF data or the proprietary XMP:LR data from a JPEG to a NEF. But what I wrote above is probably what happened and the cause for LR refusing the files...the EXIF data copied from the JPEG into the NEF just does not work.

[HansEverts]

I know Mario, I tempted the devil and I pay the price.

At this moment I am trying to recover from the camera's SD card the few hundred remaining images I am still missing. I am using testdisk/photorecovery. It is at this very moment recovering the files. Inconvenience is that it recovers them in TIF format. But CS can open them, so at least I got them back.

[BenAW]

I guess I should reconsider the workflow.

Assuming you only need the NEF's in case you want to create another JPG with different edits:

why not do all the things you did with the JPG (categories, hierarchical keywords etc.) and stop right there?
Create a buddy relation between the NEF and its XMP file to keep them together.
Create a Master/Version relation from JPG to the NEF, and don't propagate anything at all. Make the NEF read only if you want.

If you want to create a new JPG, first find the Master JPG, and with Show All versions you have found the NEF as well.

[HansEverts]

Yes Ben thanks. I have learned a lot today.

The origin of it all is that in IM3 I only included the JPG. The - perhaps false - idea was not to overload the database. So all keywords, categories, copyright, etc., were in the JPG. In IM5 I decided to include the NEF and step by step, pushing my luck until it snapped, I transferred the information from the JPG to the NEF.
Well, I reached the limit and went over it. Good thing is, I tested lots of so called free recovery softwares. Most are free until you actually start recovering. Last one was Puran, which actually recovers the NEF format.

[medgeek]

Hi BenAW.  What Hans and Mario have said makes a lot of sense.  I personally never trust original images to any software (even Mario's).  This is why I maintain untouched original images in a parallel folder structure, both date-ordered and I only work on one set.  Hard drive space is so cheap nowadays and this way, I can always go back to straight-out-of-the-camera images if/when I need to.

[Richard]

I personally never trust original images to any software

I fully agree but would go one step farther. COPY those original image files to a USB hard drive and take it with you when you are away. Chances of data loss when a file is on two HD is slim but both can be lost to a fire or burglary.

[HansEverts]

You all convinced me. I go back to my old system of separate management of the NEF originals and the JPG copies. I included them both in IM5 more for the fun of playing with the various options, than for a practical advantage.

[BenAW]

I go back to my old system of separate management of the NEF originals and the JPG copies.

As long as you don't propagate anything you can use versioning without any risk.
Your problems started when you began to propagate metadata.

[Mario]

Actually, the problem was caused by propagating EXIF data (most likely) from a version back to the master. This is a no-no in almost all cases. Replacing the EXIF data of a RAW with the EXIF data of a JPEG will break the RAW.

Propagating collections, categories, Attributes and other IMatch metadata is no problem, even when done from the version to a master. Propagating safe metadata like XMP and IPTC is also no problem. It's EXIF which causes the problems. Or maker notes. Especially when copied into a RAW.

I propagate XMP and IPTC metadata from NEF to JPEG, DNG, TIFF and PSD regularly.

[BenAW]

@Mario
This is my point as well. As long as you know what you're doing, propagation is fine, IF you have a need for it.
Current setup in preferences makes it easy to believe that you HAVE to propagate something for versioning to work.

Versioning works fine without propagating a single bit of metadata.

[Mario]

Yes. And that's why the sample rules don't propagate anything. And all new rules also don't propagate anything. The user has to actively choose to propagate metadata. I'm not sure how I can make this more safe, despite from putting more scary warning dialogs into the dialog (which are clicked away) and more exclamation mark section in the help. It's all already there. You cannot protect the user from everything.

I might change the order of the rules, to put potentially harmful metadata sets down in the list.

[Ferdinand]

FWIW, I've always included the RAW (mostly NEF) in the DB and treated them as the master, and propagated from there, and never had a problem.  My view is that doing it in the reverse direction is not to be attempted without a lot of thought.  And certainly not to be attempted on production images using beta s/w.

In my notorious synchronisation script for V3.6, you could only propagate to JPG and TIFF, unless you changed the configuration yourself.  I wonder if some limit like this involving standard formats might not be a good idea, which people would have to manually over-ride.  But this also might be hard to do without increasing complexity.

I haven't created a DB from scratch for a while - is it the case that the sample versioning rules that ship with IMatch 5 now have nothing checked for propagation, once the versioning rule is enabled?  This didn't used to be the case, and I think it's a good idea.

[BenAW]

Problem imo is the setup of the Preferences > File Relations.
Selecting the tab "Versioning" gives a lot of things to propagate!
I again propose to rename the tab Detection to Versioning.
On this tab have a checkbox "Propagate Metadata".
When selected a third tab becomes available now with the propagation options.
But only after first showing a warning "Expert users only!!" 

Another old suggestion is to have named propagation sets.
The same propagation set can be used in different version sets.
They could also be used when eg. pasting attributes etc.

[HansEverts]

Without wanting to defend I made that mistake (propagating EXIF data from JPG to NEF) I do believe that the warning could be a lot clearer.
Before selecting All metadata, I thoroughly checked the screen to see if I missed something, because I was suspicious. All I saw was the asterix with"indicates on-write attributes". This may be completely understandable by the professional, but I do not know what that means and I do not think my computer and photo knowledge are that far below that of the average user.

The screen has plenty of room to have a clear warning DO NOT ..... A second option is to prompt the user if he/she choses dangerous options with "Do you really want to do that ....".
The third option is to have "Ten things not to do" on the opening screen.

Putting warnings in the Help file is great and necessary, but not enough. The typical user will read the introduction and a few paragraphs of interest, but will only dig deeper after trying the software. I have certainly not read the complete manuals of LR ad PS before using the software.

[Mario]

I have already switched the order of the propagation sets so the least dangerous are on top.
I've also added a section on the risks of propagating the wrong metadata to the corresponding help topic.
I've also changed the configuration file to associate a warning attribute with each propagation set and I will show another warning dialog when the user enables one of these sets.

Of course the amount of potential problems not not only varies with the propagation set (XMP vs. EXIF etc.) but also with the file formats affected by the relation. I cannot cover all angles here, sorry. What works for NEF->JPEG may fail for PSD->DNG.

I, for example, have never anticipated that users would attempt to replace the EXIF data of RAW files with EXIF data from JPEG files. Thankfully, at least to my knowledge, only two users tried to do that so far. We are all learning, that's one of the things we do during this Beta test.

Plastering even more warning dialogs, scary icons and suchlike all over IMatch will not help. It will freak out some users, and other users will just ignore them anyway.

[HansEverts]

I, for example, have never anticipated that users would attempt to replace the EXIF data of RAW files with EXIF data from JPEG files. Thankfully, at least to my knowledge, only two users tried to do that so far. We are all learning, that's one of the things we do during this Beta test.

You would be surprised how many people tried that, but did not report. I wonder why?

Plastering even more warning dialogs, scary icons and suchlike all over IMatch will not help. It will freak out some users, and other users will just ignore them anyway.

I did not say plastering and did not mention scary icons and suchlike. As far as I was concerned, I tried to give, what I see as useful options, but I see the discussion is closed.

[Mario]

Discussion is still open. When I close a thread, no other user can post.
And your feedback was welcome. As I said, we are all learning. Me the most. I continuously update the help file based on questions asked here, add samples and warnings and tips if users run into problems frequently etc.

I did not mean to upset you by my comments. Just noticed that two users implemented reverse versioning, copied all metadata (or the wrong metadata) from version files to master files and in the end, broke Lightroom or other software.

I just had a 30 minute "please hold the line" session today with Adobe Germany support because LR refused to see GPS data in XMP files of NEF files. It only detected GPS data embedded in the NEF itself. I tried the usual re-synch and Metadata > Load Metadata but nothing helped. While preparing a set of samples for Adobe, I removed the folder from the catalog and re-added it. And this solved the problem as well. Strange things happen with computers. Hopefully my LR catalog is not damaged again.

As I said, I already made substantial changes to the propagation tab and the corresponding help topic. If you now attempt to propagate EXIF data or other data I consider as potentially harmful, IMatch displays an explicit warning dialog:



[attachment deleted by admin]

[HansEverts]

Thanks Mario, much appreciated.

I spared you many of the strange things I encountered since the metadata reverse versioning trick. At some moment I could click on a NEF file with the preview of a cow (example) and get to see the photo of a house: the preview did not correspond with the image. I guess something was wrong in the LR catalogue and fixed itself, but I thought it was not wise to share that with the forum. 

[Ferdinand]

I have to confess I've never really understood why people didn't include their RAW files in their V3.6 DB.  But it seems that the practice was more common than I realised.  And so I think we will see more of these reverse propagation issues as these users want to bring their RAW images into their V5 DBs.

[Sorted]

Thanks Mario, much appreciated.

I spared you many of the strange things I encountered ... At some moment I could click on a NEF file with the preview of a cow (example) and get to see the photo of a house ... 

[cytochrome]

Is the house an abattoir?

Back-versioning is a nice and almost unique possibility that IM5 offers. It should not be dropped. It is enough to put a clear warning that user may loose their raw and that they are on their own.

After much experimenting during which I annoyed Mario because I didn't really understand what I was doing, I have now settings that work for me, no raw loss with NEF and RW2.

I do it seldom, only for simple tasks (transferring only author, location, headline, description, and categories). I keep it simplest possible, only xmp, no iptc and no Exif, it is anyway idiotic to copy Exif to a raw and some like the Pana RW2 are very sensitive to any change in Exif apart from GPS data.

The whole idea of back-versioning to the raw is rather senseless when working with Adobe converters.

This is no advocacy in favor of back-versioning, I don't care if nobody but me uses it, I just try to explain why I hope Mario will keep it...

Francis

[Mario]

For IMatch, there is no normal, forward or reverse versioning. IMatch just applies the rules the user configures, and recognizes masters and versions solely based on the file extensions used in the relation rule. IMatch does not give RAW formats a special treatment. If a user wants to propagate usually safe data like IPTC, GPS or XMP from a JPEG to a RAW, this is fine with IMatch. And there are situations where this makes a lot of sense.

I've explained the risks now in the help explicitly and also put up an extra warning dialog for each potentially harmful set of data which is offered for propagation.