Read-only files with IMatch and Lightroom

[Photon]

Just like to report a strange behavior of Lightroom v5.2. This is no bug, no fault and no missing feature of IMatch!
I am using read-only DNGs with XMP sidecars in IMatch. That works good. However I identified in LR:

• Very good: When labels and ratings are defined withing IM, LR is able to read and update at least this metadata from the XMP sidecar, which is generated from IM. The LR command with right mouse button is "Metadata" -> "Read Metadata from File". That is strange, because LR does normally not support XMP sidecars for all file formats, which can embed XMP metadata. But this behavior can be very useful.
• Ok: When using LR command "Metadata" ->  "Update DNG Preview & Metadata" there is an error mesage in LR, because the DNG file is read-only. 
• Very bad: When using LR command "Metadata" -> "Save Metadata to File", the read-only DNG file attribute is disabled (Yes!). So this command in LR shall apparently better not be used for a workflow based on read-only files. This reminds me of Picasa which did in a previous version (and still does today?) not respect read-only attributes of image files.

I am still using & investigating the pure "XMP Sidecar / Read Only" based workflow with IM v3.6 & v5.0.
So far I am more an more happy with every release of v5.0.xxx. Thank you Mario!

Regards, Martin

[Mario]

Read-only files are always a problem. When not now, that maybe with another application or a future LR/CS version. An image editor or RAW processor is just not designed to work with image files which are deliberately made read-only. That's a pretty uncommon workflow and calls for trouble. Making files read-only when they are final, or storing them on a read-only media or revoking write-privileges on the file system level, all this can be done.  But after the file have been archived and when they are never touched again. Avoids a lot of potential problems, missing or out-of-synch metadata etc.

There is no real need to make DNG files read-only. If you are afraid that they become damaged, make a backup of the original RAW, the DNG after the conversion and store these backups safely on long-term archival media. Perform daily backups of all your files and keep a history of several months, with intermediate weekly and monthly backups. Keep the weekly for weeks, the monthly or quarterly for several years. That's a pro-level backup schema which does not cost you much (except some external hard disks or a NAS box). Automatic backup solutions like TrueImage etc. make such backup schemes quite easy to implement, all you need to so is swapping disks every week. And you can roll back individual files several weeks back at any time, in case something is list. These systems automatically backup only modified data, which means the daily backup volume is usually quite low.

I use similar schemes for many years, and have never lost a file since. I've also implemented such schemes on larger scales for clients. And once we'll all have truly high-speed upload Internet connections, we can store all the stuff in a encrypted containers in cloud storage and archive it for ever. I already use cloud storage for a second backup of my development machine, my web servers etc. It's to slow for mass data like images now, but that will change in a few years time.

[sinus]

Read-only files are always a problem. When not now, that maybe with another application or a future LR/CS version. An image editor or RAW processor is just not designed to work with image files which are deliberately made read-only. That's a pretty uncommon workflow and calls for trouble. Making files read-only when they are final, or storing them on a read-only media or revoking write-privileges on the file system level, all this can be done.  But after the file have been archived and when they are never touched again. Avoids a lot of potential problems, missing or out-of-synch metadata etc.

I want to stress this.
I tried some time ago such a workflow, but after troubles, I did not more go this line. In fact, if I set the flag "read only" very early, in the camera, and I transfer this file in the pc, then I had troubles.
So I decided (FOR ME) a better way:

Simply create a backup of each file BEFORE I do anything with the file. The file for working I rename later, but the original name is stored anyway and I can search the original file always. So no danger, I think.

Then I go, since "only" itpc is not more a good choice (for me), the route with xmp-files.
But ok, the real proof will come, when IM5 is out there for real! 

[Ferdinand]

Read-only files are always a problem. When not now, that maybe with another application or a future LR/CS version. An image editor or RAW processor is just not designed to work with image files which are deliberately made read-only.

It seems to me that that statement is true for DNG, as Adobe s/w regards DNG as a "standard  format", and will want to write to it.  And of course Nikon Capture will want to write to NEFs files.

But with those exceptions, I thought that read-only would not be a problem with other RAW converters and other RAW formats.  I've been using it successfully with ARW & RAF.  I wouldn't use it with standard formats.

[Photon]

...there is no real need to make DNG files read-only. If you are afraid that they become damaged, make a backup of the original RAW, the DNG after the conversion and store these backups safely on long-term archival media...

There is a need: My DNGs have no original "RAW", because my DSLR cannot produce something else than DNG or JPW or DNG+JPG. Yes, that is true! Some Lightroom experts propose to keep two DNG versions (one original and one for editing) in this case. I think that is waste of storage and nonsense. And I definitively do not want to separate my originals and my versions to different drives or folders. With IM and other DAM tools including backup tools it is very beneficial to keep all files together.

Backup is a different topic and the comments of Mario are good and a valid. But note, that use cases and tools can be quite different. E.g. with encrypted drives, Acronis Trueimage and others do NOT allow some of the usual convenient backup principles. E.g. nonstop, drive or partition backup. Due to this, it can be highly beneficial to limit file modifications, which are not really required. It is not only about backup speed it is also about ease of backup monitoring and confirmation.

I am not concerned, that tools will destroy my originals and I never experienced this. Normally it is the user itself who destroys.
But I disklike tools which modify or pollute originals with unneccessary or proprietary metadata. And this often without user confirmation.

Martin

[Richard]

Some Lightroom experts propose to keep two DNG versions (one original and one for editing) in this case. I think that is waste of storage and nonsense.

With the cost of storage being as cheap as it is, I think it is good advice but you alone know the value of your images.

[sinus]

Some Lightroom experts propose to keep two DNG versions (one original and one for editing) in this case. I think that is waste of storage and nonsense. And I definitively do not want to separate my originals and my versions to different drives or folders. With IM and other DAM tools including backup tools it is very beneficial to keep all files together.
Martin

I also would not really want to separate my originals an versions on two different drives. In my case I do have even the masters (NEFs) and Versions (jpgs, psd ...) in the same directory (monthly-system).

But in your case I would really think about the proposal to make a backup from your original DNG. Of course it depends how many images you make, but usually - with the low-cost of hard-Discs nowadays - I think, it is a good way to feel safe.
I do also backup my RAWs, would I have originally DNGs (what I do not want), I think, I would go the same line, like some LR-Experts advise.

But like Richard wrote: its up to you.

[Mario]

From your's truly: Some good advice on backup (while I wait on my compiler to finish):

Remember: Your hard disk can die any minute! Your RAID too!!

Even a RAID system can die any minute. It's not that uncommon that a disk in a RAID 5 dies. Normally this is no problem, that's what RAID systems are designed to handle. You replace the disk and the RAID starts to recover the data on the lost disks from the data on the other disks. And while doing that, it runs into a dreaded unrecoverable read error (URE), which means the replaced disk cannot be restored because the RAID cannot read data or checksum info from one of the other disks. In this case, all data on the RAID is lost. Just like that.

This type of error is becoming more and more common these days, because the disks used in the RAID boxes are so big. And when you get one (statistical) unrecoverable read error for every 1000 TB of data transfer, and you have 16 TB of disks in your RAID, it becomes much more likely that you actually run into such an error at the wrong time - for instance while the disks are under high-stress because they are fully active for 20 hours while restoring a replaced disk.

I had to restore RAID disks a number of times and I always was lucky. I think this is because because I had a full backup of the RAID on another RAID and external USB disks anyway!

You need to backup RAID systems like any other disk. And when you build your own RAID systems, don't buy the cheap desktop SATA disks. The enterprise or "RAID use" disks are about 30% to 50% more expensive, but they have much lower read failure rates (100 times lower) and handle the vibration better (packing multiple disks into one case causes a lot of tiny vibrations, and the disks need to be designed to handle that).

And of course the hardware in the NAS box itself can fail (main board, power supply, ...). Such a  hardware defect can even grill all disks in the RAID in a few seconds. Then there is nothing to restore, even if you replace the RAID box and plug in the disks from the old system!

If your files are important, spending money and time for backup is mandatory. Unfortunately.

A 'home use' NAS with 12 TB disk capacity with RAID 5 costs you maybe 800€ to 1000€, using entry-level NAS disks. Combine that with tape backup, USB disks or another NAS box for backup and you can handle a lot of images.

If you work with studio quality images shot with digital backs which output 100, 200 or more MB for each RAW file, or you work with panoramas or even video files, 10 TB storage capacity is not much anymore...and then storage and backup suddenly becomes an expensive task.

I've seen photographers mutating to IT geeks because they had to somehow handle the digital output they generate in their studios. And now IT management is often one of the key skills of a professional photographer - unless he is so well off financially that he can afford staff or external consulting for that task.

Thankfully, at least they can use IMatch 5 to manage all their files    

[Photon]

Thanks Mario,

why die any minute and not any second? :-) Since I am the original poster I take the freedom to continue this "off-topic" and like to confirm you experience with RAID. In biz IT environment I experienced recently a severe controller failure on a professional RAID system. The RAID system A controller was writing nonsense to its disk without alarm and RAID system B as a realtime backup made of course a clone of this nonsense. Only with manual recovery from a third HDD backup system the final disaster could be prevented.

Somehow I have also the theory, that a RAID does not help very much in case of theft or fire. Only the thief is happy about such RAID system and at the same time this RAID system is immediately "dead" for the original owner. Thats why I prefer HDD protection through middle and far distance multiple HDD locations and regular backup through LAN/IP and walking!

Regarding DNG files (original topic) I can accept a solution with one original DNG and a working copy for Lightroom. What I cannot imagine is the effective workflow with distributed directories for both files. The key benefit with IMatch is for me, that I can keep AND manage ALL my versions AND all my media files together. Very often I am renaming/restructuring old directories with Imatch (with separated directories it would be complicated to harmonize this). And my backup tool does detect this renaming and removing perfectly and avoids therefor useless deletions and copy process for all affected files. The RAW + JPG-Exports workflow is perfect for me, but with the DNG-Orginal + DNG-Copy + JPG-Exports/Versions workflow I am not satisfied, because there is the risk that other program (LR) modify the DNG-originals. May be my next DSLR will be a RAW one, because a RAW+DNG+JPG-Versions workflow does look totally consequent for me. The optional embedding of RAW data in DNG files would be mindless for me, because it only increases a differential backup after updating metadata/preview in DNG files.

With all this I have no problem with IMatch, but only good feelings. Thanks for it and for the good discussions in forum.

Martin


P.S.: The used hard disks in a RAID system should not have all the same age! Regular replacement of reasonable amount of RAID-HDDs can be a good idea.