Newbie needs review of buddy file / versioning setup...

[lnh]

The more I'm using IMatch, the more I'm loving it! Of course having all this flexibility is double edged, but overall it's a great product and I really appreciate all the time put into the help file. It's nice having fairly detailed docs compared to the previous database I was using. I know there isn't any one right way to setup a database, but after a bunch of trial and error on a small number of files, I think things are starting to come together. I'd like to get some critique on where I'm currently configured for buddy files and versioning.

Some of this may be entirely wrong, so correct me where I'm not understanding things right now.

My typical current workflow is to shoot RAW and JPG in camera. The files have the same name except for the extension.

1) Naming wise, I'm using IMatch to rename the files to a nomenclature based of date and time the picture was shot. The renamer is then used to move the files to a date based (Year and Month) folder hierarchy on a NAS while the database remains in a folder on a local SSD.

2) I'll usually use DxO Optics Pro for RAW conversion and to target various outputs for different uses (e.g. different target resolutions/quality levels for JPG and an 8 or 16bit TIFF if the file is headed to Photoshop (CS5 so ACR doesn't work for all my files) for additional work. Sometimes I'll use LR for RAW processing as well with similar output targeting.

3) For example, if NAME.rw2 and NAME.jpg are the renamed original files from the camera, DxO might output a target file named NAME_1024_DxO.jpg or NAME_16bit_DxO.tif and a Photoshop working file based on the tif might be named NAME_16bit_DxO_TIF_PS.psd. Likewise, a saved output from the Photoshop file might be named NAME_16bit_DxO_TIF_PS_tumblr.jpg if a file was created specifically for upload to that site. I'm trying to keep track of the work flow lineage as different tools are used. Maybe there is a better way to keep track of this workflow within IMatch short of this complex naming convention, but if so, I haven't figured it out yet.

4) After playing around, my first observation is that if you want all your associated files to act as a unit you need to both buddy them together and efficiency of display version them as well. If I want to use the camera JPG for IMatch "As Visual" use, the original RAW/JPG pair need to be buddied and versioned separate from other iterations. In addition, DxO sidecar files need to be made buddies as well.

5) One question I do have is figuring out the right metadata to propagate. Right now I just have "Categories" set, but ideally I'd also like other things like geocode/reverse geocode data to also propagate. Not sure what to set to allow this, but not something else which I might regret. Haven't fully wrapped my head around the consequences of propagation yet.

Attached are screen shots of my buddy and versioning setups. All comments welcome.

[attachment deleted by admin]

[Ferdinand]

3) ..... Maybe there is a better way to keep track of this workflow within IMatch short of this complex naming convention, but if so, I haven't figured it out yet.

I also use file names to track this kind of workflow information, but it does make for cumbersome file names.  You could also use categories or properties, but this information wouldn't be available outside IMatch.  If you used keywords to record this information then it would be, but you many not want to embed workflow keywords in converted files.

4) After playing around, my first observation is that if you want all your associated files to act as a unit you need to both buddy them together and efficiency of display version them as well. If I want to use the camera JPG for IMatch "As Visual" use, the original RAW/JPG pair need to be buddied and versioned separate from other iterations. In addition, DxO sidecar files need to be made buddies as well.

My view is that I'd be careful about specifying out-of-camera JPGs and converted versions as buddy files, which is what your setup does.  From the help file: "IMatch uses buddy relations to keep buddy files and their master files together when moving, renaming, copying or deleting files".  Be sure that this is what you always want for JPGS and other non-RAW image files.  I know that many people do set jpgs as buddy files, but you can get unexpected results.  You can find that you've renamed / moved / deleted more than you intended, especially for the files in subfolders (less so for the OoC JPGs in the same folder as the RAW).  My approach is to not have any overlap between buddy files and versions.

You seem to have duplicated your buddy relations.  You have one for JPG versions in the same folder as the master and another for versions of various filetypes in a subfolder one level down.  I don't see why you need the one just for JPGs.  Assuming that these definitions are otherwise the same, then you should only need one of these as one level down also includes the folder of the master file, i.e. zero levels down.  I.e. you shoull be able to have a link expression that covers the OoC JPGS and the converted files.

5) One question I do have is figuring out the right metadata to propagate. Right now I just have "Categories" set, but ideally I'd also like other things like geocode/reverse geocode data to also propagate. Not sure what to set to allow this, but not something else which I might regret. Haven't fully wrapped my head around the consequences of propagation yet.

It's probably wise to proceed cautiously on propagation, since it's easier to repropagate later than it is to unpropagate. 

There isn't an option to propagate geocode information specifically that I can see, so you'd need to propagate either XMP or EXIF, depending on where the geocode data is stored.

[jch2103]

It's not clear from your workflow description how you use the direct-from-camera jpg images. If they're not crucial to your workflow, it might simplify things to stick with NEF-only camera output.

[Ferdinand]

It's not clear from your workflow description how you use the direct-from-camera jpg images. If they're not crucial to your workflow, it might simplify things to stick with NEF-only camera output.

That's a good point that I didn't raise but I wondered about.  On mirrorless I shoot JPG and RAW, because the embedded thumb in the RAW in the cameras that I've used is only about 50% or less of the pixels of the RAW.  So shooting JPG is the only way to check for sharpness accurately in the camera.  But I mostly don't need them afterwards, so I usually delete them.  Sometimes if I need an image in a hurry they're convenient, and once in a while I delete the RAW, but mostly I delete them.  (The advantage of Fuji X series cameras is if you want a camera-created JPG, you can always put the RAF back on a card and do an in-camera conversion.)

[lnh]

Thank you for the feedback.

The file naming can get, a bit complex. I believe there are some DAM products that are more geared towards multi-user use which have check-out/check-in features where you can annotate what you've done. IMatch offers so much more in other ways that such a feature is a lower priority to me. Maybe this would be a good use of custom file attributes. Still without annotation or some system like the file naming it's difficult to remember what you've done.

In situations where I shoot both RAW and in camera JPG, the JPGs usually don't have a big use. Sometimes they are used if something needs to be done quickly and other times they serve as a reference when processing problematic or tricky RAW files. I'll often just use them for comparison through the process of RAW conversion. Being a digital pack rat, I'd like to keep them part of the image set. The problem I ran into when I just made them a version and not a buddy as well had to do with the associated XMP file generated by IMatch. If I didn't buddy them, and changed the name of the RAW, IMatch would change the name of the DxO sidecar and create a new XMP with the new filename also leaving the old XMP file as well. If you then tried to rename the file back to it's original name, IMatch wouldn't let you because of the existing old XMP file.

You seem to have duplicated your buddy relations.  You have one for JPG versions in the same folder as the master and another for versions of various filetypes in a subfolder one level down.  I don't see why you need the one just for JPGs.  Assuming that these definitions are otherwise the same, then you should only need one of these as one level down also includes the folder of the master file, i.e. zero levels down.  I.e. you shoull be able to have a link expression that covers the OoC JPGS and the converted files.

My mistake in configuring this stuff. All the files (RAW, JPG, PSD, XMP, DOP, etc) sit in one folder. There is no need for me to have IMatch look either up or down for the related files. I originally had all the buddies and versions in one definition, but then thought it might be useful to have the camera JPG for display which means you need to split it up. If linking the camera JPG's in this way provides no useful utility, I can easily modify one of the regular expressions.

It's probably wise to proceed cautiously on propagation, since it's easier to repropagate later than it is to unpropagate.

There isn't an option to propagate geocode information specifically that I can see, so you'd need to propagate either XMP or EXIF, depending on where the geocode data is stored.

I'm still not getting the propagation thing. As is shown, I set it to propagate all categories with no other options set. I went and assigned keywords from my thesaurus to the RAW (master) file which then automatically creates @Keywords categories. After choosing the option to "propagate data to versions," it didn't appear anything got assigned to the associated JPG file. I'm guessing this is because @Keywords aren't real whereas {File.MD.XMP::Lightroom\hierarchicalSubject\HierarchicalSubject\0} are real and would require propagating some subset of XMP which wasn't part of what I allowed. It would certainly be nice for keywords to propagate without fear of unexpected results.

On a another matter, does anyone know which Regular Expression standard IMatch uses? I'm guessing it might be Perl due to the reference to it in the help documentation. Seems like regular expressions aren't highly standardized.

That's a good point that I didn't raise but I wondered about.  On mirrorless I shoot JPG and RAW, because the embedded thumb in the RAW in the cameras that I've used is only about 50% or less of the pixels of the RAW.  So shooting JPG is the only way to check for sharpness accurately in the camera.  But I mostly don't need them afterwards, so I usually delete them.  Sometimes if I need an image in a hurry they're convenient, and once in a while I delete the RAW, but mostly I delete them.  (The advantage of Fuji X series cameras is if you want a camera-created JPG, you can always put the RAF back on a card and do an in-camera conversion.)

That is a very nice feature of the Fuji X. Makes it easy to throw away the JPGs without fear of tossing something which might be useful some day. Doesn't appear my Panasonic GM-1 or Olympus OMD-EM5 can do that in camera. I'd have to test this a bit more, but it seems like shooting RAW+JPG is actually faster than RAW alone on the Olympus. Probably has something to do with their image processing engine. It also appears you do get slightly more detailed previews when shooting both.

[Mario]

I believe there are some DAM products that are more geared towards multi-user use which have check-out/check-in features where you can annotate what you've done.
IMatch 5 automatically maintains a History for each file (see History Panel). You can also add custom entries in the History Panel. You can use metadata to record information about your files. You can use Attributes to record any amount of any type of information about your files. You can use real Annotations in the Viewer to add non-destructive annotations to your files. Just chose what you like best.

On a another matter, does anyone know which Regular Expression standard IMatch uses?

Please read the exhaustive topic on regular expressions in the IMatch help for details. Just type regular into the help index to find it. Right at the top is a big green box which tells you:

IMatch uses regular expressions based on the syntax used by the Perl language.

[sinus]

I believe there are some DAM products that are more geared towards multi-user use which have check-out/check-in features where you can annotate what you've done.

If you use for example Attributes, you can simply create one or more fields for simply put informations about the file.
And you can add remarks and more into the varioius fields of xmp.

So you can enter information WITH and IN the file, but you can also add information ONLY in the db. I can not imagine, what another DAM could offer more?

[sinus]

I believe there are some DAM products that are more geared towards multi-user use which have check-out/check-in features where you can annotate what you've done.
IMatch 5 automatically maintains a History for each file (see History Panel). You can also add custom entries in the History Panel. You can use metadata to record information about your files.

Boah! I was (still) not aware of this. Really cool, this history-stuff.

I think and hope, that history - stuff is stored in the intern db of IMatch, NOT in the file?!

Hmm, so a file tends to be, like the human, more and more transparent! 

[Ferdinand]

I'm still not getting the propagation thing. As is shown, I set it to propagate all categories with no other options set. I went and assigned keywords from my thesaurus to the RAW (master) file which then automatically creates @Keywords categories. After choosing the option to "propagate data to versions," it didn't appear anything got assigned to the associated JPG file. I'm guessing this is because @Keywords aren't real whereas {File.MD.XMP::Lightroom\hierarchicalSubject\HierarchicalSubject\0} are real and would require propagating some subset of XMP which wasn't part of what I allowed. It would certainly be nice for keywords to propagate without fear of unexpected results.

@Keywords are very real, more so that categories, since they exist in the image files and thus outside the database.  The problem here is that @Keywords are not categories.  They may look like categories, behave like categories, and be displayed in the categories view and panel, but they are different.  (Strictly speaking what you see in @Keywords is a special sort of dymanic, data-driven category.)  So when you select categories to propagate, you're not selecting @Keywords.  As you noted, they're in XMP and if you want to propagate them you have to propagate XMP.  There isn't an option to propagate just @Keywords.  I have a feeling that this may have been suggested before.  If you find an existing feature request you may wish to add your vote, or if you can't then you can start one.

[lnh]

Please read the exhaustive topic on regular expressions in the IMatch help for details. Just type regular into the help index to find it. Right at the top is a big green box which tells you:

Oops - sorry about that. I was focused on the green boxes at the bottom of that help topic where you give reference to additional resources and forgot what was on the top. There is so much to learn.

If you use for example Attributes, you can simply create one or more fields for simply put informations about the file.
And you can add remarks and more into the varioius fields of xmp.

On the annotation side, I've taken the approach you suggest and have created some custom attributes which I populate for general descriptions. That seems to work best for me, but might use the annotation layer overlays for more specific purposes while working on a project.

@Keywords are very real, more so that categories, since they exist in the image files and thus outside the database.  The problem here is that @Keywords are not categories.  They may look like categories, behave like categories, and be displayed in the categories view and panel, but they are different.  (Strictly speaking what you see in @Keywords is a special sort of dymanic, data-driven category.)  So when you select categories to propagate, you're not selecting @Keywords.  As you noted, they're in XMP and if you want to propagate them you have to propagate XMP.  There isn't an option to propagate just @Keywords.  I have a feeling that this may have been suggested before.  If you find an existing feature request you may wish to add your vote, or if you can't then you can start one.

I'll take a look at this and vote/add request as appropriate. I'd imagine there is some scripting way to do this, but I'm nowhere close to a point in my understanding of IMatch to tackle that one. It would also be nice to selectively propagate gps & location data as well. Haven't done anything complicated with templates yet, but do wonder if this desire could be solved in some way using them.

[Ferdinand]

I'll take a look at this and vote/add request as appropriate. I'd imagine there is some scripting way to do this, but I'm nowhere close to a point in my understanding of IMatch to tackle that one. It would also be nice to selectively propagate gps & location data as well. Haven't done anything complicated with templates yet, but do wonder if this desire could be solved in some way using them.

I doubt whether you can propagate using templates.  Any comments I have made about templates were meant to suggest that templates could replace scripts that wrote metadata to a file, but were not meant to suggest that you could use them to copy from one file to another.

It's possible that the ExifTool Command Processor (ECP) might be able to.  Also, today I've just released a script that does some selective propagation.  It's not as selective as you want, but it could be modified.