iMatch workflow with DXO, ON1, NIK, CS6 etc. Buddy & Versions

[Stefanjan]

I've been reading and rereading this forum and the help and starting to understand how this all works.

Couple of questions.

1. When I process an image in say DXO Photolab, If I save a finalised jpeg and make this visual proxy in a version set. If I need further versions jpeg e.g. for a competition entry (1600 dpi long side), 10cm x 15 cm print, web etc. Up to now I have always opened the original image in the RAW Processor e.g. DXO and exported a version.

I understand that many of you save a final jpeg and use this to create jpeg versions within iMatch  for various purposes. This workflow sounds easier but I've always been told not to repeatedly process a jpeg due to image degradation.

In the above scenario, the image will only ever be processed twice so I guess that the degradation might not be noticeable. Is this your workflow and what jpeg settings do you recommend?

2. While I now only shoot mainly in RAW I do have legacy photos in JPEG format plus new jpegs from other people. I'm not entirely clear what would be best practise to differentiate between JPEGS which are masters and JPEGS which are versions. What's the best way of dealing with jpeg masters as opposed to jpeg versions.

[David_H]

In the above scenario, the image will only ever be processed twice so I guess that the degradation might not be noticeable. Is this your workflow and what jpeg settings do you recommend?

Being honest, it probably doesn't matter. An export from the IMatch batch processor (to resize) or a re-process in DxO with a different set of output parameters is unlikely to make too much of a difference. Which ever works for you.

2. While I now only shoot mainly in RAW I do have legacy photos in JPEG format plus new jpegs from other people. I'm not entirely clear what would be best practise to differentiate between JPEGS which are masters and JPEGS which are versions. What's the best way of dealing with jpeg masters as opposed to jpeg versions.

I put the master files at the top of their folder tree and all versions beneath them.

[Mario]

That ^

Additionally, you can add your masters files to a specific category or give them a specific label. This allows you to always tell what's what.
If you use versioning, the master will show the master icon in IMatch (being it the RAW or the JPEG).

I keep masters in the base folder and permanent derivative files (PSD, TIFF, JPG) in sub-folders of the base folder.
This keeps things neat and tidy, and the structure works in IMatch, Windows Explorer and all applications I use now and will use in the future.

I can always tell what is what, even without IMatch (e.g. in Windows Explorer or a NAS backup).

IMatch offers a lot of features for versioning, buddy file management etc. I think it important to use the least possible complex way to do things (KISS principle).

For the re-compressing JPEGs: Yes, there should be a difference whether you create a 1600px result from the original RAW in your RAW processor or from the final JPEG via the IMatch BP.
The questions is, is it noticeable or relevant for the intended purpose and audience? If not, the BP workflow is probably much faster and easier.

[Stefanjan]

Thanks for your replies.

I put the master files at the top of their folder tree and all versions beneath them.

I couldn't figure what "I put the master files at the top of their folder tree and all versions beneath them." meant. Is this the same thing as Mario is suggesting.

I hadn't considered keeping versions in a sub folder. Makes sense but is there an easy workflow to do this?

Do you use the Renamer for this purpose? With original file name and Move. Is there a formula for subfolder?

[Mario]

Like many (most) users I organize files in folders by year/month/day.

The RAW files are always in the day folder (base). All versions and other derivative files go into sub-folders. Exports, deliverables, client files etc. also go in sub-folders.
I export the versions in my RAW processor into sub-folders. Same for composites created in PSD or final videos.

The Renamer can be used to produce sub-folders and move versions after the fact, but I usually don't need that. I prepare the folder structure before I start culling and editing.
I use the Create Multiple Folders command in IMatch (with a pre-made string I keep in the IMatch Notepad App for copy/paste) to create the whole folder hierarchy with a few keystrokes. IMatch can do so much 

If you currently just throw all files into the same folder - no problem, it will still be able to find versions (unless you also mix RAW and JPEG masters and JPEG versions in the same folder, which would complicate things).
But I find it easier in Windows Explorer and other applications to separate source files (RAW, usually) and final/output files into folders. Much tidier.

[David_H]

I put the master files at the top of their folder tree and all versions beneath them.

I couldn't figure what "I put the master files at the top of their folder tree and all versions beneath them." meant. Is this the same thing as Mario is suggesting.

Yes exactly that - so (for example) the 'master' images from a shoot from today would look like :
K:\Photography\2021\07\2021_07_16\IMG_1234.CR2

I hadn't considered keeping versions in a sub folder. Makes sense but is there an easy workflow to do this?

Do you use the Renamer for this purpose? With original file name and Move. Is there a formula for subfolder?

I copy my images to an import folder (C:\PhotoDB\Import) from the card; I'll give them an initial cull and delete any out of focus, etc.

I'll then let IMatch index them, perhaps apply Metadata templates, categories, events, then run renamer on it - which consists of precisely two processing instructions (it has been a very long time since I set this up, and it works so I don't fiddle) :

Original File Name
Move File (Destination folder - K:\Photography\{File.MD.Exif::Main\36867\DateTimeOriginal\0|format:YYYY\MM\YYYY_MM_DD}\)

In DxO, when I produce an output image, the output settings are
Destination: Custom Folder
Path: Altered\Prints\8x12
Suffix: _DxO_CE1

When the image is exported, it'll be put in K:\Photography\2021\07\2021_07_16\Altered\Prints\8x12\IMG_1234_DxO_CE1.jpg

IMatch will then pick it up, index it, and version it according to my rules (in this case, one that picks up that its in a 'Prints' subfolder)

[JohnZeman]

2. While I now only shoot mainly in RAW I do have legacy photos in JPEG format plus new jpegs from other people. I'm not entirely clear what would be best practise to differentiate between JPEGS which are masters and JPEGS which are versions. What's the best way of dealing with jpeg masters as opposed to jpeg versions.

A couple years ago I also had this situation when I decided to start using file relations with stacks, buddy files, and versions.  So first I converted all of my old legacy JPG original images to the TIFF format.  Now all of my masters are either CR2, CR3, DNG, or TIFF while all versions are JPGs.

My folder system is pretty basic, I store each image, masters, versions, and buddy files by year in one folder.  For example every photo in my IMatch database that I've taken this year is stored in my 2021 folder.  I'm not a pro and I'm not taking as many photos as I used to so by the end of the year each folder usually has less than 10,000 total files.

I understand that many of you save a final jpeg and use this to create jpeg versions within iMatch  for various purposes. This workflow sounds easier but I've always been told not to repeatedly process a jpeg due to image degradation.

In the above scenario, the image will only ever be processed twice so I guess that the degradation might not be noticeable. Is this your workflow and what jpeg settings do you recommend?

As far as re-exporting an image to use for something else, for example to post on facebook, I use the IMatch Image Batch processor which creates the copies I want from the JPG version.  This works well for me since when I create my JPG versions I save them as high quality (85%) JPGs so there's no visible quality degradation in the resulting Image Batch Processor reprocessed image.

[Mario]

As far as re-exporting an image to use for something else, for example to post on facebook, I use the IMatch Image Batch processor which creates the copies I want from the JPG version.  This works well for me since when I create my JPG versions I save them as high quality (85%) JPGs so there's no visible quality degradation in the resulting Image Batch Processor reprocessed image.

Facebook re-encodes all images anyway 

[JohnZeman]

Facebook re-encodes all images anyway 

Yeah I noticed that a few years ago when I uploaded a carefully sharpened image to facebook only to have it nowhere near sharp once it was there.

[Stefanjan]

If you currently just throw all files into the same folder - no problem, it will still be able to find versions (unless you also mix RAW and JPEG masters and JPEG versions in the same folder, which would complicate things).

I currently throw all files into a single folder e.g.
2021
2021-07-15 ¦ Cressing Temple Birds of Prey

It is likely that my folders only contain RAW masters and JPEG masters they would not be mixed.

What should I consider when setting up the versioning?

[Stefanjan]

My folder system is pretty basic, I store each image, masters, versions, and buddy files by year in one folder.  For example every photo in my IMatch database that I've taken this year is stored in my 2021 folder.  I'm not a pro and I'm not taking as many photos as I used to so by the end of the year each folder usually has less than 10,000 total files.

I'm also not a pro and probably don't need the complexity of sub folders. I do have older folders from Lightroom which are filed Year, Month, Day which I'm gradually converting using Renamer to my current system YYYY, YYYY-MM-DD ¦ Description.

As far as re-exporting an image to use for something else, for example to post on facebook, I use the IMatch Image Batch processor which creates the copies I want from the JPG version.  This works well for me since when I create my JPG versions I save them as high quality (85%) JPGs so there's no visible quality degradation in the resulting Image Batch Processor reprocessed image.

I'll give this method a try for 1600 dpi competition entries and see whether I can see a difference compared to jpegs out of DXO. Certainly would be a much simpler workflow.

[Jingo]

I think it all comes down to what you do with your images.  Personally, I gave up on the RAW->JPG train a long time ago... just too much work with little reward for my workflow.  I found I hardly never go back to the original RAW files once I have a suitable JPG created.  As such, I found no need to version or include RAW files in my DAM so my database is 100% JPG (with some videos as well).

Since my folder naming structure is identical between RAW and processed JPGs, I can find an image in my RAW editor very quickly if I ever did need to reprocess an image.  This "Keep it clean" approach works well for me as a non-professional, family/travel photographer.  But IMatch offers something for everyone!

[bekesizl]

I also have a mixed JPG/RAW master file system.

My solution is to have a folder for every camera is gather photos from (Canon MILC, various phones, older compacts).
This way I could use different file relations for different folders (but I don't have to).
I also created file relation rules, so that a JPG version always has at least one characted appended to the name of its master.
This way IMatch is able to add also a JPG version to a JPG master (e.g. a crop of a phone photo).

Here are my file relation rules.
First line is the master.
Third line is a buddy/version.

DxO Sidecar (buddy)
\.(cr3|cr2|nef|tif|tiff|jpg|jpeg|dng|nef|rw2|raf|srw|arw)$
/^_*//
^{name}{ext}\.dop$

JPG (buddy+version)
\.(jpg|jpeg)$
/^_*//
^{name}[+\-_]+.*\.(jpg|jpeg|dng|tif|tiff|afphoto|psd|heic)$

RAW (buddy+version)
\.(cr3|cr2|nef|rw2|raf|srw|arw)$
/^_*//
^(_*{name})[+\-_]*[0-9|a-z]*\.(jpg|jpeg|dng|tif|tiff|afphoto|psd|heic)$

[Stefanjan]

I also created file relation rules, so that a JPG version always has at least one characted appended to the name of its master.
This way IMatch is able to add also a JPG version to a JPG master (e.g. a crop of a phone photo).

Thanks, that sounds exactly what I need. I can easily add a character to my jpeg versions. I will need to work through your examples alongside iMatch help to make sure I understand those expressions.

Thanks again for your help

[Mario]

It looks scare, but basically

\.(jpg|jpeg)$

means:

starting with a . (dot).
(Since the . has a special meaning in regexp, you need to use \. to actually mean .

(jpg|jpeg) means jpg or jpeg

$ means end of text

So => all file with either .jpg or .jpeg at the end of the file name.

The longer expressions at the beginning at the same, just with a wider variety of extensions.

The ^(_*{name})[+\-_]*[0-9|a-z]* is a copy & paste pattern used by all sample regexp for versioning.
It just handles variations in the master file name and version file name.
In cases the user appends extra characters or letters or + or - or _ to the version file name.

The simplest case is (assuming .raw as the master and .jpg for versions and otherwise identical file names):

Master: \.raw$

Version: \.jpg$

[Stefanjan]

I'm testing this in a test database with just one or two files so I can begin to understand and testing with versions without buddy files first.

I find if I have jpg version

JPG (buddy+version)
\.(jpg|jpeg)$
/^_*//
^{name}[+\-_]+.*\.(jpg|jpeg|dng|tif|tiff|afphoto|psd|heic)$

And
RAW (buddy+version)
\.(cr3|cr2|nef|rw2|raf|srw|arw)$
/^_*//
^(_*{name})[+\-_]*[0-9|a-z]*\.(jpg|jpeg|dng|tif|tiff|afphoto|psd|heic)$

Then while JPEG master works fine, Raw files don't, if I have jpeg version disabled, then these files all stack ok
testing\IMG_8128.CR2
testing\IMG_8128.dng
testing\IMG_8128.jpg
testing\IMG_8128.tif
testing\IMG_8128_DxOFinal.jpg

When I enable jpeg version, these files are not stacked
IMG_8128.CR2
IMG_8128_DxOFinal.jpg

Am I missing a trick or should I give up on trying to stack JPEG masters?

Further complicated because I have DNG masters and jpeg version. But probably OK as I don't think I'll ever use DNG versions.

[Stefanjan]

Also could you clarify:

To enable automatic stacking. Do I have to select all the files and tap T and then CTRL SHIFT T. Or is there a setting that can automatically turn on versioning so I just have to tap the toolbar icon to turn it off?

Also where do I specify rules for a visual proxy or do I have to set it manually every version set?

[Mario]

Stacking and versioning are different concepts.
A version set automatically defines a version stack, but you can also stack files without using versioning at all.
The keyboard shortcuts you describe in your post are for normal stacks, not version stacks.

File Relations: Stacking
File Relations: Versioning and especially Version Stacks
Visual Proxy
Visual and version proxies and visual proxies are managed on a per-rule basis.

Or do you mean Auto-Stacking?

Or is there a setting that can automatically turn on versioning

Not sure what you mean by that. Versioning is always on - if you have at least one version rule defined and activated.
Versioning is a much to complex concept to make it somehow happen with a toolbar button....??

[David_H]

Then while JPEG master works fine, Raw files don't, if I have jpeg version disabled, then these files all stack ok
testing\IMG_8128.CR2
testing\IMG_8128.dng
testing\IMG_8128.jpg
testing\IMG_8128.tif

Be aware that if you end up with the xmp file testing\IMG_8128.xmp in the same folder, it applies to ALL files of the same name (even if they are meant to be embedded, like jpg, dng, etc), which can have unexpected and usually unintended consequences.

[Jingo]

Further complicated because I have DNG masters and jpeg version. But probably OK as I don't think I'll ever use DNG versions.

As you begin using the program and determining how to manage your assets - I would take a step back and ask yourself HOW you envision using the database/DAM.  When I first started out, I had versioning, stacks, visual proxies, the works... and it worked - but it was confusing and some work to maintain. 

After using the program this way for about 2 years, I realized I hardly never went back to my RAW files... and didn't use/view many of the "alternately" processed RAW images I was creating from my RAW editor.  Nor did I ever use those email versions I had created for sharing with others, etc.

In the end, I simplified things to the point where I now only use JPG's within IMatch and rarely stack or have more than a single image developed from a RAW file.  If you need to use all the functionality to support your workflow, then use it because it does work once setup.  However, I was using it mainly because it was THERE and available... which to me, was a mistake.

Free advice.. 

[Mario]

KASAP - Keep as simple as possible. My mantra 

Manual stacking is a simple way to reduce the clutter when you look at a folder.

Not every user needs versioning. This is definite an advanced concept (it says so in the help, too!).
If if one does, keeping things simple is important.
Not mixing versions and masters in the same folder.
Not using the same file formats for both masters and versions - or keep them really separate and know what you are doing.
Not copying too much metadata around because you can.
Know what visual proxies and version stack proxies do.
Etc.

The IMatch help can only add so many tips and warnings.
And most users don't read more than absolutely required to get things working - somehow...

When one is new to IMatch, I always advise to work a couple of weeks with it first.
Not use all the complex features in the first week.
Like you said.

[sinus]

This workflow sounds easier but I've always been told not to repeatedly process a jpeg due to image degradation.

In theory yes.
But I made, some years ago, quite a lot of tests. I took in Photoshop the quality of 7 and saved and stored it 10 times (each time closed Photoshop) and then I compared it with the original jpg.

I could see only with several hundert percent some differences, specially in "flat" areas. But with 100 percent I did almost see no difference.
And the best: I made some A4-high-quality-prints: no difference was to see, simply not.

Since then I store my jpgs mostly with quality 7 and do not  mind, if I store it 3-4 times.

And nowadays I have still anyway my nef (raw)-files, just in case.
But to be honest, once stored a jpg from a nef, I must use the nef again very, very seldom.

Hence I personally do not care, even if some graphic-designer wants have tif-files. 
Why? Because the know from somewhere, tif-format is lossless, use it once or twice, thats it.
I am sure, most of the time they could have used jpgs, with no difference. 

[Stefanjan]

Be aware that if you end up with the xmp file testing\IMG_8128.xmp in the same folder, it applies to ALL files of the same name (even if they are meant to be embedded, like jpg, dng, etc), which can have unexpected and usually unintended consequences.

Thanks, hadn't thought about that. Will bear in mind

[Stefanjan]

Since then I store my jpgs mostly with quality 7 and do not  mind, if I store it 3-4 times.

Thanks, that's good to know.

[Stefanjan]

In the end, I simplified things to the point where I now only use JPG's within IMatch and rarely stack or have more than a single image developed from a RAW file.

Great advice and I think I will take it.

[Stefanjan]

KASAP - Keep as simple as possible. My mantry 

Retired now, but much of my working life spent in IT so I like to know how these things work.

Rereading and testing now have a much better understanding. So for the time being:

No versions
Think I need to use buddy relations as culling photos is ongoing.

Relations - Autostack files seems to be the way to go for me. Supplemented where necessary with manual stacking.
- Using Exif Timestamp worked fine for most of my files with time set to 0 seconds.
- Didn't work for my phone photos because the DNG & JPEG are apparently taken as separate photos, increasing the time is not reliable as 1 second sometimes stacks more than 2 images. I imagine there is a solution using variables and the first 18 characters of file name just for phone photos.
- XMP DocumentID didn't seem to work on any of my files.


I could not find any shortcut for autostack or a method of saving the parameters so I suppose this has to be done via menu or right mouse click.

Now understand difference between visual proxy and version proxy. Would be nice if the visual proxy could be automated like version proxy but....

[Mario]

Think I need to use buddy relations as culling photos is ongoing.

If you work with derivative images or your RAW workflow produces auxiliary files, I recommend to setup buddy file relations.
Only then IMatch can keep the master/original files and the other files together when you move, copy, rename or delete.

I could not find any shortcut for autostack

<Alt>+<C>, <L>, <A>

All menu commands have hotkeys assigned. The hotkey shows when you open the menu via the Alt key.

Would be nice if the visual proxy could be automated like version proxy but....

You set this option for the version you want to use to represent the collapsed stack. Or not.
Version stacks require a version relation to work.
Whether or not you dedicate a specific version to be used to represent the stack when collapsed is up to you.
This is actually fully automated. If you collapse the version stack, the visual will be used if it exists.

[Stefanjan]

Thanks for all the helpful responses. After doing further testing, I have changed my mind.

Will run with a simple version setup which probably only includes two files in each version stack. The original raw and processed jpeg.

I may occasionally add further jpegs e.g. competition entries where I want to record date of entry and result. These files will usually have a different file name and I will manually add to the stack.

As recommended I will try to avoid keeping jpegs used for other purposes such as printing, email etc.

I will work with this for a while and see how I get on.

Thanks again for your help.