Workflow: How to identify/tag images processed by DXO

[WebEngel]

Hi
I shoot Raw+Jpg.  Imatch has the raw file as the master and shows the visual proxy, i.e. the file in the window is *.ARW.

I use DXO (PL4 at present, refusing to upgrade before they give me keyboard shorcuts) to process raw files.  I never process all raw files at once, so in any folder, there are either all Jpgs unprocessed, some processed and some not, or none processed.

The files processed by DXO have something like "DxO PhotoLab 4.3.3" in the "Software" and "Creator Tool" tag, the ones not processed have something like "ILCE-7M3 v3.10" in the same tags.  Also, for the processed jpgs, the file timestamp is much newer than its master's timestamp.

How can I tell in Imatch for which Raw files the jpg has been processed in DXO?  I would like to label the raw files "green", so that I immediately know which files possibly need processing.  DXO seems unable to label images, which makes the workflow in DXO tough already. Even if it labeled the output Jpg green, this would still need to be propagated to the Raw.

The easiest algorithm I can come up with is to tell Imatch to auto-label the Jpg green if the Creator tool contains the string "DXO", and then auto-propagate this change to the raw master.  No clue how to quickly implement this, however.

Anybody have a good idea for this workflow? 

Thanks
martin

[thrinn]

Sounds tricky. If you just overwrite the out-of-camera JPG with the version developed in DXO, the only option I see is Using Variables for a special versioning rule.  Combine it with the Color option for your version rule to have a visual clue that a given ARW has a developed JPG as version. Unfortunately, variables can only be used to identify a version, so you might have to trigger a relations update manually (from the ARW) after exporting the developed JPG.

However, if the JPGs exported from DXO are identifiable by name (e.g. if they have a special suffix), then you could define a special version rule which recognizes the developed JPG as a "master" for the original ARW. With this rule, you could propagate whatever you want back to the ARW. But you can not propagate what is not there: If the developed JPG does not have a label, you can not propagate a label. Again, you could use the Color option to have a visual clue that there exists a developed JPG - but the result is much less "visible" than a label color, in my opinion. 

[jch2103]

DxO PhotoLab user here also (but now at current version). If I understand your workflow correctly, you shoot RAW + JPG, and then use PL to process selected raw images to JPG. I'm not clear if you overwrite the original JPG images when you process raw to JPG. You've discovered that the "Software" and "Creator Tool" tags tell you which JPG images have been processed by PL. For the JPG images, you could use a Category color bar https://www.photools.com/help/imatch/fw_basics.htm?dl=hid-22 that's keyed to the Software tag. However, that wouldn't solve the problem of telling which raw image has been processed.

Code Select Expand

The easiest algorithm I can come up with is to tell Imatch to auto-label the Jpg green if the Creator tool contains the string "DXO", and then auto-propagate this change to the raw master. 

That should work, but I'm not an expert in propagation.

My workflow, though, is a bit different: 1) to only shoot raw and 2) to Rate my raw images in IM. I've set up PL as a Favorite Application IM, so I can select a group of images in IM and use the Favorite to open them in PL. I'm not sure about your version of PL, but for some time PL has been able to open these images as External Selections (even if they're in multiple folders). I keep both raw and processed images in the same folder but have them set up with Versions https://www.photools.com/help/imatch/rel_versions.htm?dl=hid-3 \.

Code Select Expand

Master expression: \.(nef|nrw)$
Replacement expression: /^_*//
Link expression: ^(_*{name})[+\-_]*[0-9|a-z]*[_|\-|0-9|a-z| ]*\.(jpg|jpeg|psd|tif|tiff|dng)$

That makes it fairly easy to tell which raw images have been processed. Not sure if this will fit your workflow, though.

[Mario]

How can I tell in Imatch for which Raw files the jpg has been processed in DXO? 

Maybe just create a data-driven category based on the Creator Tool or Software tag and color-code the child categories you're interested it? The files for that CreatorTool value then show the color in category color bar File Windows.

Tip: Use the part of value option to only look at the start of the value. Usually CreatorTool values include version numbers at the end, which produces new categories for each version. If you use only at the first few characters, you'll get child categories like DXO and ILC, which is perfectly fine for this task and should not vary over time. Color-code the DXO and then you see a color for each file with that CreatorTool entry.

[WebEngel]

Yes, I overwrite the ooc jpgs with the ones processed by DXO.  They have the same name, only the extension is uppercase and lowercase (which may be different with the next PL release or the next camera, so I don't want an algorithm to rely on it).

Thanks, Thorsten and John:

You both suggest the versioning.  But the ooc Jpg is already a version of the raw file.  How can I then tell the two versions apart?  Switching the master-version relationship for processed files also sounds like a disturbing change in the entire workflow.

Propagating the label is also not easy, as normally of course I want to propagate the label from the master to the version.

Yes, I could put the jpg into a data-driven category, but this will only solve half of the problem

Maybe the easiest is an Exiftool command script manually triggered.  This is the only one-step solution that I can imagine (if Jpg's creator tool contains "DXO" set the XMP's label to green).  But boy, this will take me hours to code, as I am not used to this stuff.

Thanks, Mario:

The data-driven categories can certainly help to color code the jpg buddy.  But how do I then label the master Raw file?

martin

[Mario]

But how do I then label the master Raw file?

No idea. I have only looked at the part where you wanted to visually identify the  JPG files which are still original (or the ones which have been created by your RAW processor). I assumed that you keep the RAW and JPG in the same folder, so you can tell which RAW files have been processed by the color of the JPG next to it (assuming your sort profile sorts them together).

When I understand your problem correctly, it's: Identifying JPEG files containing a specific value in a specific tag and then finding the corresponding RAW file somehow and assigning a label to it.

That's something a custom app could do easily. Or a script written in PowerShell or Python or whatever.
Request data about the files in a folder, find JPEG files, fetch metadata for these files, if a processed JPEG file is found, find the corresponding master and assign a label.
Add a favorite for the app/script and run it when needed with a single click. Running the app multiple times does no harm (while progressing with the editing). That's how I would do it.

But when it comes to something like that in my case (and I speak for myself here), I usually give my workflow a deep think and wonder why I need something so complex. Simple is always better.

I never process all raw files at once, so in any folder, there are either all Jpgs unprocessed, some processed and some not, or none processed.

Why not just move processed files in a "processed" sub-folder? That's what I do.
It allows me immediately to tell which files are finished and which need work - in IMatch, Windows Explorer and other software.

[WebEngel]

I assumed that you keep the RAW and JPG in the same folder, so you can tell which RAW files have been processed by the color of the JPG next to it (assuming your sort profile sorts them together).

They are in the same folder.  But I always "Hide Buddies" -- otherwise I see every picture twice when going thru the images.

When I understand your problem correctly, it's: Identifying JPEG files containing a specific value in a specific tag and then finding the corresponding RAW file somehow and assigning a label to it.

Correct!

But when it comes to something like that in my case (and I speak for myself here), I usually give my workflow a deep think and wonder why I need something so complex. Simple is always better.

Agreed. But I have no clue how a simpler workflow looks like.

Why not just move processed files in a "processed" sub-folder? That's what I do.
It allows me immediately to tell which files are finished and which need work - in IMatch, Windows Explorer and other software.

How does it help?  Am I working really in a completely different way than anybody else?

I always work with "Hide buddies".  All my work is with the Raw master files (that's what you guys in the forum suggested two years back, an this is why I changed the master to the raw--before I used the jpg as master).  So whenever I view files, change metadata, rate files, send files to DXO, batch process files and so on, I use the Raw file.

How then would having the jpgs in a different folder help?  I don't use jpgs at all (Imatch uses them of course for for viewing and batch processing as a visual proxy).

For example, I am viewing a person or an event in Imatch.  Of course I don't want duplicates, so either master or version, but not both.  I may then change the rating or add more metadata.  This is why I work with the raw master.  But I may also decide to send pictures to friends or a print service, in which case I would like to know whether they are processed or not.  This is where I need the processing status.  Assume the Imatch window shows me 20 pictures, so then I would like to see that 5 out of them have been processed and the others not.

Any recommendation for a simpler workflow is appreciated.

Martin

[Jingo]

I use DxO as well and also sometimes shoot RAW+JPG.. but I get around this whole process by:

• loading my images from the camera into a side drive/folder (P:\photos\<year>\<Month-Year>) in Photo Mechanic (rename files with metadata - auto add to correct month/year folders).  
• I then process the RAW files in DXO and export high resolution JPGS to a new temp folder (F:\IMatch Exports).  
• I then, copy all JPG's from the side/drive (P) to the temp folder (F) and choose NOT to overwrite - this moves in any JPG's OOC that I didn't bother to fix in RAW.
• Once complete, I move all images from F to my main IMatch photo drive (I:\photos\<year>\<Month-Year>).... which IMatch monitors and includes the new files.

In this way, only finalized images are in my IMatch database, it is fast and easy to find an original RAW in DXO if I ever want to edit the photo again (same folder structure - just different drive) and I don't worry about versions of JPG's, syncing, etc.

Hope this is helpful! - Andy.

[WebEngel]

In this way, only finalized images are in my IMatch database,

Thanks for the comment, but this means you don't utilize Imatch at all for what is in my eyes a critical part of the overall workflow and a key asset of Imatch.  I am still looking for a Workflow WITH Imatch.

Also, I never find the time to process the images immediately, mostly only when needed, which may be after months or so.  With your workflow, the Imatch DB would lack all recent pictures.  Finally, I need to enter metadata immediately after processing, because I will forget the details otherwise.

[sinus]

I guess, I cannot really help here.
But here is roughly my workflow, maybe it can give you a hint, if not, well, then not

My raws (nef) are identified with

_a at the end of the filename, when they comes from the sd-card and are renamed with the renamer.

Hence all my raws with _a are not edited. If I shoot 500 photos, there is (for me) no need all of them.

I do the choice, what files I want use, in the viewer (IMatch of course, I work only with IMatch and photoshop).

All files, what I want edit, I assign in he viewer a red pin.
After closing the viewer, I go to the categories and there I have several cats. I go the the cat with the photos I want edit.

Now I can see all the photos. Now I do (well, I know, some likes this not, but for me it worked perfectly since ages) a renamaing from these photos, change the _a to _m.
Means, this raw-file is now a master. (or will be soon).

Also assigned with the renaming will be a green pin, means, this fie is editeda and has a version (soon).

Then I edit all these files (in the same folder) in photoshop and create versions with appended _v1.
IMatch identify them as versions, and I do a propagation.

Now I have one master (raw) with _m at the end and a version with _m_v1 at the end.
Both master and version has a red and green pin.
The master has also a label called "edited".

I switch to another cat and see only the versions. I select all and with a favourites I assign a label "version", and a blue pin.

If I want I do again finetune them in Photoshop.
Now I go to the next cat, where I see only the versions with red, green and blue pin.
Means, these files I will send to the client (or private).

Here I decide often, to send not some pics, then I unassign the blue pin (and they vanishes from this cat).
Finally I send all these files with the blue pins out.

All these files are in the same folder.
Not touched pics have a _a at the end and a reddish label "not edited" or so.

All masters have _m at the end and also a red and green pin and another label.

And the versions have _m_a at the end, are propagated.

And all really used (delivered) files has a blue pin.

All of these images I do stack and I end with one file in the file window (with the 500 images stacked) and I have not problems to find all masters, or find all versions or unused images.

This is basically my workflow and works finally since a long time. Of course I made over the time here and there some changes, but basically this works in the way I described.

[Tveloso]

Sounds tricky. If you just overwrite the out-of-camera JPG with the version developed in DXO, the only option I see is Using Variables for a special versioning rule.  Combine it with the Color option for your version rule to have a visual clue that a given ARW has a developed JPG as version. Unfortunately, variables can only be used to identify a version, so you might have to trigger a relations update manually (from the ARW) after exporting the developed JPG.

This sounds like an excellent idea!...and it will fit your existing workflow.  So you would just need to update the Version Relation that you currently have, to make use of the Limit by Variables functionality, and use say, the "Regular Expression <>" option, with the literal "PhotoLab" in the Master Expression, and the variable for the creator tool tag in the Version Expression.  And maybe don't set the color option on this rule.

Then clone that rule to another name, and change the Limit by Variables option, to "Regular Expression =", and set the color option on this rule.

So the first rule will create a version pair for your ooc JPEGs, and the second for the processed ones - and it will "tag" the Master icon on the thumbnail with the color you selected, so you can see instantly which Masters have Processed JPEGs. and which ones don't.

As Thorsten says, you may need to refresh relations in the folder you just stored the new processed JPEG into, but I don't think adding that step will be very impactful.  And it may even be that, IMatch does this automatically, as a byproduct of re-loading the updated JPEG?

[RobiWan]

Anybody have a good idea for this workflow? 

Hi maybe I do not understand all what you are trying to do.
Sometimes I'm using DxO PL 6 - and there may be a difference to your version, but if not - I mean the simplest way is to show if there is a *.dop file. For all edited RAW files DxO creates always *.dop file with all edits.

[jch2103]

One follow-up question about workflow: Is there actually an advantage for you in having both raw + jpg images from the camera? One (of many) advantages of using IM is being able to work with just raw images (before developing them w/ DxO or other program). Omitting the +.jpg images would simplify image handing and save a small amount of file space. But you may have good reasons for having both raw & jpg from the camera. 

[WebEngel]

But here is roughly my workflow, maybe it can give you a hint, if not, well, then not
renamaing from these photos, change the _a to _m.
Means, this raw-file is now a master. (or will be soon).

Renaming of course has the downside for the backup: You need twice as much backup space (at least if your backup method avoids duplicates)!

Then I edit all these files (in the same folder) in photoshop and create versions with appended _v1.
IMatch identify them as versions, and I do a propagation.

Here I see the following problem for me: In my case, the workflow would not be error-proof: If I marked the raw and set the pins but forgot to create the version, all would be out of sync.

The algorithm I was looking for would be more error-proof.  It should not matter whether I WANTED to edit a file but whether I actually DID edit it.

Don't get me wrong--this may work perfectly for you, and great ideas, but I doubt it would for me.

[WebEngel]

Anybody have a good idea for this workflow?

I mean the simplest way is to show if there is a *.dop file. For all edited RAW files DxO creates always *.dop file with all edits.

No, you understand it correctly, and this is another good idea.  Problem, however, is that DXO writes that DOP file before you export the file.  So if I started some editing in DXO and never finished, my pictures would wrongly show "Edited".

Also, I guess checking for the DOP is as complicated as checking for the "Creator Tool" tag.

[RobiWan]

So if I started some editing in DXO and never finished, my pictures would wrongly show "Edited".

Also, I guess checking for the DOP is as complicated as checking for the "Creator Tool" tag.

Exported file has entry in *.DOP file 

ProcessingStatus = 3

Edited but not exported file

ProcessingStatus = 1

[WebEngel]

Is there actually an advantage for you in having both raw + jpg images from the camera? One (of many) advantages of using IM is being able to work with just raw images (before developing them w/ DxO or other program).

Just with raw, can I view high-res at pixel level with sufficient speed (which is also needed for rating to spot the blurred ones)?  Lightroom did that...

But indeed, this would solve some of my problems as Imatch graphically shows which file has versions or buddies. and which not

[Mario]

Just with raw, can I view high-res at pixel level with sufficient speed

IMatch creates cache images for RAW files automatically. Not different from what Lr does.
Typical load time for a cache image is between 0.2 and 0.6 seconds, which is fairly quick. Especially since the Viewer and Quick View Panel preload images in advance.

I can zip though 64MP+ and even Red high-res images as fast as I can mouse-wheel in the Viewer (given that the images have been cached already and the cache folder is on a SSD). Even on my laptop.

I you have a different experience, create a debug log (log file) from an IMatch session where you used the Viewer and let me know. Include details about your graphic card and Windows version.

[stefanjan2]

I'm using the latest version of Photolab Elite.

My workflow is to load a selection of RAW files from imatch using a Photolab favourite.

In Photolab I normally set the filter to unprocessed so only shows unprocessed files.

I export the processed jpegs to a "Processed" subfolder off the original folder in imatch.

In imatch I label the RAW masters green (SHIFT 3) which propagates to the processed jpeg.

I can always find jpegs processed by Photolab because they are in a "Processed" sub folder.

While I shoot RAW on card 1 and jpeg to card 2 I almost never load the jpegs into imatch but if I did I imagine I would just have them in the master folder alongside the RAW master.

I drag all RAW files off the card into a temporary "import" folder where I cull the images.

I then use RENAMER to move the files to a folder in the format \YYYY\YYYY-MM-DD ¦ Project Name

[WebEngel]

In imatch I label the RAW masters green (SHIFT 3) which propagates to the processed jpeg.

How do you identify the Raws which you have processed in PL?  That is the missing link for me.
Or do you always process a full folder?  In which case of course it is easy.

[WebEngel]

Hmm, I gave "Raw only" a try.  Had to adjust the Cache settings to obtain Hi-res cache images.  But now it seems to work.

I remember that I started Raw+Jpg in the days when I heavily relied on FastPictureViewer for Rating files.  Since Sony introduced rating in-camera, I rate and delete pictures there -- or in Imatch (which got much faster).  Geosetter, the only other tool I use, still does not handle Sony Raw files well, but you can at least see enough to judge where the image was taken.

=> no need for the Jpgs anymore.

Filtering works with the "Hide Master" and "Hide Version" filters.  So I can easily tell which files were processed and which not.

The only other downside is: At times, I delete the raw files for images that are below average, to save space, and only keep the smaller Jpg. In order to continue this, I first need to process the files in DXO (with default settings, no tweaking).  Not a big problem...

"Raw only" looks promising, I will keep experimenting.  Thanks, John, for the hint.

[stefanjan2]

How do you identify the Raws which you have processed in PL?  That is the missing link for me.
Or do you always process a full folder?  In which case of course it is easy.

If I have a large number of photos in a folder, I would select a smaller batch at a time and click on Photolab favourite in imatch to send them to Photolab. Once I have processed and exported jpegs, I would switch back to imatch where the Raw files will still be selected and simply press SHIFT 3 to colour them green.

With a smaller folder I might load the whole folder but on returning to imatch it's easy to see which photos have been processed because they will have a 2 in the top left of the image and will also display the visual proxy from the jpeg.

[stefanjan2]

The only other downside is: At times, I delete the raw files for images that are below average, to save space, and only keep the smaller Jpg. In order to continue this, I first need to process the files in DXO (with default settings, no tweaking).  Not a big problem...

If you delete the RAW in imatch you will also delete the exported jpeg if you have versioning. In this case you will need to delete the Raw and buddy files in Windows Explorer.

Puzzled why you would want to keep the jpeg for images which are below average.

[WebEngel]

If you delete the RAW in imatch you will also delete the exported jpeg if you have versioning. In this case you will need to delete the Raw and buddy files in Windows Explorer.

True, but you can temporarily switch off the buddy relations.  Works like a charm.  Of coursse this makes only sense if you do this in a big "save disk space" session for an entire calendar year or so, not for 3 pics here and there

Puzzled why you would want to keep the jpeg for images which are below average.
 

They are ok to keep, but not good enough to spend time on processing.

If I have a large number of photos in a folder, I would select a smaller batch at a time and click on Photolab favourite in imatch to send them to Photolab. Once I have processed and exported jpegs, I would switch back to imatch where the Raw files will still be selected and simply press SHIFT 3 to colour them green.

This is exactly what I did for the last two years.  However, this is error-prone.  One wrong mouse click in Imatch, selection lost.  May work for you, but I do not work as predictable unfortunately

[stefanjan2]

True, but you can temporarily switch off the buddy relations.  Works like a charm.  Of coursse this makes only sense if you do this in a big "save disk space" session for an entire calendar year or so, not for 3 pics here and there

You are probably aware that in Photolab if you have the Raw+Jpg in the same folder and load the folder you will be able to delete the RAW files you want to get rid of. You can also hide the jpeg (or Raw files). You will of course need to rescan the folder in imatch after deleting.

[sinus]

If I have a large number of photos in a folder, I would select a smaller batch at a time and click on Photolab favourite in imatch to send them to Photolab. Once I have processed and exported jpegs, I would switch back to imatch where the Raw files will still be selected and simply press SHIFT 3 to colour them green.

This is exactly what I did for the last two years.  However, this is error-prone.  One wrong mouse click in Imatch, selection lost.  May work for you, but I do not work as predictable unfortunately

just a remark to your sentence here:
"One wrong mouse click in IMatch, selection lost."

Well, undo in IMatch (Ctrl z) and the selection is again here.

[Mario]

Well, undo in IMatch (Ctrl z) and the selection is again here.

Exactly. Undo is your friend.

[WebEngel]

Well, undo in IMatch (Ctrl z) and the selection is again here.

Awesome!  Thanks, did not know you could undo selections -- neverending positive surprises in Imatch!

Still, several wrong clicks in other directories cannot be undone I guess.  And also, it is not only losing the selection, it could be forgetting about the entire thing to find later a mess of images that I simply forgot to label.