Workflow with DXO (Imatch 2020 with PL 4)

[WebEngel]

Hi all,

I have been using Lightroom for a decade but have now switched to DXO.  I am still struggling to establish a clean workflow.  All related threads in this forum seem decades old (except one which did not cover my questions), so I am starting a new one.  Questions below indicated with emoticon  .

Here is where I am:

Raw+Jpg: I always shoot Raw+Jpg. In Imatch, Master is JPG, while the Raw file and DXO's DOP as well as any XMP files are buddies.  All have the same filename for simplicity reasons, so it is Test.Jpg, Test.ARW, Test.ARW.dop, and if need be Test.xmp.  All are in the same folder.

(Why is Jpg Master?  I want to view Jpgs in full-res for rating and for viewing. For 10% of my images, I immediately process the Raw file to create a better Jpg file, replacing the out-of-camera Jpg.  For another 20% I do that later, possibly half a year later.  For another 20% I keep the Raw in case but never process it. For 50% of my images, I delete the raw file after a few years without ever having used it.  So having the Jpg as master ensures I always use the same master file -- the raw file is only there for the Raw converter.  Also, I use Imatch as well on a laptop which only has the Jpgs but not the Raw files.)

Rate/Meta/Category: Step one of my workflow is to cull, rate and generate metadata (GPS and other stuff) and rename files.  That metadata then sits in Imatch.  I can write it to the Jpg or an XMP sidecar.  I also assign the jpg to a category.

Raw processing: When I decide a picture (Test.jpg) is worth processing (i.e. the 30% mentioned before), I go to the Media&Folders view in Imatch, disable the "Hide buddy files" filter, double-click on the corresponding raw file (Test.Arw).      (I am pretty sure this can be simplified in Imatch given how powerful it is, can anybody just let me know the name of the functionality to launch the raw file from the jpg file?  RTFM comments are fine).

Export and replace in Imatch: When I am done in DXO, I export the file, thereby replacing the master file in Imatch.  Unfortunately, DXO does not seem to pick the metadata up from the xmp sidecar.  So when DXO generates the new Jpg file, it does not contain the metadata (except rating perhaps).  Of course, Imatch is smart enough to realize this is the same pic, so all categories and metadata are still there.

Face recog: Only the face annotations in Imatch are wrong if the new file is cropped.  Is there a way to automate this  (algorithm: if file dimensions have changed and if the file had face annotations before, rerun face recognition)

set label based on new meta: When the new Jpg file is digested by Imatch, I manually set it to status "green".  Given the jpg has "DXO" in metadata field "Software", Imatch can probably do that automatically  (algorithm: if file has changed and if {File.MD.Exif::Main\305\Software\0} contains "DXO", set status to green  - what is the name of the Imatch functionality to do that?)

How to keep metadata in Jpg:  How do I get the metadata into the Jpg file so that people with whom I exchange pictures (or Android devices) can view it as well?  Should I do "Metadata write-back" from Imatch?  Or is there a way to tell DXO to use the metadata in the appropriate way and create the new Jpg with the metadata in the first place?  Certainly, DXO is far behind Imatch in terms of metadata, so it is probably best to not even try DXO with metadata.    In this case, does it make sense at all to have XMP files?

Updating metadata only after processing the raw file in DXO is not an option for me: I need to update metadata immediately after taking the pictures, because I may have forgotten details later.  But I need to be able to process raw files and generate new Jpgs much later, even years later.  And I need to be able to process files again even later ("damn, why did I crop the picture this way, this looks ugly").

As you can see, I do get along, it works somehow, but it surely can be improved.  What would you do differently?  How would you automate the manual tasks?  Again, pointing me to the name of the functionality or the help page is sufficient, I can easily work from there.

Thanks for any help.

martin

[JohnZeman]

I'm also using IMatch and DxO PhotoLab 4 plus I also throw Affinity Photo into the workflow.

Rather than trying to answer your questions I'll just tell you what my workflow is, perhaps doing that may answer some of your questions.

For starters my workflow is quite different than yours.
I only shoot raw, not raw+jpg, and in IMatch my raw files are my masters.

I use FastRawViewer to send the raw file to IMatch which then imports it.
Next using IMatch I rename the raw to a YYYY_MMDD_HHMMSS.SSS format, categorize it, and enter all of my metadata, which IMatch saves to XMP sidecars.

When I'm ready to process the raw image I use an IMatch favorite to send the selected raw file(s) to PhotoLab.

I do not enter or change any metadata in PhotoLab at all, I simply use PhotoLab as a raw processor.

Once I'm done processing the raw in PhotoLab, which saves its edits to a sidecar .dop file that sits in the folder right beside the raw and XMP sidecar file, I export the raw as a temporary 16 bit tiff.  Then I open the tiff in Affinity Photo and do my final optimization.

Finally, I export the optimized tiff as a hi-res JPG and save that to IMatch, also storing it in the same folder as the original raw file.  At this point the TIFF file can be discarded.

So in the folder I have the raw file, the XMP sidecar file, the .dop buddy file, and the optimized JPG, all sitting next to each other.

In IMatch I'm using file relations and stacking to group and keep those 4 files together.

The end result is the raw is the master but in IMatch I see the optimized JPG version of the raw instead of the original raw image while the JPG file itself is hidden inside the stack.  I can easily disable this if I want but I usually don't unless I have a good reason to.

Finally using file relations IMatch automatically propagates (copies) the metadata and categories from the raw to the jpg so essentially both the raw and the jpg have the same metadata and category assignments.  It's all automatic in IMatch once you get it set up.

This system works very well for me and hopefully it might give you some ideas or at least some food for thought.

[sinus]

Thanks for any help.

martin

Martin, first I wanted answer you, but then I created an own thread (in this forum-section) to not somehow hijack your thread.
But maybe it gives you also some ideas, in the end it is you who made me write something about a workflow again. 
Good luck anyway.

[akirot]

@JohnZeman:

Just curious:

Finally, I export the optimized tiff as a hi-res JPG and save that...

This "hires JPG" is it a 1:1 export or do you scale to a certain size?

[WebEngel]

Thanks, John

When I'm ready to process the raw image I use an IMatch favorite to send the selected raw file(s) to PhotoLab.

Are you referring to application favorites or External Tool Favorites (as described here: https://www.photools.com/help/imatch/#fav_basics.htm)

Finally using file relations IMatch automatically propagates (copies) the metadata and categories from the raw to the jpg so essentially both the raw and the jpg have the same metadata and category assignments.  It's all automatic in IMatch once you get it set up.

Are you saying that the Jpg has all metadata, so that if you use Windows to copy it to a new PC, you can see the metadata there?  And are you using the versioning commands to propagate metadata to the JPG version?

It seems that metadata handling is indeed easier if you make the Raw file the master.  Do you ever delete Raw files and keep the Jpgs?  If yes, does it work, or does it create problems?  This is the main use case which made me go for JPG=Master.

[JohnZeman]

@JohnZeman:

Just curious:

Finally, I export the optimized tiff as a hi-res JPG and save that...

This "hires JPG" is it a 1:1 export or do you scale to a certain size?

It's always a full sized 1:1 export at 85% quality.

The only time I need to reduce the size and quality from that is when I'm going to send a copy of the photo to facebook or someone else.  And when I do that I drag and drop the original raw file into the IMatch Image Batch Processor.  The Batch Processor doesn't reduce the raw, instead it makes a reduced sized copy of the JPG version.

[JohnZeman]

Are you referring to application favorites or External Tool Favorites (as described here: https://www.photools.com/help/imatch/#fav_basics.htm)

The External Tool Favorites.
A partial screenshot of my favorites panel is attached.

Are you saying that the Jpg has all metadata, so that if you use Windows to copy it to a new PC, you can see the metadata there?  And are you using the versioning commands to propagate metadata to the JPG version?

It seems that metadata handling is indeed easier if you make the Raw file the master.  Do you ever delete Raw files and keep the Jpgs?  If yes, does it work, or does it create problems?  This is the main use case which made me go for JPG=Master.

Yes to your first 2 questions, no to the last one.
When I decide to delete an image I delete the entire works, the master and the raw.  However that's pretty rare for me to do since I cull all of the photos before and during the import new images process.

I do not actually propagate all metadata from a master raw image to a JPG version, just the metadata and categories I want copied.  For example I do not copy image orientation from a master raw to a JPG version.

[jch2103]

Just a note: PhotoLab is pretty good about copying pertinent metadata from the raw file to the output jpg (i.e., things like Description, Author, location, keywords, etc.). So if you use IMatch to add metadata to the raw file, you should also see it in the PL output jpg (and tiff, etc., I believe).

My workflow is similar to John Z's. It helps keep things fairly simple.

[WebEngel]

Hello

It seems that the workflow is indeed easier if the raw file is the master.  Unless you delete raw files (and not the corresponding jpg), this seems the better option.

If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

If the raw file is the master, you delete the master
If the jpg file is the master, you replace the master with a new master (when you process the file in DXO).

Maybe i need to setup a test db to see which approach fits best.  Anybody else who tried to delete master jpg files?

martin

[DigPeter]

Hello
If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

Why do you delete raw files?  I think most people keep them on their ex-camera state, in case reprocessimg is needed and a safeguard against loss or corruption of the jpeg.

[Jingo]

Hello
If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

Why do you delete raw files?  I think most people keep them on their ex-camera state, in case reprocessimg is needed and a safeguard against loss or corruption of the jpeg.

To be completely honest - since I have gone back and re-edited a RAW file only about a handful of times over the years...  if space wasn't so cheap, I too might consider deleting the RAW files after conversion. 

My JPG's are backed up to 2 locations daily as is my IMatch db so there is no concern about corruption, etc... in the end, it really is just a choice about space, complexity of workflow and ultimate use of the photos.. for most amateurs, JPG's are simply enough and with camera and software technology at a place it is at, taking JPG only might become a real option in the near future.

[jch2103]

Hello
If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

Why do you delete raw files?  I think most people keep them on their ex-camera state, in case reprocessimg is needed and a safeguard against loss or corruption of the jpeg.

DxO 4 is a case in point of why it's good to keep the raw files: The new DeepPRIME feature makes such a difference in some of my old high_ISO/small sensor shots that I'm going back to reprocess a number of them.

[JohnZeman]

DxO 4 is a case in point of why it's good to keep the raw files: The new DeepPRIME feature makes such a difference in some of my old high_ISO/small sensor shots that I'm going back to reprocess a number of them.

I agree.
One of my favorite pastimes is going back in time to optimize and reprocess raw photos I'd taken several years before.

[DigPeter]

Hello
If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

Why do you delete raw files?  I think most people keep them on their ex-camera state, in case reprocessimg is needed and a safeguard against loss or corruption of the jpeg.

It is what works for each individual.  I keep my raws on separate media from processed jpegs, so there is an additional backup - belt and braces.

[Erik]

Hi all,

I have been using Lightroom for a decade but have now switched to DXO.  I am still struggling to establish a clean workflow.  All related threads in this forum seem decades old (except one which did not cover my questions), so I am starting a new one.  Questions below indicated with emoticon  .

Here is where I am:

Raw+Jpg: I always shoot Raw+Jpg. In Imatch, Master is JPG, while the Raw file and DXO's DOP as well as any XMP files are buddies.  All have the same filename for simplicity reasons, so it is Test.Jpg, Test.ARW, Test.ARW.dop, and if need be Test.xmp.  All are in the same folder.

(Why is Jpg Master?  I want to view Jpgs in full-res for rating and for viewing. For 10% of my images, I immediately process the Raw file to create a better Jpg file, replacing the out-of-camera Jpg.  For another 20% I do that later, possibly half a year later.  For another 20% I keep the Raw in case but never process it. For 50% of my images, I delete the raw file after a few years without ever having used it.  So having the Jpg as master ensures I always use the same master file -- the raw file is only there for the Raw converter.  Also, I use Imatch as well on a laptop which only has the Jpgs but not the Raw files.)

Rate/Meta/Category: Step one of my workflow is to cull, rate and generate metadata (GPS and other stuff) and rename files.  That metadata then sits in Imatch.  I can write it to the Jpg or an XMP sidecar.  I also assign the jpg to a category.

Raw processing: When I decide a picture (Test.jpg) is worth processing (i.e. the 30% mentioned before), I go to the Media&Folders view in Imatch, disable the "Hide buddy files" filter, double-click on the corresponding raw file (Test.Arw).      (I am pretty sure this can be simplified in Imatch given how powerful it is, can anybody just let me know the name of the functionality to launch the raw file from the jpg file?  RTFM comments are fine).

I should probably read through everyone's responses, but I read through your email and wanted to add some thoughts where I think you could be helped.  I use DXO albeit quite differently (mostly because I have RAW's as masters).

I get why you are using JPG as the master, but what I think I am confused is why you are using the RAW as a buddy and not as a version.  I think setting the RAW file as a version will allow the metadata (XMP) to link up between the JPG and RAW.  You would have to play with the settings, but what would happen is that you'd want to make sure the RAW file is set up to XMP record from the JPG master.  This is a bit counter-intuitive, but assuming the JPG already matches from the original process of the RAW file, it should work.  The RAW files may not have much of an XMP record to begin with, and by using RAW's as buddy files, they are not picking up an XMP record from the JPG files.

You'll want to test some of this out within IMatch and inspecting the XMP records for the RAW and JPG files.

Once you have this set up, you can use the version panel or just open a version set to find the RAW and double-click it.

Export and replace in Imatch: When I am done in DXO, I export the file, thereby replacing the master file in Imatch.  Unfortunately, DXO does not seem to pick the metadata up from the xmp sidecar.  So when DXO generates the new Jpg file, it does not contain the metadata (except rating perhaps).  Of course, Imatch is smart enough to realize this is the same pic, so all categories and metadata are still there.

I think fixing the version may help this, but I'm not sure.  You probably need to make sure DxO shows an XMP record for the RAW file (I think version 4 allows you to inspect the metadata).  You might also need to make sure that it saves metadata (all of it) to the exported file.  Ultimately, the issue would be a DxO issue unless your RAW file doesn't have the XMP record to begin with.

This part is hard for me because I use the RAW files as masters and I've not had the issue you have, and if the exported JPG doesn't have an XMP record the file will get one from IMatch detecting a new version

Face recog: Only the face annotations in Imatch are wrong if the new file is cropped.  Is there a way to automate this  (algorithm: if file dimensions have changed and if the file had face annotations before, rerun face recognition)

I'm not sure on this one.  Problem is that I think IMatch doesn't see your file as different aside from maybe metadata.  The only possibility is to watch for whether IMatch is picking up the file as changed, triggering an update, and maybe leverage that trigger in an app.  If you have a collection for updated files, you can probably also look to that.  Automating may be a question you'll want to ask in the general forum since it isn't necessarily DxO specific.

set label based on new meta: When the new Jpg file is digested by Imatch, I manually set it to status "green".  Given the jpg has "DXO" in metadata field "Software", Imatch can probably do that automatically  (algorithm: if file has changed and if {File.MD.Exif::Main\305\Software\0} contains "DXO", set status to green  - what is the name of the Imatch functionality to do that?)

I guess the question would be how stuck are you on using a label, or would you be ok to using a category, which might be easier and definitely contained to your database.  Labels end up part of the XMP record.  If you are open, it opens up some easy solutions.  For instance, you could use a data driven category based on the {File.MD.Exif::Main\305\Software\0} variable.  You could set a color for the data driven category specific to DXO in its properties.  You'll have to read the manual, but this would be automatic no matter what.  Otherwise, you might need an app that can conditionally set the XMP label field to green when the Software field is DXO and operates when files are updated.

How to keep metadata in Jpg:  How do I get the metadata into the Jpg file so that people with whom I exchange pictures (or Android devices) can view it as well?  Should I do "Metadata write-back" from Imatch?  Or is there a way to tell DXO to use the metadata in the appropriate way and create the new Jpg with the metadata in the first place?  Certainly, DXO is far behind Imatch in terms of metadata, so it is probably best to not even try DXO with metadata.    In this case, does it make sense at all to have XMP files?

Updating metadata only after processing the raw file in DXO is not an option for me: I need to update metadata immediately after taking the pictures, because I may have forgotten details later.  But I need to be able to process raw files and generate new Jpgs much later, even years later.  And I need to be able to process files again even later ("damn, why did I crop the picture this way, this looks ugly").

As you can see, I do get along, it works somehow, but it surely can be improved.  What would you do differently?  How would you automate the manual tasks?  Again, pointing me to the name of the functionality or the help page is sufficient, I can easily work from there.

Thanks for any help.

martin

I think this end discussion speaks to a need to understand how metadata works with the various file types.  JPG images really should only operate with XMP records embedded in them.  IMatch will not work right if it finds external XMP records AND internal XMP records.  The raw files on the other hand will likely only operate with XMP records.  And that is all fine.  IMatch will handle that well, and generally so will DxO. 

To share files with people so they can see the metadata, it needs to be in the JPG.  If that is the desire, I would use the metadata write back options in IMatch.  However, because you are using JPG files as masters but then exporting what amounts to new masters, you NEED DxO to export metadata otherwise you'll lose it.

This is actually the danger in the method you are using for cataloging your files.  While you are treating the JPG file as the master in IMatch you are actually using the RAW file as the master in DxO.  You are creating a chicken and egg scenario because the exported file becomes the master for the RAW file you generated it from.  What comes first?  You could lose data that way.  I'll post some thoughts regarding how I handle my workflow in a separate post.

[Erik]

Hello

It seems that the workflow is indeed easier if the raw file is the master.  Unless you delete raw files (and not the corresponding jpg), this seems the better option.

If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

If the raw file is the master, you delete the master
If the jpg file is the master, you replace the master with a new master (when you process the file in DXO).

Maybe i need to setup a test db to see which approach fits best.  Anybody else who tried to delete master jpg files?

martin

You'll have to see the end of my extensive response to your original post.  I think the process of making the jpg a master and then replacing it with a new master (with the RAW file in between) is something that is the most dangerous of situations in terms of metadata, especially if the metadata needs to propagate from one file to another. 

An alternative scenario in your case may be for the exported file to be an additional version (different file name).  You could create a new version setup (with the same JPG master) so that the JPG that is exported is the Visual for the version stack.  For instance you might have files like follows:

Original.JPG
Original.RAW
Original-dxo.JPG

Original.JPG would be the master for both sets and would contribute metadata to the RAW file and the exported JPG (as you set it up). 

Now what you can do is ultimately delete any of the files.  You want to be careful about deleting the full set, but deleting a master on its own doesn't necessarily hurt anything.  The versions won't lose their metadata because it only gets updated when the master file's is changed or edited (or you directly edit it).  Thus, you could remove the RAW file later on.

Now as mentioned a few posts back, the recent addition of the deepPRIME noise reduction in PL 4 is a good reason to keep RAW files, but that's just a matter of preference.

[Mario]

IMatch will not work right if it finds external XMP records AND internal XMP records.

By standards, JPEG files use embedded XMP metadata (in the file).
Also by standards, an XMP sidecar file belongs to all files with the same file name as the XMP file in the same folder.

IMatch is aware of this and merges XMP metadata from the external sidecar file with XMP metadata found embedded in the JPG file - by default considering the embedded XMP metadata to be more precious.

You can thank the other software companies for all the confusion.
Every RAW processor out there deals with embedded XMP data, XMP data in sidecar files and other important metadata like EXIF and GPS as they see fit. Nobody really cares that much.
Proper metadata management may be critical - but it does not make a good source for fancy ads or magazine and blog reviews.
And most users are unaware of all these potential problems until they switch their RAW processing software, "DAM" or platform...

[Erik]

Just to add a perspective, my workflow with PL 4 (And really DxO for a while) relies predominently on IMatch.

In IMatch, the RAW files are my masters.  If I shoot in RAW+, the jpgs become the visual proxy (and are their own version set as opposed to buddy).  All the DxO sidecars are buddy files in IMatch.

For each session in DxO, I create a collection in PL.  Usually, I'll use IMatch's pins to create the collection, which I then select all and drag over to PL.  I typically work on a set of files at once, and this allows me to work in PL without being confined to folders (or its extremely rudimentary library).  Once I have files selected, I usually close IMatch (to avoid indexing intermediate files).

I process my files and export the files as either TIFF or JPG (depending on whether I have further edits).  The files out of DxO are named in the format "Original-dxo" (The original part is more complicated, but the key is the -dxo).

The finished file will come back into IMatch as a new version to the original RAW.  In IMatch, I have categories for software used, so I'll quickly go through the newly imported files and the RAW files and categorize the files for their software.  The categories are color coded, so I quickly see what images have been processed.

Now the other thing I have set up is that my database goes back to a period when my originals were JPGs or TIFFs from cameras that could only do that at best (or scanners).  Consequently, the version setups I have in IMatch are such that if there is no RAW file, a JPG or TIFF can be a master.  This works because my versions always have a suffix tacked on with a "-xxx", and it would allow for a situation where a RAW file could be deleted if I wanted to while keeping its versions with one of the versions taking over as a master.  I rarely do this since hard drive space has gotten so cheap.

[Erik]

IMatch will not work right if it finds external XMP records AND internal XMP records.

By standards, JPEG files use embedded XMP metadata (in the file).
Also by standards, an XMP sidecar file belongs to all files with the same file name as the XMP file in the same folder.

IMatch is aware of this and merges XMP metadata from the external sidecar file with XMP metadata found embedded in the JPG file - by default considering the embedded XMP metadata to be more precious.

You can the other software companies for all the confusion.
Every RAW processor out there deals with embedded XMP data, XMP data in sidecar files and other important metadata like EXIF and GPS as they see fit. Nobody really cares that much.
Proper metadata management may be critical - but it does not make a good source for fancy ads or magazine and blog reviews.
And most users are unaware of all these potential problems until they switch their RAW processing software, "DAM" or platform...

I guess i shouldn't have said it wouldn't work right.  The user may get unexpected results.  In my experience when the situation accidentally occurred, IMatch overwrote the XMP record from one with the other and didn't merge it, so data was lost.  Granted that was many versions ago now (probably version 5 time frame).  It was not pleasant, but it was a limited situation and a case of these other companies deciding to create XMP data their own way. 

[Mario]

IMatch always merged embedded XMP and sidecar data when importing XMP data. The merge happens during ingest and the results are stored in the database. This is the metadata you see in the Metadata Panel. During ingest, the embedded XMP is merged 'on top' of the XMP from the sidecar. When you write-back, IMatch stores XMP data into the JPEG, as is is standard.

[Aubrey]

Once I'm done processing the raw in PhotoLab, which saves its edits to a sidecar .dop file that sits in the folder right beside the raw and XMP sidecar file, I export the raw as a temporary 16 bit tiff.  Then I open the tiff in Affinity Photo and do my final optimization.

John,
are you aware that you can go straight to Affinity from DxO PL4?
In DxO File|Export to Application...
Select: photo.exe from C:\Program Files\Affinity\Photo

Next time just export to application, it keeps the affinity photo info.

Aubrey.

[JohnZeman]

John,
are you aware that you can go straight to Affinity from DxO PL4?
In DxO File|Export to Application...
Select: photo.exe from C:\Program Files\Affinity\Photo

Next time just export to application, it keeps the affinity photo info.

Aubrey.

Thanks Aubrey.  Yes, I have explored that option and I would be doing it that way if my IMatch versions were in TIF format instead of JPG.

But exporting as a 16 bit TIF from PhotoLab directly to Affinity Photo puts a big unneeded intermediate TIF file in my IMatch database when all I really need there is the final JPG version of it.

If PhotoLab had a way I could export to application but specify a different output folder that is not in IMatch then I would be exporting to application.

[WebEngel]

PhotoLab is pretty good about copying pertinent metadata from the raw file to the output jpg (i.e., things like Description, Author, location, keywords, etc.). So if you use IMatch to add metadata to the raw file, you should also see it in the PL output jpg (and tiff, etc., I believe).

True.  I was missing some metadata, but it seems I was using inappropriate fields ({File.MD.XMP::tiff\Artist\Artist\0} does not get propagated, but other artist fields do.  Will need to research what to use).  In a short test today, I did not notice any other field missing.

Alright, this already solves one of my problems which is: How do I get the metadata into the Jpgs!

[WebEngel]

Hello
If raw files are deleted as part of the workflow, like in my case, it is a difficult story:

Why do you delete raw files?

Let me explain my workflow a bit more:
All images get a rating and a label.  Rating means quality (the better the photo and the more unique, the more stars).  Rating refers to the potential--an underexposed picture still may get a high rating, as will a picture that will only be good after cropping.  Label "blue" means "needs processing", e.g., because of a wrong exposure or wrong white balance.  "Pink" means image is technically correct and does not need processing.  Half of the pictures are neither pink nor blue as I will later decide whether I still need to process them.

If a picture has 1 star and label pink, I will never process the raw file. I simply dont have the time, and if the Jpg already suffices for me, why should I waste my time in DXO slightly tuning exposure and shadows for an anyway mediocre image?

I NEVER delete raw files for images with 4 or 5 stars.  In fact, just this week I re-processed some old high-ISO RAWs with the DXO's amazing DeepPrime.

Also I don't delete raw files with "blue" labels even if they have only one star, because I may process them some day in the future in order to significantly improve them.  Well, in reality, I hardly ever do that, and if the picture 5 years later still bores me, I delete the raw even if it could benefit from a bit more exposure.

Does this make sense to you guys?

Do you guys really process ALL of your pictures?  I mean I can understand if you don't delete Raw files because storage is so cheap.  Just like not deleting e-mails as long as they don't have attachments.

I do understand that pros probably need to process every picture, because you cannot deliver a technically incorrect pic to a client, and you don't keep pictures that you are not going to deliver.  Make sense.  But I am an amateur.  An image may be a nice memory, but I dont' care whether the highlights need a bit more contrast or not unless the picture is really good.

Hope it explains it.

[WebEngel]

I get why you are using JPG as the master, but what I think I am confused is why you are using the RAW as a buddy and not as a version.

That is probably a VERY good question.  I started with Imatch 15 years ago and configured it in a way it would work for me.  Maybe I though it would belong this way.  This included writing JPG metadata to XMP records, thereby leaving MWG-compliance--which sets up Mario everytime he hears it (kidding).

Maybe if I make the Raw file a version, I can update the Jpg and at the same time write the XMP record.

I will probably have to create a brand new test DB to test all of that.  Will keep you updated.

Thanks a lot for this suggestion.

martin

[Mario]

When you change metadata for a master, it is propagated to versions during write-back.
See Propagation

[jch2103]

Do you guys really process ALL of your pictures?  I mean I can understand if you don't delete Raw files because storage is so cheap.  Just like not deleting e-mails as long as they don't have attachments.

No. I use IMatch to select which images I want to process (based on initial Rating in IMatch), and often don't process all of those (I mark rejects in DxO before processing a set of images). Like others who have already discussed their workflow above, the RAW file is the master and the JPG output is a version.

When I first started using DxO many years ago, one of the key things that attracted me was the efficient workflow. For many images I just use a 'standard' preset, so those images can be processed without further changes as part of the batch of selected images. For others, most of my adjustments are relatively minor and often done in groups (e.g., apply ClearView Plus to a set of landscape images). In the end, it doesn't take that much time and avoids complicated issues of jpgs as masters, etc.

[JohnZeman]

Do you guys really process ALL of your pictures?

I do.  But then I'm not a pro and I'm retired so I have time to process each individual image.  In fact I enjoy taking my time to do it.

[sinus]

Do you guys really process ALL of your pictures?  I mean I can understand if you don't delete Raw files because storage is so cheap.  Just like not deleting e-mails as long as they don't have attachments.

No. ...

So do I.
I do not process all of my pictures, maybe, hmmm, 10-50% of them (depends on the event), and of course I delete the very bad images.
But I do not delete a RAW, what is not that bad to delete.
Storage is cheap, and IMatch handles my 300'000 files simply very good and quick enough. 

[WebEngel]

Couldn't get the interface done, so asking again?

The External Tool Favorites.

@John, are you just transferring one file over?  Or multiple?  I managed to get one but not multiple.

For each session in DxO, I create a collection in PL.  Usually, I'll use IMatch's pins to create the collection, which I then select all and drag over to PL.  I typically work on a set of files at once, and this allows me to work in PL without being confined to folders (or its extremely rudimentary library).

@Erik, are you referring to "New project" in PL?  That's the only thing I found.

If yes, have you managed to use a command?  I haven't.  All I got is drag&drop.  Which is a bit tricky because then I need to have the raw files selected.  With a command, I could use the external tool as described by John above, and click on the Jpg and still transfer the Raw to DxO.  Also, Drag&Drop requires me to resize my windows.  Not bad but not extremely comfortable.

Martin

[jch2103]

(This isn't a reply from Erik or John Z.)

I created a Favorite/Application item for PhotoLap (drag and drop the desktop icon into Favorite/Application). If you then drag and drop one or more image files from the IMatch File Window onto the Favorite, PhotoLab will open with a new External Selections project in PhotoLab. Very simple and straightforward. No need to create special tags/collections for the images in IMatch, unless you want to.

[JohnZeman]

Yep, that's the way I do it too.  I just drag the desktop shortcut created by PhotoLab during the program installation to my IMatch favorites.  With that I can use IMatch to send one or several images to PhotoLab.

Maybe you might want to check the properties of the shortcut you're using in your IMatch favorite?  Below is the Target config in the properties of mine.

Code Select Expand

"C:\Program Files\DxO\DxO PhotoLab 4\DxO.PhotoLab.exe" -mode=openwith

As an aside once I have sent the files to PhotoLab I'll usually assign those files to a PhotoLab Project.  That way I don't have to process all of those files in the current PhotoLab session, I can take my time and process them as I want.  (I usually delete the external selections in PhotoLab, I prefer the versatility of the Projects instead).

[WebEngel]

Sorry for the long delay and thanks for the support so far which helped a lot.  I did not forget about the topic but experimented with JPG=Master and MWG compliance for quite some time.  This way, JPG was the master, but I wrote metadata to the JPG file directly and propagated the data to the version (Raw), thereby creating an XMP file for DXO.  The only real benefit was to be MWG-compliant.

The Experiment with MWG compliance failed, I am going to end it, mainly for three reasons.

1 The chicken-and-egg thing brought up by Erik concerned me -- the metadata in the JPG translates into the xmp sidecar, digested by DXO, put into the new master JPG.  I fear this can lead to inconsistency some day.

2 I failed to exchange data with Geosetter: If I continue to let Geosetter write geodata into XMP, it is complicated to get it into the Imatch master (my Exiftool attempts failed, and even if they succeed, they would just be an additional step that can be forgotten).  If I let Geosetter write data to the Jpg, Imatch also does not pick it up (again, some Exiftool job necessary probably to copy between segments).  Oh, and I could not get used to using Imatch for Geodata; not that it did not work, but the entire workflow turned out to be much easier in Geosetter.

3 The Raw conversion workflow is slower it seems; before sending a stack to DXO, Imatch needs to update the Jpg and create/modify the XMP sidecar.

After all, there is not much direct benefit.

martin

[WebEngel]

After ending the experiment with JPG=Master AND metadata in JPG, I tried the Raw=Master variant.  After all, this seems to be the most widespread configuration in the Imatch community.

As expected, the biggest caveat is to delete the master Raw file.  Imatch does not support this directly it seems.  Even if I externally delete the Raw master, Imatch does not "forget" it after rescanning the folder but keeps it.

For me, the visual proxy did not work -- Imatch's viewer would show the Jpg in the filmstrip, but the large window space would be the Raw file.  This can probably be repaired somehow, but as long as my key use case--deleting the master--does not work, any effort would be a waste of time.

Unless anybody has some great ideas, I guess I am going to just end this experiment as well.

[Mario]

If I let Geosetter write data to the Jpg, Imatch also does not pick it up

Mabye GeoSetter updates only the EXIF and not the existing XMP? Or only the XMP but not the existing EXIF? I don't remember, sorry. You can check the metadata in the IMatch Metadata Analyst
Maybe you have not written back the file before? Maybe you have metadata protection enabled in IMatch?
Since IMatch has the Map Panel, track import, reverse geo-coding, area-based searching and even shown/taken coordinate support, all me needs (and I guess the needs of most users) are covered and the metadata workflow stays intact and "in" IMatch. Not used GeoSetter for quite some time. Not even installed it on my new development machine.

In general, using too many applications (with more or less standard compliance) to update metadata in files can lead to issues.

[sinus]

... After all, this seems to be the most widespread configuration in the Imatch community.

I would bet, that is true. It feels (for me) simply the most natural and logic.

As expected, the biggest caveat is to delete the master Raw file.  Imatch does not support this directly it seems.  Even if I externally delete the Raw master, Imatch does not "forget" it after rescanning the folder but keeps it.

I cannot follow. The most natural thing is, if you want delete the raw, simply delete it inside IMatch and you are done.

For me, the visual proxy did not work -- Imatch's viewer would show the Jpg in the filmstrip, but the large window space would be the Raw file.  This can probably be repaired somehow, but as long as my key use case--deleting the master--does not work, any effort would be a waste of time.

I do not understand this also. In the viewer you can simply decide, if you want look the visual proxy or not (keys T,X).
If you have e.g. 2 images side by side (L2), this is very cool.

[Mario]

Even if I externally delete the Raw master, Imatch does not "forget" it after rescanning the folder but keeps it.

IMatch removes off-line files only when you to a manual rescan of the containing folder (Shift+F5). This is a safety feature.
Otherwise, if you rename or move a file in an external application, IMatch would remove the file from the database, including all unwritten metadata, category assignments etc. This would be bad.
Since IMatch keeps off-line files, you can use the relocate command to tell IMatch the new file name or folder. Or just rescan the folder to remove files you have deleted elsewhere.

To delete a file in IMatch, select it and press <Ctrl>+<Del>. The <Del> key marks a file as rejected and puts it into a special collection. This serves as a convenient undo mechanism. You can delete all rejected files in the current scope or the entire database with one command later. Or un-reject files by pressing <Del> again.

[WebEngel]

For me, the visual proxy did not work -- Imatch's viewer would show the Jpg in the filmstrip, but the large window space would be the Raw file.  This can probably be repaired somehow,

In the viewer you can simply decide, if you want look the visual proxy or not (keys T,X).
If you have e.g. 2 images side by side (L2), this is very cool.

Thanks, that was easy (and indeed cool feature).  Did not know it since I did not have versions until recently.

[WebEngel]

As expected, the biggest caveat is to delete the master Raw file.  Imatch does not support this directly it seems.

I cannot follow. The most natural thing is, if you want delete the raw, simply delete it inside IMatch and you are done.

Well, if I simply delete the raw file, Imatch would of course also delete the Jpg which it considers to be the buddy.

The only workaround I could imagine is to:
1 Propagate metadata to versions (i.e. make sure all metadata is in the Jpg)
2 Rename the jpg in order to deliberately break up the buddy relationship
3 Delete the raw file (and the xmp which is then no longer needed)
4 Rename the Jpg again (i.e. undo the rename step 2 above)

However, this does not work.  The Imatch renamer refuses to rename buddy files, and if I manually rename the buddy file (which would be a pain for hundreds of files), Imatch sometimes still deletes the Jpg file when I delete the raw file.

[Mario]

If you make the JPG a buddy of your RAW files, IMatch must move, copy, rename and delete the buddy file when you move, copy, rename or delete the RAW.
This is the sole purpose of buddy files.

If you want to delete the RAW without deleting the buddy file, just temporarily disable (uncheck the box) the corresponding buddy file relation under Edit > Preferences > File Relations.
Then delete the masters. The buddy files are not touched.
Don't forget to re-enable the buddy relation afterwards.
IMatch gives you all the tools, bells and whistles.

TEST this before applying.

[sinus]

I use nefs and jpgs, but not as buddies, but as versions.
And then I can simply delete a master (nef) and the version (jpg) stays of course, like it was.

[WebEngel]

If you want to delete the RAW without deleting the buddy file, just temporarily disable (uncheck the box) the corresponding buddy file relation

That does indeed work.  And it will not confuse the sh*** out of Imatch?  I mean it will not change many more file relations and loose them permanently (even after re-enabling the buddy relation)?

My workflow is to tag all pictures where I want to delete the raw (reasons above in this thread).  Every once in a while, probably three times a year, I will then delete all those raw files.  In other words, I would only change the preferences 6 times a year.

If that continues to work, it could indeed be my buddy+relation setup.

martin

[WebEngel]

I use nefs and jpgs, but not as buddies, but as versions.
And then I can simply delete a master (nef) and the version (jpg) stays of course, like it was.

Right, but I need the buddy relation as well to rename files, to move files and to make sure that for bad pics both the Raw and the Jpg are deleted.

[Mario]

And it will not confuse the sh*** out of Imatch?

No. Buddy file relations are always dynamically evaluated - because some of the buddy files may not even be indexed in the database.

In general, keeping things simple is much preferred.

Reverse version relations (JPG as master), the need to disable buddy file relations in order to delete RAW files but keeping the buddy etc. are all signs of a too complicated workflow.
You probably have your reasons. But KISS is always best.

[WebEngel]

No. Buddy file relations are always dynamically evaluated - because some of the buddy files may not even be indexed in the database.

That seems to be true, however, Imatch dynamically recreates the buddy and version relations based upon some triggers.  One trigger of course is "Refresh Relations".  However, other triggers I found are "rename 1 file" which causes Imatch to refresh all relations in the current folder.  There must be other triggers as well, possibly any file change in a folder causes a refresh on the entire folder.

I had hoped to retain two areas in my database:
1) Folders where Raw is Master (all new files)
2) Folders where Jpg is Master (all older files unless I find a reason to change that)

That does not seem to work, as I have no control when Imatch refreshes relations.  So suddenly I will find my old images with Raw=Master

I guess the only way to switch from JPG= Master to Raw=Master is to do a mass migration for all of my 100k pictures as follows:

1 Make Jpg =  Master again, Raw is buddy and version
2 Propagate all categories, metadata, flags from Jpg-Master to Raw buddy
3 Create face annotations for all raw files
4 Make Raw = Master
5 Remove face annotations from the Jpgs

Does this make sense to you guys?

martin

[bekesizl]

It is a similar problem I ran into my "older" JPG only photos and newer RAW+JPG photos.


Under Preferences - File relations you can limit the folders where some relations should be applied, see attached picture of my settings.


I assume your photos from case #1 and #2 are in different folders.
In this case, you can define a set of rules for your case #1 and #2 and IMatch will maintiain those relations automatically.

[JohnZeman]

I guess the only way to switch from JPG= Master to Raw=Master is to do a mass migration for all of my 100k pictures as follows:

1 Make Jpg =  Master again, Raw is buddy and version
2 Propagate all categories, metadata, flags from Jpg-Master to Raw buddy
3 Create face annotations for all raw files
4 Make Raw = Master
5 Remove face annotations from the Jpgs

Does this make sense to you guys?

martin

I'm having trouble following your proposed workflow but I'll make one suggestion followed by a comment.

The suggestion is don't do a mass migration.
At least not at first until you are 100% sure you are getting the results you want.  Process a few image pairs first, that way it'll be easy to revert your changes back if you run into unexpected problems.

The comment is in regards to my workflow again and your "older" JPG masters.
While I've been shooting raw since 2005 I have several old images in my database that were taken before that year when I was shooting in JPG format.  A year ago when I decided to use masters with versions in IMatch I needed to do something with my old JPG original images to differentiate them from the JPG versions.  So I converted all of my old JPG masters to TIF format which solved the problem.

Currently my IMatch master file types are CR2, DNG, and TIF.
All versions in my database are in JPG format.

Good luck!