Version detection clarification please

[herman]

Triggered by this topic I realized that there is one aspect of version detection I just don't get.

Maybe it is me, maybe a bug, maybe something for a feature request, or perhaps the help file needs to be more explicit.

I don't want to hijack the referenced topic, so I start a new one.

The "where to search" section of the File Relations dialog box has three choices:

• Entire database
• Master Folder (and sub-folders if specified)
• List of folders (and sub-folders if specified)

The "Entire Database" option seems clear: each and every file in the database can be a version.

The "Master Folder" seems clear as well: either master and version files are both in the same folder or the version is in a sub-folder of the master folder.

I am having troubles to understand the "List of folders" option.
The way I understand it, both from the help file and by logical thinking, is that the versions are supposed to be in the folder(s) specified here.
Consequently the masters are NOT supposed to be in the folder(s) specified here, otherwise this option would be similar to the "Master Folder" option.

This is however not how IMatch behaves.

As discussed in this topic as well as in Mantis 565 it is has been observed that Imatch can identify a file in the "List of folders" location as a master.

This happens only if master and version have exactly the same filename.

I know that using identical filenames is not considered a best practice, but that is beyond the point I am seeing here.

My question: am I misunderstanding the purpose of  the "List of folders" option?
Or is IMatch not behaving as designed / intended?

[BenAW]

Problem imo is that IM does not check (and usually CAN'T check) whether a new image is a version or not.
All images are processed as master through the relation definition(s).

Simple way to think about this is by using the link via EXIF timestamp.
Every image that has the same timestamp is related, regardless the filename or extension.
Adding a new image in a specified version folder will find all other versions of the same image.

Think of the master and link expressions as a kind of "filter" trying to separate masters from versions
(and link them of course)
As long as no clear indication exists whether an image is a version or not, ambiguities will exist.

A simple solution could be to force users to declare a folder (and its subfolders) that only contains versions.
Now IM can start with checking whether a new image is a version or not.
If yes, stop further processing, if no process it as a master and try finding versions in the specified folder.

[herman]

A simple solution could be to force users to declare a folder (and its subfolders) that only contains versions.

In my understanding that is the purpose of  the "List of folders" option.
If it is not intended this way, what is it's purpose?
What could it contain other than versions if we are telling IMatch where to look for versions?

[BenAW]

Point is that IM only LOOKS for versions in that folder.
Afaik it doesn't check if images are in that folder BEFORE processing them.
Thus a new image in this folder is still considered and treated as a master.

The list of folders is the situation were IM COULD know whether an image is a version or not.
With the other options this is not possible imo.

[herman]

The list of folders is the situation were IM COULD know whether an image is a version or not.
With the other options this is not possible imo.

It is also my understanding that this is not possible for the other options.

The question then becomes if IMatch could know or should know that the list of folders contains only versions.
If it could know this might qualify for a change request.
If it should know we have a bug at hand.

Only Mario can tell what the design requirement is for this option.

[DigPeter]

The "Master Folder" seems clear as well: either master and version files are both in the same folder or the version is in a sub-folder of the master folder.

This is an outstanding query that I have from the other thread.  How does IM know that a folder is a master folder if it is not specified, because I can find no way of specifying it?  If Master folder is selected, there is an option Specified folder.  What does that mean?

I do realise the Mario has other fish to fry, but I agree that he does need to look at this.

[BenAW]

This is an outstanding query that I have from the other thread.  How does IM know that a folder is a master folder if it is not specified, because I can find no way of specifying it?  If Master folder is selected, there is an option Specified folder.  What does that mean?

The Master folder in this case is the folder the image currently being evaluated is in.
By selecting "Specified Folder only" you tell IM to look ONLY in that specific folder.

I do realise the Mario has other fish to fry, but I agree that he does need to look at this.

This behaviour has been build into IM since a long time. I see no pressing reason to change the versioning setup in this late stage of the development.

[Ferdinand]

My understanding is that there are no such things as master folders.  The master concept applies to files.   A file is a master if IMatch can find other files that are versions of it under one of the versioning rules.  If there are several files that could be regarded as a master under a rule, only the first one found is treated as a master, which is the problem in question.

I remain unconvinced that there's a problem, although I am prepared to be convinced.  In the other thread I proposed a solution to Peter's problem, then I found a problem with that solution, and I think I solved that as well.  Have I missed something?  Was I just lucky about the order that folders and files were processed in searching for masters and versions?

[cytochrome]

I have always assumed it works like indicated by Ben. To verify I made a simple test:

- went to folder with NEF

- exited to ASP, converted one NEF. ASP is set to spit out the jpg in an /ASP subfolder to the NEF folder

- returned to IM5, did a rescan of NEF folder, the ASP subfolder with one JPG appeared, the JPG had all the metadata from the NEF

My NEF version setting for this is minimal: nothing, just default: where to search : Master folder and Direction: two  levels down.

To me the panel means: Where to search for versions?? Search starting in the folder you are and search also 2 levels of subfolders.

Francis

[DigPeter]

My understanding is that there are no such things as master folders.  The master concept applies to files.   A file is a master if IMatch can find other files that are versions of it under one of the versioning rules.  If there are several files that could be regarded as a master under a rule, only the first one found is treated as a master, which is the problem in question.

I remain unconvinced that there's a problem, although I am prepared to be convinced.  In the other thread I proposed a solution to Peter's problem, then I found a problem with that solution, and I think I solved that as well.  Have I missed something?  Was I just lucky about the order that folders and files were processed in searching for masters and versions?

@Ferdinand
Still not at my main computeer, so am unable to check your method.  I am not so much saying thing do not work, but seeking clarification.  But I get inconsistent results.  One relation definition works in one datyabase, but not in another, where the folder structure is the same!

[herman]

In the other thread I proposed a solution to Peter's problem, then I found a problem with that solution, and I think I solved that as well.  Have I missed something?  Was I just lucky about the order that folders and files were processed in searching for masters and versions?

In the other thread you could successfully use folder patterns.
That was possible because the "Originals" and "Processed" folder tree have an identical structure.

My folder structure is different, the subfolders in the "Version" tree have no equivalents in the "Masters" folder tree.

Therefore I don't use folder patterns, I simply refer to the "Versions" folder structure where the versions are located (see attached screenshot for a simplified model of the folder structure).

When I use automatic versioning (preferences - background processing - file relations - automatically refresh relations) it may happen that a file dropped in the version folder structure is identified as "master".

My point is that I assign a specific folder where IMatch can search for versions.
It defeats my logic that a file in this location, where only versions are to be expected, can be marked as "master".

The question therefore is whether this behavior is as intended.
We may have a bug when this behavior is not as intended.
When this behavior is as intended I would not mind to see it changed.
Not necessarily with high priority, as there are work-arounds available and Mario has probably more important things to do.

[attachment deleted by admin]

[BenAW]

It defeats my logic that a file in this location, where only versions are to be expected, can be marked as "master".

Your logical error is in assuming that only versions will be found in that folder.
Your Versions folder is the place where IM will LOOK for versions. Nothing precludes other users from also storing masters in that folder.
Eg.
My images
- Single images
- Masters and versions

You can now make the Specified folder the "Masters and Versions"

If eg your masters are always raw images and the versions always jpg's this can work without a problem.

So unless you specifically force users to accept that images in a specified Versions folder will always be treated as version, this is no bug.
It is designed into the current versioning setup.

[herman]

Your Versions folder is the place where IM will LOOK for versions. Nothing precludes other users from also storing masters in that folder.
Eg.
My images
- Single images
- Masters and versions

You can now make the Specified folder the "Masters and Versions"

I see your point.
As valid as it is, in a case like this I would rather select "Master Folder" in stead of "List of folders" in the "Where to search".

[BenAW]

in a case like this I would rather select "Master Folder" in stead of "List of folders" in the "Where to search"

Not sure I understand what you're saying.

As long as we have no fool proof way to decide whether an image is a version or not, ambiguities will exist.
Fool proof way can be:
- declare one or more directories that ONLY contain versions
- all raw images are always a master, and all other (jpg) images are always a version
- all versions have something specific in their filename (eg "_v" )
- ??

Point is that IM has to be able to decide whether an image is a version or not BEFORE processing it.
When the image is a version: process next one.

[herman]

in a case like this I would rather select "Master Folder" in stead of "List of folders" in the "Where to search"

Not sure I understand what you're saying.

Your example was:

My images
- Single images
- Masters and versions

I am saying that, when masters and versions share a folder, I would select "Master folder" as the place where to look for versions.

My question is about the "List of folders" as the place where to look for versions.
My understanding / logic / gut feeling is that, when one selects this option, it is highly unlikely that masters can be found in the same location as the versions.
This is because there is already an option designed to deal with the situation where masters and versions live in the same folder.
When I speak for my own setup it is impossible that a master can be found in the "List of folders", as I have a very strict separation between my original files and the processed ones.

As long as we have no fool proof way to decide whether an image is a version or not, ambiguities will exist.
Fool proof way can be:
- declare one or more directories that ONLY contain versions

Exactly.
Again, my understanding / logic / gut feeling tells me that this is the purpose of the "List of folders" option.

We know that IMatch as it is now does indeed also look for masters in the "List of folders" location.
I am wondering whether that is as intended or that this is a bug.
Only Mario knows this, as he defined the requirements for version detection.

[BenAW]

This is what the Help has to say:

Where to Search
In this section of the dialog box you specify where IMatch is looking for buddy files and versions. IMatch can search only the folder containing the master file, include sub-folders of that folder in its search (one or more levels deep) and even look in selected folders or the entire database.

Afaik nowhere is mentioned that you should ONLY have versions in the specified folders.
This makes sense if you want to accommodate as much different workflows as possible.
But you will also have lots of opportunities for strange effects and unexpected results.

I do basically the same as you do: ALL my versions are in a single folder.
I don't expect IM to treat that folder as versions only, I filter it out from the masters when manually refreshing relations.
This is why the automatic refresh screws up my version relations.

[Ferdinand]

In the other thread I proposed a solution to Peter's problem, then I found a problem with that solution, and I think I solved that as well.  Have I missed something?  Was I just lucky about the order that folders and files were processed in searching for masters and versions?

In the other thread you could successfully use folder patterns.
That was possible because the "Originals" and "Processed" folder tree have an identical structure.

My folder structure is different, the subfolders in the "Version" tree have no equivalents in the "Masters" folder tree.

Therefore I don't use folder patterns, I simply refer to the "Versions" folder structure where the versions are located (see attached screenshot for a simplified model of the folder structure).

This problem interests me since I may have had it if I hadn't made some changes to file names and folder structures some years ago.  And for that reason I was partly involved in getting the regex option introduced.   But I admit it's only of academic interest now.

I tried to replicate your structure and I still got it to work.  In my tests if the versions were all in folders covered by the list of folders then it worked. 

I did find that when ingesting new versions, the new files had wrong version relations (edit: they're both masters and versions).  But a F4,R on the entire DB fixed that.  I guess that is a little tedious each time you ingest.

I suppose I may still be getting lucky about the order images are processed during the refresh.  I tried to get IMatch to refresh in different orders, but it always worked for me, apart from immediately after a new ingest of versions.

[Ferdinand]

My question is about the "List of folders" as the place where to look for versions.
My understanding / logic / gut feeling is that, when one selects this option, it is highly unlikely that masters can be found in the same location as the versions.

I recall Mario stating in the past that by design it is possible for a file to be both a master and a version.  I have seen that, and it is in theory possible with the right (i.e. complex) versioning definitions.  I recall that he wanted to allow for that, although I can't image when I'd use it myself.  If my recollection is correct, then he isn't going to agree to exclude IMatch searching for masters in the list of folders for versions.

[cytochrome]

I had files being both master and versions (blue and orange squares) after defining JPGs as masters (to copy metadata from JPG to raw) and then trying to redefine the RAWs as masters.

To get rid of double definition (if unwanted) uncheck all master and version definitions for this file format, do F4,R check "normal" versioning, F4,R and that's it.

Francis

[BenAW]

My images
- Single images
- Masters and versions

I am saying that, when masters and versions share a folder, I would select "Master folder" as the place where to look for versions.

You still have the same ambiguity as in the Selected folder case. A version in this Master folder will also be evaluated, and can and will become a master if IM finds a matching image. Only in this case it is more difficult to filter out the versions only.

[herman]

I tried to replicate your structure and I still got it to work.  In my tests if the versions were all in folders covered by the list of folders then it worked. 

I did find that when ingesting new versions, the new files had wrong version relations (edit: they're both masters and versions).  But a F4,R on the entire DB fixed that.  I guess that is a little tedious each time you ingest.

Thank you for confirming.
It was also one of my conclusions that version identification will be correct following a forced relation refresh of either the entire database or the masters involved.
It is just the automatic version detection that gets confused in some cases.

I recall Mario stating in the past that by design it is possible for a file to be both a master and a version.  I have seen that, and it is in theory possible with the right (i.e. complex) versioning definitions.  I recall that he wanted to allow for that, although I can't image when I'd use it myself.  If my recollection is correct, then he isn't going to agree to exclude IMatch searching for masters in the list of folders for versions.

Thank you for this bit of "historical" information.
If this is intended behavior I guess we just have to live with it.
I find it hard to imagine when it would be useful, but I am sure Mario has his reasons for this.

The first time I discussed this behavior was back in July 2012.
Mario then asked to raise a Mantis bug report, which resulted in Mantis 565.
Following this I found my work-around, concentrated on other problems and "forgot" this one.
The issue surfaced in my memory again when DigiPeter hit something similar.
As the Mantis bug report is still open I think it will be automatically picked up by Mario when he goes bug-hunting again.
For completeness' sake I made a reference from the Mantis bug report to this discussion.

[sinus]

My question is about the "List of folders" as the place where to look for versions.
My understanding / logic / gut feeling is that, when one selects this option, it is highly unlikely that masters can be found in the same location as the versions.

I recall Mario stating in the past that by design it is possible for a file to be both a master and a version.  I have seen that, and it is in theory possible with the right (i.e. complex) versioning definitions.  I recall that he wanted to allow for that, although I can't image when I'd use it myself.  If my recollection is correct, then he isn't going to agree to exclude IMatch searching for masters in the list of folders for versions.

Exactly. I have, quite a long time ago  , played with this (a file can be a master AND a version).
And there are users out there, I think, for whom this scenario is possible and important.

Versioning is a great challenge for a program.
But it is also a great challenge for the users. If we are looking in different sources (web, books ...)  about master-version, then the first proposal is very often, to think about a good folder-structure and a good filenaming-strategie.

So, if IM does search for masters and versions in a folder, and we cannot say IM directly, to search there only for versions, I would try to think about my versions-structure and logic.
Or, if you have time, I would add your proposal as a feature-request.

[sinus]

Thank you for this bit of "historical" information.
If this is intended behavior I guess we just have to live with it.
I find it hard to imagine when it would be useful, but I am sure Mario has his reasons for this.

Each user has a different feeling and point of view. Sometimes such views does looking strange, but not for others.
People are different 

In my case, in the PAST, thought this:

If I have a raw and work a lot of this raw, I create a copy, say a psd or tiff.
Now, I create more versions. But in such cases, I do not go back to the raw (nef in my case), but to a specific, say psd.

Now this psd is a version of the raw, where all begun, AND it is a master for the versions.
If I have a jpg, and want change something slightly, it does not make sense in such scenarios, to know, what is the FIRST and orignal master, I want to know, what file I must open to make a change. Hence it is enough and even clever for me, to know, ahhh, the master is the psd, in this case.

If I want make BIG chances in a file, it can be, that I must go back to the original file, the raw, an it this case I must know, ok, the master of the psd is this raw.

So in this cases, an image can be FOR ME the master and the version. I am sure, other users will have other scenarios, where they wants, that a file is master AND version.

[herman]

Or, if you have time, I would add your proposal as a feature-request.

A feature request is one of the options, the other one being a documentation enhancement request.
I have plenty of time to do both 
These options can be considered when it is clear that the current behavior is as intended, that it is not a bug.
I am still hoping Mario will chime in in this discussion and share his thoughts about it.

[sinus]

Or, if you have time, I would add your proposal as a feature-request.

A feature request is one of the options, the other one being a documentation enhancement request.
I have plenty of time to do both 

Uhh, if you have plenty of time, you are very lucky or you manges your time specialy good! 

These options can be considered when it is clear that the current behavior is as intended, that it is not a bug.
I am still hoping Mario will chime in in this discussion and share his thoughts about it.

Yep, I can understand this. Sometimes only Mario can "shed the light" on a subject .... (hope this English "phrase" is correct).  ... but since Versions is a "big beast" and it works finally yet, I guess, he has other important points to work on, because he wants bring out the official IM5.    (I hope, Mario gives as some days BEFORE a tip, so that I can put a bottle of champagner in the fridge