Question regarding "Refreshing Relations"

[akirot]

The current IMatch implementation drops existing relations (except manual ones) when "Refreshing Relations".
Why is that so? Why isn't there an option to keep existing Relations?

After producing new versions I identify relations and move the versions to other folders (inside IMatch). Regularly I later produce new versions (for different use cases). If the "old" versions are in folders out of the search scope of the relation the old relation is lost and only the new one found.

Why can't IMatch optionally(!) keep existing relations when refreshing relations and only add the newly found relations?
Is there a deeper technical reason behind (and if so which one - just curious :-) )  or is this a feature request candidate? 

[Mario]

When IMatch re-creates relations it drops existing relations and then re-creates the relations from all currently activated and matching relations. This cannot be an 'additive' process.

[akirot]

Yes - currently IMatch drops existing relations, that's the problem.
Why is it not possible to skip dropping and just add newly found versions?
I've scripted this mechanism in IMatch 5 using available methods (based on "manual" versions) - works like a charm.
Unfortunately you killed IM5 scripting, leaving me alone now.
I just try to find out how to migrate to IM2017 without losing functionality.
Optionally keeping existing relations to me seems to be a minor (code) change only.

[Mario]

What would be the difference?

IMatch re-creates all relations based on the current settings after clearing the relations. There will be no difference between before and after, unless you de-activated some relation rules that were active before. There will also be no performance difference.

Unfortunately you killed IM5 scripting, leaving me alone now.

I beg to differ. I replaced an ancient and increasingly expensive and problematic BASIC scripting engine with a top-notch, modern programming environment. Scripts can now be written in virtually any language, integrated apps can be written in JavaScript and HTML and you can even run IMatch apps in your web browser or on a remote machine.

I've scripted this mechanism in IMatch 5 using available methods (based on "manual" versions) - works like a charm.

You can do that in IMatch 2017 scripting as well.
There are endpoints for creating and removing manual versions.

Check out the IMWS endpoints v1/relations/createmanualversion and  v1/relations/deletemanualversion.

[akirot]

The difference (a bug?):
An identified version can be moved to a different folder (inside IMatch) outside of the original version detection scope. One can even rename a version after being identified as a version based on e.g. file name rules. That's great and one can later always find all versions of a master. (To me it's a core functionality of a DAM.)
As soon as you produce a new version (for whatever reason, e.g.  different size, black and white etc.) and refresh relations the connection to the already existing versions (now outside the search scope) is lost!  This makes the current versioning mechanism unusable if you move versions around after detection.
You see my problem? :-(
Probably this is a bug? (I always wondered why there is no option to keep already existing relations as you can with manual versions.)
An option to keep existing relations could circumvent this problem. (Personally I would drop existing relations by intent only - never automatically.)

Thank you for making me aware of the new endpoints. (Their announcement is a bit hidden in the release notes and documentation online not yet available as this is based on version 2017.5.2.)

[Mario]

As soon as you produce a new version (for whatever reason, e.g.  different size, black and white etc.) and refresh relations the connection to the already existing versions (now outside the search scope) is lost!

And it has to. If you create the relation graph by matching file names, and you rename one of the versions so it no longer matches, a re-run of the relation will not make it a version anymore. It is generally not a good idea to rename files arbitrarily when you use filename-based versioning.

[akirot]

The filename based versioning has just been a (probably misleading) example.

As soon as a file has (rightfully) been identified as a version of a master this specific relation must be persistent as long as the master and version exist - even if one changes attributes (as e.g. the filename) afterwards.
One has to break the relation by intent.
This is how manual versions work - luckily :-)

Why not transfer this behavior to non-manual versions? Thus gaining the option to automatically identify additional versions without losing the existing ones!

One could leave "refresh relations" as is and just add an option "add relations" (without dropping existing relations).
From my point of view this should be an easy one :-)

[Mario]

persistent as long as the master and version exist - even if one changes attributes (as e.g. the filename) afterwards.
One has to break the relation by intent.

No. If you change the attributes which make a file a version of another, the next relation run will re-evaluate everything and may break the relation.
For invariant, persistent relations, use a manual relation. This is why I included them in the design.

[akirot]

Sorry, until now I still can't identify the technical need to drop existing relations if one only wants to add a new version - I kindly asked for an explanation in the original post.
Literally manually applying "manual" versions is not feasible for thousands of images (if the goal is to achieve persistent versions). 
However, "manual" versions are an option if their detection can be automated. Can this be implemented somehow please? (E.g. by defining a version as persistent [i.e. "manual"]).

[Mario]

Refresh Releations =>

1. Drop existing relations
2. Re-create existing relations based on the relation rules active at the time.

This is how I've designed it a couple of years ago. Clean, reproducible, safe.

If IMatch would not drop relations, I would need to add extensions:

For example, users will ask not to drop "some" relations, but to keep others.
This would require an additional user interface, additional per-rule settings, more documentation and even more complexity.

- What happens if existing relations conflict with the current relation rules?
For example, before A was a version of B, but the current rules make A a version of C?
Should A then become a version of both? Of none?
Should IMatch prompt the user for each file?
Irritating, prone to errors. Would cause a ton of extra documentation and support tickets.

I'm sure that if you have creates a setup which requires to retain existing relations while re-creating other relations, you can handle that nicely with manual versions. I have no intention to further complicate the already overly-complex relation processing.  So far you are the only user ever asking for this, and IMatch does versions for over three years now. Not likely that many users will ever see demand for partial relation drops.

[sinus]

akirot

Since this, what you want, will not be implemented in IMatch (the reasons why not, has Mario explained), would it not be one other possibility for you, to change your workflow?
I mean, then you do not move the version outside the scope?

Of course I do not know your folder hierarchy, but I personally have changed it several times ... and every times it has been better. 

If you cannot do this, then I think, you must "convert" your script.

I personally am also in the situation, that important scripts does not more work with IMatch2017. For some I have changed my workflow, so they are not more necessary (at least fine).

At the moment I am trying to create 2 scripts, what are important to me. It is hard to do so, but the day will come, when they are working (hopefully).

Good luck.

[akirot]

Thank you for sharing your point of view, Mario.

As far as I can see you are thinking far too complicated :-) .
If you don't change the current behavior as a whole:
1.) Not dropping relations is already implemented (for manual versions). If you add the additional option in the refresh relation window to keep "all" relations (not only the manual ones) this doesn't change complexity.
Of course this is a yes/no decision - there can't be an option to just keep "some".

2.) The conflict scenario is already there and remains unchanged: What if there are two "conflicting" rules in places? This is possible today, too. Just keep it as is.

3.) My preferred approach would be to make a relation persistent from the beginning. If you added such an option to the relation configuration window this would result in no change of the current behavior (just handle persistent relations identical to manual versions).
By such an approach you win the version detection mechanism for "manual" versions - no further adaption needed.

The implementation of the last option (together with a possibility to use "any" metadata for version detection - please see my respective feature request) would make me happy because it would replace my current IMatch 5 scripts.

[Carlo Didier]

This problem is one of the reasons why I did part of my versioning through event driven scripts, whose absence is the main reason I'm sticking with 5.8.
Doing the versioning by scripts which are triggered by events like adding or modifying images allowed me to automatically create relations which are not automatically removed by another mechanism.

[Mario]

Thank you for sharing your point of view, Mario.

As far as I can see you are thinking far too complicated :-) .
If you don't change the current behavior as a whole:
1.) Not dropping relations is already implemented (for manual versions). If you add the additional option in the refresh relation window to keep "all" relations (not only the manual ones) this doesn't change complexity.
Of course this is a yes/no decision - there can't be an option to just keep "some".

2.) The conflict scenario is already there and remains unchanged: What if there are two "conflicting" rules in places? This is possible today, too. Just keep it as is.

3.) My preferred approach would be to make a relation persistent from the beginning. If you added such an option to the relation configuration window this would result in no change of the current behavior (just handle persistent relations identical to manual versions).
By such an approach you win the version detection mechanism for "manual" versions - no further adaption needed.

The implementation of the last option (together with a possibility to use "any" metadata for version detection - please see my respective feature request) would make me happy because it would replace my current IMatch 5 scripts.

Fell free to add a feature request. If there is a sufficient number of users who vote for your request I will consider re-designing the relation manager in a future version. For now, it looks as if it works well for the majority of users as it is.