Converting from automatic versioning to manual versioning (help request)

[DavidOfMA]

I'm looking for help with a one-time migration from automatic versioning to manual versioning. Using automatic versioning, I want to set up the bulk of my master/version relations, then turn everything to manual, because my database is not set up to work well with versioning as implemented in IMatch 5 and changing the structure is not an option.

My database is set up so that all the Masters are in one folder tree, and all the Versions are in other folder trees (see below). This folder structure works for me because I do things on a project basis, and the images for each project often come from pictures I've taken over many years. Currently, for instance, I'm working on photos of leaves I've taken over the past 10 years, most of which I haven't touched until now. These files are all in my Masters folder tree organized by date, then categorized as "Leaves."

In IMatch 3, I kept track of versions using a script written, I think, by Ferdinand, and my folder structure worked well with that script. But IMatch 5's versioning logic is different, and each time I add new files, versioning gets confused. However, if I rebuild file relations only for the files in the Master file folder tree, versioning is largely correct until add more files to the database.

What I want to do is 1) rebuild file relations for only the Master folder tree files using the versioning rules I've created, and then 2) convert the file relations IMatch creates from rule-generated ones to manual ones. Then I could turn automatic versioning off and just go manual, moving forward, while still preserving the bulk of the file relations I already have.

Is it possible to automate the process of converting the automatic, rule-based file relations to manual file relations? If so, how would I go about doing so? Going through all 120,000 files and making manual file relations would take me weeks, so I hope this can be automated.

Thanks,

David

Example folder structure
Masters
- 2001
  - Folder 1
  - Folder 2
  - Folder 2
...
- 2015
  - Folder 158
  - Folder 159
Finished
- 2001
  - Folder 1
  - Folder 2
  - Folder 2
...
- 2015
  - Folder 068
  - Folder 069
Submitted
  - Museums
  - Galleries
  - Clients
...
Working
  - Project 1
  - Project 2
...

The "Masters" folders are essentially my digital negatives. The "Working" images and folders get moved to "Finished" when a project is completed. The "Submissions" folders get populated by subsets of the Finished images that I submit to galleries, etc. This arrangement makes it easy for me to keep track of what's what both inside and outside IMatch.

[herman]

[...]each time I add new files, versioning gets confused.

I don't know if I have an answer to your problem, but your folder structure is somewhat comparable to what I use, and I recall I had some versioning problems when I migrated from IM3 (also using Ferdinand's versioning script) to IM5.
Just for my proper understanding, could you please be more specific what you mean with the 'confusion' of the versioning?
When I understand what is going on on your side I may be able to offer you some advise or a work-around.

[DavidOfMA]

Hi, Herman. I figured I wasn't the only one to work this way. What I mean by confusion is that if, for instance, I copy a file to one of my project folders and then make some changes to it and save another version with a related filename, the copy becomes both master and version, or sometimes just master. So, say, File1.jpg is copied to a project folder, then File1a.jpg is created, the copy becomes the master, not the original File1.jpg. This is especially complicated because many files begin as JPEGs and the versions are also JPEGs.

I want only the files in the Masters folder tree to be Masters.  It was simple with Feedinand's script, since anything in the Masters category was a Master, and all else was a version. I'm not capable of creating a script that emulates this logic, but I would if I could. Or, if I could specify a Master folder or category, versioning would work for all but a few files that use Roman Numerals in the name. But as things are, it gets messy.

[herman]

HI David,

What I mean by confusion is that if, for instance, I copy a file to one of my project folders and then make some changes to it and save another version with a related filename, the copy becomes both master and version, or sometimes just master. So, say, File1.jpg is copied to a project folder, then File1a.jpg is created, the copy becomes the master, not the original File1.jpg. This is especially complicated because many files begin as JPEGs and the versions are also JPEGs.

This sounds very familiar to me.
I have been there, doing and thinking like you.
In the end I discovered that I failed a long time to understand the way IMatch works, how it detects versions.
There have been some discussions here, even going back to beta test phase.
During one of these discussions Mario clarified what is going on under the hood of IMatch:

If somewhere in the database a new file shows up (created in another application or by you in the batch processor), IMatch just says: "Oh, there is a new file" and adds it to the database. It then applies all file relation rules to the folder containing the new file. It does not, however, search the entire database for files which may be a master of the new file. This would take a very long, because the master could by anywhere.

So, when I understand you correctly, you drop (copy) a file 12345.jpg from your 'master' folder hierarchy to the 'project' folder hierarchy.
You then process this copy in some external software (or in the batch processor) and this results in an file 12345a.jpg in the same 'project' folder hierarchy.
IMatch will now try to find a file which matches the master expression for this new file and then applies the 'where to search' to this file. So you may end up with the copy of your master file marked as both a version (of the file in the 'master' folder hierarchy) and a master (of the derived file in the 'project' folder hierarchy).
This is how it is supposed to work, IMatch is not aware of the meaning of your folder structure.

There are a couple of ways to work around this.

First of all you may look into your file naming convention.
The phenomenon you see will not occur when there are no duplicate filenames.
So, when you copy a file from your 'master' folder hierarchy to the 'project' folder hierarchy you could rename it, for example add a suffix to it.
That is what I did actually, when the file in my 'master' folder hierarchy is called 12345.jpg the files in my equivalent of your 'project' folder hierarchy would be named 12345_D.jpg, 12345_D1.jpg etc.
The master expression I use to go with this is [0-9]{4}\.(crw|cr2|jpg|dng)$ which translates to: only when the last four positions of a name are numerical digits it is a master file.

A second way to work around could be to 'mark' the masters you are working on in your 'masters' folder hierarchy. When you are done working on these files and their derived files all you have to do is select the 'marked' masters and tell IMatch to establish the relations via <f4><R>.
To 'mark' these master-files-in-process I prefer to use categories, but collection items (pins, dots, flags, bookmarks) could be used as well.
I use myself a number of categories for this purpose, based on the traditional 'wet' darkroom workflow, see attachment.

I realize this does not answer your question, it seems that you want to switch to manual versioning.
Frankly I have no idea how to do that, but I hope my experience helps you somehow.

[attachment deleted by admin]

[DavidOfMA]

Thanks for your suggestions, and for the detail about how IMatch goes about finding masters when you add a new file. This clears up why I get the master/version problem when I add a file to the project folder. Because I have numerous file-naming schemes over 14 years, and many of the files are embedded in various projects using those names, I think it's too late for me to change the file-naming scheme for the old files, as I will break too many old projects. I can be more conscious of how I name versions moving forward, though.

I'm not sure how you are using the "marking" idea. I also "mark" my files with categories similar to yours (RAW, Captured, Intermediate, Final, Submitted). How are you using this method to get versioning to work correctly? Are you suggesting that I add a new category to the masters I'm using in my projects, then select those masters and refresh the relations when I'm finished with that image in the project? This may be the way to go for me, unless I can get Ferdinand's versioning script running on IMatch 5 or find a manual solution.

Thanks again,
David

[herman]

When renaming files is not feasible in your case (which I can fully understand) I would try the following:

First of all uncheck in Preferences - Background Processing the File Relations checkboxes for 'Automatically refresh relations' and 'Automatically search for masters for new and updated files'. I did not check this right now but I would expect this to turn off the automatic mechanism to find relations and thus to put an end to the confusing versions-shown-as-a-master phenomenon.

Next I would create a category which I would use to hold files from the 'master' folder hierarchy which I want to process. Call this category 'Masters being processed' for example.
Only when the 'masters' are assigned to this category I would do the processing, copy files to the project folders, process them in whatever program, have derived files dropped in the relevant project folder etc.
When the processing is all done return to the 'Masters being processed' category, <CTRL>+<A> to select all files in that category, then let IMatch find versions for the selected master files via <f4><R>.

Hope this helps....

[DavidOfMA]

One would expect unchecking the Background check boxes to prevent IMatch from automatically finding versions, but it doesn't do that, at least not when I use the batch processor to make a new version. If there is another way to get IMatch to not automatically look for new versions, I haven't been able to discover it (and I think Mario said there was not, earlier in this thread, though I'm not sure that's what he meant). However, using the category method you have described and then refreshing only those files when I'm finished with a project might solve the problem, as it should rebuild and correct any incorrect master/version relations. I say "should" because I tried it with one such file, and nothing I do so far seems to break the incorrect connection between a copy of a file in the project folder and a version file. I can't figure out why, but at least this one "false master" links back to the real one now. There may be something in my versioning rules I need to change.

Thanks again for your help,
David

NOTE: I can confirm that even with the version-related background stuff turned off, when I create a new file using the batch processor, IMatch still refreshes file relations on the folder containing the source image. For most people, this is probably exactly what they want, but for me it undoes the versioning I've carefully created. I can restore it by refreshing file relations on all the "master" files I've marked with a category indicating they are the source, but it would be great not to have to do that. If anyone knows an easy way to toggle file relation creation on and off, that would make working around the difference between how I handle files and how IMatch expects me to handle them much easier.

[Ferdinand]

.... unless I can get Ferdinand's versioning script running on IMatch 5 ....

I've always said that I wasn't going to do this.  Too much work and the native versioning in IMatch 5 is better, and so much easier to script with. 

However ..... I'm sympathetic to people who find themselves in this position, simply because I could easily have found myself in it as well.  Many, many years ago I did a major file rename project for other reasons, but which would have prevented these problems in IMatch 5.  That exercise was not all that hard, although still tricky.  Like David, I couldn't do it now for much the same reasons - the names are too deeply embedded in my workflow - even though a minor tweak of the naming schema would really help in IMatch 5. 

What was tricky was a major reorganisation of my folder structure, from masters and versions in separate folder trees to versions in sub-folders of the masters.  I wrote a 3.6 script to do this, but running it was a fairly hair-raising exercise.  It's not something that I'd recommend to others, unless you're both very brave and very careful.

So, perhaps I should see if I can port that script.  It's a big script, but a lot of the changes are global search and replace.

[DavidOfMA]

That would be very helpful, particularly if there's some way to integrate the results with file relations to take advantage of some of the new features, but bypass automatic file relations and instead look for masters only in one place. Mario said he thought my structure was very rare, but it does seem there are several among us who put masters in one folder tree, versions in another