Getting buddy files to batch rename properly

[rlgroff]

I have been using IMatch to manage hundreds of thousands of images for at least 10 years, and saw no need to upgrade as it met all my needs.  Then a few months ago, version 3.6 suddenly stopped batch renaming buddy files correctly, and it was to solve this problem that I finally decided to upgrade to version 5, and recently updated to 5.6.18. 

After struggling to understand the new batch Renamer (I found the old one to be easier), I find that I have the same problem all over again.  Using Nikon cameras, I sometimes shoot NEF + JPG file pairs, and sometimes unique JPG files only, so my file folders usually contain a mix of one-of-a-kind JPGs and pairs of NEFs + JPGs (buddy files).  I'm not talking about anything sophisticated here - I just need to be able to rename the camera files with a YYYYMMDD-nnnn where nnnn is a sequential number, and the destination name is the same for each NEF+JPG pair.  I don't need any other buddy file relationship, and at this point I don't want to deal with versioning. 

What the Renamer is giving me is the NEF having one number in the sequence and its corresponding JPG having the next number in the sequence.  I would have thought that the Renamer would handle this simple and very common relationship correctly by default, without the need to get into the File Relations dialog.

I have delved into the Help topic on File Relations, and I find it difficult and frustrating.  In the Configuring File Relations dialog, I have checked only the NEF Buddy File Relation Definition (I don't need any others).  I have studied the rest of the dialog and I believe the default patterns in the fields will give me the behavior I'm looking for, and I have used the Test Expressions option and it appears to give the desired results, i.e. test.nef changes to text.jpg.  I then closed the dialog with "OK", and was surprised to be told to "Refresh Relations".  (Why that isn't done automatically throughout the whole database is a mystery to me).  So, in the Media and Folders pane I selected the highest level folder in the database (the root of the drive), assuming I had to be there to assure that the changed relations would be applied to every folder in the database below that, and then I did Command | Relations | Refresh Relations.  After this, for a time, I could not open the Renamer to test the change, presumably because the Imatch was busy refreshing relations, but I can find no indication anywhere on the screen that the program is busy doing anything.  After a while, Renamer worked again but still refuses to rename NEF+JPG pairs correctly.

Thinking I may be misunderstanding the help topic, and that the program might be missing the JPG part of the pair, I tried repeating this process, changing the Master Expression to \.(nef|nrw|jpg)$ .  This made no difference - it still refuses to rename NEF+JPG pairs correctly. 

This issue has derailed my workflow and cost me 6 hours so far, and I need to get renaming done correctly to meet a number of obligations to others.  What am I missing here?

[attachment deleted by admin]

[Mario]

Your post ended up in the wrong board. This board is for giving other users tips and for posting How-to articles and FAQs.
It is not for posting questions.

Tip: Read the board description visible under each board to get an idea of what to post where.

I will more your post into the General Discussions board, which is suitable for this kind of questions.

Please help us to help you by explaining

+ List some of the file names you are using so we see what you use for input.

+ Have you tried the "Test" button in the File Relations dialog to see if your masters and buddy file names are correctly detected?

+ Attach a log file in debug mode from an IMatch session where you tried to rename files.

The Renamer automatically renames buddy files along with their originals if the buddy file relation is set up properly and IMatch can find your buddy files.
You might be new to IMatch 5, but it is available for over two years now. Much know-how has accumulated here in the board and when you give us sufficient info, we can surely help.

Maybe the problem with renaming buddy files in IMatch 5 is related to IMatch 3 suddenly, for whatever reason () stopping to rename buddy files? Did you try to find out what you have changed before this happened?

[rlgroff]

Sorry for posting this question in the wrong place!

Attached is a screen capture of my attempt to use the Renamer, showing both the settings and a partial preview of the results.  The shots are from 2 cameras whose time was synchronized, so I could merge the results in a single folder sorted by Capture Time.  In this example, I would expect the renamed files to be:
...
_D710037.JPG to 20160316-0039.JPG
_D7K4256.JPG to 20160316-0040.JPG
_D710038.JPG to 20160316-0041.JPG
_D710039.JPG to 20160316-0042.JPG
_D710039.NEF to 20160316-0042.NEF
_D710040.JPG to 20160316-0043.JPG
_D710040.NEF to 20160316-0043.NEF
_D7K4257.JPG to 20160316-0044.JPG
_D7K4258.JPG to 20160316-0045.JPG
...

From the File Relations dialog setup attached previously, I have attached two results of using the Test Expressions button.

Finally, I have attached the current IMATCH5_LOG.TXT from my \TEMP folder.

The fact that my buddy file problems might be a carryover from when the same database was in v3.6 is an intriguing one, but the files I am having difficulty renaming are from a recent trip and were just scanned into v5.6.18 this week.  Buddy files in previously scanned trips in the converted database appear to be renamed correctly.



[attachment deleted by admin]

[Mario]

I'm not sure, but your screen shot looks as if you are renaming NEF files and JPEG files at the same time, and that the JPEG files you rename are buddy files of the NEF files? Is this the case? How is this supposed to work?

When you rename a NEF file, IMatch checks for all buddy files and renames them as well.

I tested this successfully with

_D710039.NEF and _D710039.JPG

IMatch renames the NEF, finds the buddy file _D710039.JPG and renames it as well.
But I have only the NEF selected for the rename, not the JPEG buddy file as well. If you rename both the NEF and the buddy file at the same time, the results are undefined.

[rlgroff]

I'm sorry, I never guessed how difficult it would be to explain this issue clearly.  I will try again...

When I am on a day of shooting, I set the camera for "JPG only" if the pictures are intended only for projection or web use, but I change the camera setting to "NEF+JPG" when the situation looks like I will get images worthy of printing that require the highest possible quality.  The result is that I end up with a folder containing the full day's photo shoot, sorted by capture time, that is a mix of images that are JPG only and images that are NEF+JPG pairs.  IMatch has always handled this kind of mix correctly, renaming correctly after selecting every file in the folder.

Yes, I am trying to rename NEF and JPG files at the same time, wherever NEF+JPG pairs occur in the folder.  When the camera is set for NEF+JPG, it produces NEF files and corresponding JPG files of the same image, with the same file name (the part before the dot) but different extensions.  They are already buddy files coming out of the camera.  In the batch renaming process, if the renaming rules generate a sequential number (as in my submitted attachment), I need the Renamer to recognize the existing buddy file pairs and assign the same sequential number to both files in each existing buddy file pair.  That is what IMatch has always done correctly for me until recently.

If, as you suggest, I try to select only the NEF files (which has never been necessary before), it only renames the selected NEFs and not the corresponding JPGs.  And even if that method did work correctly, it would be useless to me because the folder contains a large number of JPG-only images that would not have been renamed as part of the auto-numbered sequence that should cover the whole day.

It seems to me that there has to be something wrong with the buddy file definitions in the File Relations dialog - that the Renamer is  not recognizing the existing NEF_JPG pairs.  The default setting is to have none of the Relation Definitions checked.  This type of NEF+JPG buddy file pairing is so common among Nikon shooters, that I would have thought that checking that relationship would be all that is necessary, without forcing the user to try to decipher the rest of this confusing dialog box.

One last possibility is that the File Relations are defined correctly but are not propagated throughout the database with the "Refresh Relations" command.  I assumed all I had to do is select the uppermost folder in the database and then do Commands | Relations | Refresh.  However I am wondering if the database did not get refreshed correctly.  One possibility is because the tree structure showing in the Media and Folders pane does not show the actual uppermost folder in the database (\TRAVELS_2014_ONWARD\), but instead shows the root of the G: drive.  I have attached a full IMatch screen capture that shows the Media and Folders pane, and everything else in case there are relevant clues in the rest of the screen. 

Is there a straightforward way to assure that newly defined file relations are applied throughout the database? 

[attachment deleted by admin]

[thrinn]

My understanding is that a buddy file by definition can not stand alone. It is always a buddy to some other file. In your situation, you have always a JPG, but only sometimes a corresponding NEF, right? What if you reversed the buddy file definition rule, defining the JPG as master and the NEF as buddy? This may sound a bit strange at first, but it would be easy to select all "masters" (JPG files) and let IMatch take care of the NEF buddy files, whenever they exist.
To try it out, you could create a new buddy file Relation Definition with the existing one as a template and just exchange the nef/jpg extensions in the Master and Link expression.

[Mario]

If, as you suggest, I try to select only the NEF files (which has never been necessary before), it only renames the selected NEFs and not the corresponding JPGs.

Then please double-check your settings. I tried with several of your sample file names (from your post) and it worked right out of the box. And you wrote yourself that the problem happens only in some of your folders. And that IMatch 3 also had (suddenly) problems with some of your buddy files. I guess there is something on your system that you have not noticed yet.

I used the standard definition for my test:

Type: Buddy, detection via File Name
Master: \.(nef|nrw)$
Replace: ^_*//
Link: ^(_*{name})[+\-_]*[0-9|a-z]*\.(nks|jpg|jpeg|dng|nef\.dxo|nef\.bib|nef\.dop)$

Renamed all buddy files together with the NEF, no problem.

Is there a straightforward way to assure that newly defined file relations are applied throughout the database? 

Buddy file relations are dynamic and automatically applied.
This is usually a no-brainer, really. Once you have told IMatch how to identify the buddy files of your RAW files, it just works.

But you selecting both buddy files and masters at the same time makes it impossible for the build-in logic to work. It renames buddy files twice or the behavior is undefined. This is not how this is supposed to work. Just select the master files, maybe apply a filter (File Properties: Hide budddies) before you do a Select All and rename.

If this does not work on your system with my definition from above, either the file names you have given us are wrong, or you need to check for other issues on your PC.

Tips:

Use the

{File.BuddyFiles}

variable in the App Panel, for example, to see all identified buddy files for the selected masters.

Don't mix file relation rules which make a buddy file a master as well (like in one of your earlier examples, where you included the .jpg extension in the list of master extensions).

Copy a master and a buddy into a separate folder. Use the variable above to see if the buddy file is detected. Rename the master. Is the buddy file renamed as well?

[Mees Dekker]

I did encounter a similar problem, but it can be solved very easily. To me it seems that we are mixing up "buddy" files and versions.

When you treat jpgs as "versions" of a master (i.e. the corresponding NEF) and then collapse all versionstacks before renaming, you can easily select all (ctrl A) pictures. By doing so, only the masterfiles (NEF) and the "stand-alone" jpgs are selected. Then put the renamer at work and your jpgs (versions) wille have the same new name as the master they are a version of.

[rlgroff]

As recommended, I double-checked all the settings in File Relations dialog.  I really believed there is nothing unusual about my shooting practices, so I had every reason to believe that the default definitions should allow the Renamer to work correctly.  I clicked the default button to be sure of the definitions, and selected (only) the "NEF Buddy Files" rule as before.

Your suggestion to use the "Hide Buddy Files" filter gave me the first clue about what was really going on: the file window looked the same whether "Hide Buddy Files" was selected or not.  This meant that, even before using the Renamer, the out-of-camera NEF/JPG buddy pairs were not being recognized.  This took me back to the idea that the file relations really needed to be refreshed in the database.

Buddy file relations are dynamic and automatically applied.
This is usually a no-brainer, really. Once you have told IMatch how to identify the buddy files of your RAW files, it just works.

I cannot say this was the case in my system - the relationship was not automatically applied, and a successful effort to refresh the file relations (Command | Relations | Refresh) was necessary.  And this would be consistent with the pop-up warning that comes up when you try to close the File Relations dialog after making changes.  It appears that the reason for my several nights of failure was that my previous attempts to refresh relations were not successful.

Referring to my previous attachment of a full screen capture (rlgroff_full_IMatch_screen.jpg), the Media & Folders pane shows the root of the G: drive as the direct parent folder of the database folders (trips) when, in fact, the parent folder of all the trip folders is G:\TRAVELS_2014_ONWARD, as shown (correctly) at the top of the File Window. 

My file structure for this database is
G:
  \TRAVELS_2014_ONWARD
       \Trip-specific folder
            \Date-specific folder
                  individual image files
I don't know how it happened in this database, but the 1st level folder below the root of the drive (\TRAVELS_2014_ONWARD) is missing from the Media & Folders pane.  In a different Imatch5 database on my system, the 1st level folder (parent of the trip-specific folders) shows correctly in the Media & Folders pane.

In previous attempts to refresh relations, in the Media & Folders pane, I had highlighted the root of the G: drive, because that was the folder showing directly above all the trip-specific folders.  When I attempted to do Command | Relations | Refresh, a very short pause occurred with no real indication of processing.  This time I selected the visible trip-specific folder (\2016_NEW_ZEALAND) and did Command | Relations | Refresh, and this time there was some delay and a progress pop-up so I knew something was really happening.  After doing this, using the Filter pane to toggle "Hide Buddy Files" DID make the buddy JPGs disappear and re-appear as expected!   This was my first indication that I was on the right track.

SUCCESS!
After this successful refresh of relations and selecting all files in a date-specific folder, the Renamer handled the NEF (master) + JPG (buddy) pairs correctly.  I learned that I have 2 options:

(1) Leaving the "Hide Buddy Files" filter unselected (showing all the files) -
Using the Renamer, the Preview window and the Results windows both highlight each buddy JPG with a red warning, saying the buddy file will be renamed when the master is renamed.  At the bottom of the Results window, the renamed JPG buddies are counted as "errors" but are, in fact, renamed correctly.
(2) Leaving the "Hide Buddy Files" filter selected (hiding the buddy JPGs) -
Using the Renamer, the Preview and Results show no warnings and count no errors, and afterwards when toggling the filter to show the buddy files, they are found to have been renamed correctly.  Obviously this is the preferred option.

The bottom line is, to get renaming to work correctly, I had to refresh the relations in each of my trip-specific folders, because I am unable to highlight their parent folder (\TRAVELS_2014_ONWARD) to refresh there.  Running the Database Diagnostics and Optimization does not change this, but reports no errors.  I'm hoping you might have some suggestions regarding how to get all the levels of folders to appear in the Media & Folders pane.

My thanks to thrinn and to Mees Dekker for contributing suggested work-arounds, both of which make sense and would probably have provided me with some "last resort" options.

[Mario]

From the source code in the Renamer an elsewhere I know that buddy files are determined right on the spot, by getting the relation settings and then searching the actual file system for files matching a buddy mask. Buddy files cannot be determined in advance, because they may exist outside of the database. Maybe doing a relation refresh has fixed some other issue with your setup.

because I am unable to highlight their parent folder (\TRAVELS_2014_ONWARD)

I don't understand. How can you fail to select a folder? What happens, exactly?

When you use the Refresh Relations command on a folder, the folder is processed recursively, including all child folders.

[rlgroff]

Please understand I am new to version 5.6 and just about everything I do with the program is being done for the first time, so I am constantly on a learning curve and I often jump to wrong conclusions because I just don't understand what you were thinking in developing this interface.

To answer this question

How can you fail to select a folder? What happens, exactly?

At the time of my first attempt to refresh the relations (when the Renamer would not recognize the buddy file relations), in the Media and Folders pane I selected the root directory of G: as indicated in the attached screen capture, because that was the only folder visible to me that would be above all the trip-specific folders.  When I said I failed to select the parent folder immediately above the trip-specific folders (\TRAVELS_2014_ONWARD), I meant that it was not visible, as you see in the attachment.  I concluded that the relation change was not propagating through the database when the root of G: was selected because there was little evidence of any processing going on (no appreciable delay), while later there was a very noticeable processing delay and a progress pop-up when I tried to refresh relations at the level of the trip-specific folder (which led to my renaming success).

Just tonight I finally realized that the parent folder of the trip-specific folders is hidden simply because the pane is configured to hide it (I just discovered the configuration button at the top of the pane).  This probably means I came to the wrong conclusions about my initial failure to refresh the relations, and therefore I have no clue why my subsequent refresh of the relations at the trip-specific level folders was the apparent cause of my renaming success.

If buddy file relations are processed by the Renamer dynamically as you described, and refreshing relations has nothing to do with it, then after changing relations in the File Relations dialog, why does it produce a pop-up (see my 2nd attachment) that insists that I refresh the relations and tells me where to go to do that?

[attachment deleted by admin]

[Carlo Didier]

Concerning buddy files: Isn't it recommended somewhere to not include buddy files in the database?

[Mario]

Concerning buddy files: Isn't it recommended somewhere to not include buddy files in the database?

Buddy files are usually things like configuration files produced by software, maybe .thm files produced by some camera models for videos. That buddy files are included in the database is rather uncommon, but handled properly nevertheless. Although most users would consider the JPEG files a version of the NEF and not a buddy file. And the OP mixes both combinations of RAW/JPEG buddy and standalone files, where the JPEG is the master and has no buddy. A rather complicated workflow that requires extra care - for example keeping RAW files with buddy JPEGs and stand-alone JPEG files in separate folders to easy processing.

[Mario]

Please understand I am new to version 5.6 and just about everything I do with the program is being done for the first time, so I am constantly on a learning curve and I often jump to wrong conclusions because I just don't understand what you were thinking in developing this interface.

The user interface you work with has been introduced with IMatch 5 back in 2014. Many users in this community work with IMatch 5 for one or two years, some even longer, if they participated in the public beta test at the time.

I suggest that you read the "For IMatch 3 users" help topic in the help, because it mentions the usual pitfalls for users who move up from IMatch 3. The information in that topic is from two years ago, but still valid. IMatch 5 retains the basic IMatch concepts but adds a lot on top of it. The check out the first steps chapters and maybe print out the keyboard cheat sheets of the Quick Start Guide (Help menu).

Just tonight I finally realized that the parent folder of the trip-specific folders is hidden simply because the pane is configured to hide it

Unless you use the filter panel below the Media & Folders tree, no folders are hidden. There is no option for that or anything. IMatch displays all folders you have added to the database in the Media & Folders view. It groups them by drive and media (if you have added files from multiple media in the same drive). The drive node is what you see as G: in your screen shot. When you do a refresh relations on a drive node, it updates all folders on that drive, recursively.

See "Media & Folders" view in the IMatch help for detailed information. Just click into the tree and then press <F1>.

If buddy file relations are processed by the Renamer dynamically as you described, and refreshing relations has nothing to do with it, then after changing relations in the File Relations dialog, why does it produce a pop-up (see my 2nd attachment) that insists that I refresh the relations and tells me where to go to do that?

The dialog always does that when it detects a change made to file relations. It's hard and error-prone to determine if a relation refresh is needed and thus a general warning is given.