folder rescan mode that does not wipe out offline references

[mking]

With the increasing number of source files;
just 2hrs with a GFX100s macro focus stacking I generated 500GB of raw source data....

For me the solution I would like to have is to move source files to HDD offline with fixed disk letter once I have done my initial processing on them;
then have them referenced by iMatch as an offline file/folder.
Then when I need to access the source data in the future,  I can mount the HDD and iMatch will just find it.

The only issue seems to be that rescan currently wipes out offline references - not good in this situation.

If we can have a rescan mode that does not wipe out offline references that would solve this issue.

With the increasing image size, shooting FPS and options like focus stacking / high res modes, this must be an issue more and more users are facing.

Thanks,

Kind regards,
Mike

[Mario]

Keep in mind that the mentioned behavior only happens when the user forcefully rescans a parent folder. Not when IMatch rescans the parent folderdue to Windows messages about new or updates files.

[Tveloso]

Mike, If I'm understanding correctly, you would like to move your RAW (Source) Files onto an external HDD (and do so outside of IMatch), such that as far as IMatch is concerned, they are still in their original location (a local SSD), and they therefore become off-line files. 

Then the concern is that if you should re-scan one of the parent folders of the former location on the SSD (where IMatch believes those files still are), they'll then be removed from the database.

Could you instead also index the HDD target location in IMatch, and then perform the move operation within IMatch?  Via that approach, you get another media node (for the HDD) in the Media and Folders view, and then if you remove that HDD from your PC, that entire Media will become off-line.  Then there's no danger of rescanning/removing the Files, since the top-level folder(s) on that HDD will be off-line.

But if you have some requirement for keeping them in their former location (in the Database), the "Remove non-existing files from database" flag in Edit > Preferences > Indexing may already provide that functionality.  Mario, does that control flag pertain to both automatic, and user-initiated re-scans?

[Mario]

Have you tried to disable the option Remove non-existing folders from database under Edit > Preferences > Indexing?
This will do exactly what you want.

This is such a rare situation, I'd forgotten about this particular option.

[janb83]

Keep in mind that the mentioned behavior only happens when the user forcefully rescans a parent folder. Not when IMatch rescans the parent folderdue to Windows messages about new or updates files.

I stumbled upon this topic because I also have some off-line files on purpose, and noticed the pop-up for the off-line icon in the file viewer, which says "This file will be removed from the database when you rescan this folder". Now, in the indexing settings there is a setting called "Remove non-existing files from database". I have not set this setting, so I assume the tooltip is wrong?

Could you clarify what you mean by "forcefully" rescanning. A normal rescan via the context menu of the folder? What is the behavior in this scenario with the indexing setting set or not set. The help states: "This option is also on by default. When IMatch rescans a folder it removes files which can not be found on the medium from the database. This cleans up files which have been deleted in other applications."

Am I correct in expecting my files to NOT be deleted when this setting is disabled, no matter how a rescan is triggered?

[Mario]

IMatch does not remove off-line folders and files when it detects changes in folders and rescans them automatically.
It will remove non-existing folders and files when you explicitly force a rescan of the folder containing the files or the parent folder containing offline sub-folders.

This allows users to retain offline folders unless they really want to remove them.
The typical use case is that users move/rename files/folders outside of IMatch (not recommended).
They have the opportunity to relocate the files and folders to keep them and all associated data in the database. Or they can remove them from the database by manually rescanning the contained folder.

Why would you need to manually rescan a folder that contains non-existing or off-line sub-folders or files. This sounds like a kinda very special workflow.

Tip: Make a backup copy of the database (see Backup). Then change the options you want to change and perform the operations you want to perform. If you don't like the result, change the option back, restore your database (while IMatch is not running) and you're back to the original state.

Many users setup a small test database which indexes some copies of folders to make experiments. This way you cannot do any harm and are free to try whatever you want.

[janb83]

Thanks Mario, but I'm sorry, you wrote a lot of stuff, but none of it answers my question: How does the setting work? You still claim "it will remove non-existing...", but the help for the setting states the opposite. I assume what you actually mean is "by default, it will remove non-existing, unless you deactivate this setting". Is this correct? Yes, I can test the behavior, but I could be testing a wrong specific scenario instead of the general behavior without knowing it.

You also did not answer what "force" means to you. Do you mean a normal rescan via the context menu? It is not called "force" or "forced" in the UI as far as I can see.

As for why such a rescan is necessary: Apparently, you are forgetting about folder hierarchies. The folder containing off-line files might be a subfolder, and I might need a manual rescan of a folder further up in the tree. I might not even know that any subfolder deep down has off-line files, which is why I disabled this setting, to avoid accidental deletion from the database.

[Mario]

When you disable the option to automatically remove non-existing files in Edit > Preferences > Indexing and you manually rescan a folder using Shift+F5 or force a folder rescan with Shift+Ctrl+F5, you'll get this message:



and the offline files and folders remain in the database. This is the purpose of this very rarely enabled option. I think I have not thought about this option since adding it to IMatch 5 back in 2016.

If you think that something in the help is wrong or could be explained better, please use the "Let us know" link available at the bottom of each help page.

Apparently, you are forgetting about folder hierarchies.

Nope. The option works recursively.

The entire off-line folder/file handling was added to IMatch when storing images on CD/DVD/BR-D was a common thing, many years ago. Off-line drives (not folders or files) were normal. And I keep it around since many commercial and institutional customers still need it.

Having off-line folders or files inside a hierarchy managed by IMatch is quite rare, though. Usually a user mistake or another application renaming/moving files and folders around.

I also have some off-line files on purpose, a (...)

Your workflow is your workflow. IMatch will do its best to support it.

[janb83]

When you disable the option to automatically remove non-existing files in Edit > Preferences > Indexing and you manually rescan a folder using Shift+F5 or force a folder rescan with Shift+Ctrl+F5, you'll get this message:

This is not what I'm seeing. There is no such dialog displayed for me. Maybe I did set "Don't show again" by accident in the past, I cannot be sure. But thanks for clearing this up, so I understood the setting correctly. It would be a nice improvement if the tooltip for offline files did not say "This file will be removed from the database when you rescan this folder" if this setting is disabled.

Nope. The option works recursively.

Of course it does, or rather, that statement makes no sense. The option is not a folder-specific options, so of course it applies to every folder, hierarchy or no hierarchy. What I meant is that when you said that having to manually rescan a folder is a "very special workflow", you did not think about nested folder structures. As I explained, if you have nested folders, you don't KNOW that there might be offline files deep down in the nesting. And a manual rescan on any parent folder, for whatever reason, would remove these files from the database without warning, unless you disable this deletion setting.

So I'm very glad the setting exists, even if you say it was meant for CDs etc. only. Without this setting, the behavior would be questionable for most users. Silently removing files from the db completely just because of a rescan is not expected behavior, really. I'd prefer that to be an active "clean-up" command instead.

I also have some off-line files on purpose, a (...)

Your workflow is your workflow.

That is a weirdly condescending statement, as it is true for every workflow of every user. You are selling a product. If the product does not match user needs, it won't sell. So, by definition, customer workflows are the product workflows. Of course, supporting the most common workflows best is the right focus. But you don't know how many people might have legitimate uses of offline files (in fact, this thread alone has other users, I didn't start it), so simply shrugging it off like that is weird, it's basically a "Totschlagargument".

[Mario]

Maybe I did set "Don't show again" by accident in the past, I cannot be sure.

Re-enable all prompts, info tips, explanation popups etc. again via the Help menu.

It would be a nice improvement if the tooltip for offline files did not say "This file will be removed from the database when you rescan this folder" if this setting is disabled.

Looking at telemetry data, you might actually be the only user who has disabled this option. I guess when a user deliberately disables this option, he/she will be aware of that and interpret the tooltip accordingly.

That is a weirdly condescending statement,

I beg to differ. I just remarked on the fact that your workflow is your workflow and that IMatch does its best to help. But maybe it cannot do everything you want, or in the way you want it.

Every new user brings in new ideas and has different expectations for how IMatch should work for them.
There are many different types of IMatch users and many different types of workflows. IMatch cannot support all possible workflows equally well and neither would it be financially viable or a good use of my time to cater for every fringe case affecting one, two or a handful of users.

If you find something missing in IMatch or something that could be improved, you're welcome to post a feature request in the feature request board. Other users will see your request and may Like it or discuss it. I keep an eye on the feature request board.