mixing offline and online image folders on same "disk" from imatch perspective?

[mking]

Hi, looking for some advice, please.

Although I am a 15+ years imatch user, I have run into a new challenge.
I cannot realistically keep all my images on live SSD storage anymore.

With the arrival of the 102mp GFX and focus stacking, I generated 1GB of raw + processed tiffs from just 2 hrs of shooting yesterday!
At least for a while I want to keep all the data.

Ideally I would like to keep the final tiff stacked images locally on my live storage and have the source raw and tiffs on HD offline backup.
But to make referencing easy I would really like to have them appear as one folder tree (Windows 10+) something like

D:
>Forest
>stacked images - online
>Tiff images - offline
>raw images- offline

So I guess the question is can I mix online and offline images at the folder level rather than the disk level with imatch?

Thanks in advance for any advice.

Mike

[thrinn]

Would it not be easier to use Categories for this purpose? I mean, folders represent basically the physical position of your files, while categories are much more flexible.

[Mario]

An offline image (or folder) is just an image/folder IMatch cannot find in the file system.
If you delete/move out of files of a folder managed by IMatch, these files will be shown as off-line. If you delete/move/rename a folder outside of IMatch, the same happens.
IMatch does not find the thing and marks the thing as being off-line.

That said, some words of caution:

- IMatch does not remove off-line folders or files by the database by itself. But when you do a manual rescan of a folder, it will remove the off-line files (this is how to get rid off off-line files).

- Mixing off-line and normal files in a folder is usually not a good idea, for obvious reasons.

- I recommend to use the standard "source" and "output" sub-folder hierarchy. The naming is not important, but "source" are usually the source files you used to produce the files in "output". And "source" is usually also the folder you relocate to long-term storage or cloud/off-line storage. Your original RAW files and video footage etc.

- Keeping them in separate folders below a "project" or "shooting" or "day" folder does not impact IMatch file relations or other features, but it will make your live easier because you can take the source folder off-line much easier than mixing both source and output files in the same folder. That's not an efficient workflow.

Example:
...
|- 2022-01-04
  |- source
  |- output


depending on your workflow, you might extend this to something like
...
|- 2022-01-04
  |- source
  |- output
    |- edits
    |- composites
    |- deliverables
    |- client
    |- low-res
    |- web


or similar, depending on the number and type of outputs you create. I create these folder structures with a few keystrokes using the Create Multiple Folders command in IMatch and a string of folder names I keep in IMatch Notepad. I have several, for different purposes. When I edit the files in source in Lr / Ps / Affinity / Video Editor I put the results in the output folders. IMatch picks them up, does the relation magic etc.
When the project is done, I move the source folder to long-term storage and thus make it off-line.

[BanjoTom]

- I recommend to use the standard "source" and "output" sub-folder hierarchy. The naming is not important, but "source" are usually the source files you used to produce the files in "output". And "source" is usually also the folder you relocate to long-term storage or cloud/off-line storage. Your original RAW files and video footage etc.

A very helpful suggestion!!!  Thanks, Mario!

[mking]

Hi Mario,

Many thanks for the suggestion - the only catch seems to be the issue with a manual rescan removing the offline folders.
But what does "manual" rescan mean in practice? I often right click on a folder to rescan and add newly created subfolders to the database.
Is the folder right click > rescan what you mean by manual?  And if that's the case how do I get imatch to quickly scan and add new folders to the database?

Kind regards,
Mike

[Mario]

1. You cannot rescan offline folders.
2. When you Shift+F5 or Shift+Ctrl+F5 a folder which has off-line files or off-line sub-folders, these will be removed (that's the purpose of this command).

If you add new folders to a parent folders for which you have taken off-line some sub-folders and then you rescan the parent folder as you describe, you will be in trouble.
This is not a common scenario, though. If you work that way, you will need to come up with another solution, e.g. relocating all off-line folders to a separate folder hierarchy designed only for this purpose (keeping all future off-line "source" folders in a dedicated "source" branch and all outputs in another branch. A naming convention then links the folders.

[Darius1968]

This is probably where the IMatch categories (regular/formula) would come in handy. 

[mking]

Hi Mario,

Thanks for the suggestions.
While just rescanning a specific existing subfolder is not likely, I think rescanning a top folder - like say "England" of which there are many existing subfolders is very likely.
How else do I get imatch to pick up new subfolders? Or did I misunderstand something here?

I think it would be great to be able to have the option to rescan and not delete offline files/ folders.

I do think this must be a growing issue as users do focus stacking etc. and end up with 1GB of data in a heart beat.
Just 2hrs with a GFX100s macro focus stacking I generated 500GB of raw source data....

I have 12GB of vSSD locally and only 3GB remaining - I am not going to just keep adding SSD its too expensive, so looking for a hybrid solution and I can't imagine I am the only user with this issue.

For me the solution I would like to have is to move source files to 16TB HDD offline with fixed disk letter once I have done my initial processing on them;
then have them referenced by iMatch as an offline file/folder.
Then when I need to access it I can mount the HDD and iMatch will just find it.

The only issue seems to be that rescan currently wipes out offline references - not good in this situation.

If we can have a rescan mode that does not wipe out offline references that would solve this issue.

Thanks for any further guidance.

Kind regards,
Mike

[Mario]

Feel free to add a feature request. This will tell us how many users have the same problem and if it would be worth to add support for it.

Keep in mind that the mentioned behavior only happens when the user forcefully rescans a parent folder. Not when IMatch rescans the parent folderdue to Windows messages about new or updates files.

[Mario]

Have you tried to disable the option Remove non-existing folders from database under Edit > Preferences > Indexing?
This will do exactly what you want.

This is such a rare situation, I'd forgotten about this option. But now I've remembered.