Relations, visual proxies and updates

[Carlo Didier]

I am currently testing a potential new workflow with Capture One.
Currently, I use Adobe only and all raw files are converted to DNG with full size embedded previews for simplicity (iMatch automatically shows the preview with the develop settings applied and I don't have to handle several files for each image).

As I may switch to Capture One, I will have to make another workflow like this:
- raw files stay in their original format
- by default the JPGs from the camera can be used as visual proxies. Those can be replaced by new JPGs processed by C1 (although there is a complication as C1 won't allow to simply overwrite the existing JPGs, so this implies an intermediate step with a script. Oh how simple it all is with DNGs ...)

For now, I have setup the relation shown in the screenshots which works, sort of ...
The raw files are in folders by date on the D: drive and I put the JPGs for the visual proxies in a folder on the F: drive (as they are only visual proxies, I don't want them in the same folder as the raw files).
As I said, this works, but: After importing the raw files, I add the JPGs in the folder for the proxies. iMatch detects and imports those, but it does not automatically update the relations so that they are used as visual proxies. I have to select the raw files and manually ask to update relationships. Is that normal?

[Jingo]

Hi Carlo - I use C1 exclusively and have been requesting a JPG overwrite option during export for many moons now... such an easy code change but they don't seem interested...

I can't help with the visual proxy stuff because I only use JPG's in IM for now and only use C1 for RAW Editing... should I ever need to go back and make a new JPG from a RAW, I can easily find it because my filenames are identical - just on another hard drive... and I just export that JPG to a "temp folder" and then do a cut/paste into the watched IM folder path... that allows IM to pick up the new change and display it appropriately.

Good Luck!

[Mario]

If a visual proxy exists and the proxy is modified in an external application, IMatch will detect the change and re-import the visual proxy image. This has nothing to do with file relations.
What do you mean by "but it does not automatically update the relations so that they are used as visual proxies"?

When I have a RAW with a JPEG proxy in my database and I modify the JPEG in Photoshop, the appearance of the RAW changes as soon as IMatch has processed the modified proxy.
I just tested this and it works great. Actually, I work often that way.

Or do you mean that you create a visual proxy after the master etc. was added to your database?

I tried that as well, with my standard NEF (JPEG proxy) setup. A new folder with a NEF. Open the NEF in my RAW processor ans then produce a JPEG output.
Aa soon as IMatch has ingested the JPEG, the file relation is updated and the NEF uses the proxy image to represent itself in the File Window, Quick View Panel, Viewer.

[Carlo Didier]

Or do you mean that you create a visual proxy after the master etc. was added to your database?

I tried that as well, with my standard NEF (JPEG proxy) setup. A new folder with a NEF. Open the NEF in my RAW processor ans then produce a JPEG output.
Aa soon as IMatch has ingested the JPEG, the file relation is updated and the NEF uses the proxy image to represent itself in the File Window, Quick View Panel, Viewer.

Yes, that's the part that is not working here. First, I add the raw files. Then the JPGs. iMatch ingests the JPGs, but does not recognize them as visual proxies. Only after I do a manual refresh of the relations.

[Mario]

Since it works here, I would start by double-checking your file relation setup. Start with a simple rule to easy analysis.

[Carlo Didier]

Since it works here, I would start by double-checking your file relation setup. Start with a simple rule to easy analysis.

The file relation is correct, because when I trigger a manual "Refresh Relations", the proxies are detected as such and correctly displayed. It just won't work automatically.

Attached a debug log file and screen shots of my relation definition.
You can see in the log that I first do a database analysis which detects no problems.
At 9:18, the raw files were copied to an iMatch monitored folder. iMatch correctly detected and imported the new folder ("D:\Photos\Photos Carlo\2019\06\2019-06-27 UK - Scotland") and images.
At 9:30, I started generating the proxies in folder "F:\Photos\Proxies\2019". Those are immediately detected and added to the iMatch database, but no relations detected!
At 9:55, still no detection of the releation for the proxies. I select all the files in the folder with the raw images, then, from the "Commands" menu, I start "Relations", "Update Relations". Now, iMatch detects the relations and correctly displays the proxies.

I don't understand why it's not working automatically.

[Mario]

Check Edit > Preferences > Background Processing and make sure the "Automatically refresh relations" option is still enabled.

[ChristopherFoto]

Ich würde gerne mitmachen.
Die automatische Erstellung von Dateibeziehungen funktioniert auch bei mir nicht. Ich löse das seit einiger Zeit, indem ich einen Datumsbereich auf der Zeitachse auswähle und dann F4, R auslöse.
Außerdem habe ich zuerst die NEF's in der Datenbank und danach werden die JPG's und Tiff's hinzugefügt.
Ich habe auch die Einstellungen für die Hintergrundverarbeitung aktiviert.

[Mario]

Ich habe hier keine Probleme, auf 2 Win10 Rechnern mit der aktuellen 2019.6.2 Version.

Du sucht Versionen auf einem anderen Laufwerk? Und mittels {p0}... Bist Du sicher, das diese zum Zeitpunkt des Einlesens gefunden werden?
Hast Du das mal mittels Logfile überprüft? Vielleicht klappt was nicht, weil auf einem anderen Rechner in einer ganzen Hierarchie gesucht wird, oder was mit der {p0} Maske?

Was ist, wenn Du  Versionen im gleichen oder einem Unterverzeichnis des Masters hast (entsprechende Regel anlegen und testen).

[ChristopherFoto]

Ich habe nun das Log angeschaltet und ein neues JPG erzeugt im Ordner "d:\Bilder\Bilder-Sonstige\JPG 1920 x1080\2019-06-29_14-48-45_DSC_1291.jpg"
Die NEF's liegen unter " d: \ Bilder NEF"


Wenn ich die Hilfe richtig verstanden habe, ist die [/size]{p0} Maske, das auch im Verzeichnis in der NEF-Datei liegt zu suchen ist.
[/size]
[/size]Dies Log-Datei habe ich angehängt.

[Mario]

Ich kann im Logfile keine Fehler sehen.

War das ein Test mit Versionen im gleichen oder einem Unterverzeichnis des Masters, wie von mir angefragt? Oder Dein normales, nicht funktionierendes Setup?
Wenn was nicht klappt, müssen wir solange Dinge ändern, bis wir wissen, welche Einstellung auf Deinem System das Problem verursacht. Dann kann ich versuchen, das hier nachzubauen und zu testen. Sonst sehe ich wenig Aussicht auf Erfolg. Ich nutze Relations in meiner DB jeden Tag und habe keine Probleme mit Proxies usw.

Einfach Dinge abschalten oder die Relations ändern, bis es geht. Die zuletzt geänderte Einstellung war dann das Problem und wir haben etwas, auf dem wir aufbauen können-

[ChristopherFoto]

Hallo Mario,


welche Dinge soll ich abschalten.
Und wir lange warte ich nachdem das neue JPG importiert wurde, bis ich den nächsten Versuch wage?

[ChristopherFoto]

So ich habe nun mal ein wenig getestet.
Zu erst wurden die Einstellungen in der Dateibeziehungen auf die Standardwerte zurückgesetzt.
Wenn ich in den Beziehungen dann Hauptdatei-Verzeichnis auswähle funktioniert es mit einer Datei.
Speichere ich z.B. 3 Dateinen mit dem Namen "2019-06-29_14-48-45_DSC_1291.jpg", "2019-06-29_14-48-45_DSC_1291 1.jpg" und  "2019-06-29_14-48-45_DSC_1291 2.jpg" wird nur mit erste Datei eine Dateibeziehung hergestellt, die anderen Beiden werden nicht automatisch hinzugefügt. Dies geschieht auch dann wenn die 3 jpg'S zeitgleich (2-3 Sekunde) geschrieben werden.


Eine Automatische Beziehung in einem anderen Ordner als dem Hauptordner bekomme ich überhaupt nicht hin, mit F4,R funktioniert dies.
Was kann bzw. soll ich nun noch testen?

[ChristopherFoto]

Wenn ich in den Beziehungen dann Hauptdatei-Verzeichnis auswähle funktioniert es mit einer Datei.

Hier muss ich mich korrigieren, im Hauptverzeichnis funktioniert es ohne Probleme.

Eine Automatische Beziehung in einem anderen Ordner als dem Hauptordner bekomme ich überhaupt nicht hin, mit F4,R funktioniert dies.

Dies ist noch immer so, ich bekomme es nicht hin das aus einem anderen Verzeichnis Bilder automatisch als Versionen zu kennzeichnen.
Ich habe sogar eine neue Datenbank aufgesetzt und ein Testverzeichnis mit 185 Bildern Importiert.

[Mario]

In Deinem Screen shot ist der relvante Teil abgeschnitten. Du suchst nach Versionen in einem anderen Verzeichnis/Festplatte. Wie viele Ebenen muss IMatch dort durchsuchen, um Versionen zu finden?

Ich habe hier eine ähnliche Struktur (Bilder auf C:, Versionen auf D:) und das funktioniert, weil ich IMatch dort 3 Ebenen durchsuchen lasse. Das ist natürlich für die Performance echt mies, aber es funktioniert. Die Proxies werden gefunden und auch benutzt.

Schreib mal die genauen Schritte und Einstellungen auf, die Du verwendest. Welche Dateiformate, Verzeichnisstruktur usw. Wenn ich das hier nachbauen kann, kann ich vielleicht was finden.

[Carlo Didier]

Check Edit > Preferences > Background Processing and make sure the "Automatically refresh relations" option is still enabled.

It's on (see attachment).

[Mario]

Check Edit > Preferences > Background Processing and make sure the "Automatically refresh relations" option is still enabled.

It's on (see attachment).

Yeah, well. Then I would need more details. Enough information to exactly reproduce your file and folder layout, the files you are processing. All your file relation rules in detail and in the same order you are using. As I said above, I use this myself every day. And I've already tried to reproduce the "find proxy images somewhere on a different disk" and it works. There are so many things involved in file relations, without a 100% repro case I will be unable to analyze this further. I could waste a week without any results.

[ChristopherFoto]

Guten Morgen Mario,
Also ich habe folgende VerzeichnisstruckturIm Laufwerk D: gibt es die Verzeichnisse "Bilder NEF" hier liegen alle NEF's.Ebenfalls liegen im Laufwerk D: unter "d:\Bilder\Bilder-Sonstige\" die Proxies in der Regel in einem weiteren Untzerordner.Ich habe dir von allen Einstellungen und der Verzeichnisstrucktur Screenshots angefertigt und angehängt, in der Hoffnung du siehst alles was nötig ist.

[Carlo Didier]

Check Edit > Preferences > Background Processing and make sure the "Automatically refresh relations" option is still enabled.

It's on (see attachment).

Yeah, well. Then I would need more details. Enough information to exactly reproduce your file and folder layout, the files you are processing. All your file relation rules in detail and in the same order you are using. As I said above, I use this myself every day. And I've already tried to reproduce the "find proxy images somewhere on a different disk" and it works. There are so many things involved in file relations, without a 100% repro case I will be unable to analyze this further. I could waste a week without any results.

Can't you see anything in the debug log?

I'll try to create a small test DB to reproduce this.
Meanwhile, while importing the latest images, I noticed something:
When iMatch first detects and indexes the proxies and then later the raw files, then the relations are automatically updated correctly and the proxies are displayed.

[Carlo Didier]

Guten Morgen Mario,
Also ich habe folgende VerzeichnisstruckturIm Laufwerk D: gibt es die Verzeichnisse "Bilder NEF" hier liegen alle NEF's.Ebenfalls liegen im Laufwerk D: unter "d:\Bilder\Bilder-Sonstige\" die Proxies in der Regel in einem weiteren Untzerordner.Ich habe dir von allen Einstellungen und der Verzeichnisstrucktur Screenshots angefertigt und angehängt, in der Hoffnung du siehst alles was nötig ist.

Danke Christopher. Das entspricht im Prinzip dem was ich gemacht habe. Nur halt andere Pfade und RAF statt NEF Dateien.

[ChristopherFoto]

Hallo Mario,


ich habe eben zum Testen noch mal IMatch auf einem neuen System neu aufgesetzt, so das die Voreinstellungen alle auf Standard stehen.
Dann wurden die Versionierung und die Buddy-Einstellungen wie auf dem anderen System eingetragen (copy/paste)
Zusätzlich wurde noch in der Indizierung angepasst, siehe Screenshot.
Geändert hat sich aber leider nichts, nur die Bilder die im original Verzeichnis liegen werden als Proxy's erkannt, alle anderen immer noch nicht.

[ChristopherFoto]

Hallo Mario,


hast du hier etwa gefunden bzw. können wir noch irgendwelche Informationen liefern?

[Mario]

I write my reply in English so it fits for all users:

I have tried to reproduce this, and I think there is a misconception in how this is supposed to work.

The problem, as far as I understood this long thread, is that you add versions at some point in time, for masters which are already in the database.

In my case, I created a test case as follows:

1. I've created a folder c:\images\9154 with a number of PSD master files.
2. In d:\data\proxy files\2020\9154 I saved JPEG versions created from these these PSD master files.

When I now rescan the folder containing the PSD files, all versions on the other disk are detected, links are created and the files show up correct in the version panel.

I now create a new PSD compositing from some RAW shots and save it to the PSD master folder.
IMatch rescans the folder and picks up the new PSD files. No JPEG version exists so the new PSD file is not marked as a master file.

3. I create a JPEG file from the PSD and save it to the d:\data\proxy files... folder.
This DOES NOT automatically create a link between the PSD file on the C: disk and the new JPEG file.

4. I create a JPEG version in the d:\data\proxy images folder.
The file is added to the IMatch database.

5. I now create a PSD master matching the already existing JPEG version.
IMatch adds the new PSD file to the database. Since it has detected that this is a master file, it applies the version rules and detects the already existing version.
The link is created and the PSD file becomes a master.

This is the implemented behavior, as designed.

IMatch DOES NOT rescan the entire database for potential master files when it detects a new or updated file somewhere.
This would be disastrous for the performance. For every file that comes in and that is matched by one of the extensions used for versions, IMatch would have to process every folder in the database to find files which could be a potential master for the new/updated file. It then would have to run all enabled version rules to see if the new/updated file is a version.

This would take a very long time and would make adding new files super-slow. And also the rescans when Windows notifies IMatch of new or updated files.

When you add a version, you must refresh the relations of the folder containing the master to establish the link.
This is done automatically when the folder containing the master is rescanned, which explains why this usually just works correctly, unless you put your version files in a unrelated part of the file system (like in the case of ChristopherFoto). When he changes something on his D: drive, IMatch does not rescan the folders on the C: drive and also does not check for potential master files.

[Carlo Didier]

Ok, I see. This means that my workflow with raw files and visual proxies will not work as intended, because any additions or modifications to proxy-JPGs will not cause an update of the relation and the changes will not be displayed, despite the fact that the visual proxy has been added.
This complicates my workflow, but I will see if I can find a solution anyway. I just didn't want the visual proxies to be displayed beside the raw masters.

[sinus]

Hmm,
If we use masters (psd, raw...) and versions (jpg, png..) in the same folder, there will be no problems like this?

I hold masters and versions in the same folder and had never any problems.

[Mario]

This is a specific problem when you manage versions in totally unrelated folder structures (e.g. on another disk).

If you manage versions in a normal fashion (in the same folder or sub-folders of masters) this will work because making changes to versions (or masters) causes the master folder to be rescanned and hence new versions are detected automatically.

But IMatch cannot search (for example) 100,000 files for potential masters every time a file that could be a version is added or modified. Just to find the folders which may contain masters for these files and then refresh the relations for all folders found that way so masters can pick up new versions. And that for every file!

[Carlo Didier]

If you manage versions in a normal fashion (in the same folder or sub-folders of masters) ...

Why is that normal? It's just one possible way to do it. For me, different folders are normal.

[Mario]

If you manage versions in a normal fashion (in the same folder or sub-folders of masters) ...

Why is that normal? It's just one possible way to do it. For me, different folders are normal.

I used the term normal because, from what I know, most users manage masters and versions in folder hierarchies as described.
Storing versions and proxies on another disk is also supported by IMatch, but prevents certain automatisms as described above.

[Carlo Didier]

I used the term normal because, from what I know, most users manage masters and versions in folder hierarchies as described.
Storing versions and proxies on another disk is also supported by IMatch, but prevents certain automatisms as described above.

Then maybe that should be clearly explained in the help for that case.
For me, storing proxies in a different place makes sense, because I don't need to backup them and I don't need to see them beside the original raw (see my feature request).
Someone might have the originals on a slow disk (for cost reasons) and the small proxies on a fast SSD. There may be many other reasons to do so.

[Mario]

I'm pretty sure I have explained that in the help.
As always, if you find a typo, wrong or missing information in the help, use the link available at the bottom of each help page to provide feedback.

I cannot foresee all things users try or want (years) after I have implemented a feature and hence I cannot cover everything in the help. I recall adding several warning boxes and statements explaining a suitable folder structure to the file relation configuration topic.

[Carlo Didier]

I'm pretty sure I have explained that in the help.
As always, if you find a typo, wrong or missing information in the help, use the link available at the bottom of each help page to provide feedback.

I cannot foresee all things users try or want (years) after I have implemented a feature and hence I cannot cover everything in the help. I recall adding several warning boxes and statements explaining a suitable folder structure to the file relation configuration topic.

You should update this pop-up text in "Settings", "Background processing" (attached screenshot), as it doesn't mention the limitations. As it is, it should work as I had expected it.

[Mario]

Popup texts only allow for a very limited number of characters. I assume that users read the corresponding help topic entirely until they make changes to default values or assumptions about how something is supposed to work. Sometimes users make up their mind how something should work and if it works differently, they are surprised or baffled or angry.

This works the same since IMatch 5 and apparently very well for the majority of users.
Not every feature can work for every user. Sometimes things are too complex already or enhancing them would ruin performance for the majority of users for the benefit of few.

[Carlo Didier]

Forgot the screenshot ... here it is. Maybe you could just add "see help for details" or "restrictions apply"?

[Carlo Didier]

Or add that to the text at the checkbox instead of the pop-up.