Versioned AFPHOTO files show up in unexpected spot in Timeline

[lnh]

IMatch is doing the best it can given AFPHOTO files are proprietary and unsupported by exiftool. That is old news, but nevertheless leads to some logical but unfortunate display groupings in the IMatch timeline view. Since IMatch (via exiftool) can't read/write simple things like the original creation date (exif: DateTimeOriginal), AFPHOTO files included in IMatch show up by the file creation date rather than the creation date of the file used to make that AFPHOTO file. Since I version the AFPHOTO file with a master (usually a RAW), I can mitigate some of the problems with AFPHOTO files (for me the lack of thumbnails continues, but that's another issue), but the timeline issue persists.

I understand why this is happening. What I'm wondering is if there any reasonable workaround for this problem. Certainly an IMatch created XMP sidecar file similar in concept to what is done for RAW files would make sense, but I don't see an option in the Metadata 2 file formats preferences to allow this type of setup. Besides I could imagine multiple sidecar files becoming messy. Any ideas?

Side note on AFPHOTO files:
If you look at the EXIF data using the EXIF panel in the Affinity Photo application, the proprietary file format is sucking in most available metadata. If you're using Affinity Photo as a RAW converter, it is keeping most of what is present in the RAW file. If you first import your RAW file into IMatch and have a XMP sidecar file with additional data, Affinity Photo ignores the sidecar file. If you create a JPG or TIF from your RAW (Capture One Pro in my case) and allow the RAW processor to embed metadata in the generated file, Affinity Photo will capture all embedded data as well. The bottom line is that Serif/Affinity is presenting an unnecessary point of friction for people trying to manage their files.

[Mario]

1. IMatch should be able to create an XMP sidecar file for your proprietary af* files. And when you set the created date and time in the Metadata Panel, IMatch will use that for the time line. The XMP timestamps, if set, are the anchor point for the timeline. Of course if IMatch cannot import EXIF data, it cannot initialize these tags for you.

The bottom line is that Serif/Affinity is presenting an unnecessary point of friction for people trying to manage their files.

This might be deliberate. Serif has announced that they will create (yet another) DAM. And if they can make it hard for other DAMs to support their files, they will draw more users to their DAM. Standard business practice: customer "lock-in" or "Retention". Which, as everybody knows, I don't like at all.

[lnh]

1. IMatch should be able to create an XMP sidecar file for your proprietary af* files. And when you set the created date and time in the Metadata Panel, IMatch will use that for the time line. The XMP timestamps, if set, are the anchor point for the timeline. Of course if IMatch cannot import EXIF data, it cannot initialize these tags for you.

I'm not sure if this is really a non-starter. I've established that the original date/time EXIF data does reside within the AFPHOTO files, but since IMatch via exiftool can't read them, it sounds like you are saying it won't really work anyways. If you run a AFPHOTO file through ECP asking it to display all metadata, all you get are the base Windows file properties.

Assuming I'm misunderstanding something here, the other problem in "Configure File Formats..." in Metadata 2 is that AFPHOTO only displays:
Allow create IPTC/EXIF/GPS
Use data in THM files

the option:
XMP sidecar file

is not present.

Again, assuming I'm not understanding how to approach this, the other issue would be file naming. It's very possible that I would have file names with the component to the left of the "." be the same for a master file (typically RAW), which then has an associated XMP file and an AFPHOTO file. There would have to be a unique naming or an alternative extension so the sidecars for the RAW associated XMP and AFPHOTO wouldn't conflict.

[Menace]

Offtopic about serif:

This might be deliberate. Serif has announced that they will create (yet another) DAM. And if they can make it hard for other DAMs to support their files, they will draw more users to their DAM. Standard business practice: customer "lock-in" or "Retention". Which, as everybody knows, I don't like at all.

I really like to have a well build alternativ "PS" and "Illustrator" by affinity. But, I also are skeptical about "serif". The prizing is very aggressiv (seems first good for us customers) and they want to create a broad bunch of software

- Illustrator alternative (done)
- PS alternative (done)
- QuarkXpress and Indesign
- DAM

I like it to have a competitor to Adobe, but I am note sure, if Serif will not be the same in 5-10 years: proprietary software and files (I hope, QuarkXpress can handle them); a lot of stuff insight like focus stacking, HDR, ...; this could be the death of small companies. I fear, that they will first get a market power like adobe and after they ruined the smaller competitor, they will reize the prize.

[Mario]

If you run a AFPHOTO file through ECP asking it to display all metadata, all you get are the base Windows file properties.

If ExifTool cannot extract data from these files, IMatch is at a loss. I have no information about how Serif stores metadata in their files, and I could not find any official source of information. See also this Affinity forum thread: https://affinity.serif.com/forum/index.php?/topic/31716-file-format-specification/

There has been one request for Affinity support on the ET Forum so far:

http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,8151.msg41723.html#msg41723

and Phill replied that he might look into this when there is sufficient demand. So, post!

Code Select Expand

Assuming I'm misunderstanding something here, the other problem in "Configure File Formats..." in Metadata 2 is that AFPHOTO only displays:
Allow create IPTC/EXIF/GPS


XMP is implicit.

Again, assuming I'm not understanding how to approach this, the other issue would be file naming. It's very possible that I would have file names with the component to the left of the "." be the same for a master file (typically RAW), which then has an associated XMP file and an AFPHOTO file. There would have to be a unique naming or an alternative extension so the sidecars for the RAW associated XMP and AFPHOTO wouldn't conflict.

I can't follow. Does Affinity Photo create XMP files? I don't have it, I have only Designer. Designer does not create XMP files.

[Mario]

I've just purchased Affinity Photo, just for metadata testing purposes and to be a customer and to have a better "handle" when reporting issues or demanding that Serif publishes how they store metadata in their files. Even Adobe sticks to the standards, and Serif should too. Unless lock-in is part of their plan.

[lnh]

I'll add my voice to the chorus asking for better open metadata support at both Serif/Affinity and exiftool.

Short of being open, it would be nice if Serif/Affinity did create even a optional sidecar, but they don't. My thought around the sidecar and AFPHOTO was me trying to create my own sidecar via IMatch (if possible) rather than expecting one (which doesn't exist) from Serif/Affinity. That was where my confusion around how you could name a sidecar file came from.

Thank you for getting AFPHOTO for metadata exploration. For as much as I'd like to see someone reverse engineer their file format, I could not imagine that being a worthwhile endeavor for you or anyone. If they want to compete in the DAM space at some point in the future, keeping formats proprietary is the wrong approach. They should compete by trying to build a better, more innovative and stable product rather than rely on lock-in. Not at Photoshop level, but still an OK editing program. I like some aspects better than Photoshop but also miss some capabilities either missing or a kludge in AFPHOTO.

[Mario]

Serifs online help is a bit Applish, which means that they explain only the bare minimum.
They claim to retain the full metadata imported from the original image in the .afphoto file, and to restore it when you export the file? I wonder how they make that work with maker notes and stuff. Update: They don't. When I import a NEF and export it as a JPEG, 50% of the metadata is gone.

ExifTool does not touch the file, it reports "unknown file format". That's that.

But the XMP workflow in IMatch works and is sound.
When I enter some metadata, including a create/digitized date and write-back, IMatch produces an XMP file matching the .afphoto file. IMatch then also puts the file on the timeline using the date and time I have entered.

Another observation: For a file created "fresh" in AP, I can see a thumbnail in Windows Explorer. For a file created by importing a NEF, I cannot see a thumbnail.
The thumbnail handler installed by AP seems not to handle all cases...?

[lnh]

But the XMP workflow in IMatch works and is sound.
When I enter some metadata, including a create/digitized date and write-back, IMatch produces an XMP file matching the .afphoto file. IMatch then also puts the file on the timeline using the date and time I have entered.

What settings do you need to have it create AFPHOTO XMP sidecars? Maybe the problem is I'm just using the recommended default settings in Metadata 2.
Below is a file view of a bunch of files which are related via buddies and versioning. The only XMP seems to be related to the RAW file.

Another observation: For a file created "fresh" in AP, I can see a thumbnail in Windows Explorer. For a file created by importing a NEF, I cannot see a thumbnail.
The thumbnail handler installed by AP seems not to handle all cases...?

So far I've had no trouble with either Windows File Explorer or Directory Opus showing thumbnails no matter how the AFPHOTO is created. Works when using the software as a RAW converter (extremely slow and not that great), or by opening a JPG or TIF created by another program such as Capture One. The thumbnail never shows up in IMatch, but I've accepted that due to the Serif implementation.

[Mario]

Just use the defaults. When you write-back, IMatch creates an XMP file.

[lnh]

I've figured out what was wrong.

I normally either click the yellow pencil icon for pending write back or use the collection window for pending metadata write-back. In my particular setup, that will not create a sidecar file for the AFPHOTO file. However, if I highlight that particular file (even though the yellow pencil isn't highlighted), and use the Command/Metadata Write-back/For selected files..., it does create the sidecar.

If the RAW file and the AFPHOTO file share the same file name (with the exception of the file extension), a unique sidecar isn't created. Instead the two "share" a sidecar and if something is changed it one file, it gets reflected in the other within the sidecar. Normally, my AFPHOTO files don't share the exact same name, but it's something for me to keep in mind.

[Mario]

I used the pen for my experiments. Its always the same, internally.

If the RAW file and the AFPHOTO file share the same file name (with the exception of the file extension), a unique sidecar isn't created. Instead the two "share" a sidecar and if something is changed it one file, it gets reflected in the other within the sidecar. Normally, my AFPHOTO files don't share the exact same name, but it's something for me to keep in mind.

By definition : an XMP file contains the metadata for all files with the same name in the same folder. See the XMP standard documentation for further details.

[lnh]

I've found that the XMP associated with the AFPHOTO  doesn't automatically buddy. I thought XMP was an exception and you're not suppose to explicitly buddy them. Would this be an exception I should bake into File Relations?

[Mario]

IMatch automatically link XMP files to other files with the same name in the same folder.

What do you mean by "not buddy"? What is not working?

[lnh]

IMatch created an XMP file for my AFPHOTO after doing Command/Metadata Write-back/For selected files...
The AFPHOTO had a unique file name so it wasn't in conflict with a RAW or any other file which couldn't write metadata directly to the file itself (e.g. JPG).

I then refreshed relations and had only the masters displayed and selected, and used the renamer to move the files to their permanent location (I use a yr/month/day scheme). All the files moved as expected except the XMP with the same name as the AFPHOTO. The AFPHOTO itself wasn't a master, but a buddy and version of the RAW file from which it was derived.

This is what lead me to think I might have to manually associate the XMP as a buddy even though I've not had to do that in the past with RAW file XMPs.

[Mario]

I'm not sure that I can follow...

When you move/copy/rename/delete the file beach.RAW:

1. IMatch checks if there is a beach.XMP
2. If yes, it also moves/copies/renames/deletes the beach.XMP
3. unless there is another file, e.g. beach.jpg.
In that case and with move/delete/rename operations, IMatch keeps beach.XMP and creates another .XMP for the renamed/moved/copied file.
This is independent from master / version relationships or buddy rules.

[lnh]

Mario,

Here is a sequence of operations, and what I'm seeing:

1. A single master file with a series of versions. Among the versions is an AFPHOTO file.

2. After I propagated data to Versions, the AFPHOTO XMP sidecar was created called DSC03726_8bit_100scale_TIF-v1_C1.xmp. This Directory Opus file view confirms it's there.

3) Using this renamer formula only on the master file,

4) The files are renamed as such,

5) However, the AFPHOTO XMP sidecar has disappeared.

6) If I do another propagate data to Versions, the AFPHOTO XMP sidecar file comes back and now named 2017_07_04_(15-44-37)_8bit_100scale_TIF-v1_C1.xmp.

7) Now I want to move this set of files their their final location using this renamer formula only on the master file.

8 ) And everything moved, but the AFPHOTO XMP sidecar file 2017_07_04_(15-44-37)_8bit_100scale_TIF-v1_C1.xmp has not been moved.

[Mario]

Renaming a master does not affect any of its versions. There is no propagation for renames.
I don't see how renaming a file could delete an XMP file, unless you have setup some really complicated version rules. Show us your version and buddy rules.

Do you perhaps mix version relations and buddy relations (for the same files)?
Do you have used the .XMP extension in any of your file relation rules perhaps?

Mixing version relations and buddy relations in such complex fashions can create situations where IMatch will be unable to carry out the operations.

[lnh]

Renaming a master does not affect any of its versions. There is no propagation for renames.
I don't see how renaming a file could delete an XMP file, unless you have setup some really complicated version rules. Show us your version and buddy rules.

Do you perhaps mix version relations and buddy relations (for the same files)?
Do you have used the .XMP extension in any of your file relation rules perhaps?

Mixing version relations and buddy relations in such complex fashions can create situations where IMatch will be unable to carry out the operations.

I'm guilty as accused. Call me crazy, but I love the ability of IMatch to handle renaming/moving via buddy and the capability to version the image visible (i.e. non-metadata only files). I iterated to this setup over the course of time and can't even remember the exact situations which lead me here anymore. So far I've seen very few ill side effects other than this one sidecar being left behind and I can remember seeing files in an IMatch included folder which I'd removed from the database still move with a rename.

Buddy:
Master Expression: \.(raf|rw2|orf|arw|cr2|crw|jpg)$
Link Expression: ^(_*{name}).*\.(jpg|jpeg|dng|tif|tiff|afphoto|afdesign|psd|dop|cof|cop|cos|comask|cot|exposurex2|ffproject)$

Version:
Master Expression: \.(orf|rw2|arw|cr2|crw|raf|jpg)$
Link Expression: ^(_*{name}).*\.(psd|jpg|jpeg|tif|tiff|dng|afphoto|afdesign)$