OneDrive, Sync, Metadata Writeback Problems, Possibly Related to Face Tags

[Sir Nameless]

I've searched the forums for this problem and have seen a few users report similar issues, but without definitive resolution.

I have my entire image collection stored on OneDrive. I have the "Files On-Demand" setting OFF so that all files are on both my local hard drive and on the OneDrive service.  (As an aside, I have a separate, more formal backup solution.)

My iMatch installation is set up very close to the factory defaults. I set up file relations and a few view customizations. As far as I can recall, I haven't changed any metadata preferences from the factory defaults. The iMatch database is stored only on my local device and NOT on OneDrive.

What I have observed is that when I write back metadata (in bulk or one at a time with the pencil icon), iMatch will write the file, and OneDrive will upload a new copy, as expected. However, sometimes after a few minutes, OneDrive will download a new copy of the file. After many iterations of this process it seems to me that this could and does happen to any file, but some eventually resolve themselves after a few tries (with the metadata staying written and OneDrive reaching steady state). However, some files will repeat the write-upload-download-unwritten metadata cycle after many tries and don't seem to resolve themselves. It seems to me, although I can't make a conclusive claim, that files with face annotations are prone to this problem.

Since Mario in other threads has suggested that metadata errors could be part of this problem, below I have an example of one file with several Metadata Analyst snapshots to help try to get to the bottom of the problem. I've noticed that files begin with metadata errors, and the writeback resolves them. But after the download happens, metadata errors return. I can't figure out if OneDrive is causing the errors by modifying the new version of the file, or if it is simply downloading an older copy of the file. Help/solution appreciated!

------------------------------

Image file # 0055
Date modified in Windows Explorer: 12/21/2022 8:29 PM
Run Metadata Analyst

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 2:28:58 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_1.jpg
Errors: 2
Warnings: 14

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Metadata] Warnings: '[minor] Fixed incorrect URI for xmlns:MicrosoftPhoto'
Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match IPTC keywords.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match XMP keywords.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart+PreviewImageLength runs past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] [minor] Fixed incorrect URI for xmlns:MicrosoftPhoto
Warning: [Detailed Validation] IPTCDigest is not current. XMP may be out of sync
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

Trigger metadata writeback 2:29 PM
OneDrive begins file upload
Date modified in Windows Explorer: 2/15/2023 2:29 PM
Run Metadata Analyst

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 2:31:08 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_1.jpg
Errors: 0
Warnings: 10

Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart is past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

After a few minutes, OneDrive downloads the file
Date modified in Windows Explorer: 2/15/2023 2:30 PM
Run Metadata Analyst

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 2:33:03 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_1.jpg
Errors: 2
Warnings: 13

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Metadata] Warnings: '[minor] Fixed incorrect URI for xmlns:MicrosoftPhoto'
Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match IPTC keywords.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match XMP keywords.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart+PreviewImageLength runs past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] [minor] Fixed incorrect URI for xmlns:MicrosoftPhoto
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

Triggered metadata writeback (2nd time) 2:34 PM
Date modified in Windows Explorer: 2/15/2023 2:34 PM
OneDrive uploads file
Run Metadata Analyst

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 2:34:54 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_1.jpg
Errors: 0
Warnings: 10

Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart is past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

After a few minutes, OneDrive downloads the file
Date modified in Windows Explorer: 2/15/2023 2:35 PM
Run Metadata Analyst

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 2:37:13 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_1.jpg
Errors: 2
Warnings: 13

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Metadata] Warnings: '[minor] Fixed incorrect URI for xmlns:MicrosoftPhoto'
Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match IPTC keywords.
Error: [Keywords] Flattened hierarchical XMP (embedded) keywords don't match XMP keywords.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart+PreviewImageLength runs past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] [minor] Fixed incorrect URI for xmlns:MicrosoftPhoto
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

[Sir Nameless]

Update: I thought I'd try something else.

I made a copy of the offending file used in the test above, and moved it to a local-only folder, and changed the file name. I wrote the metadata and it stayed written. I then moved it to OneDrive and it uploaded OK. So far it has not downloaded itself again.

That would suggest to me that there is some kind of synchronization problem and not that OneDrive is messing up the metadata.

Also should add that I previously tried pausing OneDrive sync while making metadata writes (for a significant duration, 2 hours, long enough that I'd expect things to settle down) and still had the reappearing unwritten metadata problem.

Here's the Metadata Analyst results for the copy of the file that was written offline, then uploaded.

Code Select Expand

Metadata Analyst Results. Version 2021.18.4. 2/15/2023 3:40:33 PM
File analyzed: C:\Users\...\MTS_20220812_York_0055_DxO_LOCTEST.jpg
Errors: 0
Warnings: 10

Warning: [GPS] Date missing.
Warning: [GPS] Time missing.
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
Warning: [Detailed Validation] [minor] MakerNotes:PreviewImageStart is past end of file
Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9000 ExifVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning: [Detailed Validation] Missing required JPEG ExifIFD tag 0xa001 ColorSpace
Warning: [Detailed Validation] Missing required JPEG IFD0 tag 0x0213 YCbCrPositioning

[Mario]

That would suggest to me that there is some kind of synchronization problem and not that OneDrive is messing up the metadata.

The workings and mechanisms of OneDrive (or GoogleDrive, DropBox, ...) are undocumented and, in my experience, somewhat erratic. I have not so far experienced issues with OneDrive or DropBox breaking any of my files, though.

There is no solid documentation about how the Windows file system and these services interact. You basically store your files on another companies' computer and you can only hope that they do it right...

IMatch has no control over this or knowledge of when and how your files, or parts of them, are uploaded into the clouds provided by Microsoft or other companies. IMatch does not even know that the folder you keep your files in are virtual folders, simulated somehow by services provided by Microsoft or others. Or what this could do to your files.

I use OneDrive myself, but only for backing up locally encrypted files. Nothing goes into US clouds without having being encrypted before. We here in Europe (and especially Germany) are very determined about privacy. OneDrive is a 3rd tier backup for me.
I don't want any Microsoft AI to analyze my personal files after upload to serve me more ads, training or sell information about me to unknown 3rd parties.

Here's the Metadata Analyst results for the copy of the file that was written offline, then uploaded.

This looks OK. Typical metadata mess.

Camera vendors and device vendors not actually respecting the EXIF and GPS standards, implementing only a convenient subset of the standard, trying to get away with as litte as they can.

ExifTool is just reporting warnings about half-assed attempts of your camera/device/software of implementing metadata standards - but failing.
Skipping a number of the mandatory metadata fields required by the standard, adding some "custom" fields not included in the standard and youknowwhat for fun...

I have no explanation.

The EXIF standard is designed mostly by Japanese camera makers. And then they (and Google, Apple, ...) ignore their very own standards and just write willy-nilly, incomplete or bad metadata - because, they don't give a sh*t. And paying customers let them get away with it.

ExifTool (and IMatch) care for metadata.
Because we understand that metadata is important. Not only now, but also in 10, 20 or 50 years time from now. Digital asset management software like IMatch manages files over decades.

For phone and camera makers, the horizon is only the next trade show or business quarter. Or shareholder conference.
They don't care about phone or cameras sold 2 years ago, or images and metadata produced by these devices.

I usually just call this the metadata mess and get on with my life.
Thankfully, I have IMatch which cares, notifies me of metadata problems and allows me to fix them.

None of the problems reported for your files are serious and ExifTool will fix some of them after the first write-back in IMatch.

[Sir Nameless]

So, if I understand correctly, storing and managing an image collection on synchronizing cloud storage services is not supported by iMatch. 

[Mario]

So, if I understand correctly, storing and managing an image collection on synchronizing cloud storage services is not supported by iMatch.  

IMatch does not care. It manages your files on built-in disks, external disks, network shares and synced folders for OneDrive/DropBox/Google Drive. As long as the files are accessible via the Windows file system, IMatch manages them.

I keep some folders with images on local synched folders on DropBox and OneDrive just for testing purposes, for one of my test databases. Just a few RAW, JPEG and video files.

I modified metadata for some of the RAW and JPEG files in the synched OneDrive folder 30 minutes ago.
After a short time, OneDrive synched the modified files into the Microsoft cloud. Nothing else happened. The changes done by me (including current time stamps in the metadata) remained. Looking at the same files on my other PC showed the changes, so OneDrive synched the files back from the cloud to the other PC.

I use the default settings with OneDrive / DropBox to always keep the files both in the cloud and local, with auto-sync.

However, sometimes after a few minutes, OneDrive will download a new copy of the file.

This is what I find puzzling. It seems OneDrive somehow decided that your local files are out-of-sync with the cloud files and that the files in the cloud are never / better? What could trigger that? I have no idea...

When I google for

onedrive replaces local files

I see some reports about OneDrive wiping or replacing local files for unknown reasons. Maybe check these to see if the problem you experiencing is related? Maybe some setting?

Do you have perhaps enabled the "Preserve date and time of original file" in Edit menu > Preferences > Metadata 2 in IMatch?
This setting is only visible when the Export Mode is enabled for IMatch (Edit > Preferences > Application).
If IMatch is actively prevented from updating the "last modified" timestamp of a file after write-back, this could maybe confuse OneDrive...?

[Sir Nameless]

Mario, thanks for trying to investigate further.

Do you have perhaps enabled the "Preserve date and time of original file" in Edit menu > Preferences > Metadata 2 in IMatch?
This setting is only visible when the Export Mode is enabled for IMatch (Edit > Preferences > Application).
If IMatch is actively prevented from updating the "last modified" timestamp of a file after write-back, this could maybe confuse OneDrive...?

No, expert mode is off and I didn't change the preserve date & time setting. I double checked by turning expert mode on, restarting iMatch, and looking at the setting. Expert mode is back off. As part of my own troubleshooting, I did verify that the file modified date/time in Windows Explorer was being updated after metadata writes (see my original post).

I use the default settings with OneDrive / DropBox to always keep the files both in the cloud and local, with auto-sync.

Same here.

When I google for

onedrive replaces local files

I see some reports about OneDrive wiping or replacing local files for unknown reasons. Maybe check these to see if the problem you experiencing is related? Maybe some setting?

I have done some of my own similar searches and do see reports of bad syncing behavior. There aren't too many OneDrive settings accessible to ordinary users. All I've changed was the setting that keeps all files both local and on the network, which should be the safer option. That said, I have not observed this behavior with other software applications so far, and with iMatch I've only seen it when writing metadata. And again, while I can't be categorically certain, I've seen a correlation between sync problems and files that have face annotations. Did the files you tried have face annotations?

I'll try to keep digging at the OneDrive angle, but this is a rather complicated issue for a layman to try to figure out.

[Mario]

ExifTool writes all metadata on "one" operation, whether or not the XMP contains face regions created from annotations. This does not make a difference.

If you work with RAW files, ExifTool will update both the XMP file (for the XMP data) and also the RAW, to ensure the XMP and native EXIF/GPS data is in sync, including timestamps, digests etc.

ExifTool uses a temporary file while writing. This is basically the original image file with the changes done by ExifTool to the metadata. When ExifTool has finished the write and verified that everything went well, it deletes the original file and renames the temporary file to the original file name.

Many applications work that way, e.g. Photoshop. This two-step procedure ensures that the original file remains untouched if something goes wrong while saving/updating a file.

The final delete/rename is of course really fast and one usually only sees the temporary file when ExifTool crashes real hard while writing.

When I keep an eye on OneDrive while IMatch is writing back, OneDrive marks the written files as out-of-sync for a couple of seconds (or longer) and then finally marks them as current when it has done synching. No files are downloaded, no metadata is replaced or whatever.

If this causes issues on your PC, maybe temporarily disable synching. Or "finish" the files in a normal folder and then move them into the OneDrive folder?

[Sir Nameless]

Thanks for the info and explanation. 

If this causes issues on your PC, maybe temporarily disable synching.

I've tried this before but gave it another go. Turned off OneDrive sync, wrote metadata on files in one folder (17). Allowed 10 minutes to settle down. All files had metadata fully written (no pencils). Turned on OneDrive sync. 17 files uploaded. Approximately one minute later, OneDrive started downloading the same files again. When it was done, all 17 files were back to having unwritten metadata.

Or "finish" the files in a normal folder and then move them into the OneDrive folder?

Yeah. This will have to be my workaround unless/until this problem gets figured out.

[Mario]

Very strange.

[stzari]

Maybe a long shot but ...
could you check your timezone settings ?  I seem to remember that Onedrive uses Pacific Time. Depending on your local file system type and timezone settings, this could lead to problems.

[Sir Nameless]

could you check your timezone settings ?

Thanks for the suggestion.

My local PC time zone and time is correct.

The modified date/time stamp in iMatch = modified date/time in Windows Explorer = modified date/time in OneDrive on the web. All the times match for files that do experience the problem as well as for those that do not.

[JonesD14]

I too had my iMatch folders in OneDrive, both the master photo collection and the staging area I use for new content. I experienced exactly the same problem with these keywords when updating keywords in a file already in OneDrive.

I moved the staging area outside of OneDrive and do all of the initial metadata work there, then move the file into the master photo collection. If a later edit messes up the keywords, I move the file back to the staging area and clean up the mess. This deletes the file from OneDrive. Moving it back to OneDrive keeps the metadata without changing it.

It turns out OneDrive doesn't mess with metadata on the initial write, but restores the Microsoft XMP Last Keyword XMP and exif XP-Keyword tags (and sometimes Last Keyword IPTC) from the copy already in OneDrive. This causes the perpetual write back loop. OneDrive does not change hierarchical keywords.

It turns out Lightroom has the exact same problem. This post (https://www.lightroomqueen.com/community/threads/lightroom-with-onedrive-keyword-metadata-conflicts.41753/) explains the problem in the context of Lightroom. It is exactly the same behavior as when you're using iMatch.

As long as you don't get the write back loop. the old keywords in these fields don't cause a problem in iMatch. As Mario explains in other posts, iMatch ignores them.

I don't think this has anything in particular to do with face annotations. OneDrive messes up keyword metadata regardless of where it comes from in iMatch (face annotations, reverse geocoding, keyword panel). I have not had problems with timestamps or other metadata being overwritten.

I use OneDrive to share photos with my wife's computer (and vice-versa) and both computers have Photoshop Elements which reads the owned and shared OneDrive folders so all of our photos are available in Photoshop on both computers. Encrypting the photos in OneDrive might solve the write back problem but would most likely destroy the sharing since OneDrive couldn't read the files, so I haven't tried that. Getting phone pictures from both phones into Photoshop, and eventually iMatch, is a real plus from using OneDrive. I am exploring OneDrive alternatives, but haven't found anything that looks better yet.

[Mario]

Why does OneDrive, a storage solution, even dare touching your files?
It should just store them and give them back exactly as you uploaded them.

I think I remember that OneDrive not only scans all photos you upload (in the US at least, for "legal reasons"), but also by default ads tags (aka keywords) using AI. Might be a setting / option somewhere and this might be country-specific.
Looking that up, I've found:

Note: OneDrive automatically creates tags based on what it thinks the image is mainly about. Some tags may not match, and some expected tags may be missing. You can always add and remove your own tags or remove the automatic tags. Some tags may be slow to load and appear in the list.

Uploading family and personal photos to a storage solution like OneDrive and then learning that this solution scans the images with AI to learn about what they contain and to (optionally) add keywords rather irritating. This is the reason why the EU has the GDPR laws.

I'm quite sure that Microsoft makes good money from the data they gather about the photos people upload to OneDrive, and that they get the permission somewhere on page 350 of the Microsoft terms of service you have to agree to before uploading to OneDrive.

[Rene Toepfer]

About what kind of Onedrive are you talking: consumer or business? Onedrive Business uses versioning automatically and cannot be swtched off. Maybe this could be a reason for the problems. In Onedrive Business once a file will be updated (changes, copying "newer" version etc) the previous version of this file is still accessible and available.