Some character(s) could not be encoded in Latin

[rolandgifford]

I've done a reverse GPS location search using Google on some images from Taiwan and am getting the warning in the subject when writing back metadata and the photos remain in a state where metadata needs to be written back.

I'm automatically creating a location Keyword based on the Reverse GPS

How do I get round this?

I have seen Thai characters against some of my Thailand photos and would have expected some Chinese against my China photos (and possibly other local characters sets elsewhere) but it only seems to be these Taiwan images causing a problem.

[Mario]

Please provide example data (coordinates) and one of the images you have used for this.
The most likely problem might be that the image contains legacy IPTC data and ExifTool was unable to encode your Thai characters. Need a lot more info on to even start figuring this out. Location data does not map into IPTC or EXIF, so I wonder which tag actually causing the problem.

Attach the requested data below or send to support email address. With a link back to this thread.
Please note that my "My metadata does not work" queue is always full and that it can take a week or two before I can look into yet another obscure metadata problem.

[rolandgifford]

I have done some experimenting and it is the generated Keyword which is causing the problem. If I remove that the metadata writes back and the image doesn't immediately pop back onto the write back list.

Latitude 24.224713 Longitude 120.895022 is an example

Reverse GPS on this single image generates the options.



Reverse GPS on a set of images automatically chooses the bottom choice with a location and generates this Keyword



If I then write back metadata, the image pops back onto the pending writeback set. If I then delete the Keyword, metadata writes back without it popping back onto the set.

I'm aware that Taiwan and China use different characters but I'm puzzled why only these images are causing a problem. I certainly have generated keywords from Reverse GPS with Thai characters, from mainland China with Chinese characters and I would expect to have some with other foreign character sets.

There are only 142 of these and I can easily fix them manually. I will probably be removing foreign characters in general, I haven't fully decided on that stage of my tidy-up.

My concern is for the future where a trip to Vietnam could hypothetically create lots more.

I've attached a debug log from a session writing back metadata for this single image.

[Mario]

I've tried to reproduce this with a JPG file.
Entered your sample coordinates, did a reverse geo-coding with Google and this yields:

Code Select Expand

City CITY Name
Location 雪山路出雲巷 27
ISO Country Code TWN
Country Taiwan
State/Province Taichung City
GPS Latitude 24.224713
GPS Longitude 120.895022
GPS Altitude 845.3
GPS Timestamp 2023:02:04 13:55:10
Hierarchical Keywords Taiwan|Taichung City|CITY Name; 雪山路出雲巷 27
(I've entered the city name by hand, Google does not deliver one).

Write-back works just fine. No warnings or errors reported.
Looking at the file with the ExifTool Command Processor shows that the data has been written correctly:

Code Select Expand

[XMP-iptcCore]  Country Code                    : TWN
[XMP-iptcCore]  Location                        : 雪山路出雲巷 27
[XMP-iptcExt]  Location Shown City            : CITY Name
[XMP-iptcExt]  Location Shown Country Code    : TWN
[XMP-iptcExt]  Location Shown Country Name    : Taiwan
[XMP-iptcExt]  Location Shown Province State  : Taichung City
[XMP-iptcExt]  Location Shown Sublocation      : 雪山路出雲巷 27
[XMP-iptcExt]  Location Shown World Region    :
[XMP-iptcExt]  Location Shown GPS Altitude    : 845.3 m
[XMP-iptcExt]  Location Shown GPS Latitude    : 24 deg 13' 28.97" N
[XMP-iptcExt]  Location Shown GPS Longitude    : 120 deg 53' 42.08" E
[XMP-dc]        Subject                        : Taiwan|Taichung City|CITY Name
[XMP-lr]        Hierarchical Subject            : Taiwan|Taichung City|CITY Name, 雪山路出雲巷 27
[XMP-photoshop] City                            : CITY Name
[XMP-photoshop] Country                        : Taiwan
[XMP-photoshop] State                          : Taichung City
Looks OK to me.
As I said, such issues are often caused by some metadata that's in the original file, like a non-tagged legacy IPTC record or similar.

[rolandgifford]

You have missed the step causing the problem.

IMatch is creating a Keyword from the reverse lookup using this statement:

WHERE|{File.MD.country|default:NO COUNTRY}|{File.MD.state|default:NO STATE}|{File.MD.city|default:NO CITY}|{File.MD.location|default:NO LOCATION}

If I write back metadata with the generated Keyword it fails, if I remove the Keyword it succeeds. I thought that I had explained that but obviously didn't make myself clear, sorry about that.

Taiwan use Traditional Chinese and mainland China use Simplified Chinese. I expect this to be the problem but only for Keywords. Hong Kong also use Traditional Chinese and I expect I would have the same problem there as well if I ever visit.

I'm OK fixing the Taiwan images manually by editing Keywords, there are only 142 of them.

I will be travelling more extensively in that part of the world in the mid-term where country/regional non-Latin character sets are the norm. It is this potential larger problem in the future which concerns me. If I can make some change so that it doesn't become a problem I will do that. It may be that Traditional Chinese is the only character set causing this problem and every other one is OK.

This is certainly not metadata that already exists for these images. These are not new images and metadata has been written back to them in the past. I can still do that if I discard (or edit) the generated Keyword, even after 'adding' the location data to other tags.

[Mario]

The keywords I've created where

Code Select Expand

Hierarchical Keywords Taiwan|Taichung City|CITY Name; 雪山路出雲巷 27
and which ended up in the file as

Code Select Expand

[XMP-lr]        Hierarchical Subject            : Taiwan|Taichung City|CITY Name, 雪山路出雲巷 27
The keywords include both Latin (English) and, I think, Chinese keywords. Whatever is delivered from Google.

[rolandgifford]

All very odd

I have removed all the the WHERE|Taiwan Keywords (but left any metadata location data) and written metadata back for all files which was successful for them all.

I then did a Reverse Geocode on all 2700 photos from Taiwan to rebuild the Keywords, write back and 381 (not 142) pop up again as still requiring write back.

I have tried copying the GPS co-ordinates to a completely unrelated file and reverse geocoded that and wrote back metadata without a problem. The same as you found.

The problem doesn't appear to be the Chinese characters which is reassuring in a way, thanks for looking at that. The problem photos have been taken on three different cameras, I can't see a pattern but I think that I can work round the problem by changing Keywords.

[Mario]

As I said in my first reply: no way to diagnose this without the original file giving you the trouble.
If ExifTool refuses to write or fails, the cause is usually some weird metadata in the file. Like untagged legacy IPTC or whatnot.

Delete the metadata in the target file with ECP or re-safe it to flush out metadata. Then write-back again. It will work.
Not worth spending more time on this from my perspective.

[rolandgifford]

Deleting the metadata using ECP did clear the problem. I needed to repeat the delete data/writeback pair more than once for some of the files which is odd. Delete followed by delete always told me that there was nothing to delete the second time and delete followed by show me the metadata always told me there is nothing there but it nevertheless didn't work first (or second/third) time for some of the files. I didn't bother looking for a pattern.

I'm still puzzled by undetected metadata problems in photos which have had metadata successfully written to them, which is what your comment implies, but will leave that as an unimportant niggle. Knowing how to fix it is all that I need and I have that now. Not worth your time as you correctly state. Thanks as always.

What does 're-safe it to flush out metadata' mean, I can't find that in the help?

[Mario]

I meant to re-save the file in your image editor.

That you have successfully written to the file before does not mean much.
This time you write keywords and other metadata with non-Latin characters. And if your files contain a non-tagged legacy IPTC record, just as one example, this might fail since ExifTool cannot know and mix different character set encodings in the same file. You can easily check for legacy IPTC records with the ECP.

Like with every of these obscure rare metadata issues, it's usually not worth spending time on them. Re-save the file in an image editor to re-create metadata from scratch or clean the mess with the ECP and then let IMatch write back a complete set of metadata to the fresh file with the next write-back.