Unable to delete "off-line" file

[Oglaroonian]

I deleted files outside IM in Capture One (erased them from disk), they were shown in IM with a yellow warning sign, but still visible. They are gone in the Windows 10 Explorer.
I did "Rescan now","Normal Rescan", " Force Update", "Reload Metadata", "Update Cache only" and "Swearing loud".
The reading said "the file is offline... will be removed from database when you rescan the folder", the picture itself vanished.
Even use of Database Tools was invain
Sure know I was to delete  in IM, but my workflow is doing that in C1
As far as I know I didn`t change the default settings.
How can I get rid of the files completely?
THX
By the way: Is the misspelling of bannnana part of the riddle?

[Mario]

IMatch removes off-line files when you rescan the folder containing the file. Just select the folder and press Shift+F5.
I've just tried and it works. No reports about related problems from other users.

If this does not work for you:

1. Switch IMatch to debug logging (Help menu > Support > Debug Logging)
2. Repeat
3. Attach log file. See log file

The misspellings in the questions are intentional. It keeps the SPAM bots out.

[Oglaroonian]

Shift F5 klappt wie bisher nicht
Hoffe, das mit dem Logfile richtig gemacht zu haben
Danke

[thrinn]

Are your Indexing options on default? Or more precisely: Do you have "Remove non-existing files from database" checked?

[Oglaroonian]

Yes, "Remove non-existing files from database" is checked.
By the way: when closing preferences now "GeoNames.org Reverse Geocoding" always pops up. Never ever used geocoding.
I´m not happy with the behavior of IMatch. But of course I`m probably the reason for the trouble.

[Mario]

This is very strange.
The database has been badly damaged.
The database system reports that critical tables, e.g. for collections, the timeline, relations, the graph are missing!
This looks like something went into the database and ripped out arbitrary tables. I have never seen such a damage before.

How did you create this database? Did you use any "tools" or other software on the database?
Did you experience any issues while creating the database like power failures, IMatch reporting errors or similar?
Since the database has only 30K files, is this a new database? Or converted from an older IMatch version?

IMatch does not even have code to delete these tables, because the are created once and then remain forever in the database.
Since the database system stumbles over critical errors all the time, nothing is working right. Like, rescanning folders etc.

This database is broken beyond repair. I recommend you restore the last working backup. Very bad. Sorry, about that.
Did IMatch not inform you during shut-down that critical errors were logged to the log file? It should have.

I have created some new databases with the current IMatch version and it creates all these tables as it should.

[Oglaroonian]

First, I´m only a user not working in IT.
I trialed IM just the other day, installed it without power loss, warnings and imported my pics, about 30k, and began to alter workspace, appearance etc.
I could not install the new update 16.12? on the trial version. On retrospection, I could not see anything in timeline, but being new in IM having no timeline could be ok. And yes, face detection did not work, but again, being new...


So I bought IM, loaded down the new version and installed it. It used the "old" database as far as I know.
The first issue with IM was that it did not handle my tiff-sidecars as expected. OK you explained, that tiff usually has no sidercars, I can only imagine the sidecars were created by DigiKam. But no warnings about critical errors etc.

Then I batch renamed some tiffs in IM, and tried to delete other in IM. I could not delete them, only mark them for delete.
I deleted them then in Explorer, but could not get rid of them in IM.

I run in trouble with malware bytes, it blocked exiftools. I tried to exclude IM in malwarebytes, still warnings in IM about not starting exiftool.
So I uninstalled IM, detected with  CCleaner some entrys that hinted, that malwarebytes had set up some rules concerning IM, which I deleted (remember I´m not in IT).

Then I reinstalled IM, used the only IMatch Database.imd5 (there was  a IMatch... .impag, which I could not relate to that time.. that is the compressed backup, I think now) I posessed (IM being new). Working oK, no problems with malware bytes.
I then tried deleting Pictures (this time raw) in IM, result see above. Deleting in Explorer worked, still remaining in IM.

I never used other tools on IM, only the database tools of IM, repair etc. After reading manuals

If I understand correct, restoring the last working backup (my only one) will not do the trick, because it is probably broken ( the .impag was probably made from the corrupt file).
So I have to install from scratch without using the backup, setting up all other things again.. ?

Merry Christmas!

[Mario]

16.12 is not an IMatch version number...?

No, this database (or backups of it which have been produced after whatever obscure event ripped critical tables out of the database) will not work either.
I have no idea how tables can be removed from your database. IMatch has no internal functionality to delete these tables. They are created once and never deleted.

I recommend you start with a fresh database. 30K files is not much and your database should be up and running in an hour.

+ If you have modified metadata in IMatch but not written back, write back the metadata so the new database can pick it up.
+ If you have created your categories, you can export them via the category export and import them into the new database. See Importing & Exporting Categories
+ Keep the damaged database around for while in case you need to export other things.

Note:

Before creating your database, go to Help menu > Support and enable debug logging. See log file
In case something goes wrong again, we'll have more information to work with.

After creating the database (before adding files), run a database diagnosis to see if there are problems.
Open the log file via Help > Support > View Logfile and search for E> to see if there are errors. If so, copy the log file (same menu) so you can attach it.

As I said, I have never had a support case where critical system tables were removed from the database. Hence I cannot give you more detailed advice.

What's about TIFF sidecar files? TIFF should never use XMP sidecar files (by the XMP and industry standards).
Even if your TIFF files have XMP sidecar files, IMatch detects and uses the data in them. It merges the XMP from the sidecar file with the XMP embedded in the TIFF (if any).
On write-back, IMatch only updates the XMP in the TIFF. You can then remove the XMP sidecar file. If DigiKam produces XMP sidecar files for TIFF images, they do it wrong and break established metadata workflows.

[herman]

[....]
I run in trouble with malware bytes, it blocked exiftools.
[....]

That is weird.
I use MalwareBytes myself (Premium version, on my system it works in tandem with Windows Security) and I have never observed MB blocking Exiftools.

What I have seen though is that Windows Security blocks Exiftools when you work with Controlled Folder Access (a way to protect from ransomware, to be found under Windows Security, Virus & threat protection, Ransomware protection).

[Oglaroonian]

Thank you for your advice!

No 16.12 is no version-number, it was the guessed release date: 16.12.2020 in DIN, sorry for the uncorrectness.
Unfortunately I already had started, when I read your additional advice.

But it does look better now:

- have entries in Categories, Timeline and People now
- Backup took 3s, not more than a minute
- IMatch database diagnosis logfile had only 4 warnings;
    Warning: File [2254] face [70] has a mismatch in face/anno sort order.
    Warning: File [3205] face [127] has a mismatch in face/anno sort order.
    Warning: File [3262] face [133] has a mismatch in face/anno sort order.
    Warning: File [7807] face [267] has a mismatch in face/anno sort order.
    dont know about that

my Test-folder was not added automatically in IM after importing raws in Capture one into my pictures folder, I hoped it would be so. Perhaps I should have waited longer, but even after a restart of IM it did not occur.
Rescan did the trick.
The file was marked red in IM after marking it in C1.
It was marked yellow in IM after deleting it from disk in C1
and
tataa
was eliminated in IM with rescan

and now I`m able to delete files from IM directly.

So everything seems fine now, I start importing the second half of my files.

(But it took approximately 3 hours the first time)

And with the tiff: these are scanned positives which I treated with Lightroom, DxO, Darktable, PhotoSupreme, Luminar, Capture One and DigiKam on my quest for the best software. I always tried to write changes to/in the file, so I dont really know, where the xmp come from.

Capture One is now (with Lightroom Abo and cloud...) by far my raw converter of choice, but its DAM capabilities suck.

So my hope was to use C1 for developing raws and IM as DAM.
Looks way better now than the last days...

THX
Michael

P.S Was bedeutet die Schraffur über den Bewertungssternen in manchen files?

[Mario]

so I dont really know, where the xmp come from.

The writer (application creating/updating) the XMP record is supposed to write it's name and version number into the XMP-xmp:CreatorTool tag.
You can see this in the Metadata Panel in Browser layout or in the ECP. IMatch, for example, writes currently photools.com IMatch 20.12.0.4 (Windows)

my Test-folder was not added automatically in IM after importing raws in Captur

Many users don't want IMatch to automatically add folders. For very good reasons.
You can either do a quick Shift+F5 on the parent folder, or enable the advanced monitoring in Indexing. Make sure to read the help to understand the possible side effects. You may also want to exclude the "stuff" folders Capture produces everywhere as shown there.

P.S Was bedeutet die Schraffur über den Bewertungssternen in manchen files?

This means that the label names you have used are non-standard and not in the built-in list in IMatch.
And this means that IMatch has no color associated with the label names you use.

Labels are just text and when you use half a dozen applications to produce metadata for your files, you will end up with proprietary label names.
You can tell IMatch about your custom label names and link them with a color Metadata Confiiguratinon.
IMatch can even scan your database and import all labels names your various applications have used. Very comfortable.
Once your labels have a color, IMatch shows them everywhere it shows label colors.

Warning: File [2254] face [70] has a mismatch in face/anno sort order.

This means that the order of the face metadata in your files does not match the order of faces.
Written by another application most likely. This is not a problem per sé, but since IMatch does a lot more with person data than other applications, it can be for you.
For example if you let IMatch list persons under the image or in the Viewer, and the list does not match the order of faces in the image.
See Ordering Persons in Images for more info.

To find out the file for the number in the [], use the handy File Data App (in the App Manager).

[thrinn]

my Test-folder was not added automatically in IM after importing raws in Capture one into my pictures folder, I hoped it would be so. Perhaps I should have waited longer, but even after a restart of IM it did not occur.

IMatch does not automatically add new folders to the database. But it will add new files if the folder itself is already indexed by IMatch. Please have a look at https://www.photools.com/community/index.php?topic=10755.msg76740#msg76740 - some other user had recently very similar questions, and Mario explained there why IMatch behaves like it does.

[thrinn]

As so often, Mario was faster and more precise than me 

[Oglaroonian]

Many thanks for all advice: I was aware, that I had to learn much, but didn´t know how much

scio ut nescio

I read the advice, the suggested links and checked some new possibilities with that mighty tool IM.

Funny enough I found this in a tiff sidecar (excerpt because of DSGVO):

tiff:Model="Nikon SUPER COOLSCAN 4000 ED"
   tiff:Software="Nikon Scan 4.0.3 W"
   xmp:CreatorTool="photools.com IMatch 20.12.0.4 (Windows)"
   xmp:MetadataDate="2020-12-19T17:16:57+01:00"
   xmp:ModifyDate="2020-12-19T17:16:57+01:00"
   xmpMM:DocumentID="xmp.did:3303087e-32e3-3746-b8d4-4c6fbbce7081"
   xmpMM:InstanceID="xmp.iid:e41788dd-69ac-45d6-abc3-f1ace18db8e0"
   xmpMM:OriginalDocumentID="xmp.did:3303087e-32e3-3746-b8d4-4c6fbbce7081"
   xmpMM:PreservedFileName="19810818_Galicien_041.tif"
   acdsee:categories="<Categories><Category Assigned="1">Brunnen</Category><Category Assigned="1">Kind</Category></Categories>"
   exif:ExifVersion="0220"
   exif:FlashpixVersion="0100"
   exif:ColorSpace="1"

So IM wrote this sidecar to a tiff?
But I read it should write only into the tiff, not into a sidecar?

Thank you all for your help

Michael

[Mario]

IMatch/ExifTool by default do not create XMP sidecar files for TIFF files.
Did you check the TIFF file itself? Does it contain an XMP record? You can check in the ExifTool Command Processor, or, less creepy, in the Metadata Analyst

[Oglaroonian]

Thank you for your patience Mario,
I´m now quite happy with IM.

But I tried to use the ExifTool Command Processor on the image in question just for curiosity.
(I was not able to find The Metadata Analyst)

So this is the reading copied from ExifTool Command Processor:

ADMIN: Removed massive block of ExifTool output and attached it as a file.

I`m quite sure, this is entierly my fault struggling with deleting etc, only I was surprised to read photools in the sidecar.

But I´m happy that everything is working smooth now, even the orientation of the photos is right again, no more crytic pop ups, my old keywords now shown as categories without my doing, same with People - some now shown because of my tries with face recognition in DigiKam? -, timeline established - but only partially for the second import -, work to do for the holidays.

Frohe Weihnacht

Michael

[Mario]

PLEASE do not just copy and paste such massive amounts of text into your posts.
It makes your post unreadable and fills the community search engine with junk.

Just copy & paste into Windows Notepad, save to disk and attach the file.
I have done this this time for you.

I was not able to find The Metadata Analyst

You can find all IMatch Apps in the App Manager.

You have two different XMP records for the same file. This is not good. Just delete the XMP sidecar file and work with the embedded XMP record.