2021 won't let me search

[Stevef48]

I'm trying to find video files, to run the 'Import to xmp' command.
It has been a trial:-
First imatch insisted that I have no video files. I discovered that both 'Description, Title, headline' and Filename were active. The search worked after I unchecked the former, the command ran, then imatch read metadata from files 3 times.
30 minutes later I altered the search term to mp4. Again there are no results.
iMatch now won't allow me to specify where to search. All of the options frrom 'Frequently Used Tags' to Location are greyed out.
I think that the options are greyed out, because it's just finished reading metadata again and is now it's adding and updating files again

What am I doing wrong?
Steve

PS Since upgrading to 2021 this morning iMatch has read metadata many times and ended suddenly 3 times. Debug logging is on, but I don't see anything relevant for the last event.

[Stevef48]

It finished adding and updating files, but won't allow me to do anything as it's reading metadata again the word Updating is visible in the keywords panel

[Mario]

30 minutes later I altered the search term to mp4. Again there are no results.

To find all video files, just use the File Format filter in the Filter Panel or set the File Window search bar to search file names and then search for .mp4 or whatever.

In both cases, remember that IMatch searches the scope (what is in the File Window). If you want to search your entire database, select the Database node in the Media & Folders View or the @All category in the Category View. See The Scope (Where to Search)

IMatch does not rescan files without reason.
It rescans folders for new and updated files when the folder "last modified" file system timestamp is different than the recorded timestamp in the database.
It rescans files when the "last modified" file system time stamp of the file is different than the timestamp recorded in the database.

The log file shows IMatch rescanning a bunch of JPEG files.
For each file, ExifTool reports IPTCDigest is not current. XMP may be out of sync, which is unusual if the data was written by ExifTool.
It seems that whatever software has updated the IPTC data in these files did not know about or bother to update the IPTC Digest (a sort of checksum).
This will go away when you write-back the files once in IMatch.

There are also warnings about the crappy xmlns:MicrosoftPhoto_1__1_ namespaces which was one of Microsoft's failed attempts do to some photo editing / metadata in some of their "home" products.

The log file covers only 20 seconds so there is not much more to tell.

[Stevef48]

I selected all 162k photos, but couldn't find a way to filter by file type. iMatch said that there were no MP4 files. For some reason Title etc. was selected as well as filename and it appears to have only searched the titles, so of course it didn't find mp4. In a brief interval between adding files and reading metadata I managed to uncheck Title and complete the search successfully.
As I said it has been busy most of the time since I installed it. When it's updating it won't allow searches, of the metadata from Title to Location, so I just left it for several hours.
It seems to have finished its adding/updating then reading metadata, which has been going since 13:30 (it's now 19:24), after several iterations.
I hope that it will be useable tomorrow.

You say that iMatch doesn't rescan without reason, but it has been the only application running on my laptop since it was installed. The only thing that changed files was iMatch, I noticed with the previous version that metadata needed to be written back immediately after I had run the command to write it back, sometimes 30-50% of the number of files that were written back. I've posted before about this issue.


I've tried to remove the crappy Microsoft data from xmp, but my attempts using exiftool haven't been successful. No idea what could have corrupted the IPTC Data, but as you say it should be OK now. 20 secs was the duration of that run of iMatch. As I said it just stopped.
Steve

[Mario]

When IMatch opens the database, it compares the folder "last modified" timestamp as reported by Windows with the database.
If there is a difference, only then IMatch will do anything.

You'll find plenty of AddorUpdate folder log file entries (which I did not see in the short log file you had attached).
IMatch does not write-back files by itself (unless you enable the immediate write-back options in the Background Processing settings).
Two write-backs are only needed if the existing metadata in the file is out of sync and IMatch needs two write-backs to fix the mess.

See Metadata for Beginners, especially the Metadata Problems and Pitfalls section in that help topic.

[Stevef48]

It still won't let me search, although I gather from the log that it tried to search albeit in the wrong metadata.
Info & Activity showed no activity, but the search options are greyed out see attachment 1. The program then stopped responding, so I waited for 5 minutes, then ended it.
Attachment 2 is the debug log.
The debug log from yesterday is 256mb, zipped it's still 8mb, do you want me to send it to you?  If so, please remind me where to send it.

FYI The recently added collection says that 7 files were added yesterday, 27,440 were updated. I expect that running with debug logging on slows things down, but all I achieved yesterday was to import the xmp data into 500 movies. Imatch also now has 42.5K files without Keywords; as far as I can see they are RAW images. iMatch 2020 showed zero last time I ran it.
Steve

[Mario]

Disable the Search Everywhere in metadata option you have enabled. This grays out all individual search options.
This is basically a global override.

[Stevef48]

I didn't enable that option! It appears to be the default for iMatch 2021. Every time I search the option resets to Search all Metadata.
I just did metadata writeback to 11K files. iMatch still says that 6,868 of them need metadata writeback.
I've also marked the RAW files that no longer have keywords attached as buddy and versions of jpegs. It hasn't made a difference still 42k RAW files don't have keywords.

I don't know what to do.
The only good news is that it has stopped endlessly reading metaadata, so I can try searching.

[Stevef48]

I managed to get the option to search filenames to stick and initiate a search.
5 minutes later there are still no mp4 files in the files window.
How do I determine whether the search has completed? The Info & Activity Window isn't displaying any activity.
I'll save the log file, turn Debug off and try again.
Steve

[Stevef48]

I clicked next to Search all Metadata to disable that option.
Clicked on Filename, the search icon confirmed that filenames would be searched. A note at the bottom of the screen said press return to search.
Pressed return, search scope change to Search all Metadata.
Tried again:- Entered MP4 in search text, Selected Filename from the dropdown list. Display cleared. Waited... 5 minutes later display is still empty.
I give up for today.
Steve

[Stevef48]

IMatch is still not finding MP4 videos.
I turned debug logging off and reduced the scope to one year, about 8K files 3 MP4s, the search scope has now stayed on filename. Not sure whether the search term should be mp4, or *.mp4, wil try both.
10 minutes later the binoculars are still being displayed on a red background in the File Window Caption Bar. Presumably this means that it's still searching.
Steve

[Stevef48]

I killed the search and selected the 8K records in the year. Search for mp4 worked.
It seems as though the search scope is defined by the records selected, not what's visible in the file window.
Now to see if it works for more files.
Steve

[Mario]

The file window search bar operates on all files in the file window, not only on the selected files.

I select the database node in a test database.
In the File Window I make sure that "File Name" is enabled and that regular expression is NOT enabled (else the . gets a special meaning - but does not matter in this particular case).
I enter .mp4 in the search bar and press <Enter>.
All 190 videos in this database are found.

If this does not work for you, double-check your search bar settings, if you have a filter enabled, maybe the videos are stacked and stacks are collapsed etc.

[Stevef48]

That's what I thought, but it didn't work for me.
The search worked when I selected files. Import XMP worked, but then iMatch read metadata 3 times, added/updated files, read metadata again (at this point I turned debug logging on again, add/updated files. I then had to leave the laptop. When I returned Updating was still displayed, but nothing was displayed in the info & activity panel. then iMatch closed without warning Debug Log attached.
The log shows lots of activity for files in 2019. I don't think that I've modified any files in that year since 2020.
There doesn't seem any explanation for the crash. There's a CEFLog from today but that just contains the following line [0904/081746.646:ERROR:gpu_init.cc(440)] Passthrough is not supported, GL is disabled.

BTW The crappy Microsoft data is still generating warnings, I'll check whether these are different files.
Steve

[Mario]

I'm not sure what you are doing.
It seems your database is stressed out to the max, and the search in the File Window is not able to create a "scope cache" in the database - because it runs in a timeout.
It seems you are trying to search your database while IMatch is constantly updating it in the background, thus invalidating the search results several times per second, causing the search to restart.
This cannot really work.

IMatch is indexing folders and files in the background.
Many files are current, many require updates because the XMP data has been updated (timestamp of the sidecar file has changed).
Search the log for File needs update to find the files.

ExifTool takes over 20 (!) seconds to extract metadata from only 5 NEF files, which is very too slow.
The warning about the broken Microsoft URi is normal, just ignore it. Microsoft messed that up big time years ago.

Is see several warning messages from the database system about databases tasks timing out. Not good.

So,

Make sure your database is not constantly scanned by your virus checker. If you use Defender you're OK.
Else make sure that you have an exception defined for the database folder.

Maybe your system is stressed out. Try to reduce the load under Edit > Preferences > Application: Process Control.
Reduce the number of threads to 4 for import and metadata import.

This should reduce the load on the system and the database and give IMatch enough space to squeeze in a search although the database is running at 100% utilization.

[Stevef48]

The main user of my PC seems to be Microsoft Search Indexer, I'm not running anything except iMatch and Microsoft Edge.
McAfee has changed and now won't let me exclude the database files, the name change on upgrading to 2021 screwed that up. McAfee support sucks.

The number of images without keywords is creeping up, it is 49k. I looked at one set of files M5290520.jpg, .orf and .xmp. The keywords have been wiped out of the .orf and the .xmp..
Steve

[Mario]

You are asking basically the same stuff now in multiple threads. Please stick to one thread.
I've just spent 10 minutes in your other thread to explain what you need to check to learn more about your problem.

https://www.photools.com/community/index.php?topic=11706.msg83693;topicseen#msg83693

Also explain if you did follow my advise from the post above.
You did not answer my questions.

All else is wasting time.

Like, please explain why IMatch needs to reload all your files?
Once in the database IMatch must never reload files, unless you modify them in other applications (!) or IMatch does a write-back.
Use only this thread now: https://www.photools.com/community/index.php?topic=11706.msg83693;topicseen#msg83693