only part-indexing of images

[sinus]

Hi Mario
Version 11.17.2 (the question is independent from the version, was also before)

I guess, I do something wrong since some versions.
Hence it is not related to versions.

It is always the same problem, I do not know, since when, but for sure since 3-4 monthes.
IMatch does not import all metadata (hm, maybe better exif?, because they are fresh out of the cam)

My steps, for example yesterday:
1) take pics with my Nikon D750, Nefs
2) transfer these pics into an IMatch-folder, auto-indexing is on (329 images)
3) IMatch imports them, fine
4) If I look at the metadata, some images are fine filled with metadata, some not Interesting is, one image can be fine, the next image, taken some seconds later, is not.
5) If I then let do a force update (Ctlr F5), after this all is fine, all images are imported correctly

Means, I have to do always a "force update" each time.
This behaviour I have almost every time, when I do import new images.

Not very important, but because I do later the renaming of such new imported files, some renaming is then wrong. Hence I must think before, to do a "force update".

I guess, I do something wrong.
Do you have any idea about this? I append some attachements.

[Mario]

This log file shows no entries for adding or updating files or reading metadata.

Please provide the log file from the session where you encountered the problem.
Add a folder and if the metadata is only partially there, copy the log file immediately.

[sinus]

This log file shows no entries for adding or updating files or reading metadata.

Please provide the log file from the session where you encountered the problem.
Add a folder and if the metadata is only partially there, copy the log file immediately.

The attached logfile was from this session.

...
11.11 09:13:16+   16 [1924] 00  I>   # Process Memory Info: WSC: 780MB, WSP: 805MB (NEW PEAK), PF: 1139751
11.11 09:13:16+    0 [1924] 00  M>   <  2 [82016ms #sl] CMainFrame::LoadDatabase
11.11 09:13:16+    0 [1924] 00  M>  <  1 [85688ms #sl] CIMatchApp::InitInstance
  _DSC4065.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4066.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4067.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4068.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4069.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4070.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4071.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4072.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4073.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4074.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4075.NEF
  a NEU importiert
  a Metas im Workflow
  _DSC4076.NEF
....

But I will just now do this a second time, maybe with only a few images.
I will set the log to debug-modus also.
Hopefully this time will be the same (Vorführeffekt), but normally it is always the same with a new import.

Hence I will add a new log, it will takes some minutes 

[sinus]

Ah.... your asking for a new folder solved MAYBE my problem. 

Attached the log from the same problem: partially metas only.

_DSC4082.NEF HAS metadata,
the next image _DSC4083.NEF has not!

BUT

I transferred the same images from my camera-card to an EMPTY folder: import perfect!!

Hence I guess, the normal folder, what I do new images copy, has too many images and too many subfolders.
I think, that this could be the problem.

If you think, this could be the case, I simply add a new folder and the problem is solved. 

It would also make sense, because this folder I use since some monthes or even years, hence there are too many images and subfolder in it.

Anyway I add the new log.

BUT I DO THE NEXT IMPORTS SIMPLY DO WITH A NEW FOLDER AND WOULD ADD HERE A POST, IF IT WORKS FINE AGAIN.

[Mario]

This log shows IMatch indexing files (tip: Search for AddOrUpdateFile to find the data, and for ET-Extract to find the data when IMatch uses ExifTool to import metadata).

_DSC4082.NEF HAS metadata,
the next image _DSC4083.NEF has not!

Interesting. I can see that ExifTool has imported data for _DSC4083.NEF but not for _DSC4082.NEF (did you remember wrong?)

The log reports that data was extracted for these 20 files:

_DSC4083.NEF
_DSC4084.NEF
_DSC4085.NEF
_DSC4086.NEF
_DSC4087.NEF
_DSC4088.NEF
_DSC4089.NEF
_DSC4090.NEF
_DSC4091.NEF
_DSC4092.NEF
_DSC4093.NEF
_DSC4094.NEF
_DSC4095.NEF
_DSC4096.NEF
_DSC4097.NEF
_DSC4098.NEF
_DSC4099.NEF
_DSC4100.NEF
_DSC4101.NEF
_DSC4102.NEF

(IMatch always processes files in batches of 20 files).


The _DSC4082.NEF is not included in that batch and there is also no other batch...?

_DSC4082.NEF is added to the database, checksum is generated, cache image is created, everything.

Metadata is only extracted for 20 files. But 42 files are added in that log and these files are also loaded into the file window.
I see a lot of activity calculating categories and collections, preview panel.
Several warnings about IMatch failing to create cache images for video files. Which is strange because ffmpeg normally has no problems with mp4 files created with Nikon bodies. Can you send me a sample?

But all this does not explain why metadata was extracted for only 20 files when 42 files are imported.
At least I have no idea why.

ET-Extract is called only once. It should have been called 3 times (20,20 and 2 files).
Does IMatch report files in the queue (check in Info & Activity panel). IMatch was still very busy when you took the log file, maybe it was not yet finished? Video processing takes a long time...

[sinus]

Sorry, Mario.

I am bit overhelmed.
It is simply outstanding, how serious you take every post, let me sometimes speechless.  Simply great, another programer (OK, not all) would has answered, fine, then there are too many images and subfolders, create a new one and problem solved.

But not Mario! 

This guy takes every single problem or irritation serious and tries to help his customer! GREAT! 

Hm, to my problem: Since I have imported my last video, this is for sure some month since!
I guess, there are some old videos in this too big folder.

I imported actually only nefs, nothing else.

I guess, this folder, where I used to import new files in IMatch, has too many files and subfolders:

gemäss Explorer:
- 66, 3 GB
-4059 files
-52 folders

I would suggest, that I create a new, fresh folder for importing (NO problem) and see, if it works.
If not, I would send a new log and describe the problem again.

What do you think? Is this ok for you to wait, if this occures also with a fresh folder?

[Mario]

5000 files is nothing. I'm quite sure that this has nothing to to with this strange issue.
Even if you have complex file relations and IMatch has to scan 5000 files every time a new file comes into the database (to detect versions/masters) it would just slow down processing a bit. But I see no warnings bout unduly slow operations in your log file so this is all OK.

Did you check the Info & Activity panel? When IMatch has finished processing it will show that the queue is empty.

[sinus]

Thanks, Mario, for your answer.

Well, I think, the Activity panel showed, that IMatch was finnished, but I am not sure.

I will try this whole stuff again, with a proper log (debug) and with better infos.
The problem is, that I have just now no time. Well, if your wife does calling you have usually better to go home. 

[sinus]

Hello Mario

FOR ME this is solved, because I am quite sure, that it is related to "too many files and/or subfolders in a folder".

I took 51 pictures (nefs, Nikon D750)
Then I copied these pictures form my SC-card into an empty folder, indexed by IMatch.
Automatically importing new images in on.

No problem, all images has been imported automatically correct, all Metadatas are here.

Then I copied the same images from my SD-card again, but this time into the normal folder, what gave me since monthes problems, because only partiell importing Metadata.

Well, then here the same, automatically import from IMatch, but roughly half of the 51 new images has only partiell infos.
Select these images again and Ctlr F5 and "force update" does then work fine, after such a "force update" all is fine.

This folder has, like explorer says:
- 66, 3 GB
-4110 files
-52 folders

So it is clear for me, if I work with not such a big folder, then all if fine!
FOR ME, you can leave this, but if you think, you want have a look, I attach the log from this session with the import of the 51 images, only partiell metadata.

I did just a second test, same result.
I will spot this during the next importing of images, and if there is also something wrong, I would post it, but I am quite sure, it is all fine now.

[Mario]

This log file shows IMatch importing one batch of 20 files, then one single file.
I don't know how this relates to your post, though.

If this is limited to one folder, create a new folder and move all files. See if the problem persists.
Otherwise, disable all file relations and see if the problem persists.
Do you use background write-back (E > P > Background Processing)? If so, disable it and retest.

[sinus]

This log file shows IMatch importing one batch of 20 files, then one single file.
I don't know how this relates to your post, though.

If this is limited to one folder, create a new folder and move all files. See if the problem persists.
Otherwise, disable all file relations and see if the problem persists.
Do you use background write-back (E > P > Background Processing)? If so, disable it and retest.

I tried again, but it is like I wrote:

It works with a new folder.
Hence the problem is this folder or (what I think), because this folder has to much files/folders.
Write-back is disabled here.

If I have more time, I will do a test with another folder with a lot of files and subfolders in it and report.

But first now I have to take pictures in the forest with x-mas-trees. 

[sinus]

Today again imported files into IMatch:

NO PROBLEMS!

Hence I think for sure, it was the folder with too many files and subfolders.

PROBLEM SOLVED!