Massive duplication of database entries - what went wrong? (v.5.6.20)

[Panther]

Something weird just happened that I hope someone can explain and help me with.  I scanned in 50 new photos and saved them into my main database image folder, business as usual.  However, when I went into iMatch, there was no activity or indication that iMatch was aware that there were new images in the folder (unlike before, when normally I would see the newly-added images in the "unassigned images" category or at least some indication that iMatch was working in the background scanning the folder).

I waited a while, and still seeing no activity I figured I would just tell iMatch to rescan the folder manually.  However, the first way I tried it (Media & Folders view, right-click and "Rescan Now") I found the "rescan" button but it was grayed out and wouldn't work.  So I looked around some more and found a place to add/update files (Commands/Add or Update Folders), browsed to my main database folder and told it to update the folder that way. 

That's when it got weirder - instead of just finding and adding to 50 new images, it appeared to be processing every file in the folder.  I was afraid to try to stop it in the middle of whatever it was doing, so I let it complete.  Then I saw that it had basically added a new, duplicate entry in the database for (almost) every file in the folder.

I closed out iMatch and checked with Windows Explorer - the 50 new files were indeed in the folder, and there was only one copy of each of the 2,416 .tif files in the folder, so the new/double entries in iMatch seem to be only spurious database entries and not actual duplicate files.  However, I also found some 2264 ".XMP" files (all with the same names as the .tif files except for the extension) in that folder, all of which were showing as last modified on the same date/time (10/18/2014 at 9:50 PM).  I don't remember ever creating or seeing those .XMP files before, and I don't know what they're used for (I've used iMatch and added other images since that 2014 date, so if they're something that iMatch is creating/using why are there not as many .XMP files as there are .tif files and why haven't any more .XMP files been created/modified since that 2014 date?)

In any event, iMatch now says there are 4,782 entries in the "All" category (when that should only be about 2,416) and 2,410 images in the "Unassigned Images" category (when that should only be the 50 new images I added today). 

Any idea what went wrong and what the best way to correct/reverse it might be?

[Mario]

Was the folder marked as off-line (yellow icon and tooltip telling you that the folder cannot be found)?
I guess so, because in that case the Rescan command is not available because IMatch cannot find the folder on disk. The tooltips tell you that.

Did you rename or more the folder outside of IMatch?

You added the folder now a second time to your database. IMatch still has it from the old location, and you added it again from the new location / disk.

You should have used the "Relocate" command to tell IMatch that you have moved the folder somewhere else!

To solve:

1. Make a backup copy of the database.
2. Remove the accidentally added folder from the database using the "Remove folder from database" command. NOT The Delete Folder command!
3. Select the original folder and choose "Relocate" from the context menu. Navigate to the new location of the folder to bring it back on-line.

Search the IMatch help index for the word Relocate to find detailed information about this.

[Panther]

Mario - thanks for the quick reply, but I had not changed anything about the location of the folder.  The database and the image files are on a removable hard disk which I keep removed from my PC unless I want to work on my database (I only work on it while offline, so I only plug it in before booting up with my Ethernet cable unplugged) - it's an internal HDD that plugs into an external SATA port on my case, and is configured to always show up as Drive D so that iMatch will always see the files in the same place.  It's always worked before, and I didn't see any offline warning (though TBH I wasn't looking for it since it has never been a problem before so I wasn't expecting anything).

The folder is in the same place as it always has been, so I'm not sure why iMatch would think it's in a new place or why it would have added it in again.

I'll go back into iMatch and try to see what it says about where it thinks the two duplicate entries are located.

[EDIT] - well, Mario, as usual you were spot on!  I thought having the drive letter/ folders be the same would be enough to mean that the files were in the "same place", but when I went back in I saw that iMatch had noticed that I was working on a test copy of my production database, which I had placed on a different physical hardware device/disk drive, and thus thought that the original files were offline as you suggested in your response (and I saw the exclamation point warning this time, now that I knew to look for it). 

So, I "removed" all the new/duplicate entries from the database, used the "relocate" command to point iMatch to the new physical HDD, and told it to rescan that "new" folder location.  Although it surprisingly processed almost all the files in the folder (some 2000 or so), perhaps because of the new exif tool and the metadata or something, this time it didn't duplicate all the entries like last time, and in the end it wound up with the right number of images in the "All" category (2,416) and the right number in the "Unassigned Images" category (50), so it looks like everything is back to working as expected at this point.

Thanks for getting me back on track!

[Mario]

IMatch identifies drives / media / folders by their name and the media id. A disk like R:, which is a CR/DVD/BR-D drive can hold different media at different times. Or when you work with external USB storage, different disks / sticks may be mounted under the same drive letter at different times.

IMatch scans folders in the background to find modifications. It does this after the database has been opened (this can take a while, but does not interfere with your work) and when Windows sends out "Folder modified" messages.

During the scan IMatch looks for files not yet in the database and for updated files. A file is updated when the "last modified" timestamp of the file reported by Windows is different from the time stamp IMatch has in the database for that file. Then the file is rescanned. Installing a new IMatch version or an updated ExifTool version does not cause rescans. This would be very unwelcome for the users with 300,000 to 500,000 files...

[Carlo Didier]

Maybe useful in this context: There are tools (from sysinternals for example) which allow to change the drive ID. I used that to put the same drive ID on my external copy drive as is on my internal drive. That way, and by assigning the same drive letter, I can use my iMatch database on the internal disk on my desktop and on the external disk on my laptop without having to do a relocate each time.
Works like a charm.