Rescan problems

[markkums]

Hi there,

I met yesterday odd thing in some subfolders under my previous year main folder (locating on a separate HD). So, I found some blue dots in some subfolders (not many, only in a couple of subfolders in a whole year, but also in the subfolders of other years). I interpreted this as a sign for rescan needed (and this was also displayed in folder properties window when clicking on that particular folder name). Odd thing was that when commanding rescan (several tries with different options), this folder was duplicated.

Originally I had backgound processing switched off. When I turned things on in preferences, duplicating continued.

I tried to recall if I have made any changes (like raw processing of images, Capture One writes its own subfolders there under that particular image folder) in folders in question, but I believe, at least,not in all cases  ( considering all blue dotted folders). I continued comparing original and duplicated folder content. They look similar, except other but data driven categories are missing from duplicate image display.Diagnostics only found some orphan files in cache (and removed those), so, no errors/warnings after that.

I already returned my last week backup db, but no help. I should have even older backup in another geo location, so I have not yet tried it. Any ideas what could have gone wrong and how to proceed?

B.r: Markku

[Mario]

With "blue dot" to you mean the "Folder needs rescsan" icon?

IMatch has no feature which would duplicate a folder (copying it under another name) except maybe the Renamer.

If you mean that a folder shows up twice in your IMatch database, the only explanation would be that the folder has been added again from another media.
Compare the physical path and media id shown in the folder properties below the folder tree.
If a folder is again added from another media (disk), it has of course no categories or other data. For IMatch, this is a new folder.

[markkums]

Hi Mario,

Thanks for your reply.

Yes, I mean the icon.

By duplicate I mean that folders are displayed twice in a folder tree of the media & folders- view, no real duplicate if I look at disk content in the explorer. Originally there is no duplicates in the folder tree, but they are created in rescan-process. However, that blue icon in the original folder did not vanished after system had completed rescan.

I think, it is obvious that db has some damages which cause this odd behaviour although diagnostics in this case does not report any error/warning.

My questions are:
1) can you propose any corrective actions (I have already taken my backup into use, but it has same problem. I typically use Syncback software for all backupping in my pc. If I had used iMatch´s own backup, had it observed the problem?)?,
and/or,
2) is there in iMatch any means to save  existing category definitions of images in those folders? (altogether there are some thousands of image files)

When trying to figure out what could have happened, couple of things came into my mind: I do post processing and raw-conversion in Capture One (as I have done for years). CO saves its own folders as subfolders of an image file. In part cases blue icons are visible in those subfolders.  I typically start CO from Favorites-window in iMatch. I do not consider this very probable reason for the problem.
Another thing is the sharing of image files in home group. I have done this (only reading ), but who knows how Windows manages files in that occasion... However, I cannot, timewise, connect this problem with any of my actual sharing action (copying a jpg-file into another pc).

B.r: Markku

[Mario]

Did you do what I asked above?
IMatch cannot index the same folder twice.
If it is added again, the original folder was moved/renamed outside of IMatch and then you added it again from the new location instead of using Relocate.
The original folder the should show as off-line in that case.

I have now idea from your description what you are seeing.
Please provide sufficient info for us to help you, which means helpful things like screen shots of what you see, the IMatch log file etc.

[markkums]

Hi Mario, yes I did.

I could not find any data that shown duplications would be from other media or manipulated by another application, so, that is why I assume damage in db.

I shall prepare screen shots to display what happened.

B.r: Markku

[Mario]

If the diagnosis runs through without a warning, the DB is OK.
I have no idea how a folder can be added twice to a database. I suggest you remove the unwanted duplicate using the "Remove from database", then rescan the original folder (the folder where files have categories etc.).
Then carefully monitor your workflow and the other software you use and try to find out if this happens again, when it happens, what you did before it happened etc.

Effects like this which happen only once, on one computer, for one user and without a clear track of events which led to the problem are almost impossible to analyze after the fact.

[markkums]

Hi Mario,

I prepared a set of screenshots to explain what I reported earlier about rescan problems. I took my older backup of db to start with, as rescans that I did in the first stage altered the actual db and I could not return to the starting status. Background indexing is turned off.My system runs under Win 7. I also tested some parts in a separate laptop running Win 8.1 (by installing Pack&Go). No difference between those two.

So, screenshot1.jpg shows the folder tree at the startup of iMatch with no external storage connected. (Drives C: and D: are desktop internal drives, there are only the current year folders on the drive D:, Drive K: contains my older files, years 2005-2014. There is a drive L: having the year 2015). All problems seem to be related with drive K:.

There is also a blue icon in 2016 main folder. Screenshot2.jpg shows the folder tree after the rescan-command on 2016. Everything looks ok concering this rescan.

The screenshot3.jpg shows the folder tree after I have connected the drive K: with the desktop. As a result K-folders turn online and there are several blue icons visible (In fact, in addition to main folders, there are blue icons on lower level folders (each shooting session that I have done has its own folder): all 2006 lower level folders, some lower level folders in years 2007-2012, nothing in 2013-2014). I am sure I have not worked with all of those folders since the previous rescan.

Screenshot4.jpg shows what happen (and this is not normal) when rescanning a folder with blue icon in drive K: (the folder "2005 12 23" as an example): The resulting additional folder is shown separately on one level higher in the folder tree  AND blue icon will not vanish from the original folder icon. (This duplication does not exist in reality in the K-drive and cannot be seen in the explorer. )

This same takes place with all K-drive folders if rescanned. The folder path of the resulting folder in the tree, however, is shown to remain same as original. If a resulting folder is removed, it disappears, but nothing else seems to be changed. If original folder will be removed from db resulting folder sits on a wrong level in the tree and categories will be lost. And, next is strange, if I attempt to move resulting folder to its logical place in the tree it will be deleted (and now also from the drive!)

Images shown in the file window of this resulting folder are as in original folder except they are without categories. Db diagnostics and testing of drive by PC tools all are passed succesfully.

Attached log file has been copied after the rescan of the folder "2005 12 23" (i.e. in the situation shown in screenshot4.jpg).

[Mario]

I'm not sure that I understand the problem.

IMatch shows the "needs rescan" icon when the "last modified" timestamp of a folder reported by Windows is older than the timestamp for that folder recorded in the database. In your case, several folders in the database need a rescan. Since you disabled background rescan (not good) you have to refresh the folders manually. If you have software which updates these folders, you will always have to rescan them manually. Windows updates folders frequently, e.g. when the background search indexer runs, Windows Explorer updates the thumbnail database files stored in the folders etc. This is usually all handled automatically by IMatch when the background indexing is enabled. Usually you won't even notice this...

[markkums]

Hi Mario,

Thanks for your reply.

Yes, I have turned indexing off due to its impact on iMatch performance and I have done refreshing manually thus far. My typical work flow contains that I load newly shot image files (typically, into a new folder, by using Nikon ViewNX) into drive D: (internal HDD in pc) and then rescan D: in iMatch. Thus far this has worked without problems. I seldom have external drives connected as those files are older (from previous years).

I use another applications (like Capture One) for raw conversion and post processing. I start CO from Favorites-window of iMatch and input image files from iMatch. Other applications are Photomatix (for HDR processing) and PTGui for panoramas. All are used in same way as CO from iMatch. After the use of another application I have typically done manual refres. Again, this has worked ok for good time already.

At the time when I first time reported about rescan problems I had connected K: into pc and found out that after manual refres something odd happened  (resulting folder was added in the tree (not in the drive), as is explained by screenshot3.jpg).

I did try automatic indexing but in the explained situation results are those additional folders in the folder tree, and, pc really slows down.

This is a bit confusing. Is there something in above written I have done totally wrong? Or, is there more info or tests which could help to understand situation?

B.r: Markku

[Mario]

Rescanning can only slow down IMatch when there is actual work to do (new or updated images are found).
The quick checks IMatch performs in the background when Windows sends notifications to applications are not noticeable. Unless IMatch finds files to process, but even processing a couple of dozen new RAW files should not interfere at all.

At the time when I first time reported about rescan problems I had connected K: into pc and found out that after manual refres something odd happened  (resulting folder was added in the tree (not in the drive), as is explained by screenshot3.jpg).

Sorry, I can't tell anything from a screen shot. You did not even include the folder properties, which would show us the full path of the folder.

For me this looks perfectly OK.
You have files from drive C: D: and K: indexed in your database.
A number of folders are outdated and need to be refreshed. Since you have disabled background indexing, IMatch can refresh these folders when you trigger it manually.

[markkums]

No, I think it is not at all ok. If I understand right having automatic indexing or doining refreshing manually are alternatives and they should do same task: to update outdated folder, or should they?

Now, If I trigger refreshing in folder "2005 12 23", for example, this yields a new branch into the folder tree (result presented in screenshot4.jpg of my previous mail). I think, that kind of result is not correct.

The folder path was visible already in the same jpg, but here you can see now the whole property window in screenshot41.jpg (the property window of original folder "2005 12 23") and, in the same way the properties of duplicated "2005 12 23" in screenshot42.jpg.

Both original and duplicated "2005 12 23" display same full path, although the duplicated folder locates one level higher in the tree!

B.r: Markku

[Mario]

If the folder was just added, did you try to press <F5> to refresh the tree? Or close and re-open the database?

One of the folders reports 6 sub-folders, but the other does not. Perhaps IMatch was confused because you have disabled the automatic indexing and was trying to make the best out of it?
If the folder has just accidentally be linked wrong in the tree, pressing <F5> or closing and re-opening the database will fix it in a sec.

Besides of that, I'm not aware of any case where IMatch had indexed the same physical folder twice.

Except, the folder was externally renamed and then added to the database instead of relocating it and then it was again renamed. This would be probably the only sequence of actions I could come up with which could cause this. But I don't test such things.  It just don't happens.

[markkums]

Hi Mario,

Thanks for your response. I cannot recall all that I have tried in this case, but probably I did not use F5, so I will try it anyway. 

Fyi, all those folders in K-drive are old (i.e. existed several years. In some cases, if I use older image files, I do raw conversion once more with the latest converter software version and, typically,I have triggered manual rescan. However,  I doubt I have done conversions in all folders with blue icon). Same concerns renaming: I have not done any renaming in those old folders.

I´ll report later about results with F5-key.

B.r. Markku

[Mario]

When IMatch displays the icon, the "last modified" timestamp of the folder is newer than in the database.
You must be aware that Windows changes folder contents all the timre, e.g. when re-indexing, search engine updates, thumbnail generation etc. All this will change the last modified timestamp of the folder and IMatch has to rescan it to check for changes relevant to your database.

If a background indexing has any negative effect on your work in IMatch, either IMatch finds many new or updated files and thus becomes real busy for a while, or something else on your system brings down IMatch performance to a crawl. It is very unusual and not recommended at all to disable background indexing. You deliberately let your database get out of synch with the file system, which can have all sorts of unwanted consequences and side effects.

[Jean-Maetso]

Hello Mario and Markkums,

This topic does not look closed/solved. But I might have a solution right now... !

I have EXACTLY the same issues, and my problem is similar and reproducible at will, as described in the thread.
To summarize the main symptoms:
- My background indexing is on
- The folders that are claimed to need rescan are indeed old. The blue icon appears after the folder has been off-line then on-line again, or if I decide to make an arbitrary (theoretically completely useless) rescan on a folder I never touch for years. The problem also comes up if I try to create a new subfolder: a blue icon appear on the original folder, the new folder does not appear here, but under a duplicate created by IMatch elsewhere in the tree (at the first level, just under the volume-level).
- When such a folder is indeed rescaned, the blue icon does NOT disappear, instead a duplicate is created (without blue icon). If several rescans, only one duplicate is created, not a new one at each rescan.
- It is actually a TRUE duplicate: if original folder contains, say 10 files, the duplicate contains obviously 10 files as well, but one easily verify that the hard disk contains 10 files, not 20. No hidden files in this for sure. Same path, same folder, same name, same extension, and same metadata.
- The full path of original (with blue icon refusing to disappear) and duplicate are strictly the same, say F:\Pictures\2001\January
- The problem is reproducible in a strongly (and desperately) deterministic way
- The diagnosis does find anything
- The files in original folder are not shown off-line, and they have their categories
- The files in duplicate folder are not shown off-line neither but they have LOST their categories !
- Metadata, size, timestamp, etc. are identical for files in original and duplicate folders.
- When "remove from database" a duplicate, the duplicate disappear, but the original is still here, with its blue icon, and files in it have their categories
- When "remove from DB" on the original folder (one may think of getting rid off the blue icon and continue the life with the duplicate), the original disappear, the duplicate is still here with correct number of files and no blue icon appearing, but... categories are lost, as said above. Hours of work potentially lost.
- When moving the resulting folder to its right place on the tree (in the original's hierarchy), the disappearance is real: the folder and all its files are DELETED from the hard disk, I can restore them from the recycle bin... and the problem is back again ! (withstanding the loss of categories and so on...)
- When restoring from a DB backup, the problem remains.

All in all, it looks like the original folder has become "obsolete" (not offline because the physical media is here, but not "refreshable" within its file hierarchy. When rescanning, the folder is "viewed" as new and different by IMatch, so the operation which is actually performed is an ADD, not an UPDATE. This would explain somehow that all metadata and stuff embedded in the file remain intact (after all, it is the same file on hard disk, untouched) but all data maintained inside the DB are lost, the categories are not copied in the duplicate if it is viewed as a "different" folder.

The original file hierarchy is seen as "obsolete", and this could even explain why dropping the duplicate back to this hierarchy makes it deleted from HDD: it means you are dropping it "nowhere", --> thus recycle bin. This is strangely close to what IMatch apparently sees.

Nevertheless, the path are identical, so why IMatch does actually see the folders as distinct ones ?

When writing this, I may have an idea, thinking what I did recently on my system. I have changed the label of my HDD, the one you see in Windows explorer when selecting "Properties": a dialog box showing free space and used space, etc, and a text field allowing you to give a human readable label. I have changed "Photos and docs" into "PHOTOS". May this be sufficient for that IMatch considered the media is no longer the same ??

YES
I have just tried to revert to the original label ("Photos and docs"), restart the computer, re-open database, and the blue icons are GONE !
I still have duplicates visible, though: folder with no categories and folder with files/categories, etc.
HOWEVER now, blue icons are appearing on the previously "duplicates" and blue icons are gone from the previously "original": the formerly duplicate folder were related to the volume labelled "PHOTOS" and we are now back to a windows volume labelled as originally.
And the behavior of IMatch is as expected when manipulating the "original". Files in them have categories, No blue icon.

Therefore, I only have to carefully go through the duplicates that have appear (and have now persistent blue icons), and removing them progressively from the DB, checking that the original remain here, complete, and stable, and so on. But this looks like it is the case.

I hope this helps. After several days of getting crazier and crazier, I suddenly think of this "Volume Label" trick. What could be done so that IMatch does not mix up the same volume when one changes its label ?

Thank you for your support.
Best regards,
JM

[Mario]

When such a folder is indeed rescaned, the blue icon does NOT disappear, instead a duplicate is created (without blue icon).

IMatch only adds a folder again to the database it it comes from a different drive/media. Are your folders on a remote media (you wrote about off-line / on-line)? In that case, your NAS box / Server implementation may have mounted the drive with a different media serial number. The path does not change, but for Match, the folder is on a different media. But in that case IMatch would show the folder as off-line (yellow icon), not for a rescan.

IMatch only does rescans on folders it can find on your media. And if IMatch shows the "need rescan" Wiindows has either sent a "something in this folder has changed" event or the "last modified" timestamp of the folder on disk is newer than the timestamp recorded in the database.

[Jean-Maetso]

Hi Mario,
my folders are not on a remote media, they are on a local HD. I have never used off-line folder indeed. In my case, the problem appeared when I wanted to add a new subfolder:
- On right-clicking the context menu and selecting "add a new subfolder", I did not see anything happening...
- Then I refreshed and rescanned, telling myself "oh, it is just a display delay or processing, it is going to show up"
- But the new folder did not appear under my folder, and instead the blue "Need Rescan" icon appeared
- Looking lower in the folder tree, I realized that a duplicate folder was now here (at the root-level), with a subdirectory named "New Folder" --> the creation actually occurred, but not in the original folder, rather in a "newly displayed", duplicate folder.

All this stuff (including your explanation Mario, regarding Windows notification messages) looks consistent with the fact that changing the Windows Volume Label is considered as mounting the drive with a different ID. And therefore adding the folders instead of refreshing them. And having no category for files in the duplicates. But having all metadata correct. Etc.

IMatch only adds a folder again to the database it it comes from a different drive/media

The case is that it is NOT a different media: never been offline, never changed drive letter, only changed volume label.

But in that case IMatch would show the folder as off-line (yellow icon), not for a rescan.

It does not show the folder as off-line. And in the original folder (whose behavior is as if it was obsolete), the files are not shown off-line.

Note: I am using IMatch 5, not 5.5. Anything change in the new version about this ?

Regards,
JM

[Mario]

I'm not sure I can follow every detail of your explanation.

I have jus´t right-clicked a folder and IMatch created a new sub-folder for that folder named "New Folder". I tested both a top-level folder and also a folder somewhere down the hierarchy. Works as designed. If this does not work on your system, there is either something special with the folder you have selected when you execute the "New Folder" command, or something on the system, the other software you run etc.

Note: I am using IMatch 5, not 5.5. Anything change in the new version about this ?

Just see here: https://www.photools.com/4439/imatch-5-5-released/

Support for IMatch 5 (your version) has been discontinued on June 1.

[Jean-Maetso]

Hi Mario,

Sorry if any confusion.

In my (long) post dated from September 12, 2016, 11:24:48 PM, I confirmed to have the problem, in a reproducible way, and offered an idea which could be the solution. It relates to changing the Volume Label of the drive in Windows explorer.

In my last post dated from this morning 08:45:31 AM, I am describing the symptoms, the unexpected behavior that I saw after changing the Volume Label of my HDD.

In your post, you say you don't follow every detail. What you describe is the normal behavior, not the symptoms. I 100% agree this is what is expected and that this would happen when you do not change the drive label.
Please, change the Volume Label of your windows drive, and re-try to create a new folder. NB: I think you may need to reboot after changing the drive label.
Please tell me if the symptom I saw on my system also appear on yours.

Best regards,
JM

[Mario]

Changing a volume label is nothing that should ever be done. Geez.

Especially not for a system drive. IMatch may consider both the media serial number and the volumne label to detect media, in order to handle roaming environments and certain SAMBA versions used for NAS boxes and stores systems. If you have changed the volume label, you need to relocate the disk so IMatch can pick up the new volumne data from Windows and update the database.

[Jean-Maetso]

Thank you Mario for your reply.

I never got any problem with the change of a Volume Label. (I am not talking about Drive Letter C:\, D:\ or so on. But about labels "SYSTEM", "DOCUMENTS", "BACKUP", etc... Just to be clear)

IMatch is the only application I can remember of which is really sensitive to the label. Given the explanations, I guess we should take this as a good news.

However, this could be the source of curious things, such as: moving files from a directory (displayed as valid by IMatch) to another directory (displayed as valid as well)... and having these files deleted from the hard drive ! Or other strange situations: a directory with 5 files (physically) but IMatch displays "My_folder (10)" as if there were 10 files, and in the file window, each file is displayed twice ! (yes with the same name). We agree this is not normal, we agree on the cause, maybe we agree this should be avoided.

Since IMatch is sensitive to the label changes, is there any way that IMatch considers the directory as "OFF-LINE" rather than "SCAN NEEDED"?

Best regards,
JM

[mking]

Hi Mario,

I think the Windows 10 anniversary update is causing this same problem for me.

Tested a few times happening every time.

Loaded up some new images in a new folder and forced a rescan of the parent folder and voila imatch creates a new duplicate parent and child folder while the old parent folder doesn't have the child folder in it. Imatch points both parent folders to the same path on the same disk with the same volume label.

So "I" have not changed any volume label or id - but I suspect I know what's screwing imatch up - I updated to the Windows 10 anniversary edition and I suspect that Windows 10 update has renamed something imatch uses to track media. I know this update changes stuff because I've had related issues in my day job with software licensing.

If that is the case how do I fix the problem?
Relocating the images from D:/england to D:/england seems to work - is that expected or might there be some nasty side effect I haven't found yet?

Update: after doing the relocation of 130,000 files and doing a database diagnosis it came up with errors that it seems to have fixed ok, as a second pass was error free. Note there were no database errors before doing the relocation.

Many thanks as always for your help.

Mike

[Mario]

I have three systems with the Anniversary update installed and none of these machines had a similar problem.
Does the original folder show as off-line (yellow icon)? This would be the case if IMatch cannot identify it.

Tip: You can see the label and the serial number of the drive containing a folder by using these variables the VarToy app in the App Panel:

{File.Folder.Media.Label}
{File.Folder.Media.SerialNumber}

Just select a file in a folder and look at the results of these variables.

[mking]

Mario,

Txs for the prompt reply.

I have three systems with the Anniversary update installed and none of these machines had a similar problem.
Does the original folder show as off-line (yellow icon)? This would be the case if IMatch cannot identify it.

No it just looks normal.

Tip: You can see the label and the serial number of the drive containing a folder by using these variables the VarToy app in the App Panel:

Unfortunately I already relocated them so not sure what it will tell me. But the log has this in it when I did the rescan - not sure what the media-id (in green) differences mean here - does that help to explain what was happening? Does the Windows 10 update change the media id info enough to confuse imatch?

  file://D:/Locations/England/Cornwall/Nov 07/Fri 1/Pano1/ Media-ID: ONE, Serial 1716600454
  file://D:/Locations/England/Cornwall/Nov 07/Fri 1/Pano1/ (ONE 1716600454)
  _D205112.TIF
  _D205113.TIF

Txs,
Mike

[Mario]

No, this looks OK. Media label is ONE, media serial number is 1716600454.