problem migrating categories

[Joe Austin]

After reading Mario's post on migrating categories from 3.6 :
https://www.photools.com/community/index.php?topic=66.0

I decided to further my testing with some actual images from my 3.6 production database.

I exported the categories form 3.6 and imported them into 5.0.   That appeared to go as expected.   I then copied a branch of the images from my 3.6 database folders (about 8k images) to a spare drive and added/indexed them to my 5.0 test database.   Upon reviewing the database I could see that the categories were not populated so I re-read the post, taking note of the fact that it seemed to rely on having the test images in place before importing the category schema.

So, I deleted all of the imported categories and re-imported them selecting "Import file assignments" and  "use Checksum" as recommended in the post for images that have been copied elsewhere.

The result was a couple of formula categories did get populated, but the vast majority of the categories have no images at all.

What could be missing from my effort?  Could the importation/deletion of the categories in my first attempt have messed up the ability to do the file assignments?

[Mario]

I then copied a branch of the images from my 3.6 database folder

Your IMatch 5 database does not hold the same files as your IMatch 3 database. You are working with copies in a different folder structure.
When you import the categories you therefore have to make sure that you use the "Map by check-sum" mode. Or else IMatch 5 will not be able to map the files.

[Richard]

When you import the categories you therefore have to make sure that you use the When you import the categories you therefore have to make sure that you use the "Map by check-sum" mode. Or else IMatch 5 will not be able to map the files.. Or else IMatch 5 will not be able to map the files.

Hi Mario,

I have used the "Map by check-sum" mode twice and none of my images were mapped to any imported category either time.

[Mario]

Since this seems to have worked until now, please double-check:

1. The images must be indexed in the database before you import the categories
2. Check-sum mode must be used if the files are in a different folder

If it still fails, send me (or ZIP/attach) your category IMPS file. I will import it here and run some tests.

[Joe Austin]

Per my original post, I did do a re-import after indexing the files and did use the checksum option.

I will attach my .imcs file.


[attachment deleted by admin]

[Mario]

I would also need a log file in debug mode (see the IMatch help for details) from a session where you imported the categories into your database.
This should tell me if there were any errors, if and which files IMatch could assign etc.

[Joe Austin]

Can I re-import the category schema over the categories that were already created by that schema?   Or do I need to delete all the existing categories first?

[Mario]

You can import multiple times. IMatch will not re-create already existing categories, just add assignments of files as needed.

[Richard]

Hi Mario,

Since I did not get any files assignments, I started doing them myself. If I were to do a third import, what should happen to the existing assignments?

[Mario]

Importing categories will not change existing assignments.

[Joe Austin]

I would also need a log file in debug mode (see the IMatch help for details) from a session where you imported the categories into your database.
This should tell me if there were any errors, if and which files IMatch could assign etc.

Ok, I did the re-import with 'file assignments' and 'checksum' option ticked.   A couple categories got all images, but most got none.

Here are the schema and log files zipped together.

[attachment deleted by admin]

[Richard]

I again imported categories and the attachments show, I hope, what I did and the results.

[attachment deleted by admin]

[Richard]

I really don't mind that my category assignments did not work when I imported the categories. I am having too much fun using Favorites to assign images to categories. What I do is select all images in @All and assign them to say What. I then select "What" and then using Shift and/or ctrl I will select images of say people. Using Favorites one click assigns all the selected images to People. Once assigned to the proper child category or categories, I then hold down Shift and click What in Favorites to un-assign them from the parent category.

In IMatch 3.6 we could use a property so that only images assigned to the Parent, not child categories, were shown and I miss that. But IMatch 5 would not be new if it did everything as it was done in IMatch 3.6.

[JohnZeman]

When you import the categories you therefore have to make sure that you use the When you import the categories you therefore have to make sure that you use the "Map by check-sum" mode. Or else IMatch 5 will not be able to map the files.. Or else IMatch 5 will not be able to map the files.

Hi Mario,

I have used the "Map by check-sum" mode twice and none of my images were mapped to any imported category either time.

I've noticed this same thing with exporting categories from IM3 and importing them into IM5 with the checksum option ticked.  Very few images are actually assigned to categories.  I've even gone so far as to manually confirm some of the IM3 and IM5 image checksums to ensure they're identical and they still fail to import correctly.  I didn't submit a bug report about it at the time because I wasn't sure I was doing things right but since Joe and Richard are seeing the same problem......

[BenAW]

I use a different approach in migrating my Category assignments from IM3 to IM5.
I work on a copy of all my files in IM5 in a totally different location from the IM3 one.

In IM3 create an export for your categories including filenames.
In IM5 import these categories INCLUDING the filenames, so now you're looking at the original IM3 files.
In IM5 now Relocate your Folders to the location of the copied images, and you're working with the copy of your files.

[gunda]

I did something similar

.  Create a copy of my image files. 
.  Create a duplicate of Imatch 3.6 database, and relocated the folders in the duplicate to point at the copies.
.  Export the category structure and file assignments from the duplicate Imatch 3.6 database
.  Ingest duplicates into Imatch 5
.  Import categories and file assignments into Imatch 5 using folder and file name

The advantage of this approach is that the original image files don't go anywhere near Imatch 5.

I wasn't convinced that import by checksum was importing everything, although with a large database it is hard to be sure.  Wasn't there a bug in checksum import that was fixed in 106?  I haven't tried a full import since then to check.  (It's an overnight job to ingest the images and several hours to import the categories.)

[JohnZeman]

I use a different approach in migrating my Category assignments from IM3 to IM5.
I work on a copy of all my files in IM5 in a totally different location from the IM3 one.

In IM3 create an export for your categories including filenames.
In IM5 import these categories INCLUDING the filenames, so now you're looking at the original IM3 files.
In IM5 now Relocate your Folders to the location of the copied images, and you're working with the copy of your files.

Ben I'm trying to follow your procedure you listed here but I'm confused by the last step.  How can you relocate your copied images to the folder they're already in?  Do you import the categories first and then import the images?

[BenAW]

Hi John

I have two complete sets of images (apart from backups  ), one for IM3 and one for IM5.
If I export the categories from IM3 they point to the IM3 images.

In IM5 I have ingested the IM5 images. When importing the IM3 categories they point to the IM3 images,
so I have to relocate them to the IM5 images to work on the copied images.

[JohnZeman]

That's what it sounded like Ben, but how do you relocate a category referencing an IM3 image to reference the IM5 image of the same name?
Sounds like I've missed a powerful tool IM5 might have.

[Richard]

Hi John,

My images are in C:\Images. My copies are in C:Test Images. The files ingested into IMatch 5 are in C:Test Images. Using Ben's system, importing the Categories from IMatch 3.6, IMatch 5 would have links to C:\Images. If I relocate C:Test Images to C:Test Images, the links should now work.

[BenAW]

That's what it sounded like Ben, but how do you relocate a category referencing an IM3 image to reference the IM5 image of the same name?

I did this also in IM3. Relocate from say C:\Images to C:\Images, but in that case they were on different drives having different drive ID's.
It seems IM5 (and 3?) doesn't care where you relocate to, it just goes ahead and changes every link in the dbase to the "new" location of the images, even if they are in the exact same location. In the process the links from the categories are changed as well.

Give it try: copy a dir from IM3, export the cat setup WITH filelinks.
Now ingest the dir in IM5, import the categories and Relocate. Should work unless my memory is failing me completely 

[JohnZeman]

Richard and Ben thanks.  Richard I thought I did exactly what you said but maybe I didn't.  I'll double check this later on.

Right now I'm banging my head against another wall but I'll start a new thread about that.

Thanks again guys, I'll revisit this later on.

[Mario]

I would also need a log file in debug mode (see the IMatch help for details) from a session where you imported the categories into your database.
This should tell me if there were any errors, if and which files IMatch could assign etc.

Ok, I did the re-import with 'file assignments' and 'checksum' option ticked.   A couple categories got all images, but most got none.

Here are the schema and log files zipped together.

Hi, Joe

you have only exported the category schema, but not which files are assigned to which category. When exporting categories in IMatch 3, make sure you use the with file links format. Otherwise IMatch will only export the categories and IMatch 5 will be unable to assign files.

See How to migrate categories from IMatch 3 to IMatch 5 for step-by-step instructions.

[Frank]

I I'm not shure if I should create a new topic or use this.

1. I exported the categories out of imatch 3 with file links
2. Modified the path with notepad ++
3. Imported it into imatch 5.
Result:
Categories are imported but files not.
I can open the files within the explorer out of the *.imcs file

I will send the log to you via mail becaus the file is too big
Frank

[Mario]

Do you use another disk for the IMatch 5 files? If so changing only the file name will not help because you will also need to change the media id and serial numbers.

Why don't you use the check-sum mode?
You don't need to change the file names manually.

[Frank]

I was using a other media as well.
Using checksum does only add a few images nat all.

Now I have started to add my origional images (but I have mounted the drive as read only)

I will see what happens

regards
Frank

[Mario]

Using checksum does only add a few images nat all.

Are you using IMatch 5 build 106?
Can you send me the IMCS file you use for review (the original one exported from IMatch 3).

[Frank]

Yes I am using Build 106
I have send the original file to you

Frank

[jelvers]

I just did a little bit of a "stress test" and put ca. 100.000 pics into my IM 5 database. Then I imported the categories from IM 3.6 (using exactly the procedure as described, i.e. using "file links" for export and "checksum" for import) into IM 5 and, whow, everything just worked fine! Not a single "mishap"!

This piece of software is just phantastic. I know, it is a beta version, but it just has already incredible working features, be it the overall workspace system, the new wonderful filter system with easy to build queries, the GPS stuff or the great slide viewer. I did not come across any real crash by now. Everything is easy and intuitive to use!

So far, extremely well done, Mario.

I liked already IM 3.6. But this is phantastic. I deeply hope that no major issue will show up!

Regards, Juergen

[Mario]

 

[Frank]

Hi Mario,
I now found a little time for further testing.
Using Imatch 5.0.112
Files are stored on the same diskfor imatch 3 and 5. For Imatch 5 the disk is read only.

In Imatch 3.6 there are 4348 files attached to the category

Now in Imatch 5

Adding one category with checksum
Result:1816 files attached

Adding the same category using filelinks
Result:4345 files attached

Adding the same category using both checksum and filelinks
Result:4345 files attached

Why is there a difference between filelinks and checksum?
How can I find the 3 files which are missing by using filelinks?


See attached all logfiles. The logfiles are separated by each run. I have also added the imatch 3 exported categories.


[attachment deleted by admin]

[Mario]

Hi, Frank

I would also need your database to analyze this.
Please upload to my FTP server and start the file name with 548_

If you have not yet used the FTP server, contact me via email for the credentials and usage details.

[Frank]

Hi Mario,
I have uploaded the file to my dropbox and I have sent you a eMail

regards
Frank

[Mario]

Thanks.
I've run some tests on that database.

The database contains about 15,000 files without a checksum and with a pixel dimension of 0 x 0 pixel.
I've found these files by applying a Dimension filter with a size of 0 x 0 on the entire database.

These files show now thumbnail as well so I assume they have been skipped on import for one of several reasons:

1. Unreadable file format or invalid file format
2. Problem with indexing. Since we don't have the log file anymore, it will be hard to tell why these files have been skipped

You can select these files and do a <Shift>+<F5> and then Force Update rescan. Once the files have been indexed properly they will also have a check sum and can be mapped when you import your category schema.

I've also tried the category import via file name. From what I can tell, most files which were not found were files which are in the IMatch 3 database but not in the IMatch 5 database. Files with file extensions like ".BIN" or ".GPX" are not included in IMatch 5 databases by default.

I assume you have modified the configuration file in IMatch 3 to include these files in your IMatch 3 database. But these files are not indexed in your IMatch 5 database and hence cannot be mapped to categories.

You can configure custom file formats under Edit > Preferences > File Formats in IMatch 5. Add .BIN, .GPX and whatever and then rescan your folders to bring these files into your IMatch 5 database. Then re-run the category import to map these files as well.

When I run a category import based on file names, IMatch 5 performs 190.665 file assignments in total.

[Frank]

Thanks Mario for the support.
I'll create a new imatch 5 database
Frank

[Mario]

Has this new database also files which have not been correctly indexed (pixel size = 0)?
If so, we have a real bug which is reproducible on your machine. This is good because we can then find it by looking at your log files.

If the new database is OK and all files are properly indexed, we can close this issue.

[Frank]

It is still running.
To create a new database lasts almost 1,5 days
Frank

[Richard]

Hi Frank,

Have you looked at the Log while it is still running. Maybe it is caught in a loop.

[Mario]

I would also be interested in a log file from a 1.5 day IMatch run.
The log file must be huge, especially if created in debug mode. Very hard to read it without special tools. There is software like "large file viewer" or so which can handle multi-GB files.

I regularly create a 120,000 files test database for testing purposes. Takes between 5 and 7 hours, depending on the moon and the settings I use. It's faster on my 32 GB USB 3.0 stick (my computer has no SSD).

Do you let IMatch create cache images when you create a new database? This will at least double the time. You should use "on-demand" under Edit > Preferences > Cache for faster database creation.

[Frank]

This time I do not create any cache image. We will see how long it lasts. Currently time is approx. 14 hours

Frank

[Frank]

Now I can tell you the result:

My computer configuration:

• i7-3770 CPU @ 3,4GHz 3.9 GHz
• Number of processors: 8
• RAM == 32GB
• OS == WIN7 64Bit
• Disc for OS                          == SSD OCZ-Vertex3                                                   120 GB
• Disc for Database and VMs == SSD Kingston SH 100s3120G                               120,0 GB
• Disc for Files                       == Hitachi HUS724040ALE640   (Serverdisc)           4000,7 GB

My settings for this run in imatch 5

• Threads for import                      == 8
• Threads for metadata import      == 8
• Cache Working Mode                 == OFF

Added one new file format:

• Geo Track Formats
• .GPX;.KML;.BIN

Time for creating a total new database lasts approx. 12 hours (as Mario mentioned faster than with creating a cache :-) )


RESULT!

In Imatch 3 there are 47198 files with a total size of 1.27TB

In Imatch 5 there were 47191 files with the identically size

So there are still 7 files missing

Importing the categories with checksum and file links end up in a different result see attached pdf.

I'm currently uploading the new database to my dropbox with all logfiles attached

I have the logfile from the creation, the import categories with checksum and import the categories with name and path.

@Mario, I will sent you an eMail if the files are uploaded.

Frank

[attachment deleted by admin]

[Frank]

Hi I found one other difference between the two databases:

The Sum of the rating is different between Imach 3 and 5. Working with stars is something we have just started to do.

I have all information written to the files with the imatch 3 menu "Tools --> XMP --> Process Pending XMP Update" The result window is empty!!

Stars                IMATCH 3                 IMATCH 5

1 Star                 6430                       4399
2 Stars               1538                       1000
3 Stars               2076                       13989
4 Stars               1466                       1077
5 Stars               1030                       651


What is wrong there?

Frank

[Mario]

If only 7 files are not assigned it would save a lot of time if you could just search the log file.
IMatch reports every file it could not assign in the log. Or just upload the log file so I can search and tell you the file names to look at.
Saves the upload of your database, if this bothers you.

I cannot analyze the rating issue here (I would need your files).
Do you use the same settings for embedded/external XMP data in both IMatch 3 and IMatch 5?
Especially usage of XMP embedded vs. sidecar files may cause a big difference.

In IMatch 3: Edit > Preferences > XMP (screen shot)
In IMatch 5: Edit > Preferences > Metadata 2 (screen shot)

ExifTool may pull in rating info from EXIF, which is covered by the latest XMP standard, IMatch 3 does not.
You'll need to find a few files which have a different or missing rating in IMatch 5.
Then we need to find out if the file contains a proper XMP rating (or the sidecar file).
The file format may be important etc.

And you need to be absolutely sure that all data has been written in IMatch 3.
The strange thing is that IMatch 5 always has fewer files per rating, except for 3 stars!
IMatch 5 has found 21116 files with rating, IMatch 3 only 12540.

We need one file which shows up in IMatch 5 with a rating but not in IMatch 3.
Then force a refresh of the file in IMatch 3 to see if IMatch 3 has the most recent data.

[Frank]

About the difference in categories I sent you an eMail. Regarding the Rating I open a new ticket.

Frank