Automatic Category: Which date field best to use?

[FrankS]

Hello,

I wish to create an automatic (data-driven) category which categorises all pictures by month when they were taken.
It basically works, however, some 10% of the pictures are listed as 'other'.

What date field should I use to make that possible? Is it possible to somhow combine two date fields in the tag selection?

Here is what I did so far:
Based on 'Tag' using tag 'XMP::exif\DateTimeOriginal\DateTimeOriginal\0'
Use part of value 'Yes' starting at 6 for 2 characters.

I understand that some are listed in the '00' category as they do not have any embedded date info, but some listed as 'other' clearly have as I coudl verify with the ExifToolCommandProcessor. BUt not necessarily in the tag which gave me the best result from all tags I used. Hence the question of possibly combining two different tags.

Thanks for any help
Frank

[jch2103]

Why not just use the Timeline?

For Original Date (in my custom metadata panel), I use XMP::photoshop\DateCreated\DateCreated\0.
If you have some files missing this tag, you're probably best off manually setting date/time for these files (using the tag you end up choosing) to make sure your metadata is consistent. If some files use the tag but aren't showing up properly in your data-driven category, you may need to check them individually (perhaps using the Browser metadata template) to see if there's a particular data glitch in them (e.g., bad format, etc.).

[JohnZeman]

Welcome to IMatch Frank.

I have a chronology data-driven category that lists my files by date taken and I've had excellent results using the Composite\DateTimeOriginal\DateTimeOriginal\0 tag.  I also use the Other element but none of my 61,000 images ever show up in Other.

Years ago when I was scanning printed images to add to my database I had to manually add the date taken metadata to those images but that was a one time thing.

I should also mention that all of the images in my database are JPGs.

[sinus]

As you can see here, users uses different tags to find out the original date of a file.
It is sometimes written wrong, with whatever program you use.
And if you have also scanned images, the mess if often here.

IMatch offers different possibilities, to write the correct date - if you know it. 

I personally try to do this correct, but it is not possible always very easy, hence I use since years for this the filename.
The date and time of my files are in the filenename. If I have different dates/times in different ! tags, I use simply the filename.

For scanned images (e.g.) the date changes sometimes, because I know more (a date will be corrected and so on), in such cases I simply change the filename.
Works very good for me since years, and hence also the sorting is correct.

[Carlo Didier]

Like sinus, I avoid this problem completely by renaming the images on import with the creation date in the filename, like D20190123...

Advantages: Any file manipulations which can change date metadata become irrelevant.
Examples: Creation date of a file changes if you make a copy (but the original date stays in the filename). Modification date(s) change at any modification (and you shouldn't suppress that).

Just my own workflow.

[herman]

Yep, like Sinus and Carlo, coding the date in the file name is the way to go for me too.

Recently I scanned a bunch of old B/W negatives from the sixties and early seventies.
Usually I don't know the date for these negatives, when I am lucky I can remember the year.
For those files I set the month and date space in the file name to 0000.

[Mario]

There is an official way to deal with image date and time, and IMatch uses that to fill its own "File DateTime" field used in the file window, for the timeline, for sorting etc.

IMatch automatically maps existing EXIF created/digitized information recorded by the camera into the corresponding XMP fields.
If your camera did not record time information in the official EXIF tags, you can set the proper date and time in the IMatch Metadata Panel (in the 'Default' layout, for example).

See How IMatch uses Date and Time Information in the IMatch help system for details about this, and how IMatch determines a usable  timestamp for video (an even bigger mess than images), Office and PDF formats.

You can easily find files without valid data in one or both standard date and time fields using a data-driven category based on the XMP::xmp\CreateDate\CreateDate or XMP::photoshop\DateCreated\DateCreated (for image files). Use these settings:

1. Enable use part of value 1,10  to get only the date portion of the tag data.
2. Enable detect hierarchies and set : as the hierarchy delimiter.
3. Enable Use Other

These settings produces a data-driven category with 3 levels, for year, month and day automatically. The Other element lists all files without a tag value (no timestamp).
You will also see some entries for invalid or incomplete date values. This is normal, every larger collection of files has some metadata problems initially. You can fix these by manually editing the date and time for these files.

For non-image files you can do the same, just look at the metadata in the files you're working with and find a timestamp that is suitable Note: IMatch manages the two XMP timestamps from above for all files, whether or not they have EXIF / XMP data. This allows you to use the timeline for all formats, and other features as well.

You can manually fill these tags for non-image files in the Metadata Panel, or you use a Metadata Template to copy date and time information from some other metadata into these tags (e.g. for video files).

[FrankS]

Thanks to all who answered. Just tried Mario's official way and it produces the best result. Just above 300 photos and videos need manual intervention. That is do'able.

@jch1203: The builtin timeline did not meet my requirement to have immediate access to all pictures I ever took in a specific month. Too often I have a rough idea at what time in a year it was taken, e.g. around Easter, in winter, or so. With this data-driven category I can now quickly limit.

Again, thanks.

[jch2103]

@jch1203: The builtin timeline did not meet my requirement to have immediate access to all pictures I ever took in a specific month. Too often I have a rough idea at what time in a year it was taken, e.g. around Easter, in winter, or so. With this data-driven category I can now quickly limit.

Aah! I missed the fact that you wanted to organize by month in your hierarchy, without year. Glad you have a solution that works for you. The amount of flexibility IMatch provides is excellent.

[Tveloso]

The Data-Driven Category Mario described here is exactly the thing I was looking for!

I knew that there were a number of files in my DataBase that lacked any Metadata at all (even though in many cases they were originally taken with a Digital Camera), so the Date Value that IMatch is using for these files is the Windows Create Date (which is usually much later than the Date the photo was taken)...so they wind up in the wrong place in the TimeLine.  I suspect that these files were all edited at some point in the past, and perhaps their Metadata was discarded as a result of that.

This post came at right about the time that I was beginning to think about how best to isolate these "bad photos".  I might have arrived at a more complex and convoluted method of identifying the "date problem"...but this is an elegant and simple solution.  Even the initial setup of this Data-Driven Category "creates itself" (thanks to the Detect Hierarchies property).

Thanks to this Data-Driven Category I was able to isolate all of the Files in my DataBase with Date Problems, and to my surprise, it was not the few hundred that I was expecting, but 1,657 files!...(so I have a bigger mess than I realized).  IMatch even let me know that I have 42 Files with "literal zeros" for the Date (these files all contain "0000:00:00 00:00:00" in the TimeStamps).

I actually created two Data Driven Categories, based upon both of the Tags that Mario suggested:

    Date Analysis
      |- XMP Create Date
            | - <the auto-generated YMD levels appear here>
      |- XMP PS Date Created
            | - <the auto-generated YMD levels appear here>

...and the Other Category inside each of those contains a different number of files (with the "Create Date" Category containing 1,657 photos, and "PhotoShop Date Created" containing 2,071).

It turns out that the difference is due to some scanned slides.  The scanning software I used populated the Date Digitized (XMP::xmp\CreateDate\CreateDate\0) as expected, but it was kind enough to leave the Date Subject Created (XMP::photoshop\DateCreated\DateCreated\0) empty...so these files have valid dates for the first Data Driven Category, but appear in Other, inside the second.

IMatch truly is a remarkable product! 

I have been an IMatch user for many years, but only intermittently so, and I have only scratched the surface...one of my New Year's resolutions is to really learn IMatch so that I can avail myself of its power and finally get my Photo Collection cleaned up.  Perhaps I can even become a contributing member of this community.

This exercise has also reminded me of something that I had forgotten about...there are a number of photos that came from my daughter's Phone, that are corrupted (this might account for some of the 1,657 photos with the bad dates).  I recall trying to help her identify an issue she was having with her Android Phone some years ago, where her Photos App was locking up.  We determined that it might have been a bad Micro SD Card, and although we got past the issue by replacing the card, and we saved most of her Photos, there were some that were not viewable.  Those Photos appear in IMatch with no thumbnail, and generate an error when we try to open them in any software.  Would anyone have a recommendation on a Utility that might be able to recover these photos?  I did Google "recover corrupted jpeg", and found several applications that claim to be able to do this, but I thought I would ask here, if someone might have a specific recommendation...thanks very much.

[Mario]

Thanks for your feedback   

Getting the metadata is shape is usually one of the first things to do once you move to a 'real' DAM like IMatch. And it's one of the most important features of a DAM to help users identify and clean the metadata.
If you have fixed all the problems and the metadata has been written back to your files, it will stay in that great shape. And it will be independent from IMatch and work with all your software now and in the future.

1000 files is not that much. When IMatch is introduced into corporate or scientific environments, the numbers are usually more like "We have 200,000 files without any usable metadata or organization - help!  )

[jch2103]

Would anyone have a recommendation on a Utility that might be able to recover these photos?  I did Google "recover corrupted jpeg", and found several applications that claim to be able to do this, but I thought I would ask here, if someone might have a specific recommendation...thanks very much.

Glad to see your success in cleaning up your images! I've said that you never know how bad your data are until you have a powerful product like IMatch to work with your images...

Regarding recovery of corrupted images, most of the software for that problem that I know about is designed to recover images from bad media (e.g., from the card manufacturer). I don't know if it can be used after-the-fact, but it's worth a try. The specifics will likely depend on the exact nature of the corruption. But my past experience with corrupted jpg images isn't very encouraging. Good luck!

[Tveloso]

Thank you John. 

I recall not really being sure whether the Micro-SD Card in my Daughter's Phone was actually bad, or if there was maybe some "software issue" with the "photo gallery" application where she was experiencing the issue (that perhaps might have corrupted the photo files themselves). 

I was able to copy the entire contents of that Card into my PC without any FileSystem Errors during the copy operation, and all the files (even the bad ones) are in the neighborhood of 1.5MB...but I don't really know if that's any indication that there wasn't a problem with the Card.  We replaced it anyway...

My Google Search turned up something called Stellar Repair for Photo, that provided a free download of the evaluation version, which is supposed to show a preview only, of the recovered photo (and the licensed version would allow it to be saved out).  I gave this a try, but alas, although it claimed to have repaired the 5 test photos I loaded into it (which were completely unviewable), the preview shown for them was just a grey background.  Only photos that I was able to see a portion of already in the Thumbnail (but that were partially covered with solid-colored bars - usually near the edges) displayed a preview when the "repaired" item was clicked on - but it didn't look much different from the original.

Oh well...this was six years ago now, and my daughter was resigned to having lost a few photos in that incident (we did save the majority of them).  I may still have the original Card, so I could maybe still try the media recovery approach you mentioned.  But I have lots of other Photos to work on with IMatch...

[jch2103]

... I gave this a try, but alas, although it claimed to have repaired the 5 test photos I loaded into it (which were completely unviewable), the preview shown for them was just a grey background.  Only photos that I was able to see a portion of already in the Thumbnail (but that were partially covered with solid-colored bars - usually near the edges) displayed a preview when the "repaired" item was clicked on - but it didn't look much different from the original.

Oh well...this was six years ago now, and my daughter was resigned to having lost a few photos in that incident (we did save the majority of them).  I may still have the original Card, so I could maybe still try the media recovery approach you mentioned.  But I have lots of other Photos to work on with IMatch...

This sounds depressingly similar to my last experience with trying to repair corrupted jpg images... Sometimes you just have to take your losses and move on.