IMatch.get('v1/files',{ not always working

Carlo Didier · December 15, 2017, 08:16:34 PM

What can make this work for some files and not for others in the same folder? Files are all DNGs converted from NEFs from the same camera and TIFFs.

Code Select


        function ProcessSelectedFiles(){

            // get the list of selected images

            IMatch.get('v1/files',{
                idlist: IMatch.idlist.fileWindowSelection
            }).then(function(response) {
                console.log(JSON.stringify(response));
                AssignFormatCategories(response.files);
            },
            function(error){
                console.log("Getting seleted files didn't work ...");
                console.log(error);
            });
        }

When I select certain files, I get the correct result:

QuoteProcess currently selected files
Setting format categories for 1 files
{"files":[{"id":104077,"ts":0,"tsmd":0,"tsCreated":"2017-09-11T18:33:54.279","tsLastModified":"2017-09-15T13:10:03.9","index":0,"folderId":2871,"folder":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\","name":"D20170901044.dng","namene":"D20170901044","ext":".dng","fileName":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\D20170901044.dng","size":18622984,"crc":1475887161,"ltid":"F092F66B-3A12-4DFC-8DFF-D42C75BB13AC","class":"image","format":"DNG","mimeType":"text/plain","lastUpdated":"2017-09-15T13:10:03","dateTime":"2017-09-01T12:48:09","offline":false,"protected":false,"readOnly":false,"width":3162,"height":4216,"longEdge":4216,"shortEdge":3162,"portrait":true,"bpp":24,"rating":2,"label":"","stackState":0,"stackId":0,"stackColor":"","versionState":0,"history":[{"event":{"type":1,"name":"added","id":104077,"id2":0,"dateTime":"2017-09-01T19:43:54","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-01T19:45:14","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-15T15:13:08","title":"","text":"","user":"Carlo"}}],"gps":{"lat":57.727527,"lon":-5.477514,"alt":785.491954},"collections":[13,34,54,10002,11001]}],"hiddenByStacks":0}
[{"id":104077,"ts":0,"tsmd":0,"tsCreated":"2017-09-11T18:33:54.279","tsLastModified":"2017-09-15T13:10:03.9","index":0,"folderId":2871,"folder":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\","name":"D20170901044.dng","namene":"D20170901044","ext":".dng","fileName":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\D20170901044.dng","size":18622984,"crc":1475887161,"ltid":"F092F66B-3A12-4DFC-8DFF-D42C75BB13AC","class":"image","format":"DNG","mimeType":"text/plain","lastUpdated":"2017-09-15T13:10:03","dateTime":"2017-09-01T12:48:09","offline":false,"protected":false,"readOnly":false,"width":3162,"height":4216,"longEdge":4216,"shortEdge":3162,"portrait":true,"bpp":24,"rating":2,"label":"","stackState":0,"stackId":0,"stackColor":"","versionState":0,"history":[{"event":{"type":1,"name":"added","id":104077,"id2":0,"dateTime":"2017-09-01T19:43:54","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-01T19:45:14","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-15T15:13:08","title":"","text":"","user":"Carlo"}}],"gps":{"lat":57.727527,"lon":-5.477514,"alt":785.491954},"collections":[13,34,54,10002,11001]}]
D20170901044.dng: 3162x4216 = vertical

On others, I get this:

QuoteProcess currently selected files
Getting seleted files didn't work ...
{
"readyState": 4,
"responseText": "{\"files\":[{\"id\":104061,\"ts\":0.000000,\"tsmd\":-nan,\"tsCreated\":\"2017-09-11T18:33:47.895\",\"tsLastModified\":\"2017-09-15T13:09:44.350\",\"index\":0,\"folderId\":2871,\"folder\":\"D:\\\\Photos\\\\Photos Carlo\\\\2017\\\\09\\\\2017-09-01 Beinn Airigh Charr - Poolewe\\\\\",\"name\":\"D20170901028.dng\",\"namene\":\"D20170901028\",\"ext\":\".dng\",\"fileName\":\"D:\\\\Photos\\\\Photos Carlo\\\\2017\\\\09\\\\2017-09-01 Beinn Airigh Charr - Poolewe\\\\D20170901028.dng\",\"size\":15300340,\"crc\":923958750,\"ltid\":\"4980F0D1-95ED-4D12-82B0-3922E87CB386\",\"class\":\"image\",\"format\":\"DNG\",\"mimeType\":\"text/plain\",\"lastUpdated\":\"2017-09-15T13:09:44\",\"dateTime\":\"2017-09-01T12:41:07\",\"offline\":false,\"protected\":false,\"readOnly\":false,\"width\":4120,\"height\":3090,\"longEdge\":4120,\"shortEdge\":3090,\"portrait\":false,\"bpp\":24,\"rating\":2,\"label\":\"\",\"stackState\":0,\"stackId\":0,\"stackColor\":\"\",\"versionState\":0,\"history\":[{\"event\":{\"type\":1,\"name\":\"added\",\"id\":104061,\"id2\":0,\"dateTime\":\"2017-09-01T19:43:54\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}},{\"event\":{\"type\":2,\"name\":\"updated\",\"id\":104061,\"id2\":0,\"dateTime\":\"2017-09-01T19:45:13\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}},{\"event\":{\"type\":2,\"name\":\"updated\",\"id\":104061,\"id2\":0,\"dateTime\":\"2017-09-15T15:12:59\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}}],\"gps\":{\"lat\":57.727540,\"lon\":-5.477497,\"alt\":785.227360},\"collections\":[13,34,54,10002,11001]}],\"hiddenByStacks\":0}",
"status": 200,
"statusText": "OK"
}

What is happening???
The "error message" doesn't tell me anything.

A database diagnostic didn't show any errors or warnings.

Mario · December 16, 2017, 08:47:02 AM

1. Use

Code Select

console.log(JSON.stringify(response,null,2));

to produce readable output in the console. Or use the Preview window in your browser's debugger.

2. ReadState 4 is often logged when the response from the server is not valid JSON.
This cannot happen, or at least is has never happened in the past 12 months since this endpoint is in use.

We would need a lot more details than a barely readable text block, starting with details about what you had selected in the file window, how you use the code, the IMatch log file, the IMatch IMWS log file etc.

Carlo Didier · December 16, 2017, 11:42:59 AM

Quote from: Mario on December 16, 2017, 08:47:02 AMWe would need a lot more details than a barely readable text block, starting with details about what you had selected in the file window, how you use the code, the IMatch log file, the IMatch IMWS log file etc.

As I already said in my first post, I selected any single (for the example) or several DNG or TIFF files which are all perfectly readable and can be opened in Photoshop or Fast Picture Viewer. As I said:

Quote from: Carlo Didier on December 15, 2017, 08:16:34 PMFiles are all DNGs converted from NEFs from the same camera and TIFFs.

The names of the files are in the results I posted: "D20170901044.dng" is one who works and "D20170901028.dng" is one who doesn't work (in the same folder, from the same camera, same post-processing ...).

I changed the console.log lines:

Code Select

        function ProcessSelectedFiles(){

            // get the list of selected images

            IMatch.get('v1/files',{
                idlist: IMatch.idlist.fileWindowSelection
            }).then(function(response) {
                console.log(JSON.stringify(response,null,2));
                AssignFormatCategories(response.files);
            },
            function(error){
                console.log("Getting seleted files didn't work ...");
                console.log(JSON.stringify(error,null,2));
            });
        }

And here is the output from a file that's not working:

QuoteProcess currently selected files
Getting seleted files didn't work ...
{
"readyState": 4,
"responseText": "{\"files\":[{\"id\":104042,\"ts\":0.000000,\"tsmd\":-nan,\"tsCreated\":\"2017-09-11T18:33:40.307\",\"tsLastModified\":\"2017-09-15T13:09:23.699\",\"index\":0,\"folderId\":2871,\"folder\":\"D:\\\\Photos\\\\Photos Carlo\\\\2017\\\\09\\\\2017-09-01 Beinn Airigh Charr - Poolewe\\\\\",\"name\":\"D20170901009.dng\",\"namene\":\"D20170901009\",\"ext\":\".dng\",\"fileName\":\"D:\\\\Photos\\\\Photos Carlo\\\\2017\\\\09\\\\2017-09-01 Beinn Airigh Charr - Poolewe\\\\D20170901009.dng\",\"size\":15126724,\"crc\":274184501,\"ltid\":\"EAA0A879-D95A-4486-8EAF-B694341B7983\",\"class\":\"image\",\"format\":\"DNG\",\"mimeType\":\"text/plain\",\"lastUpdated\":\"2017-09-15T13:09:23\",\"dateTime\":\"2017-09-01T11:21:24\",\"offline\":false,\"protected\":false,\"readOnly\":false,\"width\":4608,\"height\":3456,\"longEdge\":4608,\"shortEdge\":3456,\"portrait\":false,\"bpp\":24,\"rating\":1,\"label\":\"\",\"stackState\":0,\"stackId\":0,\"stackColor\":\"\",\"versionState\":0,\"history\":[{\"event\":{\"type\":1,\"name\":\"added\",\"id\":104042,\"id2\":0,\"dateTime\":\"2017-09-01T19:43:54\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}},{\"event\":{\"type\":2,\"name\":\"updated\",\"id\":104042,\"id2\":0,\"dateTime\":\"2017-09-01T19:45:13\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}},{\"event\":{\"type\":2,\"name\":\"updated\",\"id\":104042,\"id2\":0,\"dateTime\":\"2017-09-15T15:13:15\",\"title\":\"\",\"text\":\"\",\"user\":\"Carlo\"}}],\"gps\":{\"lat\":57.734338,\"lon\":-5.505741,\"alt\":239.580046},\"collections\":[13,34,54,10001,11001]}],\"hiddenByStacks\":0}",
"status": 200,
"statusText": "OK"
}

and from one that's working:

QuoteProcess currently selected files
Setting format categories for 1 files
{
"files": [
{
"id": 104077,
"ts": 0,
"tsmd": 0,
"tsCreated": "2017-09-11T18:33:54.279",
"tsLastModified": "2017-09-15T13:10:03.9",
"index": 0,
"folderId": 2871,
"folder": "D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\",
"name": "D20170901044.dng",
"namene": "D20170901044",
"ext": ".dng",
"fileName": "D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\D20170901044.dng",
"size": 18622984,
"crc": 1475887161,
"ltid": "F092F66B-3A12-4DFC-8DFF-D42C75BB13AC",
"class": "image",
"format": "DNG",
"mimeType": "text/plain",
"lastUpdated": "2017-09-15T13:10:03",
"dateTime": "2017-09-01T12:48:09",
"offline": false,
"protected": false,
"readOnly": false,
"width": 3162,
"height": 4216,
"longEdge": 4216,
"shortEdge": 3162,
"portrait": true,
"bpp": 24,
"rating": 2,
"label": "",
"stackState": 0,
"stackId": 0,
"stackColor": "",
"versionState": 0,
"history": [
{
"event": {
"type": 1,
"name": "added",
"id": 104077,
"id2": 0,
"dateTime": "2017-09-01T19:43:54",
"title": "",
"text": "",
"user": "Carlo"
}
},
{
"event": {
"type": 2,
"name": "updated",
"id": 104077,
"id2": 0,
"dateTime": "2017-09-01T19:45:14",
"title": "",
"text": "",
"user": "Carlo"
}
},
{
"event": {
"type": 2,
"name": "updated",
"id": 104077,
"id2": 0,
"dateTime": "2017-09-15T15:13:08",
"title": "",
"text": "",
"user": "Carlo"
}
}
],
"gps": {
"lat": 57.727527,
"lon": -5.477514,
"alt": 785.491954
},
"collections": [
13,
34,
54,
10002,
11001
]
}
],
"hiddenByStacks": 0
}
[{"id":104077,"ts":0,"tsmd":0,"tsCreated":"2017-09-11T18:33:54.279","tsLastModified":"2017-09-15T13:10:03.9","index":0,"folderId":2871,"folder":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\","name":"D20170901044.dng","namene":"D20170901044","ext":".dng","fileName":"D:\\Photos\\Photos Carlo\\2017\\09\\2017-09-01 Beinn Airigh Charr - Poolewe\\D20170901044.dng","size":18622984,"crc":1475887161,"ltid":"F092F66B-3A12-4DFC-8DFF-D42C75BB13AC","class":"image","format":"DNG","mimeType":"text/plain","lastUpdated":"2017-09-15T13:10:03","dateTime":"2017-09-01T12:48:09","offline":false,"protected":false,"readOnly":false,"width":3162,"height":4216,"longEdge":4216,"shortEdge":3162,"portrait":true,"bpp":24,"rating":2,"label":"","stackState":0,"stackId":0,"stackColor":"","versionState":0,"history":[{"event":{"type":1,"name":"added","id":104077,"id2":0,"dateTime":"2017-09-01T19:43:54","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-01T19:45:14","title":"","text":"","user":"Carlo"}},{"event":{"type":2,"name":"updated","id":104077,"id2":0,"dateTime":"2017-09-15T15:13:08","title":"","text":"","user":"Carlo"}}],"gps":{"lat":57.727527,"lon":-5.477514,"alt":785.491954},"collections":[13,34,54,10002,11001]}]
D20170901044.dng: 3162x4216 = vertical

It gives more details when it's working, but still nothing usable when not ...

Attached the IMatch log files and the app. Where do I find the IMWS log file?

You can download one of the problem files here: https://drive.google.com/open?id=1kOrFmf9Q37Qlhn0OuXVmGjq_CCGJZKUW

Carlo Didier · December 16, 2017, 12:12:04 PM

Also note that I can reproduce this problem on both my desktop PC (old Core i5, 16GB RAM, Windows 10 Home) and my laptop (recent Core i7, 16GB RAM, Windows 10 Pro).

Mario · December 16, 2017, 01:17:52 PM

I see the problem. Some of your files seem to have a corrupted or missing metadata timestamp. This leads to the -nan value returned in the result (you see this by looking at the returned JSON data) and this breaks the JSON parser. IMWS should return a 0 value in this case. I currently don't see how this can happen. The timestamp is always 0 until set by a metadata update...
Thanks for bringing it to my attention. I will look into this next year.

Until then, just request the fields you really need. Use the fields parameter to do so.
Your code requests all available fields for the file, which not only causes a massive data transfer and slow performance but also causes the tsmd field being requested which causes this problem.

If you don't request this field, you will not have problems.
If only happens if a) you don't use fields to limit the returned data or you explicitly request tsmd, b) the file has a metadata last mod that has never been set and c) the field has, for some reason, an invalid numeric value.

For the future: If possible, don't include such large bulks of text in your code. Attach them instead.
Posts of this kind flood the community search engine with noise data, causing more irrelevant search results. The search engine cannot tell.
Thanks.

Carlo Didier · December 16, 2017, 02:30:14 PM

Thank you, Mario. I'll limit the requested fields.
And sorry for posting the long texts.
I'll have a look at the files and see if I can simply update the embedded thumbnail to generate a correct timestamp. It's strange, because I processed all the files at the same time in Bridge/ACR. Maybe a bug from Adobe ...

Mario · December 16, 2017, 02:35:38 PM

This is a different timestamp. You cannot manipulate it directly in IMatch.
The timesstamp in question is currently used only by IMatch Anywhere, 2018 to support concurrent access scenarios and competing metadata updates.

Carlo Didier · December 16, 2017, 02:53:46 PM

Actually, I just tested it by changing the description of one of the problem files. After IMatch detected the change, it worked, whereas those files that didn't change still don't work.

By requesting only the fields I need, all files work.

I also tried a Ctrl-Shift-F5 "Force Rescan" on the folder and after that all files worked. So it might also be a bug while IMatch first ingested the images ...

Mario · December 16, 2017, 04:30:19 PM

This may revert when you close and re-open the database.
I have already identified the source of the problem and fixed it for the next release.

Carlo Didier · December 16, 2017, 08:14:48 PM

Wow, fast as always!