Potential asynchronicity problem?

[Carlo Didier]

I have this code:

Code Select Expand

                    arrCategories.forEach(objCategory => {

                        var strDesc  = objCategory.description
                        var strStart = strDesc.substr( 0,12)    // extract start date and number
                        var strEnd   = strDesc.substr(13,12)    // extract end date and number

                        // check if image is in range, if yes, assign it to that category

                        if ((strFileRoot >= strStart) && (strFileRoot <= strEnd)) {
                            $('#result-file').text( $('#result-file').text() + strFileName + "\n");
                            $('#result-category').text( $('#result-category').text() + objCategory.name + "\n");
                            $('#result-range').text( $('#result-range').text() + strStart + " to " + strEnd + "\n");
                            IMatch.post('v1/categories/assign',{
                                id: objCategory.id,
                                fileid: intFileID
                            }).then(function(response) {
                                if (response.result != 'ok') {
                                    console.log(JSON.stringify(response,null,2));
                                }
                            },
                            function(error){
                                console.log(JSON.stringify(error));
                            });
                        }
                    });
which, for each image in arrCategories checks if it has to be assigned to a specific event category. This works fine for a limited number of files, but when I have an array of >90000 files, it crashes IMatch.
But the 

Code Select Expand

IMatch.post('v1/categories/assign',{ will effectively be executed ~44000 times.

My question is this: Does the foreach loop just continue without the "IMatch.post('v1/categories/assign'," having finished? If so, that might cause a backlog of thousands of assignments and that might be the cause of the crash.

How can I make the loop wait for the category assignment to be done before continuing? Maybe use a flag variable and a wait loop? How do I do a wait loop in java script?

[Mario]

My question is this: Does the foreach loop just continue without the "IMatch.post('v1/categories/assign'," having finished?

You are producing 44000 simultaneous requests. Yikes! IMWS is good but not that good.
Please revise your script and don't put REST requests in a loop. This is bad programming.

Process your array on request at a time. I think already demonstrated that in one of the ~ 30 answers I gave this week to your posts. See there.
And there are of course the samples, e.g. the Process Files, which I mentioned several times already. It processes things in batches.

[Carlo Didier]

I'll try to understand what's happening in the process files sample app, but I'm struggling to understand it and to transpose it to my problem.

And I don't want to produce 44000 simultanious requests ... Just give me a way to do it properly, i.e. sequentially, and I'm happy. That's all I need. It's such a simple thing to do, after all.

But it seems to be pretty much impossible or at least extremely complicated to do anything sequentially here ...

I'm wondering if in this code

Code Select Expand

var a = some calculation
var b = a * 10
I could ever be sure that b will really be calculated after a ...

[Mario]

Code Select Expand

var a = some calculation
var b = a * 10
I could ever be sure that b will really be calculated after a ...

This is linear code, it will be processed in exactly the order in which it appears in the source code.

AJAX requests (calling endpoints in a web server, accessing HTML data on a web server, ...) are asynchronous.
In IMatch, AJAX request are used internally when you do a IMWS.get(...) or IMWS. post(...)

When you do a

IMWS.get(...);

var i = 1+2;
console.log(i);

JavaScript which perform a request against IMWS and then immediately continue in your app, with the statement following IMWS.get(...), which is
var i = 1+2;
This means it will calculate the value of i and output it to the console.

Just set a breakpoint in your debugger on var i=  and one inside the then function. You'll see how this works.

At some undetermined later time (may be 0.1 seconds or 5 seconds, if IMWS is slow) IMWS will return the result for the get request.
This is why this is called a promise. IMWS.get returns a promise that will be fulfilled (success) or rejected (failure) at some later time.
I think I have explained that in the corresponding section "Promises, Promises" in the developer documentation. There are also several thousand of AJAX / jQuery tutorials out there.

Again:

IMWS.get('v1/files',{id: 1}).then(function(response)
{
    console.log('Waiting is over.');
});

var i = 1 + 2;
console.log('Waiting...' + i);


Will output

Waiting 3

and then, after an undetermined time,

Waiting is over.

The purpose of asynchronous operations is to keep your App running while it waits for one or more (!) outstanding web requests.
Your app can animate the user interface, play some animation or whatever. When the outstanding web request is completed, it will be notified and your code will run.
This allows you to write responsive apps, which don't block. Very important these days.

For your problem - process one id at a time:
Classical divide and conquer algorithm.
(Just wrote that, not syntax checked!)


// The ids you have to process
var ids = [1,2,3,4,6,100,200,600];
// Current index into the ids array
var index = 0;

// This does the work.
function next()
{
    if (index >= ids.length) {
        // All lements in ids processed.
        someFuncToCallWhenDone();
    }
    else {
        IMWS.get('v1/files',{id: ids[index]} ).then(function(response) {
// NOTE: This code is executed when the web request is finished!
// Your app remains running all the time, maybe doing other things!
           // Process the next element in ids.
           index++;
           next();
        });
    }
}

// Start.
// Process the first element in the ids array.
next();


The next() function processes the next element in ids.
When the IMWS request has completed, it processes the next element in ids, by calling itself.

[Carlo Didier]

Hmm, I think I get what your code does (though it's still very contorted compared to the few lines of basic I had in IM5 to do the exact same thing).

Am I right to suppose that in your example if I had (at the end)

Code Select Expand

console.log("Starting ...");
next();
console.log("All done");
then those three lines will be executed in that order, i.e. the last line will only be executed when the next() function has looped through all images?

[Mario]

Hmm, I think I get what your code does (though it's still very contorted compared to the few lines of basic I had in IM5 to do the exact same thing).

You can do many things in less lines in JavaScript.
You can now program IMatch in multiple programming languages. If you don't like JavaScript, Use VB.NET. Or PowerShell.

then those three lines will be executed in that order, i.e. the last line will only be executed when the next() function has looped through all images?

Set a breakpoint in your debugger and see for yourself. The only way to learn and understand.

[thrinn]

Am I right to suppose that in your example if I had (at the end)

Code Select Expand

console.log("Starting ...");
next();
console.log("All done");
then those three lines will be executed in that order, i.e. the last line will only be executed when the next() function has looped through all images?

"those three lines will be executed in that order": Yes.
But your conclusion "i.e. ..." is wrong: When using asynchronous functions (and because next contains an IMWS.get call, it is asynchronous), executing something in some order is not the same as "... and wait until is has finished".
But Marios example already contains a "excecute when finished" function call. Therein you should put your "All done" logging.

[Mario]

One has to step through this in a debugger to really grasp what asynchronous AJAX calls really are.
This is such a powerful but initially a bit complicated concept. Especially when you come from a strict sequential language like BASIC.

[Carlo Didier]

But Marios example already contains a "excecute when finished" function call. Therein you should put your "All done" logging.

I need a "execute next" when finished. I spent half my sunday to try to get it working with a recursive function. Still not working.
I currently don't know how to debug as I can't "run" the script in Visual Studio Code and I can't run it in my web browser. Except for http://127.0.0.1:50519/info, anything else gives me an error ...

Example for "http://127.0.0.1:50519/v1/files/categories?{auth}&​id=1-20&​fields=id,path" (from the doc):

Error 400: Bad Request
Invalid URI

which is plain chinese for me. Couldn't find out from the doc how to do that correctly.

The goal is so simple:
- get a list of files (from certain categories or selected or all in database)
- for each of these files, compare to a list of categories and if certain conditions are met, assign the file to the category.

I am at 300 lines of code (including the html and comments) right now and it's still not working correctly ...
That was less than 50 lines in the IM5 basic ...

[Mario]

Open your App from the App Manager in your Browser.
This will show you the correct link and port number.

[ubacher]

Once your browser opens press F12 to get the debugger. Select sources (Chrome browser).
To reload app press F5. Set breakpoints in the source.

[Carlo Didier]

Hah, got it (with Firefox). Thanks!