How can I determine when several parallel asynchronous processes are finished

[kkae]

How can I determine when several parallel asynchronous processes are finished

I'm not no familar with javascript an the parallel processes.

I move files from a temporary folder to the destination folders.
The IMatch.fileMove is started several hundred times in a loop (see code snipped).
After all processes are finished, I want do some other code.
I'm looking for a solution, can anyone help me?

Code Select Expand



        for (i = 0; i < filenames.length; i++){
            year = filenames[i].substr(1,4);
            sourcename = SOURCEPATH + '\\' + filenames[i];
            destname = DESTINATIONPATH + '\\' + year + '\\' + filenames[i];
            params = {'source': sourcename,
                      'destination': destname};
                     
            IMatch.fileMove(params).then(function(response) {
                //console.log(JSON.stringify(response,null,2));                 
            },
            function(error) {
                console.log('Error: ' + error.status);
            });
        }



[Mario]

This is not a good idea. In theory, all your "several hundred file moves" are performed at the same time... This not only clogs up the available requests in IMWS (there is a limit of how many requests IMWS processes in parallel before returning error messages) but may also cause rece conditions in the file system.

Better change your code to move one file after the other.
Meaning, when the move endpoint in IMWS returns (then branch), proceed with the next file.
See, for example, the Process Files example script.

Note: This move operation is designed to be used for files not in an IMatch database!

[ubacher]

Welcome to the world of asynchronous programming!
You need to realize that the asynchronous programming has as its main idea that it does not want to
stop the program flow while some info is being transferred from the server. This all so that the user's interface (the browser) is not
blocked. This is great if you request info which is not critical - some banners to display on a website etc.
As soon as it is important to have the requested information first before proceeding it gets hairy.
You need to disable the user input (which buttons s/he can click etc) until the information has arrived.

For our kind of programming it is almost always necessary to wait for the return of the information before going on to the next step.
This basically means that you need to chain your program flow from each .then (function(response) {... to the next.
(Either because you need the information, or you need to make sure the operation completed successfully.)
When looping it means that you call a function which, if more calls are to be made, recurses (calls itself again with the next request).
When the last (recursive) call has completed then you chain on to the next step in your programming.

Even if you were to dispense with waiting, you need to be aware than only a certain number of
asynchronous calls can be active at the same time. The number is uncertain - 50 was mentioned somewhere in this forum. Because of this you are forced to
wait anyway for the completion of the previous call.


There is a .await functionality which would make asynchronous calls synchronous - but I have not found out how to use this.
It would simplify our programming tremendously. Maybe one day a smart Javascript programmer will be able to show us how to use
this functionality with Imatch.


PS: As Mario mentioned: the current move operation is almost certainly not the move that you want. The moved files become unknown to Imatch!
I, and several others, are waiting for Mario to code the "move ( copy, rename) within Imatch" functionality.

[Mario]

This is great if you request info which is not critical - some banners to display on a website etc.
As soon as it is important to have the requested information first before proceeding it gets hairy.

All modern web apps (Google Mail, Google Maps, ...., Spotify, Microsoft Visual Studio Code, Amazon Music Player, most web sites) use web services in the background. And naturally, all of these applications use asynchronous programming technologies to not block the browser while critical (!) data is retrieved from the server.

It's just how applications are written today. Blocking execution just to wait for some data coming in was 10 years ago.

Asynchronous programming is very easy, once you you move your head from

Copy this file from A to B.
Full stop program.
Continue when file has been copied or there was an error.

to

Copy this file and let me know when its done or when there was an error.

You don't need to use crude things like await (which may be removed from future JavaScript versions anyway, and for good reasons).

In the case of the OP, a simple solution would be something like:

var foldersToCopy = [];
var nextFolderIndex = 0;

// Now fill all folders to be copied into the foldersToCopy array...

// Copy the first folder. This function calls itself when a folder has been copied until all folders have been copied.

copyNextFolder();


function copyNextFolder()
{
    if (nextFolderIndex  >= foldersToCopy .length())  {
        return; // All done.
    }

    IMWS.endpoint(...).then(response) {
       // Folder copied. Copy the next folder
       nextFolderIndex++;
      copyNextFolder();
    });
}

[kkae]

thanks Mario.
The code works well.

@Mario:

there is a limit of how many requests IMWS processes in parallel before returning error messages

moving about 4450 files with the for loop was succussful with out error.

[Mario]

Yeah, IMWS is powerful.

When too many parallel request come in, IMWS will put them on hold until an active request is completed. But when requests are coming in faster than IMWS can handle them you will get timeouts or "no more request handles" errors in return at some point. Best not to stretch it 

The simple "perform request and when done perform next request until done" pattern I've outlined above should be nearly as fast, but only uses one request at a time. And you can abort it easily (e.g., when the user clicks a button) and you can provide live feedback via a progress bar or animation.

[ubacher]

.....and you can provide live feedback via a progress bar or animation.

I tried some of this just yesterday. Tried the progress bar sample and some stuff from the net.
Problem is to get the display updated in time - it seems it caches updates and displays them (when? when it gets time?)
Just wanted to mention this in case others run into this problem. I put this aside to solve at a later time.

Did not want to inquire at this time in this forum since we have more pressing problems...