Need expert advice on handling asynchronous javascript script

[bnewman]

I'm working on improving a script that I had written that automates the maintenance of images on my website. I'm encountering trouble having to do with waiting for an asynchronous function to complete. I'm in need of some expertise and if you think you can help me, please read on, and offer whatever advice you can. I've have tried to be as explanatory as possible below in the hopes that some combination of IMatch scripter plus javascript expert will know exactly what to do.

Thanks to whoever that might be!

Attached is a stripped down script that captures the essence of the problem, as well as the console log of what happens.

The images for my web are separated into multiple galleries starting with a root category "Web" with subcategories that are separate smaller image galleries, and in some cases, with a second tier gallery of some of those, i.e. some subcategories in IMatch themselves have subcategories.

The attached script is just a stripped down, simplified version of the real one I'm trying to use. The basic point of the script is to start by collecting in a table all the categories with indication of their hierarchical level and the images associated with each category in ascending order of date. The script later uses this table to write html files used in the actual website.

The structure of the script should be familiar to IMatch experts, but basically, the main function is "CollectFragmentData" and after getting the root category information, it makes a call to a function PopulateTable. In a completely synchronized world, the return from this call should be the completely populated table, but some async/wait is needed to ensure that - which is my problem. On entry, to PopulateTable, the file information of the input category is obtained with an "await". Since the root category response contains the complete hierarchy of generations of children, the PopulateTable function calls itself recursively for each child (and a child with children will recurse some more) until all the table elements of category and file information is populated. Each recursive entry to PopulateTable accesses the files for the given category.

Of course, with asynchronous operation, what can happen is that the a given instantiation of PopulateTable will await collecting the file data, and proceed with the subsequent recursive calls. The result of such a run is in the attached console log and it confirms that recursive calls are made completely at a different rate than file information is received.

I know that I have to complete the table populating operation with the line that comes after the initial call to at "await PopulateTable(cat)" called in CollectFragmentTable(), but I just can't figure out how to do that. I thought that the solution would be to put "await" in front of the the recursive call to PopulateTable(c) in the loop over children. However, when I do that I get a runtime syntax error and the script totally refuses to run. I have indicated that bad line. The only way to make it run is remove the "await" but then the operation gets asynchronous.

Note: The attached script will actually fail if run because of a call to "await PopulateTable" issued recursively within that same "async PopulateTable" function.

I hope someone is willing/able to help.

- Bernie

[thrinn]

Hi Bernie,
using foreach with async functions does not work . See the blue warning at Mozilla Developer Network.

But you should be able to user for await ... of instead. Something like this:

Code Select Expand


// recursively add child category objects to the table
if (FragmentTable[index].oCat.children.length > 0)
{
//==> This does not work because function(c) declares an anonymous function which is not async.
//    Do not use forEach with async function!
// FragmentTable[index].oCat.children.forEach(function(c)
//==> but you can use for await instead:
  for await (c of FragmentTable[index].oCat.children)
  {
//************************** the next line is flagged as a runtime syntax error when the "await" is uncommented: Uncaught SyntaxError: await is only valid in async functions and async generators:
    await PopulateTable(c);
    console.log("completed call for "+c.name+",id="+c.id+",parent id="+c.parentId+",level="+c.level+",index=",index);
  };
// });
}


[bnewman]

Thanks so much. That was exactly the problem - works now.

- Bernie

[Carlo Didier]

Welcome to the beautiful new world of the web ...

That's the big reason why I switched nearly all my scripts to Powershell outside of iMatch. MUCH simpler and completely linear 
It doesn't have to be Powershell either. I know, javascript is so modern and used a lot, but I just couldn't cope with all the complications it brought me without any advantage (for me) whatsoever.
Just my 2cts ...

[Jingo]

Welcome to the beautiful new world of the web ...

That's the big reason why I switched nearly all my scripts to Powershell outside of iMatch. MUCH simpler and completely linear 

The more you program with Javascript and use promises, etc the more you understand it and can realize the potential.  Granted, most of our scripts do not need parallel style processing but it is nice to be able to trigger a large retrieve or search function, let the user continue using the script while letting results process in the background.  Once complete, the user can access the results.. or choose to continue doing what they were before.

Powershell is great.. but modern javascript, react, AngularJS, etc is just so so powerful and pretty easy to learn!

[Mario]

The async/await logic introduced in modern JavaScript has removed most of the complexity of asynchronous processing.

const response = await IMWS.get('v1/files', {idlist: '@...', fields: 'id,filename'});
for (const f of response.files) {
    console.log(f.fileName);
}

is really quite straightforward and easy to understand.
But if you implement complex things like foreach iterators mixed with recursive function calls in your script/app, you may need make some extra steps.
For example, foreach just does not support asynchronous functions. It has no concept for waiting. But that stuff is pretty rare in everyday programming. And often the solution is easy, like fetching all the data once and then processing it recursively. Or simple doing a while not done logic which calls itself recursively unless done, making the async call in the function itself.

[Carlo Didier]

Welcome to the beautiful new world of the web ...

That's the big reason why I switched nearly all my scripts to Powershell outside of iMatch. MUCH simpler and completely linear 

The more you program with Javascript and use promises, etc the more you understand it and can realize the potential.  Granted, most of our scripts do not need parallel style processing but it is nice to be able to trigger a large retrieve or search function, let the user continue using the script while letting results process in the background.  Once complete, the user can access the results.. or choose to continue doing what they were before.

Powershell is great.. but modern javascript, react, AngularJS, etc is just so so powerful and pretty easy to learn!

Of course, as it depends largely on what you want to do with the script(s). In my case, all my scripts do things just for me, so there is no other user who would want to do something else while waiting (even when I am that user on the web, I very, very rarely am in a situation where I would do something else while waiting a few seconds for a result to come back; I'm male, so no multitasking  )
I tried to use await but never got it to work as expected and as I already had rewritten most of what I needed in Powershell, I saved my nerves and stuck with what works perfectly for me.

If you write web applications for use by other in the internet, that's a whole different story of course.

[Mario]

PowerShell also supports asynchronous functions (and asyn/await). Not blocking your script and doing other things while waiting for the result of a potentially slow web service call can improve performance a lot.

That said, IMatch WebServices are agnostic. You can use them from JavaScript or PowerShell or Python or curl or .NET or Java or PHP or ... even from Office Basic. Whatever works best for you.
And this will also work whether you connect to IMWS running inside IMatch on your local machine or you use IMWS running on another machine in your network - or even a computer running on a server farm in another country... 

[Carlo Didier]

Not blocking your script and doing other things while waiting for the result of a potentially slow web service call can improve performance a lot.

You are perfectly right. Just that in my case there is no slowness possible in the "web" service as everything is local on the same machine. That's why there is no advantage for me, only disadvantages by complication.

In my day job (not at home), I have also written Powershell scripts which start a number of "independent" threads/jobs/scripts and wait for them all to finish, but that was because each sub-script was started remotely on remote servers (one on each server).

There are many applications where running stuff in parallel makes perfect sense.