Can we disable or defer automatic data-driven category refreshing?

[dnovak]

This may duplicate another user's post which I cannot find now - apologies in advance.

I have several data-driven categories I'm using to sanitize my data (e.g. I have a lot of data in XMP PLUS Custom1-Custom10 fields that have data for fan art like character name, source medium or copyright, artist name and others)

This is mostly correcting typos and consolidating duplicates and variants on the same value, so I'm switching from one node in the category tree to another as I make edits.

I'm using the IMWS /metadata endpoint to do bulk updates and while this is working well, IMatch seems to freeze as it' refreshing the data driven categories.  The entire system seems to slow down, including mouse movements.

I've disabled Automatic Update on ALL my data driven categories, but this made no difference.

Updating metadata with typing or pasting in the Metadata tab doesn't seem to trigger this refresh.

Is there any way to temporarily disable or defer ALL data-driven category updating, whether updating data manually or through the API? 

I'm thinking something roughly similar to disabling "repaint" in old VB6 apps, though I realize this isn't the same thing

TL;DR: when doing updates to metadata and attributes with the API, can we disable refreshing the Category window?

PS: I have found other posts that suggest not having the category window opens as any open window may automatically refresh, but I cannot do this because I'm using the category tree to find and clean data.

Many thanks,

Dan Novak

[Mario]

If a data-driven category is set to manual it will only update on manual interaction by the user.

I understand you have written a script which updates metadata via IMWS. Did you write that script properly?

Often users don't consider that web services like IMWS and JavaScript work asynchronously and than iterating over a loop of files and doing something like

for each file in my list
  call IMWS to update metadata  ... v1/metadata...
end

actually processes all files at the same time because calling an endpoint returns immediately, while the web service is still processing the request.
And this means that the loop continues with the next file, and this can easily pile up.

I have seen scripts written which forced IMWS to process over 500 metadata updates at the same time, slowing down the system massively.

If you say that even the mouse cursor becomes slow, this is very likely the case. Not data-driven categories.

[dnovak]

>> If a data-driven category is set to manual it will only update on manual interaction by the user.

I'll re-test this.  I'm pretty sure it happens even for one image, updating one metadata field.  It doesn't do this if I edit the metadata manually.

>> Often users don't consider that web services like IMWS and JavaScript work asynchronously and than iterating over a loop of files and doing something like [omitted]

I'm accounting for that, but I'm also rather new to asynchronous programming.

>> I have seen scripts written which forced IMWS to process over 500 metadata updates at the same time, slowing down the system massively.

This may be the problem.  I'm calling in batches, but it can still be around 4 fields in 200 images at a time.

My changes don't need to happen in real time; I'll see about throttling the updates or doing smaller batches.  I'm experimenting with saving the JSON task definitions in Attributes and running them later.  This seems to be working well.

>> If you say that even the mouse cursor becomes slow, this is very likely the case. Not data-driven categories.

Thanks, I'll do some tuning and see if I can narrow down the problem.  I only suspected category refresh because of the status bar.

[Mario]

Run your script in your web browser while the developer tools are open (<F12>).
The you cannot only debug it step-by-step but you also see the requests your script makes against IMWS, and how many and how fast.

My guess is you loop over files somewhere, call IMWS endpoints and don't wait for the promises to be resolved. Resulting in tons of request being fired at the same time, slowing sown everything. This can be seen in the debugger easily.

[dnovak]

After starting a fresh database I'm not seeing freezes or slowness anymore.  I suspect the problem was a complex data-driven category I'd been using for filtering and forgotten about.  (I distinguish artwork from photos by folder and am in the process of changing my folder-based organization to use attributes or metadata.  For artwork "person in image" and "photographer" are meaningless, so I'd been filtering them out of the corresponding data-driven categories).