New endpoints in the works: Folders & Files

[Mario]

When things go according to plan, the next release will include a (more or less) complete set of endpoints to add/move/copy/rename/rescan/relocate folders and similar functions for files.

I may also duplicate or move file-system endpoints from the IMatch IMWS into general IMWS. Since all IMatch IMWS endpoints are wrapped in the IMatchLib.js this will not cause changes in scripts, unless users have directly accessed IMatch IMWS endpoints. But even then it will take a few seconds to update.

Two main problems had to be solved for implementing folders & files endpoints (actually also Attribute and Category endpoints, but I just postponed that to get things going for the 2017 edition).

1. Preventing more than one script/app/endpoint to manipulate the file system / database at a time.

For example, it should not be possible that one app is copying a folder another app is trying to rename. Maybe even the same app, since an app can run multiple endpoints at a time.

I have implemented this with a new lock manager that handles all this. Relevant endpoints now use the lock manager internally to lock "things" in the database. If two endpoints try to lock the same "thing", the first one wins and the other returns a "Cannot lock object, try later" error to the caller. The lock manager automatically retries a few times internally so the likelihood of actual collisions is minimal.  For IMatch apps anyway. It will be much more likely in IMatch Anywhere once it allows users to change the database remotely.


2. Implementing a way to provide 'live' feedback to an app during long-running endpoints.

Consider you write an app which copies or moves folders. A folder may have 10 files or 1000. In the first case the endpoint may need 2 seconds, in the second case it may run for two minutes.

jQuery AJAX functions (this is what you use when you use my wrappers IMatch.post() or IMWS.post()) have no timeout by default. This is good. You call the v1/folders/copy endpoint and whenever it is done your app does whatever it needs in the then(...) function.

What is needed, though, is a way to inform your app about the progress, errors, about what is happening while the endpoint is running. Maybe you want to block parts of your apps UI while the copy is running. Maybe you want to display a progress bar. Whatever.

The IMatch engine which does all the work already had a feedback mechanism. IMatch uses this to display progress bar dialog boxes during file operations, error messages and suchlike.

For IMWS I have no adapted this and route these events to web sockets. This means that your app receives events like "file system operation begins", "file system operation ends" or "file system operation completed to 80%". Copying files or adding a new folder to the database is now easy to do, and and apps can provide feedback to users as well.

Tomorrow I will try to find a way to allow an app to abort certain operations. It should be possible for example to cancel a copy/move or folder rescan. IMWS needs to do it when the session expires and apps should be able to do it when they need to.

And I also have to implement all new file endpoints. So far i have only the folder endpoints.

One of the next things on my list will be an image processor endpoint. This should be a replacement (of sorts) of the IMImage class (or was it IMBitmap?). Apps need a way load, manipulate and save images, for example. for gallery scripts or maybe a new KML export app. Or maybe I will add KML export to the map panel. Which requires a way to create thumbnails in specific sizes and to create a ZIP file with all the stuff. Another set of endpoints.

The good thing: If we have all that, I can start to write the new features for IMatch which I have planned. The new Map Panel in IMatch was just the beginning. And of course the file lens feature for IMatch Anywhere Web Viewer - cool stuff.

[thrinn]

Another big leap forward  For me, these file management endpoints were the last gap I missed a bit in IMatch 2017.

I really appreciate how you not only permanently add new functionality to IMatch, but also explain some of the technical background. Changing or implementing something like the internal locking mechanism sounds like a lot of work to me.

One question for my understanding:

...will not cause changes in scripts, unless users have directly accessed IMatch IMWS endpoints

With "directly accessed" you mean using "raw" AJAX calls instead of IMWS.get/post etc.? Or put the other way round: Apps using IMWS.get()/post() will not have to be changed when you move an endpoint into general IMWS?

[Mario]

I meant what you use the methods of the IMatch class like IMatch.fileMove instead of directly calling the endpoint v1/imatch/filesystem/file/move you will not be affected by any changes. Else you may need to change v1/imatch/filesystem/file/move to v1/filesystem/file/move which should not take long.

I need to decide for each 'file system' endpoint if I can move it into IMWS or need to have two implementations (in IMWS and in IMatch-IMWS).
Of course it is preferable to keep all functionality in the global IMWS implementation. Like, moving a file in the file system. But the IMWS file system endpoints in IMatch can do 'more' than IMWS, e.g. providing user feedback, disabling the IMatch UI, displaying error dialog boxes etc.

I need to ponder whether I do the split internally (making v1/filesystem/file/move behave differently in IMWS and IMatch-IMWS) or if I keep two separate endpoints to allow programmers to choose the kind of behavior they want.

Since the iMWS interfaces will be used more and more in IMatch itself (I can design & implement new features much faster as apps than native code) and also in the IMatch Anywhere™ product line, it is important to get this right.

[Mario]

Today I have implemented the endpoints to cancel long-running file operations and also added a parameter that allows apps to control how existing files are handled.

To cancel a long-running file system operation (like, /folders/copy)  you just call the /cancel/fileop endpoint. The /folders/copy then returns with an USER_ABORT error. Similar to IMatch, when the user presses Cancel in the progress dialog while copying files or folders.

Existing Files

When you copy/move folders and files, it may happen that one or more files already exist in the target folder. IMatch then displays a dialog box, asking the user what to do. This if of course not possible for IMWS. Hence the app must decide up-front what to do: replace, dontreplace, replaceifolder. IMWS then applies this setting during the current endpoint call. By default, existing files are overwritten.

An app can scan the target folder in advance to find out if files already exist and then prompt the user. This is the responsibility of the app.

[Mario]

Endpoints for adding, rescanning, copying, moving, deleting and renaming files now also work in IMWS.

They are more flexible than the corresponding methods we had in IMatch scripting too! No separate functions for moving/copying/deleting one file or multiple files.  No separate methods for moving/copying files "within" the data or to folders not in the database. It's all managed by the endpoints in a transparent fashion.

As so often, it took longer to extend my test framework to include test for all the new endpoints as it took to implement the endpoints. But as IMWS becomes more complex, I need to automate testing as much as possible. The general idea is always to press a button and then my test app runs and after a while tells me that several hundred tests have been completed successfully - or not. 100% test coverage is of course impossible, but the more the better.

Stage 3 of IMWS is well under way and the next release of IMatch and IMatch Anywhere will include it (unless something unexpected happens). I'm sure some of you are waiting for this to convert more scripts 

[jeg]

You wrote something about re-scan a folder. I'm looking for such an option.
What I'm trying to do is to copy an existing image to registered folder to create an new version of the original image. For this I have to re-scan the folder and update the master image relations.
Can I do this at the moment?

[Mario]

Do you use the files/copy endpoint?
Showing a bit of what you are doing would help to help you.