APP: IMatch Log Reader

[Jingo]

Hi all.. a few years ago, I started writing a log viewer app to be able to quickly view the log file from within IMatch.  I had never got around to finishing it nor adding some features I especially wanted - like quickly showing all warnings and errors.  I had a few minutes this week and figured with a new release, this APP might be beneficial to the user base so finally finished it up!

As such, I present you with.. the IMatch Log Reader App!  The App will allow you to:

• View any Warnings or Errors that might be present in the current Log file (note: not all warnings are bad or worth thinking about.. but they are interesting to see).
• View the entire log from within IMatch
• Search the Log for specific entries (very useful!).

Now, of course it is simple enough to do all this with a text editor.. but I figured it would be a fun thing to write.  The code, for those interested in IMatch APPs, also demonstrates how to get around local Chrome security items (the need to copy the log file into your local App folder), search an array and bind events using listeners.. so, something for everyone!

Without further Ado... here are the screenshots.  The all important App Icon:



The App Modal:



To use:

• Click the Choose File button and select the Log.txt that exists in your app folder (it is copied locally when the app is launched).  The program will display any warnings and errors.
• Click the View File checkbox to load and display the entire log.
• Type a phrase into the Search For text box and the entire log will be searched and results displayed.

As always - please unzip the file folder into your app path and the icon will load.   If you note any issues, please let me know and I'll do my best to correct. 

Enjoy!! - Andy.

[Mario]

Thanks for sharing  

[Jingo]

My pleasure.. a goal for my next app is to try and use some more advanced stuff to allow the apps to look more modern.. like the compare metadata app.  Always trying to learn and stretch my abilities!

[ubacher]

Tried it. Great.
It does what i had wanted here:https://www.photools.com/community/index.php?msg=92437

An enhancement I would like to see is a list of the various log files to select from.

%temp%Imatch6_log.txt
%temp%Imatch6_log_backup.txt
%temp%Imatch6_diagnosis_log.txt
%temp%Imatch6_PackandGoLog.txt
%temp%Imatch6_PackandGo_diag.txt

(otherwise the app just saves me searching for errors/warnings manually)

Doable?

[jch2103]

Thanks! Useful already: The app ID Favorites that no longer exist on my laptop, so I was able to delete them. 

[Jingo]

Glad you like it!  I've found it convenient as well...

I think I can add more log files for viewing - just need to finagle the code a bit to add some checks and balances for missing logs (not everyone uses Pack and go so no log would exist in that case).  Let me see what I can do! 

Andy.

[Tveloso]

Nice!

Andy, I wonder if would be possible to add a feature to allow a user message to be "posted" into the log?...(i.e. to support writing to the log in your App).

There have been times when I've been reviewing a log to see if I could identify the messages issued around a certain operation I had performed, and found it difficult to identify where in log the related messages were...(especially with a debug log - even if I had a pretty good idea of the times involved).  I thought that "it would be nice to be able to 'post' a message into the log"...something like:

    <timeStamp> USERMSG: Starting such and such operation

...and:

    <timeStamp> USERMSG: Operation such and such complete

What do you think?

[Jingo]

Hi Tony - thanks for the suggestion... I like it too!

I believe I have access to the "addlogentry" POST command so should be able to write an entry to the log.. however, there are 2 takeaway's to consider though.  

 1 - the file will need to be copied back from App folder where the entry is written to the TEMP folder where the log actually lives.. if the file is open by IMatch, it might not be available for overwriting in the app.  I'll need to check this.  
 2 - the addlogentry command probably just appends to the end of the text file with the current date/timestamp.  So, even if you pass in some text with a timestamp - it would get listed at the end of the log when the Post command is sent (I'm guessing about this).

If these are both ok.. I'll give it a shot and see what I can come up with!  Thx for playing with the app! - Andy.

[thrinn]

Wouldn't adding an user log entry be a completely different use case? From the order of steps, I mean? From my understanding, the (original) purpose of this App is to analyze an existing log file after the steps (working in IMatch) we want to analyze have been performed. The user log entries, on the other hand, should be written (basically as comments) while working with IMatch.

Let's assume I am working with IMatch for a while (which means that the log is already growing and growing). Now, I want to analyze something. It would be nice to have now a text entry box and a button available (that's Tveloso's proposal, I think) where I can enter some comment (like "Starting my test") and push the button to just write this comment to the log at this point in time. This should be possible using the addlogentry endpoint.
Now I perform the steps I want to analyze, then I put in another comment ("Test is finished"), press the button, done.
Afterwards I use the App to analyze the log, and I can find the log entries I am interested in easily by searching for my comments.

This way, it is not necessary to try inserting some log entries "by hand" into the log file itself. IMatch will write the comments into the log file at the correct position. And you don't have to think about copying a log file back to the TEMP folder (which would be a bad idea, in my opinion). IMatch should be the only application modifying the log file. The App should only analyze / read it.

[sinus]

Thanks for this app, Jingo.

This seems to be a very good idea. I will use this app for sure, when I have time to explore.
I think, if I have a long log-file, your app could be help a lot.
BTW, you have a cool icon. 

At the moment I am a bit busy with business and with searching/bujing a new car, because my Peugeot 2008 is not more a car, but a "scrap pile" - unfortunately because of an accident. 

[Tveloso]

Wouldn't adding an user log entry be a completely different use case? From the order of steps, I mean? From my understanding, the (original) purpose of this App is to analyze an existing log file after the steps (working in IMatch) we want to analyze have been performed. The user log entries, on the other hand, should be written (basically as comments) while working with IMatch.

Yes, Jingo's App is probably not the appropriate place to add this functionality...(and this should really be a different App - or maybe even a feature in IMatch?...perhaps available when Debug Logging has been activated?).

Let's assume I am working with IMatch for a while (which means that the log is already growing and growing). Now, I want to analyze something. It would be nice to have now a text entry box and a button available (that's Tveloso's proposal, I think) where I can enter some comment (like "Starting my test") and push the button to just write this comment to the log at this point in time. This should be possible using the addlogentry endpoint.
Now I perform the steps I want to analyze, then I put in another comment ("Test is finished"), press the button, done.
.
.
.
This way, it is not necessary to try inserting some log entries "by hand" into the log file itself. IMatch will write the comments into the log file at the correct position.

An excellent description of this functionality.

[Jingo]

Thorsten is correct.. because the addlogentry command only appends to the log - having the ability to add an item and then run a test to evaluate and review the results is really the only option at this point.  I'll be working on this over the next week or so when I have some time again.

I'm happy the app is showing to be useful to others as well!

[Mario]

Note: The IMatch log file uses an in-memory cache and bulk writes, plus force writes before certain tasks and in case of errors or warnings being logged. The file is write-locked, but you can copy it while it is open (but it might not reflect all the log contents at the time, due to the in-memory cache).

The addlogentry IMWS endpoint allows apps to add log file entries. These entries appear with the A> tag and are time-stamped automatically by the log file mechanism. You just supply the text you want to have logged.

[ubacher]

The addlogentry IMWS endpoint allows apps to add log file entries. These entries appear with the A> tag and are time-stamped automatically by the log file mechanism. You just supply the text you want to have logged.

I have been using this in many of my apps to log that the app was called ( and some other info). Very usefull.
I suppose one could write a simple app which asks for some text and then writes an entry to the log.

[Jingo]

I suppose one could write a simple app which asks for some text and then writes an entry to the log.

I've completed the work and just doing a bit of testing... should be able to share in a few days.

[Jingo]

Ok... I have successfully incorporated the following changes into the LogRead app:

1 - The additional Log files are now available for error/warning views, full log viewing and full log searching.  The logs are:
  IMATCH6_CEFLOG.txt
  IMATCH6_DIAGNOSIS_LOG.txt
  IMATCH6_LOG.txt
  IMATCH6_LOG_BACKUP.txt
  IMATCH6_LOG_MIGRATE.txt
  IMatchPackAndGo.txt

Loading each file will allow you to do all functions regardless of file type.  In fact, you can actually load any text file and use the view/search button functions.. a BONUS!

2 - You can now add an entry into the main IMATCH6_LOG file.  The entry will have the most recent time stamp along with
"A> APP Log Reader:" plus your free text.

NOTE: as Mario mentioned above (and I ran into during testing), your entry will not appear in the log file until IMatch forces a cache write of the data.  As such, you will want to add the text to the log, close the app, perform your tests in IMatch and then relaunch the app to see your entry and the new results.

You can just overwrite the current app folder with the new one from the attached zip file (marked internally as app version 2.0).

Enjoy and thx for the suggestions! - Andy.

[ubacher]

The additional Log files are now available for error/warning views, full log viewing and full log searching.  ....

?? I still have to find and enter the log file name - don't I? Nothing changed there as far as I can see.

And how do I trigger the writing to the logfile?

[jch2103]

I'm having trouble getting this version to work (prior one worked OK). I'm able to navigate to the LogReader folder, and I'm able to see all the log files there, but after I choose one and click 'Open' I'm just returned to the app home screen (with a message that keeps repeating - the message keeps repeating even before I try to open a file). See screenshot. 

[Mario]

I have not looked at or used this app (I never do), so I cannot comment on the details.
But trying an idea for a minimal lister I've stumbled upon the problem that the"v1/filesystem/file/readtext" endpoint fails to open the log file, despite it being opened with "allow read" sharing.
I've fixed this glitch and now a minimal log file reader that lists W> and E> on the console can be implemented like this:



Maybe this saves the need to copy the log file around before reading it.
Note that log files can become really huge and maybe copying and using FileReader or similar is then faster.

[Jingo]

Hmm.. I'm not have any difficulties here opening the text files - even with very large debug log files.  I wonder if something else is going on in your system?  Can you try to launch the app in the browser, hit F12 and using the console panel in the debugger, see if any warning or error messages show?

In the meantime, I'll be rewriting the app to use the IMatch.readtextfile function (which I somehow missed and which caused me to do all this file copy rigmarole - DOH!!!).

So, you might just want to stay tuned for V3.0!

[Mario]

In the meantime, I'll be rewriting the app to use the IMatch.readtextfile function (which I somehow missed and which caused me to do all this file copy rigmarole - DOH!!!).

This will fail for reading the log file due to locking issues.
IMatch 2023.1.14 can read the log file with this endpoint too.

As I said, when the log becomes very large, "downloading" all this data via the endpoint may be slow and a native JavaScript function may be faster. Never tested this, so...

[Jingo]

I'll play around... and see how well (and if) it works.  

I've been using the log reader a few days now and I've never had an instance where it didn't load the file.. even with a log that had 3000 lines of text so I'm curious why others might be having issues with the original code.

[Mario]

3,000 lines is nothing. A log in debug mode of a large databases from a session running some hours can be a GB in size or more.

[Jingo]

True.. I suppose I may need to try and work on a chunking system for reading the data... or come up something unique when time allows.

At work - I was having trouble loading a huge database of info into a data grid.. the data was taking minutes to load, causing massive delays and generally making the software impossible to use.  Then, I stumbled across data grid pagination which loads enough data to display the first page (say 30 entries) and then loads the rest of the data into the background.  Where there is a will - there is generally a way!

[Mario]

This is how you usually also deal with potentially huge text files.
You parse the file to find the line start/end offsets and keep that index in memory.
This allows you to specifically load individual lines when scrolling and you never need to hold more than a couple of pages of lines in-memory.

The IMWS endpoint does not support this, it just loads the text file from disk and returns it.
This is designed for small text files like config files or scripts, so apps can create and read this data.
I think this is also not possible with JavaScript's File or FileReader either.

I use a Python script to automatically analyze log files. Most of the time, Windows Notepad does the trick too ;-)

[jch2103]

Hmm.. I'm not have any difficulties here opening the text files - even with very large debug log files.  I wonder if something else is going on in your system?  Can you try to launch the app in the browser, hit F12 and using the console panel in the debugger, see if any warning or error messages show?

That worked. I didn't bother to use the debugger because no problem running it in the browser. Not sure what the issue is w/ my installation of IM. I'll check into that.