Help Statistics and Telemetry in IMatch...what do you think?

[Mario]

The IMatch 2018 online help system allows me to see how often each help topic is viewed (don't worry, I don't track you personally!).
After IMatch 2018 being out for two weeks, I've made a quick check today. The results are below.

Of course only a part of the IMatch user base has yet switched to IMatch 2018 and thus uses the online help.
I'm doing a slow roll out to fix the inevitable initial glitches and bugs.

I hope to learn from these stats which features are used most and which features require the most help.
I can then improve IMatch and the help, maybe add some more videos.



Telemetry

I'm looking into ways to make IMatch gather telemetry data. This data would allow me to see

+ How long and how often users use IMatch
+ What is the average and max database size
+ Which features are used most often
+ Which features are never used
+ Which features are 'slow' and need improvement
+ ...

IMatch could gather this info and then (with the user's explicit consent) to upload the data to my server so I can collect it and produce statistics.
Needless to say that this will not be linked to any personal data, I'm only interested in the global picture.

My problem is that only a part of the user base actively participates in this community. Most users never contact me (except they run into a problem) and so it's a bit of a black box for me.
Learning more about how users work with IMatch will allow me to enhance it in the right areas.

What do you think?

[Aubrey]

Mario,
I was part of the way there in June 2017  :
https://www.photools.com/community/index.php?topic=6727.msg46546#msg46546
Looks like it got archived. 

I would be happy to participate and have my usage being uploaded.

Obviously, it would be nice to also see how I personally use IMatch, so it would be nice for users to produce personalized graphs.

Aubrey.

[herman]

[...]
IMatch could gather this info and then (with the user's explicit consent) to upload the data to my server so I can collect it and produce statistics.
Needless to say that this will not be linked to any personal data, I'm only interested in the global picture.
[...]
What do you think?

I think that in these days, where many people have privacy concerns and data breaches are published almost daily, the best way to do this is to make it optional (in one of the application settings) with the default to off.
You may then have to draw users' attention to this new setting and explain why you wish to collect data.
It may also be wise to let the user see what you collect before it is submitted.

Just my (privacy-centric) proverbial 2 cents.

Hope this helps,

[loweskid]

Sounds like a good idea to me - I would have no objections.

[Mees Dekker]

I agree completely with Herman: make it optional and ask for an explicit agreement from your users.

This may also be a point to differentiate between IMatch and Adobe: more privacy.

[pajaro]

I agree completely with Herman: make it optional and ask for an explicit agreement from your users.

Yes, this is a good idea.

[jch2103]

I agree completely with Herman: make it optional and ask for an explicit agreement from your users.

Yes, this is a good idea.

I agree. Having more data on how IM is actually used and how it performs should be very useful, especially compared with the limited sampling available via Forum posts.

[BanjoTom]

I agree as well - and while I appreciate the idea of having IMatch users opt in for this collection of data, I also must say that Mario has repeatedly demonstrated his respect and concern for all users' security and privacy.  So I'd let HIM have my usage data whether it's "opt in" or "opt out." 

[ColinIM]

The IMatch 2018 online help system allows me to see how often each help topic is viewed (....)

This is a superbly logical idea, and I had presumed that you would (for the benefit of us all) take advantage of web-server data such as this 

I'm looking into ways to make IMatch gather telemetry data.

I am broadly in favour of this too, but I am passionate about guarding my privacy so - simply on principle - I would want to know in detail what information was being relayed back to your servers, even though you would carefully omit my personal data.

I would risk a guess that you might try as much as possible to 'distill' and consolidate the telemetry data from my IMatch (and the 'PC' environment in which it's running) locally, here on my computer before sending the 'summarised' data back to your servers.

I look forward to seeing how this idea develops.

[zematima]

I will participate.

[Jingo]

Take what you need... you have earned my trust over the years as a guardian of personal data..

[thrinn]

I agree completely with Herman: make it optional and ask for an explicit agreement from your users.

Yes, this is a good idea.

I would also participate.

[sinus]

Good idea, if it helps finally making IMatch better, then it is surely good.

Really in the same time I wrote for me some ideas, give Imatch-users some "statistics", now I did a post about them 

https://www.photools.com/community/index.php?topic=8225.0#new

[dkorman]

Mario,

I do understand your desire to REALLY know how customers are using iMatch by introducing remote session usage metrics ("telemetry data") - otherwise how can you determine what users are "actually" doing with the product (you do have this community, but I don't know what percent of your sales results in community participation).  I have some experience using automated product functional usage metrics, and sometimes (a) it results in more issues than it resolves, and (b) it helps to have a well-thought out statistical model (and goals) ahead of time (especially concerning how representative the data actually is of the user base, and how much you want heavy users to have a undo influence on "policy").  "Marketing" and "Engineering" typically have very different perspectives on the utility of this data (are you motivated to drive sales or develop exciting new technical features and/or fixing bugs?).  For example, with usage data in hand, a Marketer might ask, "Should heavy users, who are clearly more dependent and committed to the application than light users, be charged more?"  With the data, a Marketer/Product Manager might conclude: "We really have two very different sets of users, and should have two different products."  Just sayin' ...  Without the data, you really can't ask these questions - or "pretend" to answer them! 

In the stats that you have collected, analyzed, and presented on the online Help System, does a high usage mean that the subject represents the "relative" importance of subject, that the feature is, for some of the user base, confusing or not intuitive, that there is just curiosity in the subject, etc.  One would need to know "why" the user is seeking help in a given subject.  We've all seen online help systems ask users to add comments concerning the "usefulness" of the help ("Did this page address your concerns?").  Without this additional information, you don't know if users are leaving "satisfied" or "confused" (linkage data - "where did the user go next" - tends to be used to answer this question, though my experience is that it is often misinterpreted).  So, my message is, "Be careful how you interpret your data."  "Something" is not always better than "Nothing," especially if it misleads, misrepresents, or dismays.  There is also the tendency, once data is being generated, to want more and more - data always seems to want to grow ("If only we also knew ..."), and can be its own distraction - though often a fun distraction (TensorFlow?).  Despite how tempting it would be to start analyzing data when it first arrives, I would recommend waiting until you had a pre-determined statistically significant data set before drawing ANY conclusions. 

So, if you want to experiment by gathering your own "big data," I'd be happy to participate, providing you:

(a) were transparent in the data you were collecting,
(b) provided a mechanism to toggle collection on and off (this will bias the data),
(c) default to "opt out" (with easily accessible toggle for changing this setting - unlike Google) (again biasing the data), and
(d) fully anonymize the data (willing to give up modeling any specific user - a difficult, but necessary, decision, in my opinion).

[Perhaps a user might be interested in their own usage over time, or how their usage compares against other users (there could be an App for that!).] 

Mario, as I read the comments of others, it is clear that you have earned the respect and trust of this forum community, which I believe is quite a significant personal achievement.

[Mario]

I will do some very simple things. I'm no product marketing expert. I just want to make IMatch more useful and remove stuff nobody is using.
The facts I'm interested in are:

- which features are used by how many users?
For example, how many users use the Statistics Panel or Design & Print?

If features are underused this either means that a) users don't need them or b) don't know about them or c) don't understand them.
Gathering the what's used is only the first step.

- how often is IMatch used?
Considering a 'reporting period' of one month, how often has a user started IMatch and how long has he/she used it?

- What are the largest and average database sizes? Category count? Number of keywords?

- which features are too slow?

- Which apps are used?

Nothing of this will be linked to a specific person. May to a specific installation (by random number assigned during installation) so I can combine data from several month for the same user. I don't need to know which user that is, it's more than enough to know that user 4032984092384023804 uses IMatch on average 20 hours per month and has never used Design & Print or the Slide Show.

And of course is will be opt-in only.

I hope to learn which features need to be explained better (video, know-how article), which features need to be improved and which features can be removed. If a feature that costs me 1 work-day per month is used by only 5 users, it is a candidate for removal.

I guess I will use a JSON-based (text) format for the report, so users who are interested can see in-detail what is sent once per month. I really don't care about your personal data or other info because I'm not in the business of selling customer data. Many do, I don't.

I only want to make IMatch and IMatch Anywhere more useful and streamline the development process. Projects of this size, with all the surrounding machinery and features like documentation, tutorials, videos etc. are a massive undertaking and I need to automate as much as possible and don't waste time on stuff nobody uses.

Only a part of the user base participates in this community. This is normal.
If you have, dunno, 50 million users like Adobe and 5% or 20% of these users participates in a community you can learn a lot from that as a vendor.
For a niche product like IMatch, 5% or 20% is much less useful. Satisfied customers rarely post. Known fact.

I often hear from users only when they run into a real problem, and then via a direct email. It's very often like: "Hi, I'm using IMatch for 5 years and I had never a problem. But today..."

[sinus]

[Perhaps a user might be interested in their own usage over time, or how their usage compares against other users (there could be an App for that!).] 

I can only confirm that.