Improve the study system (transparency/privacy)

Firefox Development
#1

Background:
To fix the recent add-on certificate issue, which broke all add-ons: Certificate issue causing add-ons to be disabled or fail to install
I enabled studies for the hotfix.

Observations:

  • Firefox installs studies without informing you that a new study was installed
  • Studies get installed without opt-in / opt-out (the current opt-out option is a joke, because you never were informed of its installation - currently the only option for opt-out would be to constantly check about:studies for new ones)
  • Very little information about some studies (study with cryptic name, description ‘sets variable x to y’)
  • Mozilla has already made an effort to make studies transparent (shield study guidelines), but none of this makes its way to the end user.

Suggestions for privacy:
Privacy aware users might want to participate in studies, but a few things that should change:

  • Opt-in as a user setting. Only studies can set opt-in. This means if you enable studies you are automatically subscribed to opt-out studies.
    Replace the checkbox “Allow Firefox to install and run studies” to a radiobutton (those are the round single choice selections) with the title “Allow Firefox to install studies and share study relevant information” and options “Always”, “Ask for each new study”, “Never”
  • Make studies independent from “Allow Firefox to send technical and interaction data to Mozilla”.
    This is why I added the “share study relevant information” above. This would allow people to participate in studies while having a higher privacy setting.
  • Show by text or by example which information is shared/sent by keeping a study enabled (opt-out), or enabling a study (opt-in). Always display this extra information to the user, even if it is ‘None’. (This would be a new field required for every new study in the study/normandy-API.)

Suggestions for study transparency:
Apologies in advance - in this lengthy section I make a few points, which are basically identical and can be summarized as:

Put more focus on the end user.
This means making study information human readable by the average user and making it available within firefox.

  • Inform the end user when a new study has been installed.
    If you enable studies you can only opt-out when you are aware of the study being on your computer.

  • Show (more) study information in about:studies (and in a potential future opt-in dialog). - The information provided for some studies is only along the lines of "This study sets variable to value" with a cryptic name.

  • The shield study guidelines (https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guiding_Principles) state:

    All Shield Studies must be designed to answer a specific question

    Make this a requirement for all studies to formulate a specific, average user readable, goal.

    Two examples (these are elaborated further down):

  • To create a shield study you need to provide a lot of information (Example study application, push notification study: https://bugzilla.mozilla.org/show_bug.cgi?id=1491171) - none of this information is displayed in about:studies and even when the API provides a link to the bugtracker, where a lot of this information can be found, (“experimentDocumentUrl” in https://normandy.cdn.mozilla.net/api/v1/recipe/597/) it is not displayed to the user.

  • Information displayed in about:studies reads more like it was written for mozilla/developer internal record-keeping with no regard to the end user.
    For example the study to add a hotfix for add-ons:

      "name": "Hotfix: Update XPI signing intermediate [Bug 1548973]",
  "arguments": {
      "name": "hotfix-update-xpi-signing-intermediate-bug-1548973",

    In about:studies this shows as “hotfix-update-xpi-signing-intermediate-bug-1548973”. The study-name, “Hotfix: Update […]”, is not shown. From the point of view of the average user either name is a whole lot of nothing. This one has at least a somewhat reasonable description ("[…]updates an intermediate certificate used for signing add-ons[…]" - not perfect for an end user, but it mentions that it’s a hotfix for some add-on issue). But there are worse examples:

      "id": 597,
  […]"name": "Pref Flip:  Push Performance Shield Study, release 62 and 63 [Bug 1491171]",
  […]"arguments": {
      […]"slug": "prefflip-push-performance-1491171",

    All of the information about this study displays in about:study as:

      prefflip-push-performance-1491171
  This study sets `dom.push.alwaysConnect` to `true`

    Hardly something an end user can be expected to understand.
    In the API for these studies I saw that you can attach a URL with more information to the studies, but despite there being a post about this study (https://blog.mozilla.org/services/2018/10/03/upcoming-push-shield-study/) and despite there being more information about this study in the bugtracker (https://bugzilla.mozilla.org/show_bug.cgi?id=1491171) neither was attached as information visible to the end user - the API provided a “experimentDocumentUrl” to the bugtracker where detailed information about the study can be read, but it was not displayed to the user.

Semi-related, show all studies:
Using the normandy api at https://normandy.cdn.mozilla.net/api/v1/recipe/ to list all studies is guesswork as to how it would be displayed to the user.
Can you view all studies in about:studies (is there a developer switch to do this)? If the transparency towards the end user would be improved in the future allowing developers/users to view all studies (without being allowed to enable them) could help in improving the quality of the way studies are displayed to the end user (as good examples can be seen and bad examples improved).

I hope these suggestions inspired some people in charge to change some study-policies to make studies more transparent to the end user and/or allow for opt-in studies and inform the end user when a new study was installed.

#2

My thoughts on this (as a privacy-minded Firefox user):

Support:

  • Opt-in as a user setting. Only studies can set opt-in. This means if you enable studies you are automatically subscribed to opt-out studies.

  • Replace the checkbox “Allow Firefox to install and run studies” to a radiobutton.

  • Show by text or by example which information is shared/sent by keeping a study enabled (opt-out), or enabling a study (opt-in). Always display this extra information to the user, even if it is ‘None’. (This would be a new field required for every new study in the study/normandy-API.)

  • Show (more) study information in about:studies (and in a potential future opt-in dialog). - The information provided for some studies is only along the lines of "This study sets variable to value " with a cryptic name.

Comments:

  • Make studies independent from “Allow Firefox to send technical and interaction data to Mozilla”. This is why I added the “share study relevant information” above. This would allow people to participate in studies while having a higher privacy setting.

I believe that the grouping we have now is also OK. I see where you are coming from, but this allows to have a simple checkbox for studies as well.

  • Inform the end user when a new study has been installed.

I don’t see how to make this work, maybe like the Firefox update notifications that hide into the hamburger menu?

  • Personal comments:

I feel like this is not the right place to post this. Bugzilla would seem weird either, but it is the place for suggestions as well after all (?). I only found this post by looking how to enable Normandy hotfixes without enabling Shield studies. (The answer is about:config -> app.normandy.enabled)