Tails

#6 Updated by intrigeri 2019-09-14 16:53:09

> the custom theme (grep /usr/share/pixmaps/red_dot.png) is not applied

I’ll take this one as this breaks test suite steps before we can test more relevant stuff.

#7 Updated by intrigeri 2019-09-14 17:18:56

intrigeri wrote:
> > the custom theme (grep /usr/share/pixmaps/red_dot.png) is not applied
>
> I’ll take this one as this breaks test suite steps before we can test more relevant stuff.

Well, https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Themes#Documentation says that “Lightweight themes have been deprecated”. This probably explains why our custom lightweightThemes prefs are ignored.

I see two options:

• include an existing red theme: https://addons.mozilla.org/en-US/firefox/search/?platform=windows&q=red&type=statictheme
• generate our own theme for self-distribution: “Themes submitted for self-distribution will be signed immediately and an XPI file will be generated for download”

Either way, we’ll need to include a (signed) XPI in our Git tree, so there’s no benefit in creating our own theme, if we can find an existing one that does the job and has a compatible license.

#8 Updated by intrigeri 2019-09-14 19:36:19

intrigeri wrote:
> * include an existing red theme: https://addons.mozilla.org/en-US/firefox/search/?platform=windows&q=red&type=statictheme

Done, and it’s installed by default (as per about:addons) but not enabled.

#9 Updated by intrigeri 2019-09-15 12:04:53

To be clear, I’ve released the “lock” I’ve set on part of this ticket yesterday. I don’t plan to work on it again before next Thursday.

#14 Updated by segfault 2019-10-04 19:33:12

> That extensions.activeThemeID is not respected on first start must be a bug, which we should report to Mozilla. But first we should reproduce in a normal Firefox install, not Tor Browser let alone the Unsafe Browser, and also look if it is fixed in more recent (non-ESR) Firefox

I did that with Firefox 69.0.1 on sid. I did what b337af619b931a0b7ee25ad81b7cfb1c012da089 does:

ln -s config/chroot_local-includes/usr/share/tails/chroot-browsers/unsafe-browser/extensions/red-2.0-an+fx.xpi \
    ~/.mozilla/firefox/*.default/extensions/{91a24c60-0f27-427c-b9a6-96b71f3984a9}.xpi
echo 'user_pref("extensions.activeThemeID", "{91a24c60-0f27-427c-b9a6-96b71f3984a9}");' >> \
    ~/.mozilla/firefox/*.default/prefs.js

When I started Firefox afterwards, it notified me that the add-on “red” was installed by another program on my computer, and whether I want to enable it. When I click “Enable”, it does get enabled.

The same also happens if I don’t add the extensions.activeThemeID user pref.

So the behavior is different and it doesn’t matter whether extensions.activeThemeID is set, because Firefox asks the user whether they want to enable it.

#15 Updated by intrigeri 2019-10-06 06:37:20

Semi-random brain dump follows, use with care!

First, I’m worried that we’re approaching this problem — and a number of other Firefox-related matters — with a strategy that is unsupported by Mozilla, requires more and more hacks, and may be the root cause for more and more weird behavior (e.g. Bug #17121, Bug #17007). That might be related to our status as a downstream of a downstream vendor (Tor Browser) that already diverges from upstream (Mozilla) vs. the set of available (pre-)configuration “hooks”. It started to become apparent to me back when we had to start patching prefs inside omni.ja.

Tor Browser upstream already occupies some of the available customization “hooks”, and since years it’s doing so deeper and deeper in Firefox internals: setting default prefs in omni.ja, turning some add-ons to “system” ones that can’t be disabled by the user, and so on. On top of that, Tor Browser still ships a default profile (that currently boils down to bookmarks.html + 2 × XPIs). On top of that, we patch omni.ja in various ways and repack it with a tool that’s not meant to do that, add our own bits to the default profile, move add-ons elsewhere and symlink them from the profile (only to avoid duplicating add-ons when we create the Unsafe Browser profile), add our own add-ons (uBlock and langpacks), and finally do some more tweaks at runtime before we start the browser.

What I’m trying to say is that essentially, we’re trying to emulate a human user configuring their Firefox profile. I’m not surprised that this approach starts to break down in various places. And when I read segfault’s “was installed by another program on my computer”, it becomes obvious to me that if we try to emulate a human user, then we’ll get feedback from Firefox that’s targeted to human users, instead of behavior optimized for batch operations.

Mozilla provides other ways for “enterprise” deployments to pre-configure Firefox (and to lock down bits of this config). The supported tools to do so have changed a lot over the last few years so I’m glad we did not try one before: we would have had to switch technologies at least twice since. But it seems that the dust is settling now and ESR68 has support for policies set via policies.json, which essentially reimplements CCK2 directly in Firefox, and that apparently provides ways to force installing a given XPI:

• https://support.mozilla.org/en-US/kb/customizing-firefox-using-policiesjson
• https://github.com/mozilla/policy-templates/blob/master/README.md

Perhaps it would be worth trying this for the red theme add-on?

> With some luck (frankly I don’t know their merge policy) maybe we can get this fixed in ESR68 so we don’t have to ship a full addonStartup.json.lz4 for long.

Two things about this:

• If the bug is fixed in Firefox VCS upstream (which might require first reporting it), the Tor Browser team might be open to backport the fix for us.
• Back in the days, we started Firefox at build time (yes, seriously) to generate bits of the profile (for some reason I can’t recall). That’s definitely ugly but perhaps less so than including addonStartup.json.lz4 in our Git tree and then updating it regularly until this bug is fixed for real.

> Second, as I said on XMPP, we can probably avoid setting browser.uiCustomization.state, which is nice since it sets the whole state, so we will quickly desync if upstream changes. The other way to hide it is through userChrome.css, we just have to figure out the element name.

About this, also see “Hide the security level button in the unsafe browser (Bug #16735)” in 10-tbb, which might provide hints to do what you’re trying to do (or which might actually break stuff so perhaps it’s worth trying without it :)

#17 Updated by segfault 2019-10-06 12:36:12

intrigeri wrote:
> Mozilla provides other ways for “enterprise” deployments to pre-configure Firefox (and to lock down bits of this config). The supported tools to do so have changed a lot over the last few years so I’m glad we did not try one before: we would have had to switch technologies at least twice since. But it seems that the dust is settling now and ESR68 has support for policies set via policies.json, which essentially reimplements CCK2 directly in Firefox, and that apparently provides ways to force installing a given XPI:
>
> * https://support.mozilla.org/en-US/kb/customizing-firefox-using-policiesjson
> * https://github.com/mozilla/policy-templates/blob/master/README.md
>
> Perhaps it would be worth trying this for the red theme add-on?

I just tried this. It works flawlessly with my Firefox on sid, but not at all with Tor Browser. I found these relevant issues on trac.torproject.org:

• https://trac.torproject.org/projects/tor/ticket/29445

It was noticed that the policies.json doesn’t work in Tor Browser, because the update channel is not “release”. That was fixed in tor-browser-60.5.1esr-8.5-1.

• https://trac.torproject.org/projects/tor/ticket/29916

It was noticed that the policies.json could be used to bypass the the proxy. browser.policies.testing.disallowEnterprise was set to true by default and the fix from 29445 was reverted.

Maybe we could ask whether the fix from 29445 could be reintroduced, so that we can use the policies.json.

Just in case we need this later: Adding this to /etc/tor-browser/profile/prefs.js changed the pref value, setting it in /usr/share/tails/tor-browser-prefs.js did not:

user_pref("browser.policies.testing.disallowEnterprise", false);

#18 Updated by segfault 2019-10-06 13:14:27

> Maybe we could ask whether the fix from 29445 could be reintroduced, so that we can use the policies.json.

I did that here: https://trac.torproject.org/projects/tor/ticket/31978

#21 Updated by segfault 2019-10-06 19:51:15

anonym wrote:
> In related news, I actually managed to make a minimal addonStartup.json.lz4 that only contains the metadata for the red theme (so we don’t have to maintain it unless we update the theme!). Attached! So we actually have a pretty good workaround.

Great! And I found this script on stackexchange.com which can compress and extract Mozilla’s jsonlz4:

#!/usr/bin/env python3
from sys import stdin, stdout, argv, stderr
import os
try:
    import lz4.block as lz4
except ImportError:
    import lz4

stdin = os.fdopen(stdin.fileno(), 'rb')
stdout = os.fdopen(stdout.fileno(), 'wb')

if argv[1:] == ['-c']:
    stdout.write(b'mozLz40\0' + lz4.compress(stdin.read()))
elif argv[1:] == ['-d']:
    assert stdin.read(8) == b'mozLz40\0'
    stdout.write(lz4.decompress(stdin.read()))
else:
    stderr.write('Usage: %s -c|-d < infile > outfile\n' % argv[0])
    stderr.write('Compress or decompress Mozilla-flavor LZ4 files.\n\n')
    stderr.write('Examples:\n')
    stderr.write('\t%s -d < infile.json.mozlz4 > outfile.json\n' % argv[0])
    stderr.write('\t%s -c < infile.json > outfile.json.mozlz4\n' % argv[0])
    exit(1)

I just had to install python3-lz4 (and I changed the shebang to use python3).

We could use that to create the addonStartup.json.lz4 at build time.

#23 Updated by segfault 2019-10-06 20:39:01

anonym wrote:
> In related news, I actually managed to make a minimal addonStartup.json.lz4 that only contains the metadata for the red theme (so we don’t have to maintain it unless we update the theme!). Attached! So we actually have a pretty good workaround.

I can confirm that copying this addonStartup.json.lz4 to the Unsafe Browser’s profile directory enables the red theme. Do you prefer copying this as is to config/chroot_local-includes/usr/share/tails/chroot-browsers/unsafe-browser/addonStartup.json.lz4, or should we create it via the Python script in a build hook?

#24 Updated by intrigeri 2019-10-06 21:38:26

> I can confirm that copying this addonStartup.json.lz4 to the Unsafe Browser’s profile directory enables the red theme. Do you prefer copying this as is to config/chroot_local-includes/usr/share/tails/chroot-browsers/unsafe-browser/addonStartup.json.lz4, or should we create it via the Python script in a build hook?

If the extra cost is reasonable, creating it during the build would be quite nicer (as is: one can audit and review changes) than going back to tho (not-)good old times when we were putting all kinds of binary stuff in our Git tree. But that’s definitely not worth spending days on it IMO.

#25 Updated by segfault 2019-10-06 23:34:33

intrigeri wrote:
> > I can confirm that copying this addonStartup.json.lz4 to the Unsafe Browser’s profile directory enables the red theme. Do you prefer copying this as is to config/chroot_local-includes/usr/share/tails/chroot-browsers/unsafe-browser/addonStartup.json.lz4, or should we create it via the Python script in a build hook?
>
> If the extra cost is reasonable, creating it during the build would be quite nicer (as is: one can audit and review changes) than going back to tho (not-)good old times when we were putting all kinds of binary stuff in our Git tree. But that’s definitely not worth spending days on it IMO.

I implemented something, will test it tomorrow.

#26 Updated by anonym 2019-10-07 08:00:47

segfault wrote:
> Actually, in 10-tbb, we already set browser.uiCustomization.state during build time, and we use the value from upstream and just strip the security-level-button from it. So we don’t have to set it again in /usr/share/tails/chroot-browsers/unsafe-browser/prefs.js. I reverted that part of hefee’s commit.

Unless I am missing something you didn’t replace that solution with something else. Any way, instead of reading the default browser.uiCustomization.state from omni.ja and filtering it according to our needs, we can hide the security level button much more easily by dropping just this into the Unsafe Browser’s userChrome.css:

#security-level-button {display: none !important}

#27 Updated by segfault 2019-10-07 09:42:39

anonym wrote:
> segfault wrote:
> > Actually, in 10-tbb, we already set browser.uiCustomization.state during build time, and we use the value from upstream and just strip the security-level-button from it. So we don’t have to set it again in /usr/share/tails/chroot-browsers/unsafe-browser/prefs.js. I reverted that part of hefee’s commit.
>
> Unless I am missing something you didn’t replace that solution with something else.

I didn’t replace it with something else because the code is already there. See apply_prefs_hacks in 10-tbb.

> Any way, instead of reading the default browser.uiCustomization.state from omni.ja and filtering it according to our needs, we can hide the security level button much more easily by dropping just this into the Unsafe Browser’s userChrome.css:
> […]

OK, that does seem easier. I will try to do that during build instead of changing browser.uiCustomization.state.

#33 Updated by segfault 2019-10-08 10:31:58

intrigeri wrote:
> My code review yielded only one question: my understanding is that we need python3-lz4 only at build time so perhaps we should not ship it in the images we release? We could use ensure_hook_dependency_is_installed() to install/remove it.

Done

#34 Updated by segfault 2019-10-08 11:27:29

I noticed that we still set the unsupported app.update.enabled in /usr/share/tails/chroot-browsers/common/prefs.js.

I’m also wondering why we have these “common” preferences at all. IIUC, it is only used by the Unsafe Browser. Was there a plan to support other chroot browsers? Is that still the case? If not, could we simplify the structure by removing the unnecessary “chroot browsers” abstraction?

#35 Updated by anonym 2019-10-08 14:13:01

segfault wrote:
> I noticed that we still set the unsupported app.update.enabled in /usr/share/tails/chroot-browsers/common/prefs.js.

Ah, so we need to copy the prefs from commit:e43247dd2558dd391342855796e18c3186a43807 in that file too!

And one Unsafe Browser scenario (“The Unsafe Browser will not make any connections to the Internet which are not user initiated”) actually checks that we have app.update.enabled set to false. But I think that test seems kinda pointless and should be removed. I wonder if the “I configure the Unsafe Browser to check for updates more frequently” step needs an update, though, or if those prefs still do what they used to when this test was developed.

> I’m also wondering why we have these “common” preferences at all. IIUC, it is only used by the Unsafe Browser. Was there a plan to support other chroot browsers? Is that still the case?

There used to be an I2P Browser at one point. :)

> If not, could we simplify the structure by removing the unnecessary “chroot browsers” abstraction?

If you only refer to the files in /usr/share/tails/chroot-browsers, then sure, merge the common part into unsafe-browser. OTOH it seems we need a common part between Tor Browser and Unsafe Browser, e.g. for disabling the update… :)

#36 Updated by segfault 2019-10-08 14:34:38

anonym wrote:
> segfault wrote:
> > I noticed that we still set the unsupported app.update.enabled in /usr/share/tails/chroot-browsers/common/prefs.js.
>
> Ah, so we need to copy the prefs from commit:e43247dd2558dd391342855796e18c3186a43807 in that file too!

I don’t think so. The preferences we set in /usr/share/tails/tor-browser-prefs.js are also applied to the Unsafe Browser, because they are added at build time, in 10-tbb, to the omni.ja, and the Unsafe Browser uses the same omni.ja. Else, we wouldn’t have had Feature #17130, because we only set the update URL in tor-browser-prefs.js, but see the effect in the Unsafe Browser.

I think our whole Tor Browser / Unsafe Browser customization is too complex and not well structured - which becomes apparent by your confusion and the fact that both intrigeri (while implementing) and me (while reviewing) did not think about the consequences of the update URL for the Unsafe Browser. That’s why I would like to refactor a few things here.

> > I’m also wondering why we have these “common” preferences at all. IIUC, it is only used by the Unsafe Browser. Was there a plan to support other chroot browsers? Is that still the case?
>
> There used to be an I2P Browser at one point. :)

Ah, I see.

> > If not, could we simplify the structure by removing the unnecessary “chroot browsers” abstraction?
>
> If you only refer to the files in /usr/share/tails/chroot-browsers, then sure, merge the common part into unsafe-browser.

I would also like to remove the abstraction from chroot-browser.sh (and rename it to unsafe-browser.sh).

> OTOH it seems we need a common part between Tor Browser and Unsafe Browser, e.g. for disabling the update… :)

See my explanation above why /usr/share/tails/tor-browser-prefs.js is already shared between Tor Browser and Unsafe Browser.

#37 Updated by anonym 2019-10-08 17:36:41

@segfault, I will read your comments and review your commits tomorrow. I just pushed a few tiny fixes (and a few similar ones to come tomorrow) and wonder if you could review them(?).

#39 Updated by segfault 2019-10-08 19:18:03

segfault wrote:
> anonym wrote:
> > @segfault, I will read your comments and review your commits tomorrow. I just pushed a few tiny fixes (and a few similar ones to come tomorrow) and wonder if you could review them(?).
>
> Sure, will do.

LGTM

#40 Updated by intrigeri 2019-10-09 07:25:25

Hi,

segfault wrote:
> anonym wrote:
>> segfault wrote:
> I think our whole Tor Browser / Unsafe Browser customization is too complex and not well structured - which becomes apparent by your confusion and the fact that both intrigeri (while implementing) and me (while reviewing) did not think about the consequences of the update URL for the Unsafe Browser. That’s why I would like to refactor a few things here.

>> > I’m also wondering why we have these “common” preferences at all. IIUC, it is only used by the Unsafe Browser. Was there a plan to support other chroot browsers? Is that still the case?
>>
>> There used to be an I2P Browser at one point. :)

> Ah, I see.

>> > If not, could we simplify the structure by removing the unnecessary “chroot browsers” abstraction?
>>
>> If you only refer to the files in /usr/share/tails/chroot-browsers, then sure, merge the common part into unsafe-browser.

> I would also like to remove the abstraction from chroot-browser.sh (and rename it to unsafe-browser.sh).

See Feature #12264#note-29 for the next steps wrt. I2P. If we decide on Feature #16531 that we won’t reintroduce I2P, then yes, I’m all for simplifying all this!

#41 Updated by anonym 2019-10-09 09:33:41

segfault wrote:
> anonym wrote:
> > segfault wrote:
> > > I noticed that we still set the unsupported app.update.enabled in /usr/share/tails/chroot-browsers/common/prefs.js.
> >
> > Ah, so we need to copy the prefs from commit:e43247dd2558dd391342855796e18c3186a43807 in that file too!
>
> I don’t think so. The preferences we set in /usr/share/tails/tor-browser-prefs.js are also applied to the Unsafe Browser, because they are added at build time, in 10-tbb, to the omni.ja, and the Unsafe Browser uses the same omni.ja. Else, we wouldn’t have had Feature #17130, because we only set the update URL in tor-browser-prefs.js, but see the effect in the Unsafe Browser.

Derp! :)

> I think our whole Tor Browser / Unsafe Browser customization is too complex and not well structured - which becomes apparent by your confusion and the fact that both intrigeri (while implementing) and me (while reviewing) did not think about the consequences of the update URL for the Unsafe Browser. That’s why I would like to refactor a few things here.

Got it! But before we get rid of this abstraction, can we perhaps just improve the structure etc of the parts you find too complex?