Tails

#1 Updated by intrigeri 2015-08-02 07:22:26

• Assignee set to goupille

I expect frontdesk to:

1. check if uBlock is in Debian
2. check what’s the AdBlock Plus pref to disable “acceptable ads”, and what is its value in current Tails

#2 Updated by BitingBird 2015-08-02 13:42:26

• Affected tool set to Browser

#4 Updated by intrigeri 2015-08-03 04:12:39

• Status changed from New to Rejected

> as for the “acceptable ads” option, it is in add-ons > AdBlock > Preferences > filter preferences > Allow some non-intrusive advertising and it is set to ‘No’ (unchecked box) in Tails

Thanks, thus closing as this was the main argument in favour of replacing it with uBlock. Anyone, feel free to reopen once uBlock is in Debian and there are compelling arguments to switch.

#5 Updated by intrigeri 2016-02-03 13:42:26

• Status changed from Rejected to In Progress
• Assignee deleted (goupille)

“[Tails-dev] Migrate Tail’s browser AdBlock Plus to uBlock Origin”

#7 Updated by intrigeri 2016-02-05 11:08:10

• Subject changed from Remove AdBlock Plus and use uBlock to Replace AdBlock Plus with uBlock Origin
• Description updated
• Type of work changed from Discuss to Code

#9 Updated by BitingBird 2016-06-26 11:40:30

• Status changed from In Progress to Confirmed

#10 Updated by anonym 2016-09-15 13:22:28

• Assignee set to spriver

spriver volunteered to give this a try. \o/

#11 Updated by anonym 2016-12-14 18:54:41

• blocks Bug #11419: Deal with mandatory extension signing post FF45esr added

#12 Updated by anonym 2016-12-14 19:07:08

• Status changed from Confirmed to In Progress
• Priority changed from Low to Normal
• Target version set to Tails 2.10

It’s be great if this happened in Tails 2.10, the last major release before we get mandatory extension signing (Bug #11419) since this is a new feature, not bugfix. Still, if it cannot happen until 2.11 that’s ok — in that point release we’ll introduce the new ESR series, which is way crazier for a point release.

Also, because of this, this ticket is not low prio any more.

Also, I know spriver has made progress on this privately! :)

#13 Updated by anonym 2016-12-14 19:16:06

• Priority changed from Normal to Low
• Target version deleted (Tails 2.10)

Apparently I was wrong: installing uBlock Origins via the .deb still results in an unsigned add-on.

#14 Updated by anonym 2016-12-14 19:16:11

• blocked by deleted (Bug #11419: Deal with mandatory extension signing post FF45esr)

#15 Updated by spriver 2017-01-07 16:34:13

• Assignee changed from spriver to anonym
• Target version set to Tails 2.10
• QA Check set to Info Needed
• Feature Branch set to spriver:feature/9833-replace-adblock-with-ublock

My findings so far with uBlock:
Installation after adding a pinning for xul-ext-ublock-origin is going fine. I also modified the script installing the Tor Browser. It’s the same procedure as with adblock-plus, except that the symlink in ~/.tor-browser/profile.default/extensions/ wants to be called “uBlock0@raymondhill.net”. The addon is unsigned, though (see Bug #11419). So far, so good.

But there’s a thing that I’m having problems with solving, it’s the patterns and settings file of uBlock (~/.tor-browser/profile.default/extension-data/ublock0.sqlite). This file contains the ad blocking patterns and settings the addon is using and is a sqlite database blob. The patterns can be updated in a similar way as it’s being done while the release process of a Tails version is done: start Tails and TBB in it, update the file via the plugin, copy it into the git repo and commit it. However, we will pollute the git repo with a ~9MByte huge binary blob (containing EasyList, EasyList DE and EasyList FR).
There could be several approaches for solving the issue (apart from having the binary blob in our git, which I’d like to avoid):

• The blob’s created while building Tails. For doing this we need to include the EasyList text files and then create the database out ouf them, including the needed settings for the plugin (However, uBlock’s using a feature creating pre-compiled filter lists for faster startup and filtering: https://github.com/gorhill/uBlock/wiki/Launch-and-filter-lists-load-performance. But ublock.sqlite contains both non- and precompiled patterns, so I think that having them not pre-compiled will “only” result in a bit slower speed. I need to investigate that further.) I’m certainly not a database pro, so I likely would need some help here for building the whole blob file.
• While build time the blob will be created out of a database dump file which is stored as plain text. The dump can be saved in the repo without polluting it. This would require the RM to get the original sqlite file from inside a running Tails, then convert it into a text dump (which is a bash one-liner) and while build time it gets converted into the blob and included into the image being built. The git repo will then only include the text dump.
• Maybe there’s also the possibility that uBlock will fetch the filter lists after Tor Browser is being launched after each boot of Tails. uBlock has a feature for doing that, but my knowledge so far is that the update will only be applied every four days. The validity of EasyLists is four days, too, so maybe the update decision is being made upon the validity time of EasyLists (another thing I need to investigate). Maybe we can include a minimal sized blob file with outdated EasyLists that are getting updated by the addon itself after Tails is being booted and the browser was started.

I pushed now a first branch with a pre-installed uBlock (and having AdBlockPlus removed), which contains a sqlite database blob. This is only a temporary solution, as I’d like to avoid this.

#20 Updated by anonym 2017-01-10 12:29:15

• Assignee changed from anonym to spriver
• QA Check deleted (Info Needed)

spriver wrote:
> anonym wrote:
> > spriver wrote:
> > > The pre-compiled filter lists can be removed easily and with no problems for uBlock itself. uBlock will generate the pre-compiled filter lists after TBB will be started. So we could have a process like this:
> > >
> > > * Refresh the filter lists from a running Tails and copy the ublock0.sqlite to a working directory
> > > * Run a command e.g. echo '.dump' | sqlite3 ublock0.sqlite | grep -v "cached_asset_content://cache://compiled-" >> ublockdatabase.dump to convert the sqlite blob into a text file and remove all compiled filter lists after doing that
> >
> > It might be worth throwing in | sort as well since that will reduce the diff further. (This could break the checksum mentioned below, however, and in that case we don’t care about the less efficient diff, I guess.)
>
> The filter lists are stored in one list per line (the patterns’ values are being seperated by \n), so sort won’t have any effect there.

Ok, so then we need to insert newlines when extracting (so use sed and replace each filter separator with \n), and remove them when restoring (use sed to replace the newlines with the separator) — otherwise it won’t be much more line-diff friendly. Any way, the sorting will definitely break the checksum, so ignore that part.

> > > * Recompile the textdump into a sqlite database while build time with something similar to cat ublockdatabase.dump | sqlite3 ublock0.sqlite and move it into config/chroot_local-includes/etc/tor-browser/profile/extension-data/ublock0.sqlite
> > >
> > > I don’t know if this actually works and I’m aware that this is a sort of hack-ish workaround.
> >
> > It sounds both doable and like a good approach to me, not at all like a hack! :)
>
> How can this approach be done while building Tails? Where can I store the sqlite plaintext dump in the git repo?

We already store things like these in config/chroot_local-includes/usr/share/tails/. Just create a ublock-origin folder there. Remove that folder in config/chroot_local-hooks/98-remove_unwanted_files since it won’t be needed post-build.

> How do I ensure that sqlite3 is installed on the building system

Just apt install it in the hook (config/chroot_local-hooks) you’ll create for the filter list restoration. And apt purge it when done.

> (or is sqlite3 already used in the building process?).

At least in my last build sqlite3 was not installed (but libsqlite3-0 was).

> And how can do the actual conversion from plaintext dump while building

Like I said above, just do it in a new build hook (config/chroot_local-hooks) e.g. generate-ublock-origin-filter.

> and include the resulting blob into the folder .tor-browser/profile.default/extension-data/?

Just put the files in /etc/tor-browser/profile/extension-data/ (have a look at config/chroot_local-includes/usr/local/bin/tor-browser and config/chroot_local-includes/usr/local/lib/generate-tor-browser-profile).

> > > Generating the sqlite database itself from scratch while boot time seems to be a bit complicated as the there’ll be a checksum generated for each filter list.
> >
> > Let’s just be really careful when we extract the filters, and also extract the checksum — then when we restore the database the checksums should be ok. Right?
>
> Just creating the dump and removing lines, e.g. the pre-compiled filter patterns, is not a problem. I’ll research later today if the checksums are needed at all (in my first tests it seems like that, but maybe this was also because of something different change I made) and if the lines containing the patterns can be modified without breaking uBlock.

Cool!

> > > It’d also require some more of sqlite-voodoo I’m not capable of.
> >
> > If you need help, you need to be more specific! :)
>
>
> I’d for now try to implement uBlock with the sqlite dump variant with generating the blob from the sqlite dump while build time. I’ll frame the requirements for building the DB blob from scratch

Do whatever is simplest. I suspect that is to dump the whole database to plaintext and not keep a partial binary blob with only the settings. I suspect the only tricky thing then is to know when to replace a newline with the “filter separator” (for the splitted filter lines) and when to not (for everything else).

> while building Tails during the next days.

Note that the feature freeze for Tails 2.10 is on 2017-01-12 (noon CET) so if you’re still aiming at that release, time is short! I’m available to accelerate things if you think it’s possible.

> Maybe we should also contact the developer of uBlock Origin to find out, what is needed (e.g. the ublock0.sqlite contains fields that are unclear for me right now. For instance compiledMagic and selfieMagic which seem to be something like checksums).

Sounds like a good idea!

#21 Updated by spriver 2017-01-11 20:11:49

• Assignee changed from spriver to anonym
• QA Check set to Ready for QA

I pushed a branch implementing the steps from https://labs.riseup.net/code/issues/9833#note-20.
I needed to copy the dump file from /usr/share/tails/ublock-origin/ublock0.dump to /etc/tor-browser/profile/extension-data/ublock0.dump" for being able to create the sqlite blob (otherwise I got the error Error: unable to open database "/etc/tor-browser/profile/extension-data/": unable to open database file). The file will be deleted afterwards, but maybe there’s a bit “cleaner” solution for that.

Updating of contribute/release_process.mdwn is required accordingly to the creation of the dump file once the branch will be reviewed/merged.

#24 Updated by anonym 2017-01-12 14:11:26

• Status changed from In Progress to Fix committed
• Assignee deleted (anonym)
• % Done changed from 0 to 100
• QA Check changed from Ready for QA to Pass

Phew! We just barely made it for Tails 2.10. Awesome! :)

#26 Updated by spriver 2017-01-18 08:14:11

• related to Feature #12153: Generate ublock.sqlite patterns file from scratch during build time added

#27 Updated by anonym 2017-01-24 20:47:53

• Status changed from Fix committed to Resolved

#29 Updated by Anonymous 2017-07-10 12:42:24

• related to Bug #13446: uBlock Origin connects to its github added