Feature #12153
Generate ublock.sqlite patterns file from scratch during build time
0%
Description
Currently we convert a raw text dump of the ublock.sqlite database (which contains settings and patterns for the uBlock Origin Addon for Tor Browser) into the sqlite database blob during build time. This requires updating of the ublock0.sqlite from a running Tails, converting it into the dump and commit it into the repository before building Tails (see contribute/release_process.mdwn). config/chroot_local-hooks/12-generate-ublock-origin-filter will convert the dump into the needed sqlite blob during build time.
From https://labs.riseup.net/code/issues/9833#note-25:
Regarding what uBO needs in the SQL database depends on what are the specific settings you wish to be different than the ones set by default.We could also decide to not ship any filter lists (in this case, this would save us work and make the repo/iso a few MBytes smaller, but apparently it takes some minutes after starting the browser until the download of filter lists will start):
Normally it should not be necessary to include the content of filter lists in the SQL file, as uBO will fetch the content of the enabled filter list(s) from their remote location(s) or the local installation (assets/thirdparties/). Some minutes after uBO is launched, it will update the enabled filter lists if needed, and their content will then be saved in the SQL databaseSubtasks
Related issues
|
Related to Tails - |
Resolved | 2015-08-01 |
History
#1 Updated by spriver 2017-01-18 08:14:11
- related to
Feature #9833: Replace AdBlock Plus with uBlock Origin added
#2 Updated by intrigeri 2017-01-18 08:32:39
> We could also decide to not ship any filter lists (in this case, this would save us
> work and make the repo/iso a few MBytes smaller, but apparently it takes some minutes
> after starting the browser until the download of filter lists will start):
>
> Normally it should not be necessary to include the content of filter lists in the SQL
> file, as uBO will fetch the content of the enabled filter list(s) from their remote
> location(s) or the local installation (assets/thirdparties/). Some minutes after uBO
> is launched, it will update the enabled filter lists if needed, and their content
> will then be saved in the SQL database
> I don’t think that’s an option for a live system like Tails: on slow or metered connections, downloading a few MiB (on top of the Tor consensus) can be a serious problem.
#3 Updated by anonym 2017-01-18 14:31:49
intrigeri wrote:
> > We could also decide to not ship any filter lists (in this case, this would save us
> > work and make the repo/iso a few MBytes smaller, but apparently it takes some minutes
> > after starting the browser until the download of filter lists will start):
>
> > […]
>
> I don’t think that’s an option for a live system like Tails: on slow or metered connections, downloading a few MiB (on top of the Tor consensus) can be a serious problem.
Fully agreed.
#4 Updated by anonym 2017-01-18 14:48:02
- Status changed from New to Confirmed
I think the implication is that we should only talk about downloading the filter list’s at build time, and not store them in Git our at all. For reproducibility, those lists must be versioned (and we encode in Git exactly which version we want), and staying online at some static URL for a long time (= as long as we’d like a particular Tails release to be reproducible). Ideally EasyLists are versioned in Git somewhere, and we just fetch a particular checkout from there at build time.
Next, regarding the whole “generate ublock.sqlite patterns file from scratch” part, I would only consider this if the implementation on our side is simple, something like a single, straight-forward sqlite command per EasyList filter file. Anything more complex than that, and I think we need to seriously reconsider this. And I fear we do; from the brief look I’ve had at the sqlite dumps, there are values that seem to indicate post-processing of the filters. However, I would not be opposed to using the same code as uBlock Origin itself, if we can manage to call it some how.
#5 Updated by anonym 2017-01-18 15:01:15
- Priority changed from Normal to Low
I’m dropping the priority, because the current situation is pretty good, so I don’t think we should put much energy on this. And the energy put into this is probably best spent on asking the uBlock Origin developer (and community?) of how to do this, and focus a lot on framing the question in a succinct, to-the-point way without irrelevant (for them) stuff about the Tails build process.
#6 Updated by gorhill 2017-01-23 13:23:25
anonym wrote:
> I think the implication is that we should only talk about downloading the filter list’s at build time, and not store them in Git our at all.
Not sure I have been clear here. uBO ships with its default filter lists — hence why it does not need to download them immediately following first launch.
These are located in ./assets/. More specifically, EasyList (and EasyPrivacy) are under ./assets/thirdparties/easylist-downloads.adblockplus.org/. So if you wanted to ship with your own version of EasyList, this is where it would go, not in the sql file.
The script make-assets.sh[1] takes care of this when uBO is build.
So to me it seems this would work this way:
- Put your own version of EasyList in ./assets/thirdparties/easylist-downloads.adblockplus.org/easylist.txt
- Once uBO is configured as wished for Tails users, remove all entries starting with “cached_asset_content://” from the sql file. What will be left is just the settings themselves.
If you do not want uBO to download automatically the filter lists, the settings “autoUpdate” should be unchecked.
The assets management code has been refactored (only dev build currently) which will simplify greatly the customization of uBO’s filter lists.
#7 Updated by intrigeri 2017-04-20 06:54:24
- Target version deleted (
Tails_2.12)
#8 Updated by Anonymous 2018-01-16 11:34:40
- QA Check deleted (
Info Needed)