Feature #7427

Evaluate using aptly

Added by intrigeri 2014-06-21 13:20:08 . Updated 2015-10-22 09:15:07 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Tails_1.7
Start date:
2014-06-21
Due date:
% Done:

100%

Feature Branch:
Type of work:
Test
Blueprint:

Starter:
0
Affected tool:
Deliverable for:
269

Description

Using aptly for managing the packages imported from foreign APT repository may be a good alternative to reprepro, if Feature #6295 doesn’t give good results, and if the grml’s solution is not suitable.

Subtasks

History

#1 Updated by intrigeri 2014-06-21 13:20:35

  • Tracker changed from Bug to Feature

#2 Updated by intrigeri 2014-06-21 13:20:57

  • blocked by Feature #6906: Ask the Grml people how they handle the "keep lots of source packages around" problem added

#3 Updated by intrigeri 2014-06-21 13:21:15

  • blocked by Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro added

#4 Updated by intrigeri 2014-09-24 14:23:49

aptly is now in Debian sid.

#5 Updated by intrigeri 2015-05-28 15:27:03

  • blocks deleted (Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro)

#6 Updated by intrigeri 2015-05-28 15:27:07

  • blocks deleted (Feature #6906: Ask the Grml people how they handle the "keep lots of source packages around" problem)

#7 Updated by intrigeri 2015-05-28 15:27:41

  • Assignee set to intrigeri
  • Target version changed from Sustainability_M1 to Tails_2.3
  • Parent task set to Feature #9487

aptly 0.8 is in Jessie.

#8 Updated by intrigeri 2015-05-28 15:27:57

  • blocks #8668 added

#9 Updated by intrigeri 2015-05-28 15:28:15

  • blocked by Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro added

#10 Updated by intrigeri 2015-05-28 15:29:02

  • blocked by Feature #6906: Ask the Grml people how they handle the "keep lots of source packages around" problem added

#11 Updated by intrigeri 2015-05-28 15:46:46

  • Target version changed from Tails_2.3 to 246

#12 Updated by intrigeri 2015-05-30 17:42:06

The aptly feature set, semantics and user interface looks very close to what (I believe) we need:

  • aptly mirror create mirrors a remote repo; one can select distributions, components, and architectures; and then aptly mirror update actually downloads stuff. Note that:
    • Apparently the filter feature doesn’t support reading a file for input, so it’s not suitable for listing lots of packages => need to mirror entire distributions (which should be simpler anyway, by avoiding having to deal e.g. with topic branches pulling in more packages).
    • By default, aptly merges components into a single one (do we want to do that?), but it can also do multi-component publishing
    • So far, aptly always re-signs mirrored indices. Re-publishing them as-is, with the original signature, is part of upstream’s plans. Now, of course this won’t be compatible with apt snapshot pull or any other operation that modifies a snapshot.
  • aptly snapshot create creates a named snapshot from the current state of a mirror or local repo; the snapshot can then be published with aptly publish snapshot
  • aptly snapshot pull allows us to pull selected packages (optionally, with their dependencies) from a mirror into an existing (immutable) snapshot, creating a new snapshot as a result; and then aptly publish switch updates a published repo to a new snapshot. The combination of those is probably what we need in order to pull selected package upgrades from Debian during a freeze.

=> even if reprepro would work as well, I must say that aptly is very tempting. In particular, separating our own APT repo from mirrors/snapshots of remote repos is appealing: it decreases chances that the one affects the other (e.g. automatic cleanup operations).

Also:

  • aptly generates a directory structure that can be served as-is with any web server.
  • aptly task can chain commands into a single one, which can be useful when writing wrappers. It’s said to be experimental, though.
  • There’s an API and a CLI tool to interact with it
  • There’s at least one Puppet module for the initial setup and very basic functionality (mirror and local repo creation). It can also manage the aptly API’s web service, but currently only supports upstream.

#13 Updated by intrigeri 2015-05-30 17:42:20

  • blocks deleted (Feature #6295: Evaluate consequences of importing large amounts of packages into reprepro)

#14 Updated by intrigeri 2015-05-30 17:42:23

  • blocks deleted (Feature #6906: Ask the Grml people how they handle the "keep lots of source packages around" problem)

#15 Updated by intrigeri 2015-05-30 18:36:06

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Type of work changed from Sysadmin to Test

Next step is to actually test the typical workflow we would use.

#16 Updated by intrigeri 2015-08-26 06:04:26

  • Deliverable for set to 269

#17 Updated by intrigeri 2015-10-22 09:15:08

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • Target version changed from 246 to Tails_1.7
  • % Done changed from 10 to 100

We’ve quickly hit one problem that reflects what I see as a blocker (lack of maturity): the generated Release files have no Valid-Until field. The list of bugs, including ones that cause data loss, we’ve seen on GitHub, is a bit scary. We’ve decided to stick to reprepro for now. Too bad, the interface looked very nice.