Tails

#4 Updated by segfault 2017-12-12 15:51:58

intrigeri wrote:
> > * Be based on the development version of these packages. For example, the dialog to format a partition in Disks has been rewritten since the version in Debian Stretch.

We don’t have to base our work on the development versions at all cost. I think the new Disks formatting dialog is the only change I encountered yet(!) which affects our work. For the dialog we could create patches for both the old and new design (given that we do this work at all, because it is part of the optional Feature #15050).

> > * As a consequence, not be easy to backport in Debian Stretch.
>
> It seems that these two bullet points, and the costs of whatever strategy we’ll pick here, greatly impact the cost (Feature #14473) / needs (Feature #14474) analysis that leads to deciding what set of features we can implement with the allocated budget. My understanding is that this analysis was done last week, so from what you write I understand that despite the bonus work it requires, creating a VeraCrypt volume is included in that set of features. If there’s still doubt on this I would recommend factoring in the apparently unexpected bonus costs (e.g. backporting features for Stretch) into this analysis.
>
> > So we could take different approaches to have our changes land in Tails:
>
> > * A. Create backports our changes to make them work in Debian Stretch and maintain these backports during the Stretch cycle. But I don’t see this being considered in the budget.
> > * B. Build custom packages of our changes to have them in Tails only and maintain these custom packages during the Stretch cycle.
>
> If critical low-level foundations of the GNOME stack such as GTK+ and gvfs really have to be involved,

Yeah, and you can add GNOME Shell to this list. GTK+ is needed at least for Feature #15036, and GVfs and GNOME Shell for Feature #15044.

> I don’t think it’s realistic to backport the corresponding stacks to Stretch and maintain it (be it as a Tails-only backport or as an official backport) for the next 1.5+ years. For example, some apps from Stretch may not be compatible with a newer GTK+ API, some may need to be rebuilt against the newer GTK+ (in case of ABI incompatibility), etc. In other words, the GNOME stack is meant to be upgraded all at once. I’m convinced that trying to run GNOME 3.22 on top of a newer GTK+ will quickly become a maintenance nightmare.

Right, this was also my concern.

> So I think that cherry-picking only the new bits we’ve upstreamed & need on top of the Stretch packages is the only sensible way to go. The resulting packages would be custom to Tails, until we upgrade to Buster. (I’ve always assumed this was the plan segfault and anonym had in mind: the way I see it, once we’ve switched to an upstream first strategy, the budget we have for “Porting to Debian 10 (Buster)” kinda automatically translates to “Backporting to Debian Stretch”.)

Yes, it was also my understanding that the plan was to backport our changes to Stretch, but I did not think about backporting other changes we might require. I don’t know yet how much work this will be, but given how many packages are affected, I would prefer it if we found another solution.

> > * C. Build custom packages and only use them in a beta version for user testing and community feedback. Then wait until Tails is based on Debian testing or Stretch to have them in Tails. That would change our plans regarding the delivery of these changes in our official version.
>
> I’ll assume you mean “based on Debian Buster (either while it’s testing or once it’s stable)” and not Stretch here.
>
> I see a few major problems with delaying the official release of these new features in Tails:
>
> * I expect we’ll get much more feedback once the new features are in an official Tails release than about a beta version. I’m worried that if we delay the official release, we’ll get feedback too late, and then it may be hard or impossible to get fixes implemented, merged upstream, and included in time for the Debian Buster release. I suspect that as a result:

I think we will already get a lot of feedback with the GNOME release after we upstreamed our work, since Debian testing and basically all other distros using GNOME will already ship this, right? So if we get our work in GNOME 3.30, we will have from early September 2018 until the Buster freeze deadline to apply fixes. Based on the freeze deadlines of Jessie and Stretch, I expect that this will not be before December 2018.

> We will likely have to maintain custom packages for Tails/Buster with some bugfixes that we did not manage to sneak into Debian stable. So this seems like a dangerous bet i.e. instead of doing the “maintain custom packages for a while” work now (while it’s budgeted and we know who’s responsible), we’re postponing to “in 1-2 years” (that is a time when it’s unclear who’ll do the work), in the hope that it becomes much cheaper in 1-2 years than it is now. Granted, it will very likely be cheaper, because backporting a few bugfixes is easier than backporting an entirely new feature-set, but in general we’re not very good at dealing with this kind of stuff after a contract is over and I’d rather not take the risk. What is done, is done, what is postponed, well, you know.
> Whatever we’ll have merged upstream in GNOME will be of worse quality than something we would dog-food in Tails official releases. This feels like a risky bet wrt. our relationship with GNOME developers. It’s the first time we want to deliver such a big new thing and upstream it in GNOME, I’d rather see us optimize for quality and thus acceptance.

But we would also upstream our fixes to GNOME, it would only be the packages in Debian Buster which would have worse quality.

> * We’ll be able to invoice “C.1 Major release including objective A” only once it’s done, which might cause budget forecasting problems for our Core work.

Ok, that could be bad, I don’t know about how much money that is and what our budget for 2018 currently looks like, so I’ll let you decide on whether we could afford this.

> … so I’d rather not do that, unless we switch Tails stable releases to Debian testing no later than 2018Q4.
>
> > having Tails based on Debian testing would solve this
>
> Right, it would solve this if all the following conditions are met:
>
> * Everything has been upstreamed in time to be included in GNOME 3.30 that will be released early September 2018 (I don’t know when they freeze the code before the release BTW); that’s the plan according to Redmine.
> * Everything we need has been uploaded to sid, and has migrated to testing, by the time we freeze Tails 3.10 (probably around 2018-10-09); https://www.0d.be/debian/debian-gnome-3.26-status.html and the Debian PTS show that big parts of the GNOME 3.26 stack were uploaded to sid, and migrated to testing, very fast, but in the past it has sometimes taken much longer. Now that Ubuntu is using GNOME again, I’m quite confident this work will be done quickly.
> * Tails 3.10 (or whatever it’s called), scheduled for 2018-10-23, is based on Debian testing; that’ll be only 3 months before the Buster freeze starts so it doesn’t seem totally crazy :)
>
> I propose segfault initially tries option B (cherry-picking only the new bits we need and maintaining custom packages for Tails/Stretch), which we’ll need anyway for the automated testing we are committed to implement (we have no functional Tails/Buster at this point). We stay in touch, segfault keeps me updated, and whenever it starts to look like backporting requires too much work and does not fit into the budget we had for forward-porting (and can re-allocate to backporting), we discuss this again, see what the Tails/Buster schedule plans are and whether it’s feasible to speed them up in order to ease segfault’s job.

Sounds like a reasonable plan to me.

#5 Updated by anonym 2017-12-13 16:03:46

intrigeri wrote:
> Hi!
>
> Meta: I’ve added anonym as a watcher. The questions I’m being asked here are somewhere in a grey area between his mentoring/rubber-duck job and my higher-lever manager job. I’m fine with providing my own input but I really want anonym to be in the loop: if he cannot easily follow such implementation/release strategy discussions, I’m worried he cannot do his mentoring/rubber-duck job which would be a problem.

Thanks! I would have been sad to miss this discussion!

> > Our work on VeraCrypt will likely:
>
> > * Affect many Debian packages (udisks, gnome-disk-utility, nautilus, gtk, libgtk, gvfs, etc.)
>
> Wow. I’m not following closely the implementation details so I had no idea (or I conveniently forgot) about GTK+ and gvfs.

And libgtk is a surprise for me. :)

> > * Be based on the development version of these packages. For example, the dialog to format a partition in Disks has been rewritten since the version in Debian Stretch.
> > * As a consequence, not be easy to backport in Debian Stretch.
>
> It seems that these two bullet points, and the costs of whatever strategy we’ll pick here, greatly impact the cost (Feature #14473) / needs (Feature #14474) analysis that leads to deciding what set of features we can implement with the allocated budget. My understanding is that this analysis was done last week, so from what you write I understand that despite the bonus work it requires, creating a VeraCrypt volume is included in that set of features. If there’s still doubt on this I would recommend factoring in the apparently unexpected bonus costs (e.g. backporting features for Stretch) into this analysis.

Agreed. Alternative: do it as volunteer work, while trying to be very mentally conscious about it so it doesn’t infringe on the paid work.

> > So we could take different approaches to have our changes land in Tails:
>
> > * A. Create backports our changes to make them work in Debian Stretch and maintain these backports during the Stretch cycle. But I don’t see this being considered in the budget.
> > * B. Build custom packages of our changes to have them in Tails only and maintain these custom packages during the Stretch cycle.
>
> If critical low-level foundations of the GNOME stack such as GTK+ and gvfs really have to be involved, I don’t think it’s realistic to backport the corresponding stacks to Stretch and maintain it (be it as a Tails-only backport or as an official backport) for the next 1.5+ years. For example, some apps from Stretch may not be compatible with a newer GTK+ API, some may need to be rebuilt against the newer GTK+ (in case of ABI incompatibility), etc. In other words, the GNOME stack is meant to be upgraded all at once. I’m convinced that trying to run GNOME 3.22 on top of a newer GTK+ will quickly become a maintenance nightmare.
>
> So I think that cherry-picking only the new bits we’ve upstreamed & need on top of the Stretch packages is the only sensible way to go. The resulting packages would be custom to Tails, until we upgrade to Buster. (I’ve always assumed this was the plan segfault and anonym had in mind: the way I see it, once we’ve switched to an upstream first strategy, the budget we have for “Porting to Debian 10 (Buster)” kinda automatically translates to “Backporting to Debian Stretch”.)

It was my impression that we (during Tails Summit 2017) already had outlined a vague plan that resembles B: in summit-2017/notes/OTF.mdwn I see that one of the (last) activities is “Port to the latest version of GNOME”, so it seems we thought of an approach where we first get it working in the version of GNOME used in Tails, and then forward-port it when it’s time for upstreaming. However, given how important it is this is upstreamed in time (for GNOME 3.30) I think upstream-first is the right approach this time.

> > * C. Build custom packages and only use them in a beta version for user testing and community feedback. Then wait until Tails is based on Debian testing or Stretch to have them in Tails. That would change our plans regarding the delivery of these changes in our official version.
>
> I see a few major problems with delaying the official release of these new features in Tails:
>
> […]
>
> … so I’d rather not do that, unless we switch Tails stable releases to Debian testing no later than 2018Q4.

I only remember us talking about this option as a fallback emergency plan if the delta of Stretch’s GNOME vs upstream GNOME is so big that backporting the VC support to Stretch doesn’t fit our budget. So let’s not plan for this, but keep it as a last option.

> > having Tails based on Debian testing would solve this
>
> Right, it would solve this if all the following conditions are met:
>
> * Everything has been upstreamed in time to be included in GNOME 3.30 that will be released early September 2018 (I don’t know when they freeze the code before the release BTW); that’s the plan according to Redmine.

Thanks for making this clear! It seems that the GNOME 3.30 feature freeze is an important deadline for you, segfault: no matter what, at some point Tails will be based on buster, so no matter the plan this deadline is valid. I urge you to ask for clarifications from the gnomes ASAP!

> I propose segfault initially tries option B (cherry-picking only the new bits we need and maintaining custom packages for Tails/Stretch), which we’ll need anyway for the automated testing we are committed to implement (we have no functional Tails/Buster at this point). We stay in touch, segfault keeps me updated, and whenever it starts to look like backporting requires too much work and does not fit into the budget we had for forward-porting (and can re-allocate to backporting), we discuss this again, see what the Tails/Buster schedule plans are and whether it’s feasible to speed them up in order to ease segfault’s job.
>
> anonym, what do you think?

I like it, and it 100% fits in the plan I thought we were following! :)

#7 Updated by segfault 2017-12-13 18:28:47

anonym wrote:
> segfault wrote:
> > intrigeri wrote:
> > > > * Be based on the development version of these packages. For example, the dialog to format a partition in Disks has been rewritten since the version in Debian Stretch.
> >
> > We don’t have to base our work on the development versions at all cost.
>
> I fear that if we do it the other way, then we risk not having upstream happy to accept your changes for GNOME 3.30. When I think about the costs of missing that deadline, well, then “at all cost” actually makes sense.

Of course we will also have to apply our changes to current development version of GNOME. What I meant is that we don’t have to base the version we ship in Tails on the development version (and backport it), but instead on the version in Stretch. Sorry for being unclear.

> > I think the new Disks formatting dialog is the only change I encountered yet(!) which affects our work.
>
> That sounds promising! But let’s not count on it being the only thing, e.g. there could be big under-the-hood refactorings or similar that won’t know until you see the code.

Full ack.

> > > So I think that cherry-picking only the new bits we’ve upstreamed & need on top of the Stretch packages is the only sensible way to go. The resulting packages would be custom to Tails, until we upgrade to Buster. (I’ve always assumed this was the plan segfault and anonym had in mind: the way I see it, once we’ve switched to an upstream first strategy, the budget we have for “Porting to Debian 10 (Buster)” kinda automatically translates to “Backporting to Debian Stretch”.)
> >
> > Yes, it was also my understanding that the plan was to backport our changes to Stretch, but I did not think about backporting other changes we might require.
>
> I don’t understand: what “other” (as opposed to “our”) changes are you talking about? Please clarify!

I mean changes between GNOME in Stretch and GNOME 3.30, which affect our work, for example the new format dialog.

#8 Updated by intrigeri 2017-12-13 19:46:37

> I think we will already get a lot of feedback with the GNOME release after we upstreamed our work, since Debian testing and basically all other distros using GNOME will already ship this, right?

I hope this shiny new feature is as popular as you’re expecting among the kind of people who run bleeding-edge distros. Otherwise, I don’t think it’ll be “a lot of feedback”.

>> I propose segfault initially tries option B (cherry-picking only the new bits we need and maintaining custom packages for Tails/Stretch), […]

> Sounds like a reasonable plan to me.

:)

#9 Updated by intrigeri 2017-12-13 19:50:17

> Thanks for making this clear! It seems that the GNOME 3.30 feature freeze is an important deadline for you, segfault: no matter what, at some point Tails will be based on buster, so no matter the plan this deadline is valid.

Right. And perhaps even more importantly, that’s what we have to do in order to get the vast majority of this work paid: upstreaming is part of our deliverables, and GNOME 3.30 is our last chance within the contract.

#12 Updated by intrigeri 2018-01-25 15:35:15

Goal for the 3.6 cycle: re-read this discussion, identify what’s left to be done on this ticket and when it shall be done. Then proceed accordingly, e.g. postpone to the time when the next action must be taken, or finish whatever needs to be done before we can close this ticket.