Feature #10085

Port Tails Installer to Python 3

Added by alant 2015-08-24 06:25:29 . Updated 2020-02-07 17:37:57 .

Status:
In Progress
Priority:
Normal
Assignee:
sascha.markus@gmail.com
Category:
Installation
Target version:
Start date:
2015-08-24
Due date:
% Done:

0%

Feature Branch:
https://gitlab.com/saschamarkus/liveusb-creator/tree/feature/10085-Port-Tails-Installer-To-Python3
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Installer
Deliverable for:

Description

Liveusb-creator should be migrated to python3.

Subtasks

Related issues

Related to Tails - Feature #7036: Move custom software to our main Git repository In Progress
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by alant 2015-08-24 06:25:46

#2 Updated by alant 2015-08-24 06:28:41

  • Tracker changed from Bug to Feature

#3 Updated by intrigeri 2015-08-24 23:51:03

  • Subject changed from Migrate liveusb-creator to python3 to Port Tails Installer to Python 3
  • Category set to Installation
  • Status changed from New to Confirmed
  • Affected tool set to Installer

#4 Updated by Anonymous 2017-06-29 13:31:19

Any takers?

#5 Updated by Anonymous 2018-08-18 10:46:45

#6 Updated by Anonymous 2018-08-18 10:48:30

It seems we need to discuss the future of tails-installer WRT to our aim to distribute a USB image.
Tails installer could still be used as a tool to clone Tails. Or could we find a possibility to make this happen differently?

#7 Updated by intrigeri 2018-08-18 14:10:50

u wrote:
> Tails installer could still be used as a tool to clone Tails.

That’s the plan we came up with: keep Tails Installer but only in Tails. If this needs more discussion, please file a dedicated ticket (sorry if you did and I missed it so far :)

#8 Updated by intrigeri 2018-08-18 14:11:01

#9 Updated by sascha.markus@gmail.com 2019-04-29 08:37:38

  • Status changed from Confirmed to In Progress
  • Assignee set to sascha.markus@gmail.com

#10 Updated by sascha.markus@gmail.com 2019-04-29 14:59:32

  • Feature Branch set to https://gitlab.com/saschamarkus/liveusb-creator/tree/feature/10085-Port-Tails-Installer-To-Python3

Removed implementation for Windows

Removed implementation to download images from Fedora
-> we don’t depend on python-urlgrabber anymore

Removed implementation to read config files (there were non in tails 3.13.1)
-> we don’t depend on python-configobj anymore
-> we can remove the placeholder /etc/tails-installer/deprecated-outside-of-Tails

Executed 2to3

Replaced usage of python2’s statvfs.F_BSIZE and statvfs.F_BAVAIL

#11 Updated by sascha.markus@gmail.com 2019-05-01 21:36:04

  • Assignee changed from sascha.markus@gmail.com to kurono
  • QA Check set to Ready for QA

Hi @kurono I think I’m done with porting the installer to python3.
I wrote a couple of times Tails to an USB stick using downloaded ISOs (the current release and a nightly) and cloning the running Tails.
Seems to work fine.

The Fedore and Windows stuff is gone.

I checked the code with flake8. There are still long lines but I think it would be better to add a configuration for flake8 to the setup.py to allow 100 or 120 chars.

There was a mix of ’ and " as string delimiters. I changed all of them to ’ because I’m used to do it that way at work too. I hope that’s OK.

Cheers
Sascha

#12 Updated by intrigeri 2019-06-02 14:42:56

  • Status changed from In Progress to Needs Validation

#13 Updated by sajolida 2019-07-22 16:16:25

  • Assignee deleted (kurono)
  • Target version set to Tails_3.16

I understand that Sascha did some work on this 3 months ago (thanks a bunch!) but never got a review from kurono. I think that we should stop waiting for kurono and find other reviewer in time for the next release, no?

#14 Updated by intrigeri 2019-07-30 23:26:47

  • Target version changed from Tails_3.16 to Tails_3.17

sajolida wrote:
> I understand that Sascha did some work on this 3 months ago (thanks a bunch!) but never got a review from kurono.

I understand the same.

> I think that we should stop waiting for kurono and find other reviewer in time for the next release, no?

Agreed, modulo this is too invasive to land in a bugfix release, so re-targeting to 3.17. Still, the earlier a first review is done, the better :)

@segfault, maybe?

#15 Updated by segfault 2019-08-01 10:35:15

  • Assignee set to segfault

#16 Updated by segfault 2019-08-01 11:15:13

sascha.markusgmail.com: Could you create a request to merge your branch in to master in my gitlab repo (https://gitlab.com/segfault3/liveusb-creator)? Then I can use gitlab to review your changes.

#17 Updated by intrigeri 2019-08-07 12:01:57

  • Target version changed from Tails_3.17 to Tails_4.0

If 3.17 is released, it’ll be a bugfix release.

#18 Updated by segfault 2019-08-14 12:49:57

This is a huge change, there are 643 additions and 1516 deletions. It’s impractical to do the review here, without inline comments, so I won’t taker a deeper look at the changes without a gitlab MR.

I’m also hesitant to spend a lot of time on this review because with Feature #16906 and Feature #16907 solved, we could remove a big part of the Tails Installer code (and should refactor the remaining parts).

#19 Updated by intrigeri 2019-10-09 19:48:13

  • Target version changed from Tails_4.0 to Tails_4.1

4.0 is now frozen so I’m postponing this. Note that 4.1 is a bugfix release so such a large change might not qualify.

#20 Updated by intrigeri 2019-11-14 08:28:12

  • related to Feature #7036: Move custom software to our main Git repository added

#21 Updated by intrigeri 2019-12-01 08:21:45

  • Target version changed from Tails_4.1 to Tails_4.2

Hi!

(I’ve tried to discuss this with kurono & segfault on XMPP but the stars did not align properly. I already have too many meetings that independently try to be scheduled in December, and it’s hard to coordinate all this, so it’s not a good time for me to start yet another meeting scheduling process. Therefore, I’m falling back to async’ communication.)

segfault wrote:
> This is a huge change, there are 643 additions and 1516 deletions. It’s impractical to do the review here, without inline comments, so I won’t taker a deeper look at the changes without a gitlab MR.

I understand this. If we decide to go ahead with this review, I will set up things so that our canonical installer.git is mirrored to Salsa, and then Sascha or the reviewer can create a MR for this branch.

> I’m also hesitant to spend a lot of time on this review because with Feature #16906 and Feature #16907 solved, we could remove a big part of the Tails Installer code (and should refactor the remaining parts).

I don’t understand this: what part of the code will Feature #16906 and Feature #16907 allow us to remove? As I understand it:

  • Feature #16906 is about replacing code for ISO images with code that deals with USB images.
  • Feature #16907 is a mere rename that should affect only a few bits of the GUI and the .desktop file.

I have a few arguments in favour of proceeding with this review right away:

  • A Python 3 code base could make it easier and more pleasurable to do Feature #16906, Feature #16907, and potential follow-up code deletion & refactoring.
  • This branch was submitted 7 months ago. If I were Sascha, I might be kinda discouraged of contributing to Tails. I think we can, and should, do better :)
  • Sascha’s branch removes cruft that we don’t need anymore, which allows dropping a few dependencies.
  • Feature #16906 and Feature #16907 are somewhat low priority as sajolida thinks this will only improve things for uncommon use cases. It would feel wrong to me to see this very ticket, which is FT work and has to be done in time for 5.0, blocked by lower priority tasks.

OTOH, if we first moved the Installer to tails.git (Feature #7036), it would be easier to review & test this branch, without having to go through a awkward process (I don’t know if this doc still works; the fact it has “liveusb-creator” in its title makes me wonder if anyone used it recently). But that’s not necessarily a blocker: if it helps unlock the situation, I’m happy with doing the little bit of extra work needed to have Sascha’s changed turned into a .deb that’s installed when building a Tails image from the relevant topic branch :)

What do you think?

#22 Updated by intrigeri 2019-12-01 08:27:09

#23 Updated by segfault 2019-12-07 13:51:22

  • Assignee deleted (segfault)

intrigeri wrote:
> segfault wrote:
> > This is a huge change, there are 643 additions and 1516 deletions. It’s impractical to do the review here, without inline comments, so I won’t taker a deeper look at the changes without a gitlab MR.
>
> I understand this. If we decide to go ahead with this review, I will set up things so that our canonical installer.git is mirrored to Salsa, and then Sascha or the reviewer can create a MR for this branch.
>
> > I’m also hesitant to spend a lot of time on this review because with Feature #16906 and Feature #16907 solved, we could remove a big part of the Tails Installer code (and should refactor the remaining parts).
>
> I don’t understand this: what part of the code will Feature #16906 and Feature #16907 allow us to remove? As I understand it:
>
> * Feature #16906 is about replacing code for ISO images with code that deals with USB images.

Yes, and I think that will involve removing a lot of code from the Installer, basically everything that the create-usb-image-from-iso script does.

> I have a few arguments in favour of proceeding with this review right away:
>
> * A Python 3 code base could make it easier and more pleasurable to do Feature #16906, Feature #16907, and potential follow-up code deletion & refactoring.

Well, if I first review a lot of changes to code that I then delete afterwards, that’s not very pleasurable to me.

> * This branch was submitted 7 months ago. If I were Sascha, I might be kinda discouraged of contributing to Tails. I think we can, and should, do better :)

That I totally agree on, and I’m very sorry about it. IMO we should have made it clear on this ticket that we will do major changes to the Installer code and that we should postpone porting the code until that is done.

> * Sascha’s branch removes cruft that we don’t need anymore, which allows dropping a few dependencies.
> * Feature #16906 and Feature #16907 are somewhat low priority as sajolida thinks this will only improve things for uncommon use cases. It would feel wrong to me to see this very ticket, which is FT work and has to be done in time for 5.0, blocked by lower priority tasks.
>
> OTOH, if we first moved the Installer to tails.git (Feature #7036), it would be easier to review & test this branch, without having to go through a awkward process (I don’t know if this doc still works; the fact it has “liveusb-creator” in its title makes me wonder if anyone used it recently). But that’s not necessarily a blocker: if it helps unlock the situation, I’m happy with doing the little bit of extra work needed to have Sascha’s changed turned into a .deb that’s installed when building a Tails image from the relevant topic branch :)

Sure, having the Installer in the main repo would make it a lot easier to test the code. But I think that would also complicate merging this branch, because we would have to move both liveusb-creator.git:master and this branch to tails.git.

> What do you think?

I still think that reviewing this branch would take me a lot of time, and that the next time I have to dive that deep into the Installer code, I also want to get Feature #16906 done and refactor the code (probably quite a lot). And I don’t have time for that right now, but I hope that I can find the time for that in the next few months. I will unassign myself for now in case that anyone else wants to take over the review in the meantime.

#24 Updated by kurono 2019-12-08 12:51:18

  • Assignee set to kurono

I am really sorry that I have abandoned this review that I committed to do in the first place. I’ll do my best to complete the review.

#25 Updated by intrigeri 2019-12-15 08:21:29

>> I don’t understand this: what part of the code will Feature #16906 and Feature #16907 allow us to remove? As I understand it:
>>
>> * Feature #16906 is about replacing code for ISO images with code that deals with USB images.

> Yes, and I think that will involve removing a lot of code from the Installer, basically everything that the create-usb-image-from-iso script does.

I understand you think the partitioning & formatting code can go away via Feature #16906.

My understanding is that after Feature #16906 we’ll support 4 operation modes:

  • upgrade from USB image or by cloning → no need to partition/format (unchanged)
  • install from USB image → no need to partition/format anymore (this is new and I guess that’s what you had in mind)
  • install by cloning → it depends on the actual implementation; I can imagine ways in which we can avoid partitioning/formatting entirely, but I doubt it’s trivial and it could make UX worse; the cheapest way to get Feature #16906 done probably requires keeping the partitioning/formatting code around

Are we in agreement?

> IMO we should have made it clear on this ticket that we will do major changes to the Installer code and that we should postpone porting the code until that is done.

ACK, this is a good point. Let’s try to keep this in mind: when we make new plans, we should check if they invalidate or block pre-existing plans.

> I still think that reviewing this branch would take me a lot of time, and that the next time I have to dive that deep into the Installer code, I also want to get Feature #16906 done and refactor the code (probably quite a lot). And I don’t have time for that right now, but I hope that I can find the time for that in the next few months. I will unassign myself for now in case that anyone else wants to take over the review in the meantime.

Fair enough, fully understood!

#26 Updated by intrigeri 2019-12-15 08:21:37

> I’ll do my best to complete the review.

Thanks a lot, kurono!

#27 Updated by segfault 2019-12-15 11:29:31

intrigeri wrote:
> >> I don’t understand this: what part of the code will Feature #16906 and Feature #16907 allow us to remove? As I understand it:
> >>
> >> * Feature #16906 is about replacing code for ISO images with code that deals with USB images.
>
> > Yes, and I think that will involve removing a lot of code from the Installer, basically everything that the create-usb-image-from-iso script does.
>
> I understand you think the partitioning & formatting code can go away via Feature #16906.

Yes, and also the installation of the bootloader.

>
> My understanding is that after Feature #16906 we’ll support 4 operation modes:
>
> * upgrade from USB image or by cloning → no need to partition/format (unchanged)
> * install from USB image → no need to partition/format anymore (this is new and I guess that’s what you had in mind)
> * install by cloning → it depends on the actual implementation; I can imagine ways in which we can avoid partitioning/formatting entirely, but I doubt it’s trivial and it could make UX worse; the cheapest way to get Feature #16906 done probably requires keeping the partitioning/formatting code around

OK, yes, maybe it’s cheaper to handle this case with partitioning/formatting code in the Installer, but I would definitely spend some thought on this to possibly get rid of that code.

> > IMO we should have made it clear on this ticket that we will do major changes to the Installer code and that we should postpone porting the code until that is done.
>
> ACK, this is a good point. Let’s try to keep this in mind: when we make new plans, we should check if they invalidate or block pre-existing plans.

ACK.

> > I still think that reviewing this branch would take me a lot of time, and that the next time I have to dive that deep into the Installer code, I also want to get Feature #16906 done and refactor the code (probably quite a lot). And I don’t have time for that right now, but I hope that I can find the time for that in the next few months. I will unassign myself for now in case that anyone else wants to take over the review in the meantime.
>
> Fair enough, fully understood!

:)

#28 Updated by intrigeri 2020-01-06 07:58:28

  • Target version changed from Tails_4.2 to Tails_4.3

#29 Updated by kurono 2020-01-31 00:10:02

  • Status changed from Needs Validation to In Progress
  • Assignee changed from kurono to sascha.markus@gmail.com

kurono wrote:
> I am really sorry that I have abandoned this review that I committed to do in the first place. I’ll do my best to complete the review.

I had a little of trouble on configuring the alternative test suite on nested virtual machines. Finally I could test the Sascha’s changes and I have found a first problem. When “Installing Tails to a used USB drive” I get this error: “No such file or directory /sbin/sgdisk —print /dev/sda”. Maybe the popen is taken the command string as a file instead of a command with parameters.

#30 Updated by intrigeri 2020-02-07 17:37:58

  • Target version deleted (Tails_4.3)