Feature #8183

Ship a 64-bit (x86_64) instead of 32-bit userspace

Added by Dr_Whax 2014-10-28 22:40:10 . Updated 2016-11-26 11:43:06 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Hardware support
Target version:
Start date:
2016-10-11
Due date:
% Done:

100%

Feature Branch:
feature/8183-64bit-userspace
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Currently, Tails ship an x86 (32-bit) userland but will load an x86_64 (64-bit) kernel if the system you’re using has 64-bit support.

Supporting the “32-bit userspace on 64-bit hardware” combination has historically caused lots of trouble, both for developers and for users (e.g. Bug #11518, Bug #9969, Feature #5606). Also, software built for 64-bit processors is more interesting from a security standpoint (e.g. it’s harder to bruteforce offsets/addresses, ASLR becomes stronger in that sense as is PIE support).

So, we have a few good reasons to consider switching our userspace to 64-bit. This implies to drop support for 32-bit hardware. Is it acceptable to do that in Tails 3.0, that we will release at some point between 2017Q2 and 2018Q1?

32-bit vs. 64-bit kernel stats among WhisperBack bug reports:

32-bit % 64-bit %
2014Q2 31 15 171 85
2014Q3 53 18 244 82
2014Q4 34 13 226 86
2015Q1 30 10 243 89
2015Q2 27 15 155 85
2015Q3 36 14 213 86
2015Q4 17 7 210 92
2016Q1 32 8 349 91
2016Q2 14 6 201 93
2016Q3 18 7 215 92

Note that a good share of the 32-bit systems are virtual machines: e.g. in 2016Q1, 11 of the 32 32-bit systems were VirtualBox and VMware. It seems safe to assume that the hardware able to run Tails in a VM is most likely 64-bit, and is running a 64-bit host OS (this seems plausible given our current hardware requirements, and e.g. the VirtualBox ones are probably due to https://www.virtualbox.org/ticket/11037 that forces us to tell users to set up a 32-bit VM). So we should just ignore the 32-bit VMs when looking at these stats.

Other than those, we have (32 - 11) / (32 + 349) = 5.5% of bare metal 32-bit systems. On Feature #8183#note-29 we have analyzed these systems, and to sum up, among these 21 bare metal systems:

  • 4 supports only 64-bit CPU so will still work once we switch to full 64-bit (let’s blame syslinux CPU auto-detection) => no regression
  • 1 supports max. 512MB of RAM => is not supported currently
  • 3 unknown
  • 10 will be 10+ years old when we release Tails 3.x, and support max. 2GB of RAM => we can be that this hardware won’t last much longer
  • the 3 remaining systems are from 2009 or 2012, and support max. 2GB of RAM

=> even including the 10+ years old systems in the equation, we’re talking of dropping support for 16 (4.2%) of systems that currently report bugs about Tails.


Files

arch.sh (418 B) sajolida, 2015-04-10 11:43:12

Subtasks


Related issues

Related to Tails - Feature #5606: Add VirtualBox host software Rejected
Related to Tails - Bug #9969: Re-enable dkms modules building with Linux 4.x Resolved 2015-08-11
Related to Tails - Feature #11638: Tell 32-bit users why Tails cannot start Resolved 2016-08-15
Related to Tails - Bug #11663: Clarify the scope of hardware support Resolved 2016-08-19
Related to Tails - Feature #11734: Test the 64-bit ISO with 32-bit UEFI hardware Resolved 2016-08-26
Related to Tails - Bug #11873: "Upgrade from ISO" fails from 32-bit Tails with 64-bit ISO Resolved 2016-10-11
Related to Tails - Feature #12163: Announce in advance that 3.0 will be 64-bit only Resolved 2017-01-30
Blocks Tails - Bug #7505: Video is broken with switchable graphics Rejected 2014-07-06
Blocks Tails - Feature #11829: Switch to the Debian-packaged aufs kernel module Resolved 2016-09-23
Blocks Tails - Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org Resolved 2016-06-08

History

#1 Updated by BitingBird 2014-10-29 10:14:59

  • Tracker changed from Bug to Feature
  • Category set to Hardware support
  • Status changed from New to Confirmed

#2 Updated by intrigeri 2014-10-29 13:40:55

I’ve asked frontdesk if they want to collect stats about what kernel (32 vs. 64-bit) is used on machines that report bugs.

#3 Updated by intrigeri 2014-10-30 09:31:28

Stats for 2014 Q2-Q3:

vmlinuz % vmlinuz2 %
2014Q2 31 15 171 85
2014Q3 53 18 244 82

Note that I’ve dropped the 2014Q1 stats, as they do not mean anything wrt. 32-bit vs. 64-bit: before Tails 0.23 (March 18, 2014), vmlinuz2 was a 686-pae kernel, not a 64-bit one.

#4 Updated by intrigeri 2014-10-31 15:04:44

These statistics were generated with:

find -iname "*.eml" | while read email ; do
   echo "${email}"
   gpg --quiet --output "${email%.eml}.asc" --decrypt "${email}"
done

for folder in 2014Q{1,2,3} ; do
   echo "${folder}"
   echo "vmlinuz"
   grep --files-with-matches "BOOT_IMAGE=/live/vmlinuz " ${folder}/*.asc | wc -l
   echo "vmlinuz2"
   grep --files-with-matches "BOOT_IMAGE=/live/vmlinuz2 " ${folder}/*.asc | wc -l
done

#5 Updated by Dr_Whax 2014-11-03 22:11:58

I think that 18% of users on only X86 hardware is quite a lot and we can’t just simply drop them. There’s simply no excuse to do that.

Perhaps we should drop this for now, revisit this idea when we have reached our Tails 3.0 goals and see how much work it would be at that point in time?

#6 Updated by intrigeri 2014-11-03 22:29:17

> I think that 18% of users on only X86 hardware is quite a lot and we can’t just simply drop them. There’s simply no excuse to do that.

Agreed. My point, in starting to gather these stats now, was not to make a decision immediately. It was rather to provide data points when we come back to it, say in 6-10 months, when Tails/Jessie is almost ready and (oh, surprise) has hardware requirements that are unlikely to be satisfied by more that a third of these 32-bits systems.

I wouldn’t be surprised if the 32-bit/64-bit ratio dropped quite a bit until then anyway: hardware gets old, hardware dies. The remaining 32-bit boxes used in production around me won’t live another full year.

> Perhaps we should drop this for now, revisit this idea when we have reached our Tails 3.0 goals and see how much work it would be at that point in time?

I agree, assuming you meant 2.0.

#7 Updated by intrigeri 2015-01-01 18:41:08

  • related to Bug #8485: Consider including an x86 kernel with SMP support added

#8 Updated by intrigeri 2015-01-01 18:46:51

  • Type of work changed from Code to Research

#9 Updated by sajolida 2015-02-02 21:47:24

  • related to #8837 added

#10 Updated by intrigeri 2015-04-01 12:42:54

  • Assignee set to sajolida
  • Target version set to Tails_1.4
  • QA Check set to Info Needed

sajolida, may you please compute these stats for 2014Q4 and 2015Q1? Then, you can drop the milestone, the assignee and qa check.

#11 Updated by sajolida 2015-04-10 11:43:13

Here you go:

./2014Q4

32-bit 34 13%
64-bit 226 86%

./2015Q1

32-bit 30 10%
64-bit 243 89%

With the script in attachment.

#12 Updated by intrigeri 2015-04-14 13:10:07

OK, thanks! It seems that 32-bit is slowly losing ground.

#13 Updated by intrigeri 2015-04-14 13:10:47

  • Assignee changed from sajolida to intrigeri
  • Target version changed from Tails_1.4 to Tails_1.7
  • QA Check deleted (Info Needed)

#14 Updated by intrigeri 2015-10-05 13:15:09

  • Assignee changed from intrigeri to sajolida
  • QA Check set to Info Needed

sajolida, as promised I’m nagging you six months later: may you please run your stats script again, report back here, and reassign to me?

#15 Updated by sajolida 2015-10-06 10:45:08

Here you go:

./2015Q2

32-bit 27 15%
64-bit 155 85%

./2015Q3

32-bit 36 14%
64-bit 213 86%

#16 Updated by Anonymous 2015-10-06 14:06:47

sajolida wrote:
> Here you go:
>
> ./2015Q2
>
> | 32-bit | 27 | 15% |
> | 64-bit | 155 | 85% |
>
> ./2015Q3
>
> | 32-bit | 36 | 14% |
> | 64-bit | 213 | 86% |

woah!

#17 Updated by sajolida 2015-10-08 00:28:46

  • Assignee changed from sajolida to intrigeri
  • QA Check deleted (Info Needed)

#18 Updated by intrigeri 2015-10-12 05:48:47

  • Target version changed from Tails_1.7 to 2016

Thanks! So in a year there’s been no statistically meaningful drop of 32-bit usage. Let’s revisit 6 months after Tails/Jessie is out.

#19 Updated by sajolida 2016-02-17 12:35:17

#20 Updated by sajolida 2016-02-17 12:38:35

./2015Q4 is better:

32-bit 17 7%
64-bit 201 92%

#21 Updated by sajolida 2016-04-29 04:20:16

./2016Q1 is stable despite 2.0:

| 32-bit | 32 | 8% |
| 64-bit | 349 | 91% |

#22 Updated by sajolida 2016-04-29 04:24:02

But actually a good share of these are virtual machines :)

grep -rl "BOOT_IMAGE=/live/vmlinuz " * | while read file ; do grep -A 1 system-product-name $file ; done | grep -v system-product-name | sort | uniq -c | sort -rn

     10 VirtualBox
      3 Satellite A100
      3 MS-7728
      2 Inspiron 510m                   
      2                                 
      1 VMware Virtual Platform
      1 TravelMate C310              .     
      1 Stylistic ST5112
      1 MM061                           
      1 Hi-Fi A85S3
      1 dynabook SS S31 120S/2W
      1 Aspire 5610Z    
      1 Aspire 2020
      1 AOA110
      1 AMILO Pro Edition V3405
      1 1101HA
      1 1015CX

#23 Updated by Dr_Whax 2016-04-29 05:23:38

The question in my opinion that gets raised now is, at which % of 32-bit users are we going to say, this is low enough that we should consider shipping an X86_64 iso. Will we wait another month to see if usage of 32-bit kernels are staying the same and then move to make a decision? Since then, it’s been 8/9 months since Tails/jessie.

#24 Updated by intrigeri 2016-04-29 07:29:36

sajolida wrote:
> But actually a good share of these are virtual machines :)

I think it’s safe to assume that the hardware able to run Tails in a VM is most likely 64-bit, and is running a 64-bit host OS (this seems plausible given our current hardware requirements, and e.g. the 10 VirtualBox ones are probably due to https://www.virtualbox.org/ticket/11037 that forces us to tell users to set up a 32-bit VM). So IMO we should just ignore the 32-bit VMs from these stats.

Other than those, we have (32 - (10+1)) / (32 + 349) = 5.5% of bare metal 32-bit systems.

DrWhax wrote:
> The question in my opinion that gets raised now is, at which % of 32-bit users are we going to say, this is low enough that we should consider shipping an X86_64 iso. Will we wait another month to see if usage of 32-bit kernels are staying the same and then move to make a decision?

I think that at about 5% (given the above), we can already seriously start considering the switch to 64-bit: waiting for the long tails of 32-bit hardware to die can very well take years, and IMO we’ve already reached the acceptable limit to move on. And we already have 2 years of data, which sounds more than enough to have confidence in our estimates.

Still, IMO it would be nice to combine the switch to 64-bit with another, easier to understand, change in hardware requirements. I’m thinking e.g. of the move to Debian Stretch. I would not be ready to commit to it formally, but it would be nice to informally make the “if I get a ‘new’ box to be able to upgrade to Tails 2.0, then it’ll work for all 2.x releases and I won’t have to get another computer before 3.x” reasoning valid. Granted, that’s quite a weak argument. I’m not sure. I could also live with switching now (as in: whenever any of us has time to make it happen, which would be ~2016Q4 I guess).

#25 Updated by intrigeri 2016-06-07 12:16:51

Note that shipping both 32-bit and 64-bit kernels makes e.g. Feature #10298 much harder than it should be (Bug #9969); that’s one argument in favour of switching to 64-bit, instead of spending lots of time to support two kernels now, and then dropping all that work in 6-12 months.

#26 Updated by intrigeri 2016-06-07 12:19:47

  • related to Bug #9969: Re-enable dkms modules building with Linux 4.x added

#27 Updated by intrigeri 2016-06-08 08:30:19

  • related to Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org added

#28 Updated by intrigeri 2016-06-08 08:53:16

  • related to deleted (Bug #8485: Consider including an x86 kernel with SMP support)

#29 Updated by intrigeri 2016-06-09 07:54:46

sajolida wrote:
> But actually a good share of these are virtual machines :)

I’ve looked at the bare metal systems on that list.

Some of those are irrelevant for the discussion at hand:

  • dynabook SS S31: 2009, unknown by memory upgrade wizards, probably soldered RAM given form factor
  • Hi-Fi A85S3 and MS-7728: supports only 64-bit CPU, no idea why it’s listed there, let’s blame syslinux auto-detection
  • AOA110: 2007-2008, max. 512MB RAM => not supported

Relevant for this discussion:

  • Aspire 2020: 2004, max. 2GB RAM
  • Dell Inspiron 510m: 2004, max. 2GB RAM
  • TravelMate C310: 2005, max. 2GB RAM
  • Toshiba Satellite A100: 2006, max. 2GB RAM
  • Dell Inspiron 6400 MM061: 2006, max. 2GB RAM
  • Aspire 5610Z: 2007, max. 2GB RAM
  • Fujitsu AMILO Pro V3405: 2007, max. 2GB RAM
  • Asus Eee PC 1101HA: 2009, max. 2GB RAM
  • Eee PC 1015CX: 2012, max. 2GB RAM

My next question would be: do we actually support these systems in practice? I.e. were the bug reported about issues that prevent one from using Tails, and if yes, were we able to fix it? We’ve seen issues with old graphics cards vs. GNOME Shell (Bug #11096), that we were not able to solve.

sajolida, if it’s not too painful for you, it would be great if you could have a look at the bug report threads regarding these (relevant) systems, to give us an idea of whether they actually work on Tails currently. If you want to save time, only look at the ones that are from 2007 or later (I suspect that the 10+ years old hardware among those won’t last much longer).

#30 Updated by intrigeri 2016-07-30 10:15:09

FTR sajolida sent me some data, but he did it using a sharing service that expires stuff after a while, and sadly I didn’t look at it early enough so I could not access it. Anyway, I think we have enough info to make a decision wrt. Tails 3.0 (I could do Feature #10298 without blocking on this topic so it’s less an emergency than what I thought a couple months ago), so I’ll sum this up and will send a proposal on tails-dev@, and add the issue to the next monthly meeting’s agenda.

#31 Updated by intrigeri 2016-07-30 10:52:28

  • Subject changed from Investigate shipping an x86_64 iso to Investigate shipping an x86_64 ISO
  • Description updated

#32 Updated by intrigeri 2016-07-30 11:05:59

  • Subject changed from Investigate shipping an x86_64 ISO to Investigate shipping 64-bit (x86_64) instead of 32-bit userspace

#33 Updated by intrigeri 2016-07-30 11:10:14

  • Type of work changed from Research to Discuss

#35 Updated by intrigeri 2016-08-15 09:16:59

  • Subject changed from Investigate shipping 64-bit (x86_64) instead of 32-bit userspace to Ship a 64-bit (x86_64) instead of 32-bit userspace
  • Assignee deleted (intrigeri)
  • Priority changed from Low to Normal
  • Target version changed from 2016 to Tails_3.0
  • Type of work changed from Discuss to Code

The people attending the last monthly meeting agreed with this proposal, and at the summit we agreed too.

#36 Updated by intrigeri 2016-08-15 09:19:37

  • related to Feature #11638: Tell 32-bit users why Tails cannot start added

#37 Updated by intrigeri 2016-08-26 12:03:38

  • Feature Branch set to feature/8183-64bit-userspace

#38 Updated by intrigeri 2016-08-27 08:57:03

  • blocks Bug #7505: Video is broken with switchable graphics added

#39 Updated by intrigeri 2016-08-28 03:56:00

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri

#40 Updated by intrigeri 2016-08-28 09:54:45

http://nightly.tails.boum.org/build_Tails_ISO_feature-8183-64bit-userspace/lastSuccessful/archive/ has 64-bit ISOs now. Due to Bug #11694 one has to manually switch to VT 2 (with CTRL-ALT-F2) after clicking “Login” in the Greeter. At first glance, ISOs built from this branch work just as fine as those built from feature/stretch, but the state of the test suite on these branches is not good enough yet to be sure.

#41 Updated by sycamoreone 2016-09-03 13:19:00

  • related to Bug #11663: Clarify the scope of hardware support added

#42 Updated by intrigeri 2016-09-23 10:38:02

  • blocks Feature #11829: Switch to the Debian-packaged aufs kernel module added

#43 Updated by intrigeri 2016-11-15 10:00:14

  • related to Feature #11734: Test the 64-bit ISO with 32-bit UEFI hardware added

#44 Updated by intrigeri 2016-11-15 10:00:29

  • related to Bug #11873: "Upgrade from ISO" fails from 32-bit Tails with 64-bit ISO added

#45 Updated by intrigeri 2016-11-15 11:42:08

  • related to deleted (Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org)

#46 Updated by intrigeri 2016-11-15 11:42:21

  • blocks Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org added

#47 Updated by sajolida 2016-11-15 15:28:15

I removed the FAQ we had on 64-bit proc with 89539bb.

#48 Updated by intrigeri 2016-11-15 17:39:34

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset commit:b04913ad9bd307408d2da98ecf51679b28fe55b9.

#49 Updated by sajolida 2016-11-26 11:43:06

  • Description updated

#50 Updated by intrigeri 2017-01-30 14:46:17

  • related to Feature #12163: Announce in advance that 3.0 will be 64-bit only added