Bug #12735: live/initrd.img not reproducible in some environments

Bug #12735

live/initrd.img not reproducible in some environments

Added by anonym 2017-06-19 10:12:34 . Updated 2017-10-13 08:50:15 .

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Build system

Target version:

Tails_3.3

Start date:

2017-06-19

Due date:

% Done:

100%

Feature Branch:

Type of work:

Research

Blueprint:

Starter:

Affected tool:

Deliverable for:

289

Description

See ~~Feature #12608#note-18~~ for arnaud’s diffoscope report. There are also .torrent:s for the good ISO, and for arnaud’s.

The diff is huge, so the uncompressed data likely contains several differences.

Subtasks

History

#1 Updated by anonym 2017-06-19 10:24:37

Assignee set to lamby
QA Check set to Info Needed

Could you have a look?

#2 Updated by intrigeri 2017-06-22 13:49:54

Parent task set to ~~Feature #5630~~

#3 Updated by lamby 2017-06-23 15:57:18

I think I’ll need to resolve (or it would definitely be worth resolving) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820631 to debug this now and in the future.

#4 Updated by anonym 2017-08-18 10:11:13

lamby wrote:
> I think I’ll need to resolve (or it would definitely be worth resolving) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820631 to debug this now and in the future.

Is this an (implicit) question if you can work on this diffoscope improvement and bill Tails? :P If so, how much time are we talking about?

#5 Updated by lamby 2017-08-18 17:08:14

anonym wrote:
> lamby wrote:
> > I think I’ll need to resolve (or it would definitely be worth resolving) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820631 to debug this now and in the future.
>
> Is this an (implicit) question if you can work on this diffoscope improvement and bill Tails? :P If so, how much time are we talking about?

1 day max.

#6 Updated by intrigeri 2017-08-24 13:17:21

Target version changed from Tails_3.1 to Tails_3.2

#7 Updated by intrigeri 2017-09-07 12:12:59

Assignee changed from lamby to anonym
QA Check deleted (~~Info Needed~~)

This issue was not reproduced by anyone during the 3.1 call for reproduction :)

#8 Updated by anonym 2017-09-07 12:56:16

Assignee changed from anonym to lamby

intrigeri wrote:
> This issue was not reproduced by anyone during the 3.1 call for reproduction :)

Despite this we don’t feel confident it won’t reappear later, and then we’ll regret not having a tool making investigation of this easier, so…

lamby wrote:
> anonym wrote:
> > Is this an (implicit) question if you can work on this diffoscope improvement and bill Tails? :P If so, how much time are we talking about?
>
> 1 day max.

… please go ahead! Please use any remaining time of that “1 day” to investigate this issue. If you need more time, please let us know!

Here are the initrd’s in question:

initrd.img.good: https://share.riseup.net/#ZvKc9aFEDkQ4Vdq5Qb5BwQ
initrd.img.bad: https://share.riseup.net/#_41tlqhgsXY_AWsrJDIefg

#9 Updated by lamby 2017-09-07 15:09:26

> … please go ahead!

ACK :)

#10 Updated by lamby 2017-09-09 10:20:11

lamby wrote:
> > … please go ahead!
>
> ACK :)

Done in https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1c26813019c09bf4b2e82da9668564cec58b504e

#11 Updated by intrigeri 2017-09-09 10:50:12

Status changed from Confirmed to In Progress
Assignee changed from lamby to anonym
Target version changed from Tails_3.2 to Tails_3.3
QA Check set to Ready for QA

> Done in https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1c26813019c09bf4b2e82da9668564cec58b504e

Woohoo!

anonym, perhaps you want to test it with real-world initrds and then close this ticket as resolved?

#12 Updated by anonym 2017-10-02 11:59:04

% Done changed from 0 to 70

I finally tested this through version 87 from Debian sid.

So I compared a good vs bad initrd from the 3.0~rc2 reproducibility test. The resulting diff is 124 MiB and looks like this:

--- initrd.img.good
+++ initrd.img.bad
├── 0.cpio
│┄ No file format specific differences found inside, yet data differs (ASCII cpio archive (SVR4 with no CRC))
│ @@ -878420,15 +878420,15 @@
│  00d67530: cfbf 3873 57d4 2f5b e9a4 edc9 6276 1e00  ..8sW./[....bv..
│  00d67540: 649d d9dd 1e24 cf0b 8ac7 867d 14b7 829b  d....$.....}....
│  00d67550: acf4 0299 83f0 36ed 72f6 bb1d e408 b216  ......6.r.......
│  00d67560: a92f a403 8afd a0a1 d85b 7d5d bfff 922a  ./.......[}]...*
│  00d67570: 5411 b940 7829 7deb 28d9 2308 73ef 6029  T..@x)}.(.#.s.`)
│  00d67580: 0fc4 b5d0 7068 bcd2 fa3c a8d5 8740 0a4d  ....ph...<...@.M
│  00d67590: 0b09 3ece 0448 c008 a587 8a86 b953 b900  ..>..H.......S..
│ -00d675a0: 840f 34ef fe00 7139 b350 49e3 4bc4 fd69  ..4...q9.PI.K..i
│ +00d675a0: 8417 35ef fe00 7139 b350 49e3 4bc4 fd69  ..5...q9.PI.K..i
│  00d675b0: ce12 9240 92ec caaf 1f53 be2c 3d91 23f9  ...@.....S.,=.#.
│  00d675c0: dc52 0792 1d13 bb9d 2248 e9ab a00f 3d08  .R......"H....=.
│  00d675d0: 4da2 22a3 70b1 c755 b21c 6d7b 5d97 f2f5  M.".p..U..m{]...
│  00d675e0: 0aee a6a4 1843 4b73 424c b08f f8ff 3e02  .....CKsBL....>.
│  00d675f0: 7e2c 1469 f240 cc5b e212 df14 e56b 6e94  ~,.i.@.[.....kn.
│  00d67600: 4057 e103 6ba2 0f9a 5634 5876 2300 2341  @W..k...V4Xv#.#A
│  00d67610: feec 5322 69cf 6aa8 18b1 ef50 9d9f 31cb  ..S"i.j....P..1.
│ @@ -879909,435308 +879909,427172 @@
[... binary data: tons of differences ...]
├── 6200.cpio
│┄ No file format specific differences found inside, yet data differs (ASCII cpio archive (SVR4 with no CRC))
│ @@ -876852,15 +876852,15 @@
│  00d61330: cfbf 3873 57d4 2f5b e9a4 edc9 6276 1e00  ..8sW./[....bv..
│  00d61340: 649d d9dd 1e24 cf0b 8ac7 867d 14b7 829b  d....$.....}....
│  00d61350: acf4 0299 83f0 36ed 72f6 bb1d e408 b216  ......6.r.......
│  00d61360: a92f a403 8afd a0a1 d85b 7d5d bfff 922a  ./.......[}]...*
│  00d61370: 5411 b940 7829 7deb 28d9 2308 73ef 6029  T..@x)}.(.#.s.`)
│  00d61380: 0fc4 b5d0 7068 bcd2 fa3c a8d5 8740 0a4d  ....ph...<...@.M
│  00d61390: 0b09 3ece 0448 c008 a587 8a86 b953 b900  ..>..H.......S..
│ -00d613a0: 840f 34ef fe00 7139 b350 49e3 4bc4 fd69  ..4...q9.PI.K..i
│ +00d613a0: 8417 35ef fe00 7139 b350 49e3 4bc4 fd69  ..5...q9.PI.K..i
│  00d613b0: ce12 9240 92ec caaf 1f53 be2c 3d91 23f9  ...@.....S.,=.#.
│  00d613c0: dc52 0792 1d13 bb9d 2248 e9ab a00f 3d08  .R......"H....=.
│  00d613d0: 4da2 22a3 70b1 c755 b21c 6d7b 5d97 f2f5  M.".p..U..m{]...
│  00d613e0: 0aee a6a4 1843 4b73 424c b08f f8ff 3e02  .....CKsBL....>.
│  00d613f0: 7e2c 1469 f240 cc5b e212 df14 e56b 6e94  ~,.i.@.[.....kn.
│  00d61400: 4057 e103 6ba2 0f9a 5634 5876 2300 2341  @W..k...V4Xv#.#A
│  00d61410: feec 5322 69cf 6aa8 18b1 ef50 9d9f 31cb  ..S"i.j....P..1.
│ @@ -878341,435308 +878341,427172 @@
[... binary data: tons of differences ...]
├── CF600.xz
│ ├── CF600
│ │ ├── 0.cpio
│ │ │ ├── file list
│ │ │ │ @@ -1352,17 +1352,15 @@
│ │ │ │  -rw-r--r--   1        0        0    20168 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/usb/storage/ums-sddr55.ko
│ │ │ │  -rw-r--r--   1        0        0    20584 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/usb/storage/ums-usbat.ko
│ │ │ │  -rw-r--r--   1        0        0   150648 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/usb/storage/usb-storage.ko
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/virtio
│ │ │ │  -rw-r--r--   1        0        0    17408 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/virtio/virtio.ko
│ │ │ │  -rw-r--r--   1        0        0    35408 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/virtio/virtio_pci.ko
│ │ │ │  -rw-r--r--   1        0        0    30608 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/drivers/virtio/virtio_ring.ko
│ │ │ │ -drwxr-xr-x  26        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs
│ │ │ │ -drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/aufs
│ │ │ │ --rw-r--r--   1        0        0   696496 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/aufs/aufs.ko
│ │ │ │ +drwxr-xr-x  25        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/btrfs
│ │ │ │  -rw-r--r--   1        0        0  1746104 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/fs/btrfs/btrfs.ko
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/configfs
│ │ │ │  -rw-r--r--   1        0        0    70168 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/fs/configfs/configfs.ko
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/crypto
│ │ │ │  -rw-r--r--   1        0        0    43944 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/fs/crypto/fscrypto.ko
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/fs/ext4
│ │ │ │ @@ -1445,20 +1443,20 @@
│ │ │ │  drwxr-xr-x   3        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/sound
│ │ │ │  drwxr-xr-x   3        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/sound/core
│ │ │ │  drwxr-xr-x   2        0        0        0 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/kernel/sound/core/seq
│ │ │ │  -rw-r--r--   1        0        0    14648 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/sound/core/seq/snd-seq-device.ko
│ │ │ │  -rw-r--r--   1        0        0    53136 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/sound/core/snd-rawmidi.ko
│ │ │ │  -rw-r--r--   1        0        0   132384 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/sound/core/snd.ko
│ │ │ │  -rw-r--r--   1        0        0    18904 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/kernel/sound/soundcore.ko
│ │ │ │ --rw-r--r--   1        0        0   295021 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.alias
│ │ │ │ --rw-r--r--   1        0        0   265696 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.alias.bin
│ │ │ │ +-rw-r--r--   1        0        0   295002 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.alias
│ │ │ │ +-rw-r--r--   1        0        0   265675 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.alias.bin
│ │ │ │  -rw-r--r--   1        0        0     3910 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/modules.builtin
│ │ │ │  -rw-r--r--   1        0        0     5211 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.builtin.bin
│ │ │ │ --rw-r--r--   1        0        0    53340 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.dep
│ │ │ │ --rw-r--r--   1        0        0    78966 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.dep.bin
│ │ │ │ +-rw-r--r--   1        0        0    53316 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.dep
│ │ │ │ +-rw-r--r--   1        0        0    78931 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.dep.bin
│ │ │ │  -rw-r--r--   1        0        0      195 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.devname
│ │ │ │  -rw-r--r--   1        0        0   133543 2017-05-02 15:21:44.000000 lib/modules/4.9.0-3-amd64/modules.order
│ │ │ │  -rw-r--r--   1        0        0      310 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.softdep
│ │ │ │  -rw-r--r--   1        0        0   180574 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.symbols
│ │ │ │  -rw-r--r--   1        0        0   218652 2017-05-21 18:33:20.000000 lib/modules/4.9.0-3-amd64/modules.symbols.bin
│ │ │ │  drwxr-xr-x   4        0        0        0 2017-05-21 18:33:20.000000 lib/systemd
│ │ │ │  -rw-r--r--   1        0        0  2217104 2017-04-29 19:47:47.000000 lib/systemd/libsystemd-shared-232.so
│ │ │ ├── lib/modules/4.9.0-3-amd64/modules.alias
│ │ │ │ @@ -5691,8 +5691,7 @@
│ │ │ │  alias acpi*:VMBus:* hv_vmbus
│ │ │ │  alias acpi*:VMBUS:* hv_vmbus
│ │ │ │  alias char-major-14-* soundcore
│ │ │ │  alias char-major-116-* snd
│ │ │ │  alias net-pf-16-proto-16-family-devlink devlink
│ │ │ │  alias rpc_pipefs sunrpc
│ │ │ │  alias fs-rpc_pipefs sunrpc
│ │ │ │ -alias fs-aufs aufs
│ │ │ ├── lib/modules/4.9.0-3-amd64/modules.alias.bin
│ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ -00000000: b007 f457 0002 0001 2004 0d76 3a2a 0000  ...W.... ..v:*..
│ │ │ │ +00000000: b007 f457 0002 0001 2004 0d61 3a2a 0000  ...W.... ..a:*..
│ │ │ │  00000010: 0000 0100 0001 6173 6468 6369 5f61 6370  ......asdhci_acp
│ │ │ │  00000020: 6900 3a2a 0000 0000 0100 0001 6173 6468  i.:*........asdh
│ │ │ │  00000030: 6369 5f61 6370 6900 3436 c000 000c 0000  ci_acpi.46......
│ │ │ │  00000040: 0000 c000 0022 313a 2a00 0000 0001 0000  ....."1:*.......
│ │ │ │  00000050: 0148 6932 635f 6465 7369 676e 7761 7265  .Hi2c_designware
│ │ │ │  00000060: 5f70 6c61 7466 6f72 6d00 4600 3134 2000  _platform.F.14 .
│ │ │ │  00000070: 0038 0000 0000 0000 0000 c000 0046 3243  .8...........F2C
│ │ │ │ @@ -495,16112 +495,16111 @@
[... binary data: tons of differences ...]
│ │ │ ├── lib/modules/4.9.0-3-amd64/modules.dep
│ │ │ │ @@ -605,8 +605,7 @@
│ │ │ │  kernel/net/ceph/libceph.ko: kernel/lib/libcrc32c.ko
│ │ │ │  kernel/lib/crc-ccitt.ko:
│ │ │ │  kernel/lib/crc16.ko:
│ │ │ │  kernel/lib/crc-itu-t.ko:
│ │ │ │  kernel/lib/libcrc32c.ko:
│ │ │ │  kernel/lib/raid6/raid6_pq.ko:
│ │ │ │  kernel/lib/lru_cache.ko:
│ │ │ │ -kernel/fs/aufs/aufs.ko:
│ │ │ ├── lib/modules/4.9.0-3-amd64/modules.dep.bin
│ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ -00000000: b007 f457 0002 0001 2001 3354 345f 6373  ...W.... .3T4_cs
│ │ │ │ +00000000: b007 f457 0002 0001 2001 3331 345f 6373  ...W.... .314_cs
│ │ │ │  00000010: 0000 0000 0100 0000 716b 6572 6e65 6c2f  ........qkernel/
│ │ │ │  00000020: 6472 6976 6572 732f 6e65 742f 6574 6865  drivers/net/ethe
│ │ │ │  00000030: 726e 6574 2f33 636f 6d2f 3363 3537 345f  rnet/3com/3c574_
│ │ │ │  00000040: 6373 2e6b 6f3a 206b 6572 6e65 6c2f 6472  cs.ko: kernel/dr
│ │ │ │  00000050: 6976 6572 732f 7063 6d63 6961 2f70 636d  ivers/pcmcia/pcm
│ │ │ │  00000060: 6369 612e 6b6f 206b 6572 6e65 6c2f 6472  cia.ko kernel/dr
│ │ │ │  00000070: 6976 6572 732f 7063 6d63 6961 2f70 636d  ivers/pcmcia/pcm
│ │ │ │ @@ -439,4498 +439,4496 @@
[... binary data: tons of differences ...]

So, at least one of the differences was a consequence of ~~Bug #12741~~ (our test results support the implication ~~Bug #12741~~ => ~~Bug #12735~~, so this makes sense!). I don’t really know what 0.cpio and 6200.cpio are about though, but hopefully it just yet another consequence. Any ideas?

Any way, it seems to me like we got the improvement for diffoscope we asked for, so unless someone can point out that those other changes are caused by something else than ~~Bug #12741~~, let’s close this ticket. I’ll gather the opinions of the participants of the reproducibility status meeting we have in a few minutes…

#13 Updated by lamby 2017-10-02 14:57:55

> I don’t really know what 0.cpio and 6200.cpio

This CPIO file actually contains two concatentated initrd images; 0 and 6200 are the byte-offsets into them. This is, of course, very badly named and displyed.

I have filed this in Debian as https://bugs.debian.org/877525

#14 Updated by anonym 2017-10-02 16:13:28

lamby wrote:
> > I don’t really know what 0.cpio and 6200.cpio
>
> This CPIO file actually contains two concatentated initrd images; 0 and 6200 are the byte-offsets into them. This is, of course, very badly named and displyed.
>
> I have filed this in Debian as https://bugs.debian.org/877525

Ok. Why isn’t diffoscope recursively comparing these “contained” initrds’ contents, then? I just got gigantic binary diffs for them, which I had snip out above (the two first occurrences of “[... binary data: tons of differences ...]”). If I understand things correct, this would be a bug, no?

#15 Updated by lamby 2017-10-02 17:06:10

> Why isn’t diffoscope recursively comparing these “contained” initrds’ contents then

It is, it just cannot find any “nice” differences to display.

#16 Updated by lamby 2017-10-03 16:16:13

lamby wrote:

> I have filed this in Debian as https://bugs.debian.org/877525

Fixed here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877525#10

#17 Updated by anonym 2017-10-13 08:50:15

Status changed from In Progress to Resolved
Assignee deleted (~~anonym~~)
% Done changed from 70 to 100
QA Check changed from Ready for QA to Pass

lamby wrote:
> lamby wrote:
>
> > I have filed this in Debian as https://bugs.debian.org/877525
>
> Fixed here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877525#10
>
>

Looks nicer! But since we don’t get any “nice differences” for these two other initrd:s I cannot really validate my hypothesis from ~~Bug #12735#note-12~~ that all initrd differences is just another fallout from the now fixed ~~Bug #12741~~. Any way, the hypothesis still seems reasonable, so I’m gonna assume it’s true until proven otherwise => closing this ticket!