Feature #8722

Create MIME/Multipart message with attachments in WhisperBack

Added by alant 2015-01-18 18:38:21 . Updated 2019-10-09 19:49:51 .

Status:
In Progress
Priority:
Normal
Assignee:
sascha.markus@gmail.com
Category:
Target version:
Start date:
2015-01-18
Due date:
% Done:

0%

Feature Branch:
https://gitlab.com/saschamarkus/whisperback/tree/feature/8722-Create-MIME-Multipart-attachments
Type of work:
Code
Blueprint:

Starter:
Affected tool:
WhisperBack
Deliverable for:

Description

Files

test_8722_attached.eml (2133699 B) sascha.markus@gmail.com, 2019-02-04 20:22:25
test_8722_inline.eml (2187205 B) sascha.markus@gmail.com, 2019-02-04 20:22:25
Screenshot_View_attachments_inline.png (285929 B) sascha.markus@gmail.com, 2019-02-25 19:00:34
0001-Attach-debug-info-as-individual-files.patch (5935 B) sascha.markus@gmail.com, 2019-04-27 08:26:35

Subtasks

Related issues

Blocked by Tails - Feature #8514: Replace WhisperBack.mail_appended_info with a dictionary Resolved 2015-01-02

History

#1 Updated by alant 2015-01-18 18:39:27

  • blocked by Feature #8514: Replace WhisperBack.mail_appended_info with a dictionary added

#2 Updated by intrigeri 2015-01-18 19:12:44

  • Subject changed from Create MIME/Multipart message with attachements to Create MIME/Multipart message with attachements in WhisperBack
  • Status changed from New to Confirmed

#3 Updated by sascha.markus@gmail.com 2018-06-18 09:36:44

  • Subject changed from Create MIME/Multipart message with attachements in WhisperBack to Create MIME/Multipart message with attachments in WhisperBack
  • Assignee changed from alant to intrigeri
  • QA Check set to Info Needed

I could start to work on this ticket soon. But I have some questions:

1) I assume the mail body will be text text the user enters in the first tab, right?

2) We changed the append_info to create single attachments.
Should the prepended data still be part of the body or also turned into an attachment?

3) How should the names of the attachments look like?
For command output like
/usr/sbin/dmidecode’, ‘-s’, ’system-manufacturer
I’d suggest
usr_sbin_dmidecode_-s_system-manufacturer.txt
and for paths like
/live/persistence/TailsData_unlocked/persistence.conf
or
/var/lib/live/config/tails.physical_security
something like
_live_persistence_TailsData_unlocked_persistence.conf.txt

4) The attachment could contain the header currently in the text output of command xyz =
or just contain the ‘real’ content.

#4 Updated by intrigeri 2018-06-19 16:03:33

  • Target version set to Tails_3.9

#5 Updated by sascha.markus@gmail.com 2018-06-20 10:02:58

I sent a mail to the helpdesk two days ago with the following implementation:

The body of the mail looks like before. I didn’t remove the appended info yet to have the opportunity to check the attachments contents. If the attachments are fine

The attachments are added without giving an encoding. MIMEText sets the correct encoding on it’s own.

The attachments all start with the same header as before.

For the filename all ‘/’ and blanks are replaced with an underscore and ‘.txt’ is added.

In the mail I asked to comment on the ticket to have feedback if everything worked as expected.
But I’m not sure is the mail bypassed spam filters because of the unexpected attachments.

#6 Updated by intrigeri 2018-06-28 16:08:14

  • Assignee changed from intrigeri to sascha.markus@gmail.com
  • QA Check changed from Info Needed to Dev Needed

> 1) I assume the mail body will be text text the user enters in the first tab, right?

Yes.

> 2) We changed the append_info to create single attachments.
> Should the prepended data still be part of the body or also turned into an attachment?

I think the prepended data should remain on top of the body.
And by the way, at some point we’ll want to make it machine-readable.

> 3) How should the names of the attachments look like?
> For command output like
> /usr/sbin/dmidecode’, ‘-s’, ’system-manufacturer
> I’d suggest
> usr_sbin_dmidecode_-s_system-manufacturer.txt
> and for paths like
> /live/persistence/TailsData_unlocked/persistence.conf
> or
> /var/lib/live/config/tails.physical_security
> something like
> _live_persistence_TailsData_unlocked_persistence.conf.txt

Looks good.

> 4) The attachment could contain the header currently in the text output of command xyz =
> or just contain the ‘real’ content.

I would include the header.

#7 Updated by intrigeri 2018-06-28 16:11:02

Also, I’m not sure what kind of attachments we want: inline vs. “real” attachments. I’m not sure how Thunderbird implements support for “inline” attachments. Ideally one should be able to read all the attachements without doing special actions, and to read one specific attachment by double-clicking on it. If this feasible?

#8 Updated by goupille 2018-06-28 16:34:47

I forgot your report, I’m really sorry about that…

so we did received a full whisperback report with, attached, the logs splitted in 22 files.

note that before Tails 3.8 it was really painfull to open an email like that in thunderbird.

#9 Updated by sascha.markus@gmail.com 2018-07-03 20:19:40

intrigeri wrote:
> Also, I’m not sure what kind of attachments we want: inline vs. “real” attachments. I’m not sure how Thunderbird implements support for “inline” attachments. Ideally one should be able to read all the attachements without doing special actions, and to read one specific attachment by double-clicking on it. If this feasible?

I sent a second mail with inline attachments instead of “real” ones. I hope goupille can tell us if this is better to handle.

#10 Updated by sascha.markus@gmail.com 2018-07-03 20:21:23

goupille wrote:
> I forgot your report, I’m really sorry about that…
>
> so we did received a full whisperback report with, attached, the logs splitted in 22 files.
>
> note that before Tails 3.8 it was really painfull to open an email like that in thunderbird.
I sent a second mail with inline attachments. Please check if this is easier to use.

#11 Updated by goupille 2018-07-06 17:33:49

when opening your second report in thunderbird, it looks just like a usual one with no attachments, and if I enable View>Display Attachments Inline, the attachments are just pasted after the email body and it is not possible to see only one by “double clicking on it” (at least, I don’t know how).

Since thunderbird is OK with opening emails with heavy attachments now, I’d say that your first try seems to be a better answer to the concerns intrigeri raised in #7

#12 Updated by sascha.markus@gmail.com 2018-07-16 23:55:01

  • File <del>missing: 0001-Feature-8722-Create-MIME-Multipart-message-with-atta.patch</del> added
  • Assignee changed from sascha.markus@gmail.com to intrigeri
  • QA Check deleted (Dev Needed)

#13 Updated by sascha.markus@gmail.com 2018-07-19 20:46:47

  • File deleted (0001-Feature-8722-Create-MIME-Multipart-message-with-atta.patch)

#14 Updated by sascha.markus@gmail.com 2018-07-19 20:48:04

Here is a branch on top of your branch:
https://github.com/saschamarkus/whisperback/commits/feature/8722-Create-MIME-Multipart-message-with-attachments

it deserializes the json like before. But instead of just creating the text it creates a list of MIMETexts and adds them as attachment like in #8
If the output from debug_info is no json the whole output will be attached in on file. So this will still work if WhisperBack is used with another linux distribution where the script to create the info to append is not yet updated.

#15 Updated by intrigeri 2018-08-01 08:23:38

  • Assignee changed from intrigeri to sascha.markus@gmail.com
  • QA Check set to Info Needed

I’m not sure why this was assigned to me. Did you reach an agreement with help desk wrt. what we want?

#16 Updated by intrigeri 2018-09-05 16:26:47

  • Target version changed from Tails_3.9 to Tails_3.10.1

#17 Updated by mercedes508 2018-10-12 10:21:09

We don’t really get the point of this ticket wrt help desk. Because using a simple ctrl+f then look for specific part of logs in WhisperBack reports works fine for us, ans maybe is simplier than having different attachments.

#18 Updated by intrigeri 2018-10-12 13:27:50

> We don’t really get the point of this ticket wrt help desk. Because using a simple ctrl+f then look for specific part of logs in WhisperBack reports works fine for us, ans maybe is simplier than having different attachments.

Point taken. As someone who processes some WhisperBack reports as well (although way fewer than you folks!), what I would like is:

  • Ability to quickly go to a specific part (attached file, output of command) of the bug report without having to know its name/heading by heart; it might be because I get less muscle training than you folks at this sort of things, but for me the ability to see graphically parts of the bug report as separated and to fold/unfold those I want to check would help a lot.
  • Ability to search for a text string in the entire bug report at once. We already have this in the current version and it would be sad to lose it. And presumably it also preserves backwards compatibility with the way you’ve been doing things so far.
  • Increased chances that the bug reports help desk members forward me are complete and pristine (i.e. not rewrapped). It took the current team months, if not years, to get used to do it right most of the time, and even now there are still occasional hiccups. I’d rather not go through the same process with the next help desk member. I think that having logs in attachments would give this for free.

So, dear help desk, assuming this ticket is implemented in a way that gives us all of the above, would it be fine with you or would it harm your work in any way?

Sascha Markus, could you please send to me, somehow, an email generated with the new code with inline attachments and another one with non-inline attachments? I’d like to check how they look in my own email client (rationale: if they’re rendered in a way that does not give me the expected advantages I’ve described above, and help desk is happy with the current situation, indeed it’s maybe not worth spending more time on this). Note that the SMTP relay used by WhisperBack won’t allow relaying to any other address than help desk’s so I guess you need to make the code temporarily 1. encrypt to my key (7C84A74CFB12BC439E81BA78C92949B8A63BB098); 2. save the generated message to a file instead of sending it via SMTP.

Thank you all in advance!

#19 Updated by emmapeel 2018-10-20 11:04:21

I am not sure why this will be better, but as long as we can use ctrl+f I am OK with the changes….

#20 Updated by mercedes508 2018-10-22 10:54:21

intrigeri wrote:
> * Ability to quickly go to a specific part (attached file, output of command) of the bug report without having to know its name/heading by heart; it might be because I get less muscle training than you folks at this sort of things, but for me the ability to see graphically parts of the bug report as separated and to fold/unfold those I want to check would help a lot.
> * Ability to search for a text string in the entire bug report at once. We already have this in the current version and it would be sad to lose it. And presumably it also preserves backwards compatibility with the way you’ve been doing things so far.
> * Increased chances that the bug reports help desk members forward me are complete and pristine (i.e. not rewrapped). It took the current team months, if not years, to get used to do it right most of the time, and even now there are still occasional hiccups. I’d rather not go through the same process with the next help desk member. I think that having logs in attachments would give this for free.
>
> So, dear help desk, assuming this ticket is implemented in a way that gives us all of the above, would it be fine with you or would it harm your work in any way?

Described this way it looks fine, because we won’t lose our current way of digging in bug reports, but I’d like to see it in practice to really say if it fits me or not.

#21 Updated by intrigeri 2018-10-24 17:03:30

  • Target version changed from Tails_3.10.1 to Tails_3.11

#22 Updated by CyrilBrulebois 2018-12-16 14:11:35

  • Target version changed from Tails_3.11 to Tails_3.12

#23 Updated by anonym 2019-01-30 11:59:10

  • Target version changed from Tails_3.12 to Tails_3.13

#24 Updated by sascha.markus@gmail.com 2019-02-04 20:29:12

mercedes508 wrote:
> intrigeri wrote:
> > * Ability to quickly go to a specific part (attached file, output of command) of the bug report without having to know its name/heading by heart; it might be because I get less muscle training than you folks at this sort of things, but for me the ability to see graphically parts of the bug report as separated and to fold/unfold those I want to check would help a lot.
> > * Ability to search for a text string in the entire bug report at once. We already have this in the current version and it would be sad to lose it. And presumably it also preserves backwards compatibility with the way you’ve been doing things so far.
> > * Increased chances that the bug reports help desk members forward me are complete and pristine (i.e. not rewrapped). It took the current team months, if not years, to get used to do it right most of the time, and even now there are still occasional hiccups. I’d rather not go through the same process with the next help desk member. I think that having logs in attachments would give this for free.
> >
> > So, dear help desk, assuming this ticket is implemented in a way that gives us all of the above, would it be fine with you or would it harm your work in any way?
>
> Described this way it looks fine, because we won’t lose our current way of digging in bug reports, but I’d like to see it in practice to really say if it fits me or not.

Attached are two files. One with inline attachments and one with attached files.
I opened both in the thunderbird that comes with the current tails and I assume the inline version is the one the helpdesk might prefer. It’s not possible to search in the attached files.

Cheers
Sascha

#25 Updated by intrigeri 2019-02-22 09:08:16

  • Assignee changed from intrigeri to sascha.markus@gmail.com
  • QA Check changed from Info Needed to Dev Needed

Sorry for the delay!

> Attached are two files. One with inline attachments and one with attached files.
> I opened both in the thunderbird that comes with the current tails and I assume the inline version is the one the helpdesk might prefer. It’s not possible to search in the attached files.

Indeed, the version with attached files (Content-Disposition: attachment) won’t fly.

The other version (inline attachments) looks good to me: help desk should be able to keep their workflow and other people/software who want to work with structured data can do it (e.g. we often use these reports to compute stats; once we have more structured data, the implementation of scrappers will be much less hackish and the results more reliable).

Just one thing wrt. the version with inline attachments: when their content is folded/hidden, there’s no way to tell which is which, which makes the whole thing a bit moot (both for humans and automated processing). To make the data more usable and structured, I think we need to give them a name, with something like Content-Disposition: inline; filename=SOMETHING_CLEVER. Is this possible? I see that in the other version we have filename="_proc_cmdline.txt" so I guess it should be?

#26 Updated by sascha.markus@gmail.com 2019-02-25 19:01:43

intrigeri wrote:
> Sorry for the delay!
>
> > Attached are two files. One with inline attachments and one with attached files.
> > I opened both in the thunderbird that comes with the current tails and I assume the inline version is the one the helpdesk might prefer. It’s not possible to search in the attached files.
>
> Indeed, the version with attached files (Content-Disposition: attachment) won’t fly.
>
> The other version (inline attachments) looks good to me: help desk should be able to keep their workflow and other people/software who want to work with structured data can do it (e.g. we often use these reports to compute stats; once we have more structured data, the implementation of scrappers will be much less hackish and the results more reliable).
>
> Just one thing wrt. the version with inline attachments: when their content is folded/hidden, there’s no way to tell which is which, which makes the whole thing a bit moot (both for humans and automated processing). To make the data more usable and structured, I think we need to give them a name, with something like Content-Disposition: inline; filename=SOMETHING_CLEVER. Is this possible? I see that in the other version we have filename="_proc_cmdline.txt" so I guess it should be?
The inline version also had filenames set in the code. But the are ot transfered.
msg.add_header(’Content-Disposition’, ’inline’, filename=filename)
But if thunderbird is the tool used by the helpdesk:
In the thunderbird that comes with tails in the menue there is “View -> Display Attachments Inline”.
With this I see the attachments like in the inline version but with the filenames.
Please have a look at the attched screenshot

Cheers
Sascha

#27 Updated by intrigeri 2019-02-26 08:47:20

  • Assignee changed from intrigeri to goupille

> In the thunderbird that comes with tails in the menue there is “View -> Display Attachments Inline”.
> With this I see the attachments like in the inline version but with the filenames.

Confirmed with test_8722_attached.eml. With this Thunderbird option enabled, I think all our requirements are met: direct access to an arbitrary attachment for me, full text search across the entire message including attachments for help desk, and as a bonus we get structured data we can process programmatically. This sounds awesome!

@goupille, can you please open /code/attachments/download/2282/test_8722_attached.eml in your Thunderbird, enable “View → Display Attachments Inline”, and tell us if it would make your workflow worse in any way? Thanks in advance! :)

#28 Updated by goupille 2019-02-26 12:58:06

intrigeri wrote:

>
> @goupille, can you please open /code/attachments/download/2282/test_8722_attached.eml in your Thunderbird, enable “View → Display Attachments Inline”, and tell us if it would make your workflow worse in any way? Thanks in advance! :)

I gave it a quick look, and at first it looks good to me (it seems that “display attachments inline” does not display the huge images that some users are attaching to their emails, and that’s a good thing for me). however, to forward that kind of report to you through tails-bugs@, it seems I’d need to save the attachments, then attach them again to the reply (nightmare), or copy/paste the whole report in the body (no gain for you). I can forward it directly but that’ll make it harder for me to see what my co-worker are aware of and what not

#29 Updated by goupille 2019-02-26 12:58:32

  • Assignee changed from goupille to intrigeri

#30 Updated by intrigeri 2019-03-04 14:21:54

> I gave it a quick look, and at first it looks good to me (it seems that “display attachments inline” does not display the huge images that some users are attaching to their emails, and that’s a good thing for me).

Great! So far so good, I’m happy it works for you wrt. reading reports :)

> however, to forward that kind of report to you through tails-bugs@, it seems I’d need to save the attachments, then attach them again to the reply (nightmare), or copy/paste the whole report in the body (no gain for you). I can forward it directly but that’ll make it harder for me to see what my co-worker are aware of and what not

You could forward them directly (without going through Schleuder) and Cc your team’s mailing list. I’ve just verified that when forwarding a message, Thunderbird adds the In-Reply-To header that should make the forwarded message be part of the thread on your mailing list. Can you please try this for one report and confirm that the forwarded email, once you’ve received it via tails-bugs, is in the correct thread?

Would this be worse than your current “forwarding but not quite” workflow? I expect that typing Schleuder X-* pseudo-headers is more painful than adding your team to Cc.

As a bonus, I would know who I’m talking to, which could allow me to follow up on XMPP, to know if they’re still on duty and whether I can expect fast answers if I need more info (depending on that, I would often act in a different manner).

#31 Updated by intrigeri 2019-03-04 14:59:52

  • Assignee changed from intrigeri to goupille

@goupille, see my above questions :)

#32 Updated by intrigeri 2019-03-08 14:55:38

  • Assignee changed from goupille to intrigeri
  • QA Check deleted (Info Needed)

@goupille, thanks!

#33 Updated by intrigeri 2019-03-10 13:44:33

  • Assignee changed from intrigeri to goupille
  • QA Check set to Info Needed

Next step is for you is: “confirm that the forwarded email, once you’ve received it via tails-bugs, is in the correct thread”.

Assuming 1. you confirm this and 2. my proposal that I directly receive the WhisperBack reports is accepted:

  • You folks won’t need to forward me the full contents of such reports anymore. You’ll only have to reply to them with me in copy and I’ll have the identifier that I need to find the corresponding report (that I’ll already have on my side, formatted as I want it :)
  • You’ll still need to forward stuff to other developers, especially once the Foundations Team “frontdesk” rotation is in place in a few months. If you do this as you’ve tested (“Forward” + Cc your team), as per your tests they won’t get the benefits of attachments: too bad, but they’ll get basically the same thing that what they would get today, without the changes proposed here, i.e. no regression.

So this seems good to me: things improve for me, presumably Help Desk needs to do a tiny bit less work when forwarding reports to developers, and nothing changes for anyone else.

The only downside I can thing of is: some of sajolida’s scripts might need to be adjusted to extract text from attachments. Happy to help with this!

Does anyone need a more detailed summary of the proposal and its consequences for various people, in order to share their thoughts on this matter, before we go forward?

#34 Updated by goupille 2019-03-12 13:10:49

intrigeri wrote:
> Next step is for you is: “confirm that the forwarded email, once you’ve received it via tails-bugs, is in the correct thread”.

I confirm that the forwarded emails with tails-bugs@ in Cc: are displayed in the correct thread.

#35 Updated by intrigeri 2019-03-12 14:30:47

  • Assignee changed from goupille to intrigeri
  • Target version changed from Tails_3.13 to Tails_3.14

> I confirm that the forwarded emails with tails-bugs@ in Cc: are displayed in the correct thread.

Thank you :)

Let’s give others until the end of the month to comment or ask details before I move this thing forward.

#36 Updated by intrigeri 2019-03-12 14:38:51

  • QA Check deleted (Info Needed)

#37 Updated by intrigeri 2019-04-26 13:09:40

  • Assignee changed from intrigeri to sascha.markus@gmail.com

sascha.markusgmail.com, it looks like we have the go ahead to do this! If you’re still interested, please read the last discussions and propose a branch that implements whatever we settled on :)

#38 Updated by sascha.markus@gmail.com 2019-04-27 08:28:27

  • File 0001-Attach-debug-info-as-individual-files.patch added
  • Assignee changed from sascha.markus@gmail.com to intrigeri
  • QA Check set to Ready for QA
  • Feature Branch set to https://gitlab.com/saschamarkus/whisperback/tree/feature/8722-Create-MIME-Multipart-attachments

The implementation is available as a patch or in the feature branch at gitlab

#39 Updated by intrigeri 2019-04-29 10:32:27

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to segfault

hi @segfault! I’m happy to do everything else (building, releasing, testing, communicating with affected parties, etc.) but could you please do a code review? Your Python is way better than mine :)

#40 Updated by CyrilBrulebois 2019-05-23 21:23:18

  • Target version changed from Tails_3.14 to Tails_3.15

#41 Updated by segfault 2019-06-01 22:52:29

sascha.markusgmail.com I would prefer to do this review in a GitLab merge request. Could you open a MR to merge the feature branch into whisperback’s master?

The first issue I found (beside some Python style issues) is that in whisperback.py:__get_debug_info, you append a dictionary with the key "appended_info" to the list of debug info. That will cause an error in line 132, where you assume that all items in the list are dictionaries with keys "key" and "content".

#42 Updated by segfault 2019-06-01 23:09:56

  • Assignee changed from segfault to sascha.markus@gmail.com

#43 Updated by segfault 2019-06-01 23:10:04

  • QA Check changed from Ready for QA to Info Needed

#44 Updated by intrigeri 2019-06-02 15:28:12

  • QA Check deleted (Info Needed)

#45 Updated by CyrilBrulebois 2019-07-10 10:33:55

  • Target version changed from Tails_3.15 to Tails_3.16

#46 Updated by CyrilBrulebois 2019-09-05 00:05:26

  • Target version changed from Tails_3.16 to Tails_3.17

#47 Updated by intrigeri 2019-09-12 14:25:05

  • Target version changed from Tails_3.17 to Tails_4.0

#48 Updated by intrigeri 2019-10-09 19:49:51

  • Target version deleted (Tails_4.0)

sascha.markusgmail.com, are you still interested in leading this to a conclusion.

Either way, 4.0 is now frozen so I’m removing the target version → feel free to set it to whichever you’d like :)