Tails

#6 Updated by intrigeri 2015-06-05 15:36:02

We now have a Python (3) library: https://git-tails.immerda.ch/pythonlib. But it’s not installed in Tails yet, and only used by infrastructure bits and during the release process.

#7 Updated by intrigeri 2015-10-14 12:05:43

kurono volunteered to help. I’ll let you folks discuss it.

#8 Updated by anonym 2015-10-16 04:46:16

I have a few additional ideas for such a Python library (plus steps towards a grander Python vision for Tails in general, see below). For starters, parts of the Shell library should migrate into the Python library, so the shell functions are thin wrappers around the Python library. The priority should be:

• Anything that heavily uses lists! (Think: whitespace issues in the shell)

• Anything complex should move there, e.g. hardware.sh:mod_rev_dep() (recursion in the shell? why not, but please no :)), and I think we can get common.sh:wait_until()/try_for() functions implemented in python with real timeouts, instead of the “sort of”-timeouts we have now.

• Many things in tor.sh can be implemented reliably and trivially via stem.

• Python probably has proper localization stuff, instead of our sed hack in localization.sh.

However, maybe we can convert everything in the Shell library, and even aim for replacing more or less all our Shell scripts in the future? There are several Python modules that could make this easier for us. We could use e.g. plumbum for this:

• It can import any command-line tool in your $PATH and integrate it into python like a function. It’s super easy to extract the exit code, stdout and stderr. There’s a simple interface to use such imported commands through Popen too, when needed. There’s a simple interface to use such imported commands through Popen too, when needed. All this works so well that I’m afraid we might continue to use the Shell tools instead of the more appropriate Python equivalents. :)

• It provides simple ways to do piping, backgrounding and similar convenient shell concepts.

• We get the equivalent of set -eu by default, and can work with it more easily since the program won’t exit, but throw an exception, so exception handling will make things much more easier for us.

• It makes command nesting with sudo, ssh etc much simpler and clearer than in the shell.

Overall, plumbum is quite brilliant. :) There are also alternatives, like sh, scriptine and so on, but I do not know what is best for our needs (I suspect plumbum, though). Also, for more complex subprocess interaction we may want to look at e.g. sarge as a replacement for the standard (but IMHO awkward) subprocess Python module.

I’m digressing, a bit. I guess we really need another ticket. :)

#9 Updated by intrigeri 2015-10-16 05:54:20

> Overall, plumbum is quite brilliant. :) There are also alternatives, like sh,

FYI we already use sh in our Python lib.

#10 Updated by anonym 2015-10-16 06:25:34

intrigeri wrote:
> > Overall, plumbum is quite brilliant. :) There are also alternatives, like sh,
>
> FYI we already use sh in our Python lib.

After a brief look, it looks good, too (and indeed very similar; apparently plumbum has sh as an inspiration). A major difference is how piping works:

• Normal shell: ls -1 | wc -l
• Python plumbum: (ls['-1'] | wc['-l'])() or chain = ls['-1'] | wc['-l']; chain()
• Python sh: wc(ls('-1'), '-l')

So while plumbum works more similarly to the shell, the (...)() stuff is ugly. The sh way, i.e. function composition, is actually how it’d be done in most programming languages, which might be a good thing: instead of making us thing that we are “writing shell scripts in python”, it may promote the feeling that we are “writing python scripts with an easy way to invoke the shell”. ;)

Any way, most of the time we use pipes in the shell it’s for text processing, and Python has much better facilities for that. On the top of my head, we do some piping into nc for control port stuff, but we should use stem any way. I’m sure there’s more but, again, I digress.

#11 Updated by alant 2015-10-18 10:17:36

intrigeri wrote:
> kurono volunteered to help. I’ll let you folks discuss it.

kurono, feel free to ask any question or start working on this. You can assign the ticket to you if you want. I’ll be happy to give any help you wish and to review your work. I’ll try not to let you wait for a whole week, but don’t expect me to answer daily.

#14 Updated by intrigeri 2016-02-18 16:23:05

This is one of the last two remaining tickets on the obsolete “Sustainability_M1” (that doesn’t mean much anymore, since we have our new roadmap). kurono, please consider moving this ticket to e.g. the 2016 or 2017 milestone, or just emptying the Target version field entirely.

#17 Updated by intrigeri 2016-02-21 18:57:28

Note that another idea (for build -time stuff) is to migrate to Ansible, so perhaps converting run -time stuff to Python is more likely to be useful on the long run. But of course, at runtime we need to be more careful about memory usage.