Feature #8281

Consider replacing Florence with GNOME's own on-screen keyboard

Added by intrigeri 2014-11-20 23:05:31 . Updated 2018-04-08 16:47:24 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Tails_3.2
Start date:
2014-11-20
Due date:
% Done:

100%

Feature Branch:
bugfix/8281-replace-florence-with-gnome-osk
Type of work:
Code
Blueprint:

Starter:
Affected tool:
On-screen keyboard
Deliverable for:

Description

Florence is badly integrated into GNOME Shell. Some tricks can probably save us (see notes about it on the parent ticket) for the time being, but on the long run, using the virtual keyboard shipped by default in GNOME could be the best solution. Let’s evaluate it.

  1. Install the last Tails release.
  2. Boot from this ISO on bare metal (some virtualization environment make it hard to display Caribou)
  3. Start caribou : Accessibility menu -> Screen keyboard
  4. Try it out and report back

Subtasks

Related issues

Related to Tails - Bug #8010: Florence icon is badly integrated within GNOME Shell Resolved
Related to Tails - Feature #6064: Handheld Tails Confirmed 2014-07-10
Related to Tails - Feature #6202: Evaluate Caribou Rejected 2013-07-28
Related to Tails - Bug #11062: Florence can't be used when GNOME asks for a password Resolved 2016-02-05
Related to Tails - Bug #12385: The virtual keyboard in the Greeter allows to type only a limited set of keys Resolved 2017-03-19
Related to Tails - Bug #14675: GNOME on-screen keyboard is broken without the X11 XTEST extensions Resolved 2017-09-16
Related to Tails - Feature #14713: Discuss and evaluate feedback coming from the Florence removal Resolved 2017-10-03
Related to Tails - Bug #16628: Remove caribou Resolved
Blocks Tails - Feature #13234: Core work 2017Q3: Foundations Team Resolved 2017-06-29

History

#1 Updated by intrigeri 2014-11-20 23:05:44

  • Tracker changed from Bug to Feature

#2 Updated by intrigeri 2014-11-23 16:16:51

#3 Updated by intrigeri 2014-11-23 16:18:37

  • related to Bug #8010: Florence icon is badly integrated within GNOME Shell added

#4 Updated by sajolida 2014-11-24 18:17:25

  • Assignee set to sajolida

#5 Updated by intrigeri 2014-11-26 10:02:17

#6 Updated by alant 2014-12-07 00:15:16

I tested caribou but it is qwerty even when I set another kekboard layout in GNOME. I failed to find how to configure caribou keyboard layout. https://git.gnome.org/browse/caribou/tree/data/layouts/fullscale suggests only qwerty is supported, but there is code that reads X keyboard configuration in https://git.gnome.org/browse/caribou/tree/libcaribou/xadapter.vala. gnome-shell code displaying the virtual keyboard is in https://git.gnome.org/browse/gnome-shell/tree/js/ui/keyboard.js

#7 Updated by sajolida 2014-12-11 11:16:58

  • Assignee deleted (sajolida)

Alan: I’m glad to see that you were faster than me on this one. So I’m deassign it from me. Do you think that there is anything else to be tested here?

#8 Updated by sajolida 2014-12-11 11:17:49

Apparently GNOME has plans to integrate a custom screen keyboard in GNOME Shell but they are not there yet:

https://wiki.gnome.org/Design/OS/ScreenKeyboard

#9 Updated by intrigeri 2014-12-18 11:25:53

  • Assignee set to alant
  • Priority changed from Normal to Low
  • QA Check set to Info Needed
  • Type of work changed from Test to Research

> Apparently GNOME has plans to integrate a custom screen keyboard in GNOME Shell but they are not there yet:

In jessie caribou is integrated in GNOME Shell (the UI is Clutter drawn by the shell, but libcaribou is behind the scene)

> Alan, do you want to research this any further? (probably not a blocker for Tails/Jessie, now that Bug #8010 is solved)

I don’t think it’s worth spending more hours reading its code and searchine for its nonexistant documentation. So I see two options:

- we give up and keep current status
- we (possibly me) ask the right GNOME developpers for help

The 1st option is the lazy one. The 2nd, if it succeedes, is more integrated into the desktop and eases the integration of a virtual keyboard in the greeter (it might event git it for free).

What do you think?

#10 Updated by anonym 2014-12-18 11:28:43

  • Assignee deleted (alant)
  • Priority changed from Low to Normal
  • QA Check deleted (Info Needed)
  • Type of work changed from Research to Test

My experience with Caribou wasn’t so great. It covers the lower half the screen (at least on screens with small vertical size) and may very well block the text field you are editing. It doesn’t switch from bottom to top in that case, which otherwise would have been an improvement, possibly.

#11 Updated by intrigeri 2014-12-18 13:42:50

  • Assignee set to alant
  • Priority changed from Normal to Low
  • QA Check set to Info Needed
  • Type of work changed from Test to Research

#12 Updated by intrigeri 2015-03-09 02:04:19

Indeed, the layout issue seems to be well-known. There’s been some work on it recently, but it’s still probably not as good as what Florence gives us currently:

#13 Updated by intrigeri 2015-03-09 02:08:55

#14 Updated by alant 2015-07-06 15:00:08

  • Assignee deleted (alant)

Unassigning this ticket from me as I don’t plan to work on it shortly.

#15 Updated by BitingBird 2015-08-25 12:04:14

#16 Updated by intrigeri 2015-11-10 10:17:58

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Jessie to Evaluate the Caribou virtual keyboard in Tails/Stretch
  • Priority changed from Low to Normal
  • Target version changed from Tails_2.0 to Tails_3.0
  • QA Check deleted (Info Needed)

3 of us have evaluated it on Jessie and agree that it’s not good enough => postponing to Stretch.

#17 Updated by intrigeri 2016-02-09 19:03:32

  • related to Bug #11062: Florence can't be used when GNOME asks for a password added

#18 Updated by intrigeri 2016-08-26 09:17:01

  • Description updated

#19 Updated by intrigeri 2016-08-26 10:29:52

  • Assignee set to intrigeri
  • Target version changed from Tails_3.0 to Tails_4.0

On the Git side, very little has changed in https://git.gnome.org/browse/gnome-shell/log/js/ui/keyboard.js and in Caribou itself, since last time we checked. No progress on the GNOME bug reports listed above. Still, I’ve tested it again on feature/stretch. It required installing these packages, so that GNOME’s “Screen Keyboard” accessibility feature started working: caribou libcaribou-gtk-module libcaribou-gtk3-module. The keyboard layout used in the screen keyboard still does not reflect the one used elsewhere, except for a very small number of those such as German and French. The positioning problem that anonym reported earlier on this ticket is still present.

So I’m postponing this to Buster.

#20 Updated by intrigeri 2016-08-26 10:30:03

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Stretch to Evaluate the Caribou virtual keyboard in Tails/Buster

#21 Updated by intrigeri 2016-08-26 10:30:13

  • Assignee deleted (intrigeri)

#22 Updated by intrigeri 2016-08-29 01:19:13

I’m basically done with Bug #8312 so this ticket can’t be a subtask of it anymore.

#23 Updated by intrigeri 2017-03-19 18:55:19

  • related to Bug #12385: The virtual keyboard in the Greeter allows to type only a limited set of keys added

#24 Updated by intrigeri 2017-06-28 17:31:09

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Buster to Evaluate the Caribou virtual keyboard in Tails
  • Assignee set to intrigeri
  • Target version changed from Tails_4.0 to Tails_3.2

I’ll give it another try, because Bug #12385#note-14 proved that it’s much better than I thought, and we’re having lots of issues with Florence itself + its poor integration in GNOME. I’m now confident we could drop Florence entirely in 3.2 :)

#25 Updated by intrigeri 2017-06-29 10:25:16

#26 Updated by intrigeri 2017-07-25 08:10:45

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails to Consider replacing Florence with GNOME's own on-screen keyboard
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30
  • Type of work changed from Research to Discuss

So I’ve looked into this again.

Advantages of dropping Florence (and relying on GNOME’s own on-screen keyboard):

  • We get rid of the many issues we have with Florence and its lack of integration into GNOME, e.g. one can type passwords in GNOME Shell password prompts with the on-screen keyboard (Bug #11062).
  • Better support for touch devices: the GNOME on-screen keyboard is enabled by default on such devices, and it pops up automatically when needed instead of having to toggle it on/off manually.
  • We get rid of Florence bugs that affect us, such as https://bugs.debian.org/805895.
  • One less icon in the top bar => leaner, slicker desktop UI.
  • Smaller delta with other GNOME-based distros.
  • One less third-party application that our friends have to maintain in Debian, and for which we have to deal with a not-very-active upstream.

Drawbacks of GNOME’s own on-screen keyboard, compared to Florence:

  • The on-screen keyboard layout does not reflect the physical keyboard layout except for very few languages (but accented and other special chars can be input); that’s pretty bad, but if it’s good enough for other GNOME-based distros, perhaps it’s good enough for us? Actually there are two cases:
    • using the on-screen keyboard as a counter-measure against hardware keyloggers: then this is a serious problem as one has to context-switch between the layout of the physical keyboard and the layout of the on-screen keyboard;
    • using the on-screen keyboard for universal access reasons: in this case, one will simply get used to the QWERTY layout, and no context switch is involved, so it’s OK-ish IMO.
  • Toggling the on-screen keyboard on/off requires two clicks instead of one; here again, this is a problem only for those who use this tool only occasionally, as a counter-measure against hardware keyloggers; I think we should file a bug report in GNOME to ask for a keyboard shortcut for toggling the on-screen keyboard accessibility feature.
  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it’s background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it’s not a serious blocker.

So all in all, IMO the UX and maintenance cost benefits of switching to GNOME’s own on-screen keyboard outweigh the drawbacks. I think we should do it. Let’s make a decision during our next monthly meeting and if we decide to do what I’m proposing, I’ll implement it in Tails 3.2.

#27 Updated by intrigeri 2017-07-25 08:16:53

  • Description updated

#28 Updated by alant 2017-07-31 11:08:48

I think it would be great to use GNOME OSK. I’ve discussed this with GNOME people at GUADEC, and I’ve looked at it. So here we go:

intrigeri wrote:
> Drawbacks of GNOME’s own on-screen keyboard, compared to Florence:
>
> * The on-screen keyboard layout does not reflect the physical keyboard layout except for very few languages (but accented and other special chars can be input); that’s pretty bad, but if it’s good enough for other GNOME-based distros, perhaps it’s good enough for us?

That’s more a bug than a choice. Actually, GNOME OSK is drawn by the shell, but still uses Caribou for getting its layout. And caribou is unmaintained. But there are gnome-shell developpers that plan to work on dropping the caribou dependency. I investigated the layout issue, and we should meet to discuss that shortly. So on the long term, this has chances to be fixed.

On the short term, there is a script tools/convert_cldr.py in caribou sources that can import layouts from http://www.unicode.org/repos/cldr/trunk/keyboards/android/. Current version misses a few things, but I’ve turned to something that’s close to actually automatically import layouts. I can work on that if we want to import all these layouts for the Tails before we ship the gnome-shell version including the fixes.

> * Toggling the on-screen keyboard on/off requires two clicks instead of one; here again, this is a problem only for those who use this tool only occasionally, as a counter-measure against hardware keyloggers; I think we should file a bug report in GNOME to ask for a keyboard shortcut for toggling the on-screen keyboard accessibility feature.

I’m gonna ask that, stay tuned.

> * The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it’s background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it’s not a serious blocker.
>
I don’t think that this will change, it’s a design decision.

> So all in all, IMO the UX and maintenance cost benefits of switching to GNOME’s own on-screen keyboard outweigh the drawbacks. I think we should do it.

I agree with you.

#29 Updated by intrigeri 2017-08-01 15:52:15

> I can work on that if we want to import all these layouts for the Tails before we ship the gnome-shell version including the fixes.

Sounds great to me. I wouldn’t call it a blocker personally, but it’s reassuring that we can rely on this to be fixed some day :)

>> * The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it’s background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it’s not a serious blocker.

> I don’t think that this will change, it’s a design decision.

If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we’re gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

#30 Updated by sajolida 2017-08-02 10:43:59

I agree with all your reasoning and I’m fine with replacing Florence with GNOME Screen Keyboard. I tried to test it myself but it fails to load on 3.0.1.

I get in journalctl -f:

Aug 02 10:38:50 amnesia keepassx[20197]: Failed to load module "gail"
Aug 02 10:38:50 amnesia Web Content[20130]: Failed to load module "caribou-gtk-module"
Aug 02 10:38:50 amnesia firefox[14797]: Failed to load module "gail"
Aug 02 10:38:50 amnesia Web Content[20130]: Failed to load module "gail"
Aug 02 10:38:50 amnesia gnome-shell[11251]: caribou_group_model_create_group_name: assertion 'group != NULL' failed
Aug 02 10:38:50 amnesia gnome-shell[11251]: caribou_keyboard_model_populate_group: assertion 'group != NULL' failed

How did you test it?

#31 Updated by sajolida 2017-08-02 10:52:28

  • Affected tool set to On-screen keyboard

#32 Updated by sajolida 2017-08-02 11:01:39

  • QA Check set to Info Needed

Actually, it’s there but only shows up or works in rare places.

  • Places it shows up (expected):
    • Activities overview
    • SSH key password prompt
  • Places it doesn’t show up (unexpected):
  • Places where it shows up but I cannot type (unexpected):
    • OpenPGP Applet password prompt, pinentry
    • Password prompt from GPG command line

All these ones involving password prompts are clear blockers to me. I’m fine doing more tests but first I want to check whether I’m missing something obvious here.

#33 Updated by alant 2017-08-03 13:40:57

sajolida wrote:
> * Places it doesn’t show up (unexpected):
> KeePassX password prompt
> gedit
> Password fields in Firefox (eg. https://mail.riseup.net/)
>

Both works on fresh debian/stretch

> * Places where it shows up but I cannot type (unexpected):

[…]

> Password prompt from GPG command line
>
Works in debian/stretch with pinentry-gnome3

> All these ones involving password prompts are clear blockers to me. I’m fine doing more tests but first I want to check whether I’m missing something obvious here.

Tails seems to be missing libcaribou-gtk-module and pinentry-gnome3.

#34 Updated by alant 2017-08-03 13:44:12

> >> * The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it’s background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it’s not a serious blocker.
>
> > I don’t think that this will change, it’s a design decision.
>
> If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we’re gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

See https://wiki.gnome.org/Design/OS/ScreenKeyboard

#35 Updated by sajolida 2017-08-03 17:47:03

Good that it work in Debian/Stretch, so that’s a problem in Tails and can be solved.

I tried to install libcaribou-gtk-module and restart GNOME Shell and the Screen Keyboard would still not show up in all those places.

#36 Updated by intrigeri 2017-08-03 18:02:26

> All these ones involving password prompts are clear blockers to me.

Note that some other password prompts work better than with Florence, so IMO we should weigh them against each other and not look only at regressions in isolation.

#37 Updated by intrigeri 2017-08-03 18:03:45

> Tails seems to be missing libcaribou-gtk-module and pinentry-gnome3.

We made a collective decision (against my opinion but whatever) to ship another pinentry implementation. IIRC the main reason was supporting copy’n’pasting from KeePassX after having been requested for the OpenPGP passphrase. We can certainly revisit this, but it’s more complicated than one missing package.

#38 Updated by emmapeel 2017-08-03 18:57:43

We receive many bug reports of Florence, is a feature peple uses and it is quite buggy…

I think the change will be a good idea.

#39 Updated by nodens 2017-08-03 19:20:48

course of action decided on last monthly meeting:

  1. intrigeri tries to prepare a branch without florence that doesn’t introduce too many regressions, and let people test it early enough before the 3.2 freeze.
  2. intrigeri ensures we discuss the remaining regressions (e.g. pinentry) and what solutions we should implement later on (not blocking on that to drop Florence though.)

#40 Updated by sajolida 2017-08-04 10:33:44

  • QA Check changed from Info Needed to Dev Needed
  • Type of work changed from Discuss to Code

Discussed during the August 2017 monthly meeting:

https://tails.boum.org/contribute/meetings/201708/

  • emmapeel has seen several reports on buggy Florence
  • It’s mostly unmaintained
  • does the pros outwight the cons ?
    • some issues with GNOME OSK, but according to intrigeri, fixable appart from pinentry issue
    • pinentry issue might be worked around by revisiting previous project decision to ship pinentry-gtk2 and not pinentry-gnome3, after weighting the consequences carefully.
  • Course of action:
    • intrigeri tries to prepare a branch without florence that doesn’t introduce too many regressions, and let people test it early enough before the 3.2 freeze.
    • intrigeri ensures we discuss the remaining regressions (e.g. pinentry) and what solutions we should implement later on (not blocking on that to drop Florence though.)

#41 Updated by alant 2017-08-13 21:42:45

There is developpment in GNOME Shell based on the unicode.org CLDR layouts we worked on at GUADEC : https://git.gnome.org//browse/gnome-shell/log/?h=wip/carlosg/osk-cldr

Let me know if you want me to backport these layouts to be used with current Tails.

#42 Updated by intrigeri 2017-09-01 13:23:48

> There is developpment in GNOME Shell based on the unicode.org CLDR layouts we worked on at GUADEC : https://git.gnome.org//browse/gnome-shell/log/?h=wip/carlosg/osk-cldr

Thanks!

> Let me know if you want me to backport these layouts to be used with current Tails.

I want to focus on blockers, and I think the layouts issue is not one. If you have time to help in time for 3.2, I suggest you instead (explicitly) pick one of the serious issues mentioned above. I’m not sure when exactly I’ll come back to this (probably at some point in the 1st half of September); let’s try to coordinate and avoid duplicating work :)

#43 Updated by intrigeri 2017-09-01 13:33:29

Alan wrote:

>>>> * The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it’s background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it’s not a serious blocker.

>>> I don’t think that this will change, it’s a design decision.

>> If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we’re gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

> See https://wiki.gnome.org/Design/OS/ScreenKeyboard

Sorry, I was not able to find what I was asking for there :/ Could you please point me to the exact bits you were referring to?

The only related thing I’ve seen is When the OSK is displayed, the view should be shifted in order to keep the text cursor visible. So at least they’re aware that there is a real-world problem that’s not addressed by the current design :)

FWIW the https://github.com/schuhumi/gnome-shell-extension-caribou-resize-workspace extension is intended to workaround this problem. It could be an option for us on the short term.

#44 Updated by intrigeri 2017-09-06 15:38:50

  • Feature Branch set to bugfix/8281-replace-florence-with-gnome-osk

The topic branch repairs GNOME’s on-screen keyboard in:

  • pinentry-gtk2, OpenPGP Applet password prompt, password prompt from gpg command line: installing some missing libraries was enough to fix it, and now GNOME’s OSK can read the passphrase prompts.
  • Password fields in Firefox (eg. https://mail.riseup.net/): installing the caribou package did the trick
  • KeePassX

The only other regression that was reported here was about gedit, which works for me flawlessly on Tails 3.1. I expect this core GNOME app to work fine with a11n so I’m surprised it has failed => sajolida, please reproduce and send me the Journal, if you want. Anyway, I don’t think this is a blocker.

So we’ve reached the state when no serious regression is known. I’ll call for testing, will update the test suite and documentation, and hopefully will submit for QA in time for 3.2 :)

#45 Updated by intrigeri 2017-09-06 18:10:57

  • QA Check changed from Dev Needed to Ready for QA
  • Starter deleted (Yes)

Sent call for testing. I’ll wait a few days and will send to anonym for review if no serious issue is reported.

Meanwhile:

  • anonym: you might want to do a first quick code review.
  • sajolida: you might want to have a look at the doc update.

#46 Updated by intrigeri 2017-09-07 11:51:00

The branch passed a full test suite run (modulo Bug #14586 of course) on my personal Jenkins.

#47 Updated by intrigeri 2017-09-10 12:07:08

  • Assignee changed from intrigeri to anonym

No test report (not very surprising), let’s go ahead! Please reassign to sajolida once merged so he can review the doc (not a blocker for the merge IMO as I think my draft is not that bad that it justifies delaying this change by N more months).

#48 Updated by anonym 2017-09-13 13:41:13

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 30 to 100

#49 Updated by intrigeri 2017-09-18 08:22:40

  • related to Bug #14675: GNOME on-screen keyboard is broken without the X11 XTEST extensions added

#50 Updated by intrigeri 2017-09-24 08:51:15

  • related to Feature #14713: Discuss and evaluate feedback coming from the Florence removal added

#51 Updated by anonym 2017-09-28 18:49:10

  • Status changed from Fix committed to Resolved

#52 Updated by intrigeri 2018-04-08 16:47:24

  • QA Check deleted (Ready for QA)

#53 Updated by intrigeri 2019-05-24 11:30:33