Running additional command line tools on Ubuntu Touch

On April 5th, 2017 Mark Shuttleworth founder of the Ubuntu project and of Canonical, the company that sponsors the project, dropped, what many in the Ubuntu community would consider, a bombshell. He announced that the company would be ending their investment in Unity8, the phone and development of a convergent experience for the user across multiple devices and form factors

The news left me reeling. I felt left out in the cold, not to say betrayed, by Canonical. I had wholeheartedly embraced the vision Mark had pitched. And when the first Ubuntu Phone came out, severely underspecc'd and under-powered (even then) I bought it, partly to support the effort, partly to experience the journey first hand. I did the same when the first (and only) tablet came out. To me the fledgling mobile OS was much better suited as a tablet OS than a phone OS, as I didn't use WhatsApp on tablets, so I didn't miss it on one either.

But the very next day Marius Gripsgård, founder of the UBports project, posted a message to his G+ page that the UBports team would continue development of Ubuntu Touch, and Unity 8. I think I became aware of this initiative a few days later, and even then I had more of a wait and see attitude.

Fast forward to today and Ubuntu Touch is very much alive. A community has formed around the project, the likes of which Canonical had envisioned, but never managed to garner. The UBports team have released four  OTA's since April 2017, the last of which, OTA-4, was released on August 26, 2018. This is an important milestone for the project, because it brings the base of the OS from 15.04 to 16.04, which will be supported till April 2021, and frees up the team to focus on features instead of having to backport security patches and work around ancient versions of libraries and other software components.

Needless to say, all my original devices are running Ubuntu Touch powered by UBports, and I even managed to source a Meizu Pro 5 from amazon.it, as that was a device that would eventually support Anbox (or at least it was, back then ...). I am currently running the release candidate of 16.04 on my BQ Aquaris M10 FHD tablet. And part of the charm of running a regular(-ish) Linux OS on the tablet is that I can open a terminal and poke around at stuff, and even do most of the things I would do on a laptop or a desktop computer.

One main difference between a regular PC install and the device image is that the root filesystem on the device is read-only. This is means that additional packages cannot be installed from the repositories out of the box. Of course this being Linux, you could remount the root file system as writable and sudo apt install to your hearts content, but there are drawbacks to that path.

• Firstly it would put your device in a state that is undefined, and that could cause the regular update mechanism to fail the next time an update of the system image is made available for the device. I have personally never seen the mechanism fail even after modifying the root system, but it is something to be aware of. 
• Secondly, in case the system does update correctly, which it always has for me, then any packages you installed would have to be reinstalled after applying the update. As far as I understand the process, the system's root image is overwritten by the update, undoing any changes applied by the user. 
• Lastly, and this is what led me to search for alternatives, the current system image for the M10 hardly has any free space for installing additional packages in the first place.

So what could I do if I wanted additional command-line tools available on my tablet, like htop, nmap and screenfetch? The answer, as in so many other instances these days, would be: containers! No, I am not talking Docker or LXD here, but Libertine containers. This is the technology that makes it possible to install desktop applications like Firefox and LibreOffice on your Ubuntu Touch device. It does not magically make them usable on a 4.5" to 10" high dpi screen, though. 

Starting with 16.04 there is a fancy section in the system settings where you can create and administer Libertine containers. But I will focus on how to do it from the command line because that is where I have had the most consistent success.

First you create your container. I am calling mine Xerus and giving it the id xenial:

$ libertine-container-manager create-container -i xenial -n "Xerus"

This might take a while to complete. Next install the command line tools we want to use in the newly created container.

$ libertine-container-manager install-package -i xenial -p htop
$ libertine-container-manager install-package -i xenial -p nmap
$ libertine-container-manager install-package -i xenial -p screenfetch

To invoke the tools use libertine-launch. I will illustrate with screenfetch:

$ libertine-launch -i xenial screenfetch

: DISPLAY environment variable not set!
[[ ! ]] awk: fatal: cannot open file `/sys/devices/system/cpu/cpu0/
cpufreq/scaling_max_freq' for reading (Permission denied)
                          ./+o+-       phablet@android
                  yyyyy- -yyyyyy+      OS: Ubuntu 16.04 xenial
               ://+//////-yyyyyyo      Kernel: aarch64 Linux 3.10.93+
           .++ .:/++++++/-.+sss/`      Uptime: 2m
         .:++o:  /++++++++/:--:/-      Packages: 679
        o:+o+:++.`..```.-/oo+++++/     Shell: 21829
       .:+o:+o/.          `+sssoo+/    CPU: MT8163
  .++/+:+oo+o:`             /sssooo.   RAM: 1324MiB / 1962MiB
 /+++//+:`oo+o               /::--:.  
 \+/+o+++`o++o               ++////.  
  .++.o+++oo+:`             /dddhhh.  
       .+.o+oo:.          `oddhhhh+   
        \+.++o+o``-````.:ohdhhhhh+    
         `:o+++ `ohhhhhhhhyo++os:     
           .o:`.syhhhhhhh/.oo++o`     
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.      

OK, so it's not perfect, as every command you run in this way will always give that : DISPLAY environment variable not set! warning. But this a command line tool, so that doesn't bother me. The fatal error in awk is more serious. It only happens with the screenfetch command, but that undoubtedly has to do with the command running from inside a container, so I ignore that as well.

However, typing libertine-launch -i xenial <program name> every time I want to run one of the tools gets very tedious very quickly. But this being essentially a fully capable Linux OS running on a mobile device, a trick from the desktop transfers seamlessly to the device. Just edit (or create) the .bash_aliases file in the $HOME folder

$ nano .bash_aliases

And add the following lines

alias htop='libertine-launch -i xenial htop'
alias nmap='libertine-launch -i xenial nmap'
alias screenfetch='libertine-launch -i xenial screenfetch'

Log out, log in and then you can use the short form like so:

$ htop

Having a fully functional terminal in the palm of my hand never ceases to amaze me. And with this neat trick I can extend the toolset available to me and tailor it to my specific needs.

Setting up an encrypted /home partition with LVM and LUKS

My shiny new Slimbook Katana II laptop recently arrived. And I am very pleased with its looks. I got the black model.
Mine is fitted with an Intel i7 8th gen processor, 16 GB of RAM, a 120 GB SSD and a secondary HD of 1 TB.
Lots of room to play around with VM's, LXD containers and Docker containers. I also particularly like that it comes with a penguin on the SUPER key, instead of a Windows logo. It also has a badge that says: "Powered by GNU/Linux".

This being a laptop and us living in this day and age, I decided I wanted two things. I wanted the OS to have one logical pool of storage to draw from and I wanted the /home partition to be encrypted. The first requirement can be met with LVM, the second with LUKS. But I had never used the two combined.

Setting up LVM at install time for Ubuntu is quite straight forward. It is one of the options in the setup wizard. Unfortunately the wizard does not provide any obvious way to customize the LVM setup in
a similar way that the Anaconda installer does on Red Hat based distros. Ubiquity only lets you choose which hard disk you want to install the LVM on. I chose the SSD.

At the end of the installation the LVM situation looks as follows:

root@ubuntu:~# pvs
  PV         VG        Fmt  Attr PSize    PFree 
  /dev/sdb1  ubuntu-vg lvm2 a--  <120,00g 44,00m
root@ubuntu:~# lvs
  LV     VG        Attr       LSize   ...
  root   ubuntu-vg -wi-ao---- 119,00g                                                    
  swap_1 ubuntu-vg -wi-ao---- 976,00m
root@ubuntu:~#

The installer allocated one physical volume, /dev/sdb1, to LVM, and assigned it to the ubuntu-vg volume group. Within that volume group it created two logical volumes, named root and swap_1. The root file system / is mounted under root and swap space is mounted under swap_1.

But what about my 1TB disk? Well, I will need to make LVM aware of it myself. Here I will go into some more detail as to how I want the final system to be configured. I want to dedicate  500 GB of disk space to the var logical volume as that is where all the data for the virtual machines and containers will reside. I also thought I would start out with 100GB of space for my home logical volume. Notice that this would not use all of the 1 TB of space I have available. I am using LVM, so I figure I can always resize the logical volume if I need more space. (That should be possible to do, even if the file system on the home logical volume will be encrypted. But I will cross that bridge when I reach it. If it gives me trouble, that will be the subject of another blog post.) I will mostly just jot down the commands that I used without explaining them in depth. Your preferred search engine and the man pages for the fine grained usage details of each command.

Let's get to it then. My SSD was assigned to /dev/sda. I had assigned some space at the beginning for a recovery partition of sorts.
I still need to decide how I want that to work exactly. But the rest of the 1 TB of space I want to add to the ubuntu-vg volume group. So first I booted the 18.04 installation media and elected to enter the live session. There I fired up gparted (because I like me a graphical user interface if I can use one) and created a partition filling the rest of the disk and assigned it the file system lvm2 pv.

Now the physical volume situation looks as follows:

root@ubuntu:~# pvs
  PV         VG        Fmt  Attr PSize    PFree 
  /dev/sda2            lvm2 a--  <916,61g <315,61g
  /dev/sdb1  ubuntu-vg lvm2 a--  <120,00g 44,00m
root@ubuntu:~#

Notice that, while LVM is aware of the new physical volume, it has not been assigned to any volume group (VG) as yet. Let's remedy that:

root@ubuntu:~# vgextend ubuntu-vg /dev/sda2
  Volume group "ubuntu-vg" successfully extended
root@ubuntu:~# pvs
  PV         VG        Fmt  Attr PSize    PFree 
  /dev/sda2  ubuntu-vg lvm2 a--  <916,61g <315,61g
  /dev/sdb1  ubuntu-vg lvm2 a--  <120,00g 44,00m
root@ubuntu:~#

That's better. Now we create our two logical volumes. First we'll tackle var since that just needs to be created and formatted:

root@ubuntu:~# lvcreate -L 500G -n var ubuntu-vg
  Logical volume "var" created
root@ubuntu:~# mkfs -t ext4 /dev/ubuntu-vg/var
  # Output of this command omitted for brevity
root@ubuntu:~#

The making of the home volume group is just slightly more involved, since I want to have encrypted additionally:

root@ubuntu:~# lvcreate -L 100G -n home ubuntu-vg
  Logical volume "home" created
root@ubuntu:~#

Now we will encrypt the volume. First we scramble the contents:

root@ubuntu:~# shred --verbose --random-source=/dev/urandom \
 --iterations=3 /dev/ubuntu-vg/home
   ...

By performing this operation we fill the whole partition with random bytes, making it impossible for a snooper to see what parts of the volume contain data and what parts dont. But be patient. This might take a while to complete. Next we setup the encryption:

root@ubuntu:~# cryptsetup --verbose --cipher aes-xts-plain64 \
 --key-size 512 --hash sha512 --iter-time 5000 \
 --use-random luksFormat /dev/ubuntu-vg/home
WARNING!
========
This will overwrite data on /dev/ubunut-vg/home irrevocably.
 
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.
root@ubuntu:~# 

Be sure to remember the passphrase you entered here. Write it down on a slip of paper and store that in a safe. Or use a password manager. But do not loose it!! You will be in a world of pain if you do.

Now that the encryption is in place, we have to unlock the volume to be able to format it and copy over the contents of /home to it.

root@ubuntu:~# cryptsetup open --type luks \
 /dev/ubuntu-vg/home encrypted_home
Enter passphrase for /dev/ubuntu-vg/home:
root@ubuntu:~# mkfs -t ext4 /dev/mapper/encrypted_home
...

Right... Almost there. Now we make temporary mount points for the root, var and home volume groups to populate the latter two with the contents from the former. Remember that we are doing all of this from a live session:

root@ubuntu:~# mkdir /mnt/{root,var,home}
root@ubuntu:~# mount /dev/ubuntu-vg/root /mnt/root
root@ubuntu:~# mount /dev/ubuntu-vg/var  /mnt/var
root@ubuntu:~# mount /dev/mapper/encrypted_home /mnt/home
root@ubuntu:~# rsync -avz /mnt/root/var/ /mnt/var
...
root@ubuntu:~# rsync -avz /mnt/root/home/ /mnt/home
...
root@ubuntu:~# umount /mnt/{root,var,home}
root@ubuntu:~# cryptsetup close encrypted_home

Notice that I did not clean up the old home and var directories under /mnt/root. As this was a clean install they don't take up so much space. And leaving them put guarantees I have a bootable system if for whatever reason the logical volumes cannot be mounted.

What have we achieved thus far?

• We've extended the volume group to encompass both disks.
• We've added var and home logical volumes to house our /var and /home partitions. 
• We've setup LUKS encryption for the home logical volume.
• We've copied the contents from the old /home and /var directories to the designated logical volumes.

Looks like we're about ready for a reboot, right? Wrong. We still need to have our logical volumes mounted at boot time. We also need to let the kernel know that one of the volumes is encrypted, so that we will be prompted for the password at boot time to unlock it.

First we will notify the kernel of the encrypted volume. We do this by editing /etc/crypttab and adding the following encry:

root@ubuntu:~# nano /etc/crypttab

encrypted_home  /dev/mapper/ubuntu--vg-home none luks,discard

And finally we edit /etc/fstab and add our new mount points:

root@ubuntu:~# nano /etc/fstab

/dev/mapper/ubuntu--vg-var  /var    ext4  errors=remount-ro  0  1
/dev/mapper/encrypted_home  /home   ext4  errors=remount-ro  0  1

The complete file looks like this on my system:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
#                
/dev/mapper/ubuntu--vg-root /       ext4  errors=remount-ro  0  1
/dev/mapper/ubuntu--vg-swap_1 none  swap  sw                 0  0

/dev/mapper/ubuntu--vg-var  /var    ext4  errors=remount-ro  0  1
/dev/mapper/encrypted_home  /home   ext4  errors=remount-ro  0  1

And we're done! Reboot your machine and see if it all works as it should. Mine didn't but that was because I misspelt the name of the /etc/crypttab file. After I corrected the typo all went as it should.

Where have all the comments gone?

I changed the domain of the blog, from arubislander.blogspot.com to blog.arubislander.nl There was a warning that changing the domain would "adversely affect" the existing comments. So I had been warned ...

Rebooting the Blog

It has been five months since the last post to this blog. In the intervening time, I got to thinking what I wanted to do with this space. So I decided to switch focus.

Originally I had intended this blog to document the "journeys of the Starship Enterprise", or at least my journey of discovery in the realm of software development. But I have to face the fact that, either I don't do much discovering I think worth blogging about, or I am very very bad at documenting my discoveries. On the other hand, there have been things I have come accross that have been helpful at the time. But when I want to go back and review them, or re-apply them or dig deeper, I would have trouble finding the resource again.

Because of the above I am going to reboot this blog by repurposing it. The primary audience for it will be myself, future me to be exact. This will be the place where I jot down stuff I find and want to be able to return to for reference.

If someone else also derives some benefit of it, all the better!

Embarking on a journey of exploration and discovery

For the first time ever I will attempt to implement an IETF specification.

Over the next few weeks I hope to document the journey of implementing the Json Web Token specification in Go. I have not checked to see if there already are any existing implementations because that is immaterial to my aims.
I am using this process to gain experience in two areas: Go library creation and specifications implementation.

I will add  a post as I progress in this journey. Hope not the be shipwrecked along the way.

Removing Diacritical Marks

I recently found myself in the situation where I needed to remove the diacritical marks from a string. The poor man's way would have been to substitute the individual letters with their unadorned equivalents (é, è, ê => e; à, á, â => a; etc.) But I was looking for a more elegant solution.

A little googling found me this little gem in C#. The article is in Dutch, but the code speaks for itself:

string q = "Wûnseradiel"; 
char[] normalised = q.Normalize(NormalizationForm.FormD).ToCharArray(); 
q = new string(normalised.Where(c => (int) c <= 127).ToArray()); 
// q == "Wunseradiel" 

The issue is that in Unicode two characters that look the same can have different binary representations (look here for the full details.) The process of giving all equivalent characters the same representation is called normalization. In .NET the String class has a built in method to do just that. The form that is chosen is Normalisation Form D (NFD) which separates each 'accented' character into the unadorned character followed by the diacritical mark. This makes it easy to iterate over the resulting character array and skip all the diacritical marks, leaving us with just the unadorned characters. Elegant and concise. But what if I wanted to apply the same solution in Go?

The heart of the above solution is the ability to normalize a Unicode string as NFD. So I checked the standard libraries for normalization support for Unicode. At the time of writing such support is not included in the standard Go libraries. But it is available as a third party package at: https://code.google.com/p/go.text. The documentation is available here.

To make use of this package, simply cd into the ./src of your GOPATH directory and execute:

hg clone https://code.google.com/p/go.text/

Now the unicode/norm package is available to be used in your own project. The following code illustrates how to achieve the same effect as with the C# snippet above, but fleshed out into an executable project:

package main

import (
 "code.google.com/p/go.text/unicode/norm"
 "fmt"
)

func StripDiacritics(value string) string {
 normalized_value := norm.NFD.String(value)
 var buffer []rune
 for _, char := range normalized_value {
  if char < 128 {
   buffer = append(buffer, char)
  }
 }
 return string(buffer)
}

func main() {
 message := "Buén día, mundo!"
 fmt.Println("message: ", message)
 fmt.Println("stripped: ", StripDiacritics(message))
}

The output:
message: Buén día, mundo!
stripped: Buen dia, mundo!

Go Proxy

For a personal project I'm currently doing I needed to write a proxy server for some audio I had hosted on my server.

Authentication is needed to access the audio on the server. But for a variety of reasons I wanted to grant access to some of the audio without exposing the credentials I use to authenticate to the server. So I figured if I wrote a little Internet facing server to process the requests for audio I could then relay them to my audio server with the appropriate credentials added.

As I am exploring Go (golang.org), I decided I would try my hand at writing what I needed using that. It took some trial, error and exploring the documentation. (If I had done it the other way around, there would have been way less error in the trials, but oh well...)

I wanted to make the audio available for streaming. So I was already dreading having to implement some sort of buffering and what not. But I was pleasantly surprised. Turns out that the part that does the heavy lifting is really quite straight forward thanks to the Go's excellent standard library of packages. All it takes is the code below:

func streamUrl(w http.ResponseWriter, url string) {
 response, _ := http.Get(url)
 defer response.Body.Close()
 reader := io.TeeReader(response.Body, w)
 _, err := ioutil.ReadAll(reader)
 if err != nil {
  log.Println("Error: ", err.Error())
 }
}

This function is passed an http.ResponseWriter stuct and a url. The http.Get() function is invoked which returns a pointer to an http.Response struct the Body of which implements the io.Reader interface. The magic happens thanks to the io.TeeReader function, which takes an io.Reader and an io.Writer, and returns an io.Reader. The documentation states:

TeeReader returns a Reader that writes to w what it reads from r. All reads from r performed through it are matched with corresponding writes to w. There is no internal buffering - the write must complete before the read completes. Any error encountered while writing is reported as a read error. (http://golang.org/pkg/io/#TeeReader)

Which is exactly what I was looking for!

Edit: Thanks to +Michael Gebetsroither for pointing out that io.Copy does the job just as well, while providing cleaner code too!

func streamUrl(w http.ResponseWriter, url string) {
 response, _ := http.Get(url)
 defer response.Body.Close()
 _, err := io.Copy(w, response.Body)
 if err != nil {
  log.Println("Error: ", err.Error())
 }
}