Alrighty! Let’s start doing this again. The plan is to get back to doing these roughly once a month again.

Work I did (1 Jan 2021 - 26 Feb 2021)

Technical

• Published the last version of pip that supports Python 2! 🎉
• Published a few releases of Furo – a Sphinx theme I wrote.
• Made some (unreleased) changes to sphinx-autobuild.
• Made some more progress on installer – a reusable library for Python (wheel) package installation.
• Rewrote get-pip.py and the generation pipeline for it.
• Started a rewrite of pip’s documentation. I’d love to get some feedback on this.

Communication

• I’m experimenting with a new thing: social weekends!
• I presented 2 talks at FOSDEM: in the Python devroom and Open Source Design devroom. Shout-out to Bernard Tyers, for all the help and the bazillion reminders to make sure I do all the things on time. :)
• Collaborating on designing a lockfile format for Python, that can hopefully be standardised for interoperability.

General notes

Onboarding in a new company, relocating internationally, settling into a new space has been… well, it’s all been a very interesting learning experience.

Given the fairly strict lockdown and the percentage of people wearing masks in my locality, I’ve spent a lots of time indoors. Looking forward to the social weekends experiment I’m doing.

What next?

Technical

• pip: Work on the documentation rewrite, hopefully to get it ready in time for the next release.
• pip: Clear some of the backlog on the pull request front.
• pip: General discussions for new features and enhancements.
• TOML: Work on writing that the compliance test suite.
• TOML: Bring toml for Python back from the dead.
• Furo: Make the first stable release.
• Start work on the other Sphinx theme I have in mind.

Communication

• Spend more time looking into the Python lockfile standardisation effort.
• Catch up on the Python-on-Debian saga, and see how I can contribute constructively.

Other commitments

Oh, I have a full time job at Bloomberg now. :)

Our story dates all the way back to 2006, believe it or not. The first steps were taken towards what we know today as containers. We'll discuss their history, how to build them and how to use them. Stick around! you might enjoy the ride.

Concepts

cgroups

What are cgroups ?

Let's find out ! Better yet, let's use the tools at our disposal to find out together…

Open a terminal and run the following command.

man 7 cgroups

This should open the man pages for cgroups.

Control groups, usually referred to as cgroups, are a Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored. The kernel's cgroup interface is provided through a pseudo-filesystem called cgroupfs. Grouping is implemented in the core cgroup kernel code, while resource tracking and limits are implemented in a set of per-resource-type subsystems (memory, CPU, and so on).

What does this all mean ?

This can all be simplified by explaining it in a different way. Essentially, you can think of cgroups as a way for the kernel to limit what you can use.

This gives us the ability to give a container only 1 CPU out of the 4 available to the kernel. Or maybe, limit the memory allowed to 512MB to the container. This way the container cannot overload the resources of the system in case they run a fork-bomb, for example.

But, cgroups do not limit what we can "see".

namespaces

Namespaces to the rescue !

As we did before, let's check the man page for namespaces

man 7 namespaces

A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. One use of namespaces is to implement containers.

Wooow ! That's more mumbo jumbo ?!

Is it really simple ?

Let's simplify this one as well.

You can think of namespaces as a way for the kernel to limit what we see.

There are multiple namespaces, like the cgroup_namespaces which virtualizes the view of a process cgroup. In other words, inside the cgroup the process with PID 1 is not PID on the system.

The namespaces manual page lists them, you check them out for more details. But I hope you get the gist of it !

Linux Containers

We are finally here! Let's talk Linux Containers.

The first topic we need to know about is images.

What are container images ?

We talked before that Docker came in and offered tooling around containers.

One of those concepts which they used, in docker images, is layers.

First of all, an image is a file-system representation of a container. It is an on-disk, read-only, image. It sort of looks like your Linux filesystem.

Then, layers on top to add functionality. You might ask, what are these layers. We will see them in action.

Let's look at my system.

lsb_release -a

LSB Version:	n/a
Distributor ID:	ManjaroLinux
Description:	Manjaro Linux
Release:	20.2.1
Codename:	Nibia

As you can see, I am running Manjaro. Keep that in mind.

Let's take a look at the kernel running on this machine.

uname -a

Linux manjaro 5.10.15-1-MANJARO #1 SMP PREEMPT Wed Feb 10 10:42:47 UTC 2021 x86_64 GNU/Linux

So, it's kernel version 5.8.6. Remember this one as well.

• neofetch
  

  I would like to test a tool called neofetch. Why ?

  • First reason, I am not that creative.
  • Second, it's a nice tool, you'll see.

  We can test neofetch

  neofetch
  

  fish: Unknown command: neofetch
  

  Look at that! We don't have it installed… Not a big deal. We can download an image and test it inside.

Pulling an image

Let's download a docker image. I am using podman, an open source project that allows us to use containers.

Note

You might want to run these commands with sudo privileges.

podman pull ubuntu:20.04

f63181f19b2fe819156dcb068b3b5bc036820bec7014c5f77277cfa341d4cb5e

Let's pull an Ubuntu image.

As you can see, we have pulled an image from the repositories online. We can see further information about the image.

podman images

REPOSITORY                TAG     IMAGE ID      CREATED      SIZE
docker.io/library/ubuntu  20.04   f63181f19b2f  5 weeks ago  75.3 MB

Much better, now we can see that we have an Ubuntu image downloaded from docker.io.

What's a container then ?

A container is nothing more than an instance of an image. It is the running instance of an image.

Let's list our containers.

podman ps -a

CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

We have none. Let's start one.

podman run -it ubuntu:20.04 uname -a

Linux 57453b419a43 5.10.15-1-MANJARO #1 SMP PREEMPT Wed Feb 10 10:42:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

It's running the same kernel as our machine… Are we really inside a container ?

podman run -it ubuntu:20.04 hostname -f

6795b85eeb50

okay ?! And our hostname is ?

hostname -f

manjaro

Hmm… They have different hostnames…

Let's see if it's really Ubuntu.

podman run -it ubuntu:20.04 bash -c 'apt-get update && apt-get install -y vim'

Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
...
Setting up libpython3.8:amd64 (3.8.5-1~20.04.2) ...
Setting up vim (2:8.1.2269-1ubuntu5) ...
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vim (vim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vimdiff (vimdiff) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rvim (rvim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vi (vi) in auto mode
...
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in auto mode
...
Processing triggers for libc-bin (2.31-0ubuntu9.1) ...

This should not work on my Manjaro. apt-get is not a thing here. Well, the output is a bit large so I truncated it a bit for readability but we seem to have installed vim successfully.

Building a container image

Now that we saw what an image is and what a container is. We can explore a bit inside a container to see it more clearly.

So, what can we do with containers? We can use the layering system and the docker created tooling to create them and distribute them.

Let's go back to our neofetch example.

I want to get an Ubuntu image, then install neofetch on it.

First step, create a Dockerfile in your current directory. It should look like this.

FROM ubuntu:20.04

RUN apt-get update && \
    apt-get install -y neofetch

This file has two commands:

• FROM designates the base image to use. This is the base image we will be building upon. In our case, we chose Ubuntu:20.04. You can find the images on multiple platforms. To mention a few, we have Dockerhub, Quay.io and a few others.

  By default, this downloads from Dockerhub.

• RUN designates the commands to run. Pretty simple. We are running a couple of commands that should be very familiar to any user familiar with debian-based OS's.

Now that we have a Dockerfile, we can build the container.

podman build -t neofetch-ubuntu:20.04 -f Dockerfile.ubuntu .

STEP 1: FROM ubuntu:20.04
STEP 2: RUN apt-get update &&     apt-get install -y neofetch
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
...
Fetched 17.2 MB in 2s (7860 kB/s)
Reading package lists...
...
The following additional packages will be installed:
  chafa dbus fontconfig-config fonts-dejavu-core fonts-droid-fallback
  fonts-noto-mono fonts-urw-base35 ghostscript gsfonts imagemagick-6-common
  krb5-locales libapparmor1 libavahi-client3 libavahi-common-data
  libavahi-common3 libbsd0 libchafa0 libcups2 libdbus-1-3 libexpat1
  libfftw3-double3 libfontconfig1 libfreetype6 libglib2.0-0 libglib2.0-data
  libgomp1 libgs9 libgs9-common libgssapi-krb5-2 libicu66 libidn11 libijs-0.35
  libjbig0 libjbig2dec0 libjpeg-turbo8 libjpeg8 libk5crypto3 libkeyutils1
  libkrb5-3 libkrb5support0 liblcms2-2 liblqr-1-0 libltdl7
  libmagickcore-6.q16-6 libmagickwand-6.q16-6 libopenjp2-7 libpaper-utils
  libpaper1 libpng16-16 libssl1.1 libtiff5 libwebp6 libwebpmux3 libx11-6
  libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxml2 poppler-data
  shared-mime-info tzdata ucf xdg-user-dirs
Suggested packages:
  default-dbus-session-bus | dbus-session-bus fonts-noto fonts-freefont-otf
  | fonts-freefont-ttf fonts-texgyre ghostscript-x cups-common libfftw3-bin
  libfftw3-dev krb5-doc krb5-user liblcms2-utils libmagickcore-6.q16-6-extra
  poppler-utils fonts-japanese-mincho | fonts-ipafont-mincho
  fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai
  fonts-arphic-uming fonts-nanum
The following NEW packages will be installed:
  chafa dbus fontconfig-config fonts-dejavu-core fonts-droid-fallback
  fonts-noto-mono fonts-urw-base35 ghostscript gsfonts imagemagick-6-common
  krb5-locales libapparmor1 libavahi-client3 libavahi-common-data
  libavahi-common3 libbsd0 libchafa0 libcups2 libdbus-1-3 libexpat1
  libfftw3-double3 libfontconfig1 libfreetype6 libglib2.0-0 libglib2.0-data
  libgomp1 libgs9 libgs9-common libgssapi-krb5-2 libicu66 libidn11 libijs-0.35
  libjbig0 libjbig2dec0 libjpeg-turbo8 libjpeg8 libk5crypto3 libkeyutils1
  libkrb5-3 libkrb5support0 liblcms2-2 liblqr-1-0 libltdl7
  libmagickcore-6.q16-6 libmagickwand-6.q16-6 libopenjp2-7 libpaper-utils
  libpaper1 libpng16-16 libssl1.1 libtiff5 libwebp6 libwebpmux3 libx11-6
  libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxml2 neofetch poppler-data
  shared-mime-info tzdata ucf xdg-user-dirs
0 upgraded, 66 newly installed, 0 to remove and 6 not upgraded.
Need to get 36.2 MB of archives.
After this operation, 136 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 fonts-droid-fallback all 1:6.0.1r16-1.1 [1805 kB]
...
Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 neofetch all 7.0.0-1 [77.5 kB]
Fetched 36.2 MB in 2s (22.1 MB/s)
...
Setting up ghostscript (9.50~dfsg-5ubuntu4.2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.1) ...
STEP 3: COMMIT neofetch-ubuntu:20.04
--> 6486fa42efe
6486fa42efe5df4f761f4062d4986b7ec60b14d9d99d92d2aff2c26da61d13af

Note

You might need sudo to run this command.

As you can see, we just successfully built the container. We also got a hash as a name for it.

If you were careful, I used the && command instead of using multiple RUN. You can use as many RUN commands ase you like. But be careful, each one of those commands creates a layer. The more layers you create, the more time they require to download*/*upload. It might not seem to be a lot of time to download a few extra layer on one system. But if we talk about container orchestration platforms, it makes a big difference there.

Let's examine the build a bit more and see what we got.

STEP 1: FROM ubuntu:20.04
STEP 2: RUN apt-get update &&     apt-get install -y neofetch

The first step was to download the base image so we could use it, then we added a layer which insatlled neofetch. If we list our images.

podman images

REPOSITORY                 TAG     IMAGE ID      CREATED        SIZE
localhost/neofetch-ubuntu  20.04   6486fa42efe5  5 minutes ago  241 MB
docker.io/library/ubuntu   20.04   f63181f19b2f  5 weeks ago    75.3 MB

We can see that we have localhost/neofetch-ubuntu. If we examine the ID, we can see that it is the same as the one given to us at the end of the build.

Running our container

Now that we created a brand-spanking-new image, we can run it.

podman images

REPOSITORY                 TAG     IMAGE ID      CREATED        SIZE
localhost/neofetch-ubuntu  20.04   6486fa42efe5  6 minutes ago  241 MB
docker.io/library/ubuntu   20.04   f63181f19b2f  5 weeks ago    75.3 MB

First we list our images. Then we choose which one to run.

podman run -it neofetch-ubuntu:20.04 neofetch

Text

neofetch is installed in that container, because the image has it.

We can also build an image based on something else, maybe Fedora ?

I looked in Dockerhub (Fedora) and found the following image.

FROM fedora:32

RUN dnf install -y neofetch

We can duplicate what we did before real quick. Save file, run command to build the image.

podman build -t neofetch-fedora:20.04 -f Dockerfile.fedora .

Then, run the container.

podman run -it neofetch-fedora:20.04 neofetch

Text

So the new OnionShare is out and it has a bunch of exciting new features and some improvements in the UI/UX designs of the tool. One of the main new features that I helped build was the anonymous chat feature in OnionShare. Just like the other modes (share, receive, and website), there is now a chat mode. So if you want to start a chat service, you just start the chat server, share the onion address of the server with people you want to chat with, everyone opens this onion address in Tor Browser and voila! You have an anonymous chat.

Let's dive in a little deeper into the feature.

Why do we need an anonymous chat?

A common question that we got during developing this feature is what's the use of an anonymous chat room since we already have end-to-end encrypted messaging apps. It leaves a lot fewer traces.

The way we achieve this is very simple. There is no form of storage whatsoever in OnionShare chat mode. The chat is not persistent. The chat server stores no information at all (not even the usernames of people chatting). So once the chat server is closed, and the Tor Browser tab with the chat client is closed, there is no data (or metadata) related to chat that remains, even in the person's system who started the server. Hence, it leaves much less trace compared to other chat applications.

A good example of the above as mentioned by Micah in his blog is:

If, for example, you send a message to a Signal group, a copy of your message ends up on each device (the devices, and computers if they set up Signal Desktop of each member of the group). Even if disappearing messages is turned on it’s hard to confirm all copies of the messages are actually deleted from all devices, and from any other places (like notifications databases) they may have been saved to. OnionShare chat rooms don’t store any messages anywhere, so the problem is reduced to a minimum.

Given that the OnionShare chat feature works over the onion network, so it also has the additional anonymity feature. Also, adding to the anonymity feature, OnionShare chat doesn't need any form of signing in. Hence, people chatting can stay anonymous, and everything happens inside the tor network. One can just start a chat server, share the link via some disposable way, and then wait for the other people to join while maintaining anonymity.

Because it's an onion service, there is no way for an attacker to eavesdrop on the messages. The closest they can get is if they run a malicious Tor rendezvous node that's chosen for the onion service, they'll be able to spy on encrypted onion traffic. So, there's no capturing ciphertext to decrypt later on.

So what happens under the hood?

The chat feature is dependent on flask-socketio and eventlet for the WebSocket server implementation, and socket.io client js for the frontend implementation of the chat client. So when a chat server is started, the WebSocket is started in a namespace "/chat". Whenever a new user joins the link, they are given a randomly generated username and they are added to the room "default". There is only one room, and the actual name of the room can be set from the OnionShare settings-related code, but it doesn't really impact anything in the implementation. Both the room name and the randomly generated username are stored in a flask session. But that information is also completely gone once the chat server is stopped. The room and username information are only there to emit the messages properly.

You can also change the randomly generated username to a username (or pseudo username) of your choice for that particular session.

There are two main types of messages:

1. status messages - these are sent from the client to the server only when a new user joins or someone updates their username. The status message is then broadcasted to all the other connected clients, who will then see it as a form of a status message in the chat window.

2. user messages - these are sent when a user sends a message. All messages are broadcasted, so in case you share the link to multiple users, there is no concept of private message and everyone connected to the room can view your messages. Hence, sharing the onion link securely is important.

All of these WebSocket communication happens over the Tor onion services. OnionShare in itself doesn't implement any encryption algorithm to the chat and heavily relies on the Tor onion service's encryptions for the same. The message from the client to the OnionShare server is E2EE as it goes via Tor's onion connection. Then the OnionShare server broadcasts the message to all the other clients connected to the chat room through their E2EE onion connection, over WebSockets.

So what now?

I feel, as of now, the OnionShare anonymous chat is great if you quickly want to have an anonymous, secure, non-persistent conversation with someone or a group of people. It is also great if a whistleblower wants to share some details over chat with a journalist and then remove all traces of that conversation completely. But I feel if someone needs to host a chat server for a long time where people can connect anonymously, this is probably not the best solution for that.

There are still some issues that we will hopefully improve in the next releases. Firstly, we need to try and make it a bit more asynchronous. Right now, if left inactive for a long time, the Tor connection over WebSocket is sometimes dropped which isn't a great user experience. We are working on improving that.

Also, we will improve the UI for the chat more to give a better user experience.

With the new tabs feature, one can have all different modes (even multiple servers of same mode) running at the same time. So you can have a chat server and share mode server running at the same time. All the modes are very independent of each other and hence don't affect one another in any way.

I hope you all enjoy the new chat feature and leave feedbacks/suggestions on how to improve it. You can also read more about this and other features at docs.onionshare.org

Background

I recently get really pissed quite often when I browse the web; I'm nuked with ads and pop-ups or - should I dare to use an adblocker - will not be able to see the content I was looking for at all. With the GDPR in place, living in Europe also reveals the incredible amount of Cookies used for profiling and tracking my browsing behavior. I'm being tracked everywhere, always.

Sure, I can opt-out. That's what they say. And it's a bad joke.

I cannot opt-out of what is called "essential to make the site run", which is obviously a whole lot. Technically it's forbidden to make the opt-out process harder then the opt-in, but I have never seen a single of these modal dialogs that would have offered me an "opt-out of everything" button that was just as easy to find as the "I agree to everything" button. Now even if I opt-out of everything, sites hide their "legitimate interest" settings on another page where I often need to manually de-select zillion items.

That's not the web I learned to love and I want to use as a major source of daily information.

I also recently learned about a new internet protocol that aims to make things different than HTTP/S, called gemini. If you're familiar with the gopher protocol, which has been around for some time now, you can put gemini somewhere between gopher and HTTP/S. By intention it does not much. It will let you display text, basically, that you type in a markdown-like syntax called gemtext. The function set is very limited on purpose, so content counts. It does not support images or inline links, but three levels of headings and code blocks. gemini does also TLS by default and explicitly welcomes self-signed certificates.

Hosting a Gemlog

To serve a gemlog, a server is needed; since TLS is a requirement, it's not sufficient to just serve the file structure. There are a couple of lightweight servers available, I'm using gemserv here, which I found fairly easy to compile and configure.

Generating a Certificate

I use openssl to generate a private key and a certificate signing request (CSR):

openssl req -new -newkey rsa:4096 -nodes \
  -keyout private_key.txt \
  -out csr.txt \
  -subj "/CN=gemini.schubisu.de"

This will generate an X.509 certificate signing request with a new RSA 4096 bit key. The nodes option will skip encryption of the key. You can put more information in the subject name, it's important to have the CN right, however, to let it match your domain.

With the CSR we can generate the certificate:

openssl x509 -req -sha256 \
  -days 365 \
  -in csr.txt \
  -signkey private_key.txt \
  -out certificate.txt \
  -extensions req_ext

The private_key.txt and certificate.txt will need to be on our server.

Configure gemserv

If available in your packet manager, install gemserv from there, or clone the repository and follow the build instructions. The configuration can be as simple as the following config.toml file:

port = 1965
host = "0.0.0.0"

[[server]]
hostname = "gemini.schubisu.de"
key = "/path/to/private_key.txt"
cert = "/path/to/certificate.txt"
dir = "/path/to/content"

and simply run

/path/to/gemserv config.toml

For local testing you don't even need to have a registered domain, just add a line like this to your /etc/hosts file:

127.0.0.1    gemini.schubisu.de    localhost

Converting my lektor blog to gemtext

I write my blog with a lightweight CMS for static websites written in python, called lektor. I always liked about lektor that it's file structure mirrors the resulting web page. Every page I write is represented by a folder that contains at least one content.lr file which consists of a bit yaml and markdown.

This is very much like what gemini expects, which will either show you the subfolders of a directory, or - if present - renders the index.gmi (or index.gemini) file.

I started by renaming all contents.lr files to index.gemini to get the quickest possible working gemlog and an initial idea of what work lies ahead of me. It works, of course, but there is still plenty of room for improvement. What needs to be adjusted are

• Meta data: The contents.lr starts with a yaml block of meta data which could be anything. In my case it's the title, publication date and some tags.
• Inline links and links in general: Gemini does not allow inline links, so my links need to be converted to the correct format.
• Navigation: I want to allow quick navigation to previous and next blog posts, as well as returning to an overview.

I decided to write a small python script, to use lektor's API to access all meta information and contents. I also reviewed all my blog posts to bring the links into a standardized format:

 # inline links:
 [some text][identifyer:my_link_name]

 # and at the bottom of the document:
 [identifyer:my_link_name]: https://...

The script will convert that into valid gemini links that are numerically indexed inline when rendered:

# inline links:
some text[1]

# and just below the paragraph:
[1] my_link_name [some text]

I also edited a few minor things in my contents.lr to make the markdown comply with gemtext:

• gemtext does not allow nested bullet lists, so I need to reduce lists to one level wherever I used more
• Different than with when I convert markdown to HTML, gemtext will break a line wherever I inserted a line break. If I want something to show as a nice paragraph, it should not be interrupted by line breaks.
• Inline text formatting to emphasize some words is not possible in gemtext. However, I find markdown with inline formatting still readable enough, so I left that as is.
• I found a whole lot of typos :)

I've uploaded the script to convert the blog here. It's pretty much tailored to my needs. However, it may be useful if you happen to run into the same or similar idea :)

For what it's worth, I can now simply run my conversion script, rsync the files to my gemini server and that's it.

You can grab a gemini browser and checkout the results of my conversion at gemini://gemini.schubisu.de

Background

So being in lockdown means my son is into homeschooling for some weeks now. The school decided to use a school information system (SIS) that combines lots of free software projects, which is good. The video conferencing tool that's implemented is BigBlueButton which works very well. The children aged 6-8 all have learnt how to activate or deactivate their cameras, mute or unmute their microphones, use the chat or set their status. They follow invitations to break-out rooms and the classes in general are progressing really well.

It it wasn't for unexpected outage of the whole SIS; the company recently released a statement that explains why the servers have been unreachable for a couple of days. Apparently the infrastructure suffers from DDoS (destributed denial of service) attacks that eventually brings down the servers.

For most schools in our city, the solution is obvious: change the service to Microsoft Teams. I could write a book about why switching to Teams is a bad idea and why Microsoft should be banned from all schools everywhere. However, what I want to do is to show how a backup video conferencing solution can be provided easily within a couple of minutes.

I'm going to use the same software BigBlueButton, hosted on Hetzner (a German hoster) virtual machines and managed with Ansible.

Single Server Setup

To set up a single BigBlueButton server, we start with creating a VM. The documentation recommends an Ubuntu 16.04 server with 16GB RAM and 8 cores minimum for production. The documentation also recommends using a provided shell script, that will do the full installation for you which can be configured with several flags and switches. However, there are also some Ansible roles out there, that will allow me to have more control in my initial configuration. The Ansible roles are not maintained by the official BigBlueButton project.

For development, I'm going to use a 4CPU machine with 8GB RAM. With the hcloud command line tool you can create the VM with

hcloud server create \
    --image ubuntu-16.04 \
    --name bbb \
    --ssh-key ~/.ssh/id_rsa.pub \
    --type cx31 \
    --datacenter nbg1-dc3

3s [=====================================] 100.00%
Waiting for server xxxxxx to have started 
   ... done

Server xxxxxx created
IPv4: x.x.x.x

So there is our VM, with my ssh key already injected. Next we need a domain to reach the server properly and have the scripts create Let's Encrypt SSL certificates. I've bought my domain at a German provider and set-up a sub-domain to use dynamic dns which I'm going to update with the ddclient tool.

My ddclient.conf file looks similar to this:

protocol=dyndns2
ssl=yes
server=dyndns.strato.com/nic/update

use=cmd
cmd="hcloud server ip bbb"
login=schubisu.de
password='***********'
bbb.schubisu.de

use=cmd
cmd="hcloud server ip -6 test.debian"
login=schubisu.de
password='***********'
bbb.schubisu.de

As you can see, I've named my sub-domain bbb.schubisu.de. The ip I'm going set in the A and AAAA record are obtained through the hcloud command. To set both records, I simply copy the blocks; ddclient can automatically recognize the ip address format and will set the IPv4 and IPv6 correctly.

To update the dns records I run the command with

ddclient --file ddclient.conf --cache ddclient.cache

we need to specify the cache file separately to be able to run ddclient without root, since the default cache location is only writable as root.

Finally we can set up our Ansible scripts. I'm going to use n0emis bigbluebutton role. To obtain the role and dependencies let's clone some git repositories:

git clone https://github.com/n0emis/ansible-role-bigbluebutton n0emi.bigbluebutton
git clone https://github.com/geerlingguy/ansible-role-nodejs geerlingguy.nodejs
git clone https://github.com/geerlingguy/ansible-role-docker geerlingguy.docker

I put the following in my inventory file

[bbb]
bbb.schubisu.de

and also create an ansible.cfg with the following content

[defaults]
remote_user = root

because my ssh key has been injected to the root user of the VM.

Finally I can create the playbook.yml:

---
- hosts: bbb
  roles:
    - role: n0emis.bigbluebutton
      bbb_hostname: bbb.schubisu.de
      bbb_coturn_secret: 123456789012345678901234567890
      bbb_greenlight_secret: 123456789012345678901234567890
      bbb_greenlight_db_password: 123456789012345678901234567890
      bbb_letsencrypt_email: john.doe@email.com
      bbb_greenlight_accounts: false
      bbb_greenlight_users:
        - name: 'John Doe'
          email: 'john.doe@email.com'
          password: '**********'
          type: 'admin'

Most options have default values and I don't need to modify them for now. Secrets however need to be set. To have the Let's Encrypt certificates I also set the correct hostname and email address.

Finally I configure greenlight, the optional login manager used by BigBlueButton, to not accept new registrations but have me as single admin user to spawn new rooms etc.

Let it run!

ansible-playbook -i server playbook.yml

This will run for about half an hour, but when it's finished we have our BigBlueButton server available at our domain. Once this is set up we can easily delete and create our VM whenever we want. Setting up the full-blown server will always take that 15-30 minutes but does not need any interaction on our end. That's cool!

Scale it up

Using the minimum server requirements with 16GM RAM and 8CPUs, the BBB server can handle up to 200 concurrent users. That said, it's clear that this is not the end. I will write a follow-up to this post, describing how to set up a set of BigBlueButton back end servers with a load-balancing front end.

Yesterday Alex Birsan posted a blog explaining how he made a supply chain attack on various companies via dependencies. I was waiting for this blog from last August when we noticed the mentioned packages on PyPI (and removed). I reached out to Alex to figure out more about the packages, and he said he will write a blog post.

This is the same time when @rsobers also tweeted about how any similar attack works via DNS.

At SecureDrop project, we spend a lot of time figuring out a way to defend against similar attacks. My PyCon US 2019 talk explained the full process. In simple words, we are doing the following:

• Verify the source of every python dependency before adding/updating them. This is a manual step done by human eyes.
• Build wheels from the verified source and store them into a package repository along with OpenPGP signed sha256sums.
• Before building the Debian package, make sure to verify the OpenPGP signature, and also install only from the known package repository using verified (again via OpenPGP signature) wheel hashes (from the wheels we built ourselves).

Now, in the last few months, we modified these steps, and made it even easier. Now, all of our wheels are build from same known sources, and all of the wheels are then built as reproducible and stored in git (via LFS). The final Debian packages are again reproducible. Along with this and the above mentioned OpenPGP signature + package sha256sums verification via pip. We also pass --no-index the argument to pip now, to make sure that all the dependencies are getting picked up from the local directory.

Oh, and I also forgot to mention that all of the project source tarballs used in SecureDrop workstation package building are also reproducible. If you are interested in using the same in your Python project (or actually for any project's source tarball), feel free to have a look at this function.

There is also diff-review mailing list (very low volume), where we post a signed message of any source package update we review.

The format for chattr: $ chattr [+|-|=mode] filename

The general format for mkfs: mkfs [-t fstype] [options] [device-file]
where [device-file] is usually a device name like /dev/sda3 or /dev/vg/lvm1

fsck [-t fstype] [options] [device-file]
where [device-file] is usually a device name like /dev/sda3 or /dev/vg/lvm1.

• You rarely need to scan the entire partition for errors, as everything but the very last transaction has been logged and confirmed, so it takes almost no time to check.​
• Even if you do check the whole filesystem, newer filesystems have been designed with fast fsck in mind; older filesystems did not think much about this when they were designed as sizes were much smaller.

The followind command can be used : $ sudo fsck -t ext4 /dev/sda10

The general format for mount is : $ mount [options] (source) (directory)

$ umount [device-file | mount-point]

Content Security Policy (or CSP) is a way of avoiding certain types of website-related attacks like cross-site scripting and malicious data injections. It is a way by which website developers can tell the browser what content origins are approved so that everything else is blocked. One needs to add a Content-Security-Policy HTTP header mentioning the sources which they allow for loading scripts, styles, images, etc.

To read in detail about CSP, check Content Security Policy Level 3 working draft.

We are going to discuss here why sha256 hashes often don't let inline styles to not pass in chromium browsers. Chromium browser console complains about the style-src hashes mismatch even though it shows them to be the same. Why? And how to solve it?

TL;DR: If using <style>, use style-src. If using style="" attribute in HTML tag, use style-src-attr

Now, if you are interested in more information, let's dive a little deeper into what's going on.

Hashes to allow inline styles & scripts

The usual practice of having a tight, secure CSP is to not allow any inline style or inline scripts. This helps mitigate malicious scripts entered via data injection from getting executed.

When I say inline scripts, one might understand 2 different scenarios:

<!-- Scenario 1 -->
<script>alert('Hello world');</script>

or

<!-- Scenario 2 -->
<button onclick="alert('Hello world');">
    Click me!
</button>

Now, the easiest way to allow this would be to add unsafe-inline in script-src of the CSP. But then we are back to the problem of executing malicious scripts entered by data injection. There are two ways to still allow only these scripts to work: nonce and sha256 hashes. We are going to talk about sha256 hashes here.

The idea is to get the sha256 hash of the entire script and add it to the script-src. So in this case, it would be something like this:

script-src 'self' 'sha256-DUTqIDSUj1HagrQbSjhJtiykfXxVQ74BanobipgodCo='

You can get the hash from https://report-uri.com/home/hash. Also, chromium browsers will usually show the hash that should be added for a particular inline script.

The Problem

Now, all this sounds good, and in Firefox, just adding the above to your CSP will make both the scripts to work. However, in chromium, the above CSP will work only in Scenario 1 but not in Scenario 2. You can read more about the discussion here: https://bugs.chromium.org/p/chromium/issues/detail?id=546106#c8

In JavaScript, I think in general scenario 1 will be much more encouraged than scenario 2. So scenario 2 might not be encountered that often. However, the situation changes, when it comes to styles (or CSS)

In case of inline styles, following are the scenarios:

<!-- scenario 1 -->
<style>p{color: blue;}</style>

and

<!-- scenario 2 -->
<p style="color: blue;">This is a text</p>

In CSS, the second scenario is much more common when someone does inline styles than scenario 1. But again, in this case, adding a sha256 hash to style-src won't execute the scenario 2 in chromium browsers.

This is because styles added in scenario 2 are part of the style attribute in the HTML tag which in CSP terms are essentially event handlers. According to w3c CSP draft, the hash in style-src allows the inline styles mentioned inside <style> tag to pass but doesn't allow event handlers (as is the case in scenario 2). There's more on this discussion here.

So it's a feature?

Yes, it is a feature. In chromium browsers, adding a hash to style-src only allows any inline style written inside the <style> tags to execute. This is by design. If you need to execute the inline styles present in style= attribute of HTML tags, you need to use another directive in CSP called style-src-attr. Similarly, script-src-attr should be used if you are doing JavaScript event handling in the HTML tag itself.

So, for example, if you want to only allow an inline CSS such as this:

<p style="color: blue;">This is a text</p>

all you need to do is put the sha256 hash in style-src-attr along with 'unsafe-hashes'. This will tell the browser to allow any inline style, with the hashes that you added in style-src-attr to be executed.

So the CSP will have something like this:

style-src-attr 'unsafe-hashes' 'sha256-C8uD/9cXZAvqgnwxgdb67jgkSDq7f8xjP8F6lhY1Gtk='

And, that's it! That will do the trick in any chromium browser. The related code for chromium can be found here. According to caniuse.com, all chromium browsers above 75 supports this behaviour.

Even though firefox still doesn't have support for style-src-attr but it allows inline styles and scripts of all types to pass based on style-src and script-src hashes. So as long as the hash is mentioned in both style-src and style-src-attr, it should work in most of the browsers.

As for the explanation behind why 'unsafe-hashes', there is a pretty good explainer document written by andypaicu talking about exactly this.

Also, read more about style-src-attr in detail in the w3c draft to understand exactly what's happening and what kind of risk it may still pose.

PS: Inline JavaScript event handlers using script-src-attr can be very risky given an attacker can trigger a passing javascript from within an unrelated HTML tag.

Generating OpenPGP keys in an offline air-gapped system and then moving them into a smart card is always a difficult task for me. To remember the steps and command-line options of gpg2 correctly and then following them in the same order is difficult, and I had trouble enough number of times in doing so when I think about someone who is not into the command line that much, how difficult these steps are for them.

While having a chat with Saptak a few weeks ago, we came up with the idea of writing a small desktop tool to help. I started adding more features into my Johnnycanencrypt for the same. The OpenPGP operations are possible due to the amazing Sequoia project.

Introducing Tumpa

The work on the main application started during the holiday break, and today I am happy to release 0.1.0 version of Tumpa to make specific OpenPGP operations simple to use. It uses Johnnycanencrypt inside, and does not depend on the gpg.

Here is a small demo of the application running in a Tails (VM) environment. I am creating a new OpenPGP key with encryption and signing subkeys, and then putting them into a Yubikey. We are also setting the card holder's name via our tool.

We can also reset any Yubikey with just a click.

You can download the Debian Buster package for Tails from the release page from Github. You can run from the source in Mac or Fedora too. But, if you are doing any real key generation, then you should try to do it in an air-gapped system.

You can install the package as dpkg -i ./tumpa_0.1.0+buster+nmu1_all.deb inside of Tails.

What are the current available features?

• We can create a new OpenPGP key along with selected subkeys using Curve25519. By default, the tool will add three years for the expiration of the subkeys.
• We can move the subkeys to a smart card. We tested only against Yubikeys as that is what we have.
• We can set the name and public key URL on the card.
• We can set the user pin and the admin pin of the smart card
• We can reset a Yubikey.
• We can export the public key for a selected key.

What is next?

A lot of work :) This is just the beginning. There are a ton of features we planned, and we will slowly add those. The UI also requires a lot of work and touch from a real UX person.

The default application will be very simple to use, and we will also have many advanced features, say changing subkey expiration dates, creating new subkeys, etc. for the advanced users.

We are also conducting user interviews (which takes around 20 minutes of time). If you have some time to spare to talk to us and provide feedback, please feel free to ping us via Twitter/mastodon/IRC.

We are available on #tumpa channel on Freenode. Come over and say hi :)

There are a lot of people I should thank for this release. Here is a quick list at random. Maybe I miss many names here, but you know that we could not do this without your help and guidance.

• Sequoia team for all the guidance on OpenPGP.
• Milosch Meriac for providing the guidance (and a ton of hardware).
• Vincent Breitmoser, for keep explaining OpenKeyChain codebase to me to understand smart card operations
• Anwesha Das for fixing the CI failures for Johnnycanencrypt, and documentation PRs.
• Harlo and Micah, for all the amazing input for months.
• Saptak Sengupta for being the amazing co-maintainer.

I’m actually really good at procrastinating things when I’m uncomfortable.
- me, back in Feb 2020

Prelude

As you’re likely aware, the human world has been quite a different place since my last post here. My transition to this change wasn’t super smooth, but hey, it could’ve been a lot worse. My mental health has definitely had a really fun roller coaster ride. I have been fortunate enough to be able to stay safe and also work remotely with some wonderful folks over this time.

I wrapped up my 3 month GSoC project, after 3 years

As you might know, I’ve been working on pip’s dependency resolver since 2017, which was originally my Google Summer of Code project. There is now a public pip release that uses a new, written-from-the-ground-up dependency resolver, that replaces the old pseudo-dependency-resolver it had. 🎉

A large part of why this project was finally pushed over the line was that the Packaging-WG at the Python Software Foundation was able to secure funding toward this, from Mozilla (through its Mozilla Open Source Support Awards) and the Chan Zuckerberg Initiative.

Huge thanks to Bernard Tyers, Ernest W. Durbin III, Georgia Bullen, Nicole Harris, Paul Moore, Sumana Harihareshwara and Tzu-Ping Chung for being amazing colleagues (and friends!) as we worked on this together.

I have a college degree now

Hurray! I made it out of that place. As a faculty once told me, “you got to where you are despite this college, not because of it”.

My college has made me sign paperwork that prohibits me from speaking out against university management (something that should speak for itself). Here’s a link to a relevant section on Wikipedia.

Update: I had to change the link above to a permalink to a specific version of the document, because the linked section was edited out the day I put up this post.

I mentored a GSoC student

I mentored Raphael McSinyx, who worked on pip, exploring speedups to pip’s dependency resolution process. You can read more about their work in his final GSoC report.

I relocated for a job

I am now working at Bloomberg Engineering as, currently, a part of the Python Infrastructure team in London.

I’m officially no longer living in my parent’s home. Onward to new adventures, I guess.

I am doing Sphinx-y things now

I made a Sphinx documentation theme: Furo, modernised sphinx-themes.org and am collaborating with the amazing folks of the Executable Books project.

I went to San Francisco

CZI conducted an EOSS kickoff meeting, bringing lots of members of their first cohort of 32 EOSS grantees into one room. This included me!

It was a really well conducted event, and the room was filled with brilliant people. I felt like a squirrel in a room full of elephants.

This happened right before COVID-19 was deemed serious enough to care about it. It was the last trip I had in the before-times, and it was definitely a good one.

The blog post "Email setup with isync, notmuch, afew, msmtp and Emacs" prompted a few questions. The questions were around synchronizing email in general.

I did promise to write up more blog posts to explain the pieces I brushed over quickly for brevity and ease of understanding. Or so I thought !

$ ls
cur/  new/  tmp/

IMAPAccount Personal
Host email.hostname.com
User personal@email.hostname.com
Pass "yourPassword"
# One can use a command which returns the password
# Such as a password manager or a bash script
#PassCmd sh script/path
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore personal-remote
Account Personal

MaildirStore personal-local
Subfolders Verbatim
Path ~/.mail/
Inbox ~/.mail/Inbox

Channel sync-personal-inbox
Master :personal-remote:"Inbox"
Slave :personal-local:Inbox
Create Slave
SyncState *
CopyArrivalDate yes

Channel sync-personal-archive
Master :personal-remote:"Archive"
Slave :personal-local:Archive
Create Slave
SyncState *
CopyArrivalDate yes

Channel sync-personal-sent
Master :personal-remote:"Sent"
Slave :personal-local:Sent
Create Slave
SyncState *
CopyArrivalDate yes

Channel sync-personal-trash
Master :personal-remote:"Junk"
Slave :personal-local:Trash
Create Slave
SyncState *
CopyArrivalDate yes

Group Personal
Channel sync-personal-inbox
Channel sync-personal-archive
Channel sync-personal-sent
Channel sync-personal-trash

Thunderbird is the free and open source email client by Mozilla Foundation. I have been using it for some years now. Till now the Thunderbird users had to use an extension Enigmail to use GnuPG. Thunderbird 78 now uses a different implementation of OpenPGP called RNP.

Since RNP library still does not support the use of secret key on smartcards, to use Yubikey or any other GnuPG enabled smartcards, we need manually configure Thunderbird with GnuPG. The steps as said are the following :

Install GPGME

dnf install GPGME

GPGME, GnuPG Made Easy library makes the GnuPG easily accessible by providing a high level crypto API for encrypt, decrypt, sign, verify and key management. I already have GnuPG installed in my Fedora 33 machine and my Yubikey ready.

Modify Thunderbird configuration

Go to the Preferences menu then click on the config editor button at the very end.

Click on the I accept the risk.

Search for mail.openpgp.allow_external_gnupg and switch to true.

Remember to restart the Thunderbird after that.

Configure the secret key usage form Yubikey

Now go to the Account Settings and then go to the End-To-End-Encryption at the sidebar. Select the Use your external key through GnuPG(e.g. from a smartcard) option and click on continue.

Type your Secret Key ID in the box and click on Save key ID.

Now open the OpenPGP Key Manager and import your public key and then verify.

Now you can start using your hardware token in Thunderbird.

In this case we have to use 2 keyrings - GnuPG and RNP’s keyring (internal in Thunderbird). This is an extra step, which I hope in future can be avoided.

Mailvelope is an extension on web browsers to send end to end encrypted emails. This is a good option available to the users to send end to end encrypted without changing the email service they use. It is licensed under AGPL v3, making it Free and Open Source software. The code is there in Github for the community to have a look. This can be added as an extension to the - Chrome, Firefox and Edge browsers to securely encrypt emails with PGP using your email providers.
​
Mailvelope does provide end to end encryption for the email content but does not protect the metadata (subject, IP address of the sender) from third parties. As most of the email encryption tools, it does not work on the mobile browser. There is a detailed user guide on Mailvelope from the Freedom of the Press Foundation, which is really helpful for the new users.
​
By default, Mailvelope uses its own keyring. To use my Yubikey along with GnuPG keyring, I had to take the following steps:
​

Install gpgme

​
We need gpgme installed. On my Fedora 33 I did
​

$ sudo dnf install gpgme -y

​

For Chrome browser

​
​
We have to create gpgmejson.json .json file in the ~/.config/google-chrome/NativeMessagingHosts directory write the following json in there.

{
    "name": "gpgmejson",
    "description": "Integration with GnuPG",
    "path": "/usr/bin/gpgme-json",
    "type": "stdio",
    "allowed_origins": [
        "chrome-extension://kajibbejlbohfaggdiogboambcijhkke/"
    ]
}

​

For Firefox

​

mkdir -p ~/.mozilla/native-messaging-hosts
​

After creating the native-messaging-hosts directory inside the Mozilla directory, add gpgmejson.json file there with the following content.
​

vim ~/.mozilla/native-messaging-hosts/gpgmejson.json
​

​

{
    "name": "gpgmejson",
    "description": "Integration with GnuPG",
    "path": "/usr/bin/gpgme-json",
    "type": "stdio",
    "allowed_extensions": [
        "jid1-AQqSMBYb0a8ADg@jetpack"
    ]
}
​

​
Remember to restart the respective browser after you add the .json file. Then go to the Mailvelope extension to select the GnuPG keyring.
​

I recently read this paper titled, Understanding Real-World Concurrency Bugs in Go (PDF), that studies concurrency bugs in Golang and comments on the new primitives for messages passing that the language is often known for.

I am not a very good Go programmer, so this was an informative lesson in various ways to achieve concurrency and synchronization between different threads of execution. It is also a good read for experienced Go developers as it points out some important gotchas to look out for when writing Go code. The fact that it uses real world examples from well known projects like Docker, Kubernetes, gRPC-Go, CockroachDB, BoltDB etc. makes it even more fun to read!

The authors analyzed a total of 171 concurrency bugs from several prominent Go open source projects and categorized them in two orthogonal dimensions, one each for the cause of the bug and the behavior. The cause is split between two major schools of concurrency

Along the cause dimension, we categorize bugs into those that are caused by misuse of shared memory and those caused by misuse of message passing

and the behavior dimension is similarly split into

we separate bugs into those that involve (any number of ) goroutines that cannot proceed (we call themblocking bugs) and those that do not involve any blocking (non-blocking bugs)

Interestingly, they chose the behavior to be blocking instead of deadlock since the former implies that atleast one thread of execution is blocked due to some concurrency bug, but the rest of them might continue execution, so it is not a deadlock situation.

Go has primitive shared memory protection mechanisms like Mutex, RWMutex etc. with a caveat

Write lock requests in Go have ahigher privilege than read lock requests.

as compared to pthread in C. Go also has a new primitive called sync.Once that can be used to guarantee that a function is executed only once. This can be useful in situations where some callable is shared across multiple threads of execution but it shouldn't be called more than once. Go also has sync.WaitGroups , which is similar to pthread_join to wait for various threads of executioun to finish executing.

Go also uses channels for the message passing between different threads of executions called Goroutunes. Channels can be buffered on un-buffered (default), the difference between them being that in a buffered channel the sender and receiver don't block on each other (until the buffered channel is full).

The study of the usage patterns of these concurrency primitives in various code bases along with the occurence of bugs in the codebase concluded that even though message passing was used at fewer places, it accounted for a larger number of bugs(58%).

Implication 1:With heavier usages of goroutines and newtypes of concurrency primitives, Go programs may potentiallyintroduce more concurrency bugs

Also, interesting to note is this observation in tha paper

Observation 5:All blocking bugs caused by message passing are related to Go’s new message passing semantics like channel. They can be difficult to detect especially when message passing operations are used together with other synchronization mechanisms

The authors also talk about various ways in which Go runtime can detect some of these concurrency bugs. Go runtime includes a deadlock detector which can detect when there are no goroutunes running in a thread, although, it cannot detect all the blocking bugs that authors found by manual inspection.

For shared memory bugs, Go also includes a data race detector which can be enbaled by adding -race option when building the program. It can find races in memory/data shared between multiple threads of execution and uses happened-before algorithm underneath to track objects and their lifecycle. Although, it can only detect a part of the bugs discovered by the authors, the patterns and classification in the paper can be leveraged to improve the detection and build more sophisticated checkers.

TLDR; Trying to learn new things I tried writing a URL shortner called shorty. This is a first draft and I am trying to approach it from first principle basis. Trying to break down everything to the simplest component.

I decided to write my own URL shortner and the reason for doing that was to dive a little more into golang and to learn more about systems. I have planned to not only document my learning but also find and point our different ways in which this application can be made scalable, resilient and robust.

A high level idea is to write a server which takes the big url and return me a short url for the same. I have one more requirement where I do want to provide a slug i.e a custom short url path for the same. So for some links like https://play.google.com/store/apps/details?id=me.farhaan.bubblefeed, I want to have a url like url.farhaan.me/linktray which is easy to remember and distribute.

The way I am thinking to implement this is by having two components, I want a CLI interface which talks to my Server. I don’t want a fancy UI for now because I want it to be exclusively be used through terminal. A Client-Server architecture, where my CLI client sends a request to the server with a URL and an optional slug. If a slug is present URL will have that slug in it and if it doesn’t it generates a random string and make the URL small. If you see from a higher level it’s not just a URL shortner but also a URL tagger.

The way a simple url shortner works:

Flow Diagram

A client makes a request to make a given URL short, server takes the URL and stores it to the database, server then generates a random string and maps the URL to the string and returns a URL like url.farhaan.me/<randomstring>.

Now when a client requests to url.farhaan.me/<randomstring>, it goest to the same server, it searches the original URL and redirects the request to a different website.

The slug implementation part is very straightforward, where given a word, I might have to search the database and if it is already present we raise an error but if it isn’t we add it in the database and return back the URL.

One optimization, since it’s just me who is going to use this, I can optimize my database to see if the long URL already exists and if it does then no need to create a new entry. But this should only happen in case of random string and not in case of slugs. Also this is a trade off between reducing the redundancy and latency of a request.

But when it comes to generating a random string, things get a tiny bit complicated. This generation of random strings, decides how many URLs you can store. There are various hashing algorithms that I can use to generate a string I can use md5, base10 or base64. I also need to make sure that it gives a unique hash and not repeated ones.

Unique hash can be maintained using a counter, the count either can be supplied from a different service which can help us to scale the system better or it can be internally generated, I have used database record number for the same.

If you look at this on a system design front. We are using the same Server to take the request and generate the URL and to redirect the request. This can be separated into two services where one service is required to generate the URL and the other just to redirect the URL. This way we increase the availability of the system. If one of the service goes down the other will still function.

The next step is to write and integrate a CLI system to talk to the server and fetch the URL. A client that can be used for an end user. I am also planning to integrate a caching mechanism in this but not something out of the shelf rather write a simple caching system with some cache eviction policy and use it.

Till then I will be waiting for the feedback. Happy Hacking.

I now have a Patreon open so that you folks can support me to do this stuff for longer time and sustain myself too. So feel free to subscribe to me and help me keeping doing this with added benefits.

TLDR; Link Tray is a utility we recently wrote to curate links from different places and share it with your friends. The blogpost has technical details and probably some productivity tips.

Link Bubble got my total attention when I got to know about it, I felt it’s a very novel idea, it helps to save time and helps you to curate the websites you visited. So on the whole, and believe me I am downplaying it when I say Link Bubble does two things:

1. Saves time by pre-opening the pages
2. Helps you to keep a track of pages you want to visit

It’s a better tab management system, what I felt weird was building a whole browser to do that. Obviously, I am being extremely naive when I am saying it because I don’t know what it takes to build a utility like that.

Now, since they discontinued it for a while and I never got a chance to use it. I thought let me try building something very similar, but my use case was totally different. Generally when I go through blogs or articles, I open the links mentioned in a different tab to come back to them later. This has back bitten me a lot of time because I just get lost in so many links.

I thought if there is a utility which could just capture the links on the fly and then I could quickly go through them looking at the title, it might ease out my job. I bounced off the same idea across to Abhishek and we ended up prototyping LinkTray.

Our first design was highly inspired by facebook messenger but instead of chatheads we have links opened. If you think about it the idea feels very beautiful but the design is “highly” not scalable. For example if you have as many as 10 links opened we had trouble in finding our links of interest which was a beautiful design problems we faced.

We quickly went to the whiteboard and put up a list of requirements, first principles; The ask was simple:

1. To share multiple links with multiple people with least transitions
2. To be able to see what you are sharing
3. To be able to curate links (add/remove/open links)

We took inspiration from an actual Drawer where we flick out a bunch of links and go through them. In a serendipitous moment the design came to us and that’s how link tray looks like the way it looks now.

Link Tray

Link Tray was a technical challenge as well. There is a plethora of things I learnt about the Android ecosystem and application development that I knew existed but never ventured into exploring it.

Link Tray is written in Java, and I was using a very loosely maintained library to get the overlay activity to work. Yes, the floating activity or application that we see is called an overlay activity, this allows the application to be opened over an already running application.

The library that I was using doesn’t have support for Android O and above. To figure that out it took me a few nights , also because I was hacking on the project during nights . After reading a lot of GitHub issues I figured out the problem and put in the support for the required operating system.

One of the really exciting features that I explored about Android is Services. I think I might have read most of the blogs out there and all the documentation available and I know that I still don't know enough. I was able to pick enough pointers to make my utility to work.

Just like Uncle Bob says make it work and then make it better. There was a persistent problem, the service needs to keep running in the background for it to work. This was not a functional issue but it was a performance issue for sure and our user of version 1.0 did have a problem with it. People got mislead because there was constant notification that LinkTray is running and it was annoying. This looked like a simple problem on the face but was a monster in the depth.

Architecture of Link Tray

The solution to the problem was simple stop the service when the tray is closed, and start the service when the link is shared back to link tray. Tried, the service did stop but when a new link was shared the application kept crashing. Later I figured out the bound service that is started by the library I am using is setting a bound flag to True but when they are trying to reset this flag , they were doing at the wrong place, this prompted me to write this StackOverflow answer to help people understand the lifecycle of service. Finally after a lot of logs and debugging session I found the issue and fixed it. It was one of the most exciting moment and it help me learn a lot of key concepts.

The other key learning, I got while developing Link Tray was about multi threading, what we are doing here is when a link is shared to link tray, we need the title of the page if it has and favicon for the website. Initially I was doing this on the main UI thread which is not only an anti-pattern but also a usability hazard. It was a network call which blocks the application till it was completed, I learnt how to make a network call on a different thread, and keep the application smooth.

Initially approach was to get a webview to work and we were literally opening the links in a browser and getting the title and favicon out, this was a very heavy process. Because we were literally spawning a browser to get information about links, in the initial design it made sense because we were giving an option to consume the links. Over time our design improved and we came to a point where we don’t give the option to consume but to curate. Hence we opted for web scraping, I used custom headers so that we don’t get caught by robot.txt. And after so much of effort it got to a place where it is stable and it is performing great.

It did take quite some time to reach a point where it is right now, it is full functional and stable. Do give it a go if you haven’t, you can shoot any queries to me.

Link to Link Tray: https://play.google.com/store/apps/details?id=me.farhaan.bubblefeed

Happy Hacking!

http://bikeshed.org/

What color should I paint the bike-shed?

[Published in Open Source For You (OSFY) magazine, October 2017 edition.]

This article is the eighth in the DevOps series. In this issue, we shall learn to set up Docker in the host system and use it with Ansible.

Introduction

Docker provides operating system level virtualisation in the form of containers. These containers allow you to run standalone applications in an isolated environment. The three important features of Docker containers are isolation, portability and repeatability. All along we have used Parabola GNU/Linux-libre as the host system, and executed Ansible scripts on target Virtual Machines (VM) such as CentOS and Ubuntu.

Docker containers are extremely lightweight and fast to launch. You can also specify the amount of resources that you need such as CPU, memory and network. The Docker technology was launched in 2013, and released under the Apache 2.0 license. It is implemented using the Go programming language. A number of frameworks have been built on top of Docker for managing these cluster of servers. The Apache Mesos project, Google’s Kubernetes, and the Docker Swarm project are popular examples. These are ideal for running stateless applications and help you to easily scale them horizontally.

Setup

The Ansible version used on the host system (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. Internet access should be available on the host system. The ansible/ folder contains the following file:

ansible/playbooks/configuration/docker.yml

Installation

The following playbook is used to install Docker on the host system:

---
- name: Setup Docker
  hosts: localhost
  gather_facts: true
  become: true
  tags: [setup]

  tasks:
    - name: Update the software package repository
      pacman:
        update_cache: yes

    - name: Install dependencies
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - python2-docker
        - docker

    - service:
        name: docker
        state: started

    - name: Run the hello-world container
      docker_container:
        name: hello-world
        image: library/hello-world

The Parabola package repository is updated before proceeding to install the dependencies. The python2-docker package is required for use with Ansible. Hence, it is installed along with the docker package. The Docker daemon service is then started and the library/hello-world container is fetched and executed. A sample invocation and execution of the above playbook is shown below:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=setup
SUDO password: 

PLAY [Setup Docker] *************************************************************

TASK [Gathering Facts] **********************************************************
ok: [localhost]

TASK [Update the software package repository] ***********************************
changed: [localhost]

TASK [Install dependencies] *****************************************************
ok: [localhost] => (item=python2-docker)
ok: [localhost] => (item=docker)

TASK [service] ******************************************************************
ok: [localhost]

TASK [Run the hello-world container] ********************************************
changed: [localhost]

PLAY RECAP **********************************************************************
localhost                  : ok=5    changed=2    unreachable=0    failed=0   

With verbose ’-v’ option to ansible-playbook, you will see an entry for LogPath, such as /var/lib/docker/containers//-json.log. In this log file you will see the output of the execution of the hello-world container. This output is the same when you run the container manually as shown below:

$ sudo docker run hello-world

Hello from Docker!

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Example

A Deep Learning (DL) Docker project is available (https://github.com/floydhub/dl-docker) with support for frameworks, libraries and software tools. We can use Ansible to build the entire DL container from the source code of the tools. The base OS of the container is Ubuntu 14.04, and will include the following software packages:

• Tensorflow
• Caffe
• Theano
• Keras
• Lasagne
• Torch
• iPython/Jupyter Notebook
• Numpy
• SciPy
• Pandas
• Scikit Learn
• Matplotlib
• OpenCV

The playbook to build the DL Docker image is given below:

- name: Build the dl-docker image
  hosts: localhost
  gather_facts: true
  become: true
  tags: [deep-learning]

  vars:
    DL_BUILD_DIR: "/tmp/dl-docker"
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Download dl-docker
      git:
        repo: https://github.com/saiprashanths/dl-docker.git 
        dest: "{{ DL_BUILD_DIR }}"

    - name: Build image and with buildargs
      docker_image:
         path: "{{ DL_BUILD_DIR }}"
         name: "{{ DL_DOCKER_NAME }}"
         dockerfile: Dockerfile.cpu
         buildargs:
           tag: "{{ DL_DOCKER_NAME }}:cpu"

We first clone the Deep Learning docker project sources. The docker_image module in Ansible helps us to build, load and pull images. We then use the Dockerfile.cpu file to build a Docker image targeting the CPU. If you have a GPU in your system, you can use the Dockerfile.gpu file. The above playbook can be invoked using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=deep-learning

Depending on the CPU and RAM you have, it will take considerable amount of time to build the image with all the software. So be patient!

Jupyter Notebook

The built dl-docker image contains Jupyter notebook which can be launched when you start the container. An Ansible playbook for the same is provided below:

- name: Start Jupyter notebook
  hosts: localhost
  gather_facts: true
  become: true
  tags: [notebook]

  vars:
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Run container for Jupyter notebook
      docker_container:
        name: "dl-docker-notebook"
        image: "{{ DL_DOCKER_NAME }}:cpu"
        state: started
        command: sh run_jupyter.sh

You can invoke the playbook using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=notebook

The Dockerfile already exposes the port 8888, and hence you do not need to specify the same in the above docker_container configuration. After you run the playbook, using the ‘docker ps’ command on the host system, you can obtain the container ID as indicated below:

$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND               CREATED             STATUS              PORTS                NAMES
a876ad5af751        floydhub/dl-docker:cpu   "sh run_jupyter.sh"   11 minutes ago      Up 4 minutes        6006/tcp, 8888/tcp   dl-docker-notebook

You can now login to the running container using the following command:

$ sudo docker exec -it a876 /bin/bash

You can then run an ‘ifconfig’ command to find the local IP address (“172.17.0.2” in this case), and then open http://172.17.0.2:8888 in a browser on your host system to see the Jupyter Notebook. A screenshot is shown in Figure 1:

TensorBoard

TensorBoard consists of a suite of visualization tools to understand the TensorFlow programs. It is installed and available inside the Docker container. After you login to the Docker container, at the root prompt, you can start Tensorboard by passing it a log directory as shown below:

# tensorboard --logdir=./log

You can then open http://172.17.0.2:6006/ in a browser on your host system to see the Tensorboard dashboard as shown in Figure 2:

Docker Image Facts

The docker_image_facts Ansible module provides useful information about a Docker image. We can use it to obtain the image facts for our dl-docker container as shown below:

- name: Get Docker image facts
  hosts: localhost
  gather_facts: true
  become: true
  tags: [facts]

  vars:
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Get image facts
      docker_image_facts:
        name: "{{ DL_DOCKER_NAME }}:cpu"

The above playbook can be invoked as follows:

$ ANSIBLE_STDOUT_CALLBACK=json ansible-playbook playbooks/configuration/docker.yml -K --tags=facts 

The ANSIBLE_STDOUT_CALLBACK environment variable is set to ‘json’ to produce a JSON output for readability. Some important image facts from the invocation of the above playbook are shown below:

"Architecture": "amd64", 
"Author": "Sai Soundararaj <saip@outlook.com>", 

"Config": {

"Cmd": [
   "/bin/bash"
], 

"Env": [
   "PATH=/root/torch/install/bin:/root/caffe/build/tools:/root/caffe/python:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 
   "CAFFE_ROOT=/root/caffe", 
   "PYCAFFE_ROOT=/root/caffe/python", 
   "PYTHONPATH=/root/caffe/python:", 
   "LUA_PATH=/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/root/torch/install/share/lua/5.1/?.lua;/root/torch/install/share/lua/5.1/?/init.lua;./?.lua;/root/torch/install/share/luajit-2.1.0-beta1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua", 
   "LUA_CPATH=/root/torch/install/lib/?.so;/root/.luarocks/lib/lua/5.1/?.so;/root/torch/install/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so", 
   "LD_LIBRARY_PATH=/root/torch/install/lib:", 
   "DYLD_LIBRARY_PATH=/root/torch/install/lib:"
], 

"ExposedPorts": {
   "6006/tcp": {}, 
   "8888/tcp": {}
}, 

"Created": "2016-06-13T18:13:17.247218209Z", 
"DockerVersion": "1.11.1", 

"Os": "linux", 

"task": { "name": "Get image facts" }

You are encouraged to read the ‘Getting Started with Docker’ user guide available at http://docs.ansible.com/ansible/latest/guide_docker.html to know more about using Docker with Ansible.

[NOTE: This post originally appeared on deepsource.io, and has been posted here with due permission.]

In the early part of the last century, when David Hilbert was working on stricter formalization of geometry than Euclid, Georg Cantor had worked out a theory of different types of infinities, the theory of sets. This theory would soon unveil a series of confusing paradoxes, leading to a crisis in the Mathematics community  regarding the stability of the foundational principles of the math of that time.

Central to these paradoxes was the Russell’s paradox (or more generally, as we’d talk about later, the Epimenides Paradox). Let’s see what it is.

In those simpler times, you were allowed to define a set if you could describe it in English. And, owing to mathematicians’ predilection for self-reference, sets could contain other sets.

Russell then, came up with this:

\(R\)  is a set of all the sets which do not contain themselves.

The question was "Does \(R \) contain itself?" If it doesn’t, then according to the second half of the definition it should. But if it does, then it no longer meets the definition.

The same can symbolically be represented as:

Let \(R = \{ x \mid x \not \in x \} \), then \(R \in R \iff R \not \in R \)

Cue mind exploding.

“Grelling’s paradox” is a startling variant which uses adjectives instead of sets. If adjectives are divided into two classes, autological (self-descriptive) and heterological (non-self-descriptive), then, is ‘heterological’ heterological? Try it!

Epimenides Paradox

Or, the so-called Liar Paradox was another such paradox which shred apart whatever concept of ‘computability’ was, at that time - the notion that things could either be true or false.

Epimenides was a Cretan, who made one immortal statement:

“All Cretans are liars.”

If all Cretans are liars, and Epimenides was a Cretan, then he was lying when he said that “All Cretans are liars”. But wait, if he was lying then, how can we ‘prove’ that he wasn’t lying about lying? Ein?

⊕

This is what makes it a paradox: A statement so rudely violating the assumed dichotomy of statements into true and false, because if you tentatively think it’s true, it backfires on you and make you think that it is false. And a similar backfire occurs if you assume that the statement is false. Go ahead, try it!

If you look closely, there is one common culprit in all of these paradoxes, namely ‘self-reference’. Let’s look at it more closely.

Strange Loopiness

If self-reference, or what Douglas Hofstadter - whose prolific work on the subject matter has inspired this blog post - calls ‘Strange Loopiness’ was the source of all these paradoxes, it made perfect sense to just banish self-reference, or anything which allowed it to occur. Russell and Whitehead, two rebel mathematicians of the time, who subscribed to this point of view, set forward and undertook the mammoth exercise, namely “Principia Mathematica”, which we as we will see in a little while, was utterly demolished by Gödel’s findings.

The main thing which made it difficult to ban self-reference was that it was hard to pin point where exactly did the self-reference occur. It may as well be spread out over several steps, as in this ‘expanded’ version of Epimenides:

The next statement is a lie.

The previous statement is true.

Russell and Whitehead, in P.M. then, came up with a multi-hierarchy set theory to deal with this. The basic idea was that a set of the lowest ‘type’ could only contain ‘objects’ as members (not sets). A set of the next type could then only either contain objects, or sets of lower types. This, implicitly banished self-reference.

Since, all sets must have a type, a set ‘which contains all sets which are not members of themselves’ is not a set at all, and thus you can say that Russell’s paradox was dealt with.

Similarly, if an attempt is made towards applying the expanded Epimenides to this theory, it must fail as well, for the first sentence to make a reference to the second one, it has to be hierarchically above it - in which case, the second one can’t loop back to the first one.

Thirty one years after David Hilbert set before the academia to rigorously demonstrate that the system defined in Principia Mathematica was both consistent (contradiction-free) and complete (i.e. every true statement could be evaluated to true within the methods provided by P.M.), Gödel published his famous Incompleteness Theorem. By importing the Epimenides Paradox right into the heart of P.M., he proved that not just the axiomatic system developed by Russell and Whitehead, but none of the axiomatic systems whatsoever were complete without being inconsistent.

Clear enough, P.M. lost it’s charm in the realm of academics.

Before Gödel’s work too, P.M. wasn’t particularly loved as well.

Why?

It isn’t just limited to this blog post, but we humans, in general, have a diet for self-reference - and this quirky theory severely limits our ability to abstract away details - something which we love, not only as programmers, but as linguists too - so much so, that the preceding paragraph, “It isn’t … this blog … we humans …” would be doubly forbidden because the ‘right’ to mention ‘this blog post’ is limited only to something which is hierarchically above blog posts, ‘metablog-posts’. Secondly, me (presumably a human) belonging to the class ‘we’ can’t mention ‘we’ either.

Since, we humans, love self-reference so much, let’s discuss some ways in which it can be expressed in written form.

One way of making such a strange loop, and perhaps the ‘simplest’ is using the word ‘this’. Here:

• This sentence is made up of eight words.
• This sentence refers to itself, and is therefore useless.
• This blog post is so good.
• This sentence conveys you the meaning of ‘this’.
• This sentence is a lie. (Epimenides Paradox)

Another amusing trick for creating a self-reference without using the word ‘this sentence’ is to quote the sentence inside itself.

Someone may come up with:

The sentence ‘The sentence contains five words’ contains five words.

But, such an attempt must fail, for to quote a finite sentence inside itself would mean that the sentence is smaller than itself. However, infinite sentences can be self-referenced this way.

The sentence
    "The sentence
        "The sentence
                                ...etc
                                ...etc
        is infinitely long"
    is infinitely long"
is infinitely long"

There’s a third method as well, which you already saw in the title - the Quine method. The term ‘Quine’ was coined by Douglas Hofstadter in his book “Gödel Escher, Bach” (which heavily inspires this blog post). When using this, the self-reference is ‘generated’ by describing a typographical entity, isomorphic to the quine sentence itself. This description is carried in two parts - one is a set of ‘instructions’ about how to ‘build’ the sentence, and the other, the ‘template’ contains information about the construction materials required.

The Quine version of Epimenides would be:

“yields falsehood when preceded by it’s quotation” yields falsehood when preceded by it’s quotation

Before going on with ‘quining’, let’s take a moment and realize how awfully powerful our cognitive capacities are, and what goes in our head when a cognitive payload full of self-references is delivered - in order to decipher it, we not only need to know the language, but also need to work out the referent of the phrase analogous to ‘this sentence’ in that language. This parsing depends on our complex, yet totally assimilated ability to handle the language.

The idea of referring to itself is quite mind-blowing, and we keep doing it all the time — perhaps, why it feels so ‘easy’ for us to do so. But, we aren’t born that way, we grow that way. This could better be realized by telling someone much younger “This sentence is wrong.”. They’d probably be confused - What sentence is wrong?. The reason why it’s so simple for self-reference to occur, and hence allow paradoxes, in our language, is well, our language. It allows our brain to do the heavy lifting of what the author is trying to get through us, without being verbose.

Back to Quines.

Reproducing itself

Now, that we are aware of how ‘quines’ can manifest as self-reference, it would be interesting to see how the same technique can be used by a computer program to ‘reproduce’ itself.

To make it further interesting, we shall choose the language most apt for the purpose - brainfuck:

>>>>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++[->++++++++<]>--....[-]<<[<]<<++++++++[->+++++>++++++++<<]>+++>-->[[-<<<+>.>>]<.[->+<]<[->+<]>>>]<<<<[<]>[.>]

Running that program above produces itself as the output. I agree, it isn’t the most descriptive program in the world, so written in Python below, is the nearest we can go to describe what’s happening inside those horrible chains of +’s and >’s:

THREE_QUOTES = '"' * 3

def eniuq(template): print(
  f'{template}({THREE_QUOTES}{template}{THREE_QUOTES})')

eniuq("""THREE_QUOTES = '"' * 3

def eniuq(template): print(
  f'{template}({THREE_QUOTES}{template}{THREE_QUOTES})')

eniuq""")

The first line generates """ on the fly, which marks multiline strings in Python.

Next two lines define the eniuq function, which prints the argument template twice - once, plain and then surrounded with triple quotes.

The last 4 lines cleverly call this function so that the output of the program is the source code itself.

Since we are printing in an order opposite of quining, the name of the function is ‘quine’ reversed -> eniuq (name stolen from Hofstadter again)

Remember the discussion about how self-reference capitalizes on the processor? What if ‘quining’ was a built-in feature of the language, providing what we in programmer lingo call ‘syntactic sugar’?

Let’s assume that an asterisk, * in the brainfuck interpreter would copy the instructions before executing them, what would then be the output of the following program?

*

It’d be an asterisk again. You could make an argument that this is silly, and should be counted as ‘cheating’. But, it’s the same as relying on the processor, like using “this sentence” to refer to this sentence - you rely on your brain to do the inference for you.

What if eniuq was a builtin keyword in Python? A perfect self-rep was then just be a call away:

eniuq('eniuq')

What if quine was a verb in the English language? We could reduce a lot of explicit cognitive processes required for inference. The Epimenides paradox would then be:

“yields falsehood if quined” yields falsehood if quined

Now, that we are talking about self-rep, here’s one last piece of entertainment for you.

The Tupper’s self-referential formula

This formula is defined through an inequality:

\({1 \over 2} < \left\lfloor \mathrm{mod}\left(\left\lfloor {y \over 17} \right\rfloor 2^{-17 \lfloor x \rfloor - \mathrm{mod}(\lfloor y\rfloor, 17)},2\right)\right\rfloor\)

If you take that absurd thing above, and move around in the cartesian plane for the coordinates \(0 \le x \le 106, k \le y \le k + 17\), where \(k\) is a 544 digit integer (just hold on with me here), color every pixel black for True, and white otherwise, you'd get:

This doesn't end here. If \(k\) is now replaced with another integer containing 291 digits, we get yours truly:

The Tex User Group 2019 conference was held between August 9-11, 2019 at Sheraton Palo Alto Hotel, in Palo Alto, California.

I wanted to attend TUG 2019 for two main reasons - to present my work on the “XeTeX Book Template”, and also to meet my favourite computer scientist, Prof. Donald Knuth. He does not travel much, so, it was one of those rare opportunities for me to meet him in person. His creation of the TeX computer typesetting system, where you can represent any character mathematically, and also be able to program and transform it is beautiful, powerful and the best typesetting software in the world. I have been using TeX extensively for my documentation and presentations over the years.

Day I

I reached the hotel venue only in the afternoon of Friday, August 9, 2019, as I was also visiting Mountain View/San Jose on official work. I quickly checked into the hotel and completed my conference registration formalities. When I entered the hall, Rishi T from STM Document Engineering Private Limited, Thiruvananthapuram was presenting a talk on “Neptune - a proofing framework for LaTeX authors”. His talk was followed by an excellent poetic narration by Pavneet Arora, who happened to be a Vim user, but, also mentioned that he was eager to listen to my talk on XeTeX and GNU Emacs.

After a short break, Shreevatsa R, shared his experiences on trying to understand the TeX source code, and the lessons learnt in the process. It was a very informative, user experience report on the challenges he faced in navigating and learning the TeX code. Petr Sojka, from Masaryk University, Czech Republic, shared his students’ experience in using TeX with a detailed field report. I then proceeded to give my talk on the “XeTeX Book Template” on creating multi-lingual books using GNU Emacs and XeTeX. It was well received by the audience. The final talk of the day was by Jim Hefferon, who analysed different LaTeX group questions from newbies and in StackExchange, and gave a wonderful summary of what newbies want. He is a professor of Mathematics at Saint Michael’s College, and is well-known for his book on Linear Algebra, prepared using LaTeX. It was good to meet him, as he is also a Free Software contributor.

The TUG Annual General Meeting followed with discussions on how to grow the TeX community, the challenges faced, membership fees, financial reports, and plan for the next TeX user group conference.

Day II

The second day of the conference began with Petr Sojka and Ondřej Sojka presenting on “The unreasonable effectiveness of pattern generation”. They discussed the Czech hyphenation patterns along with a pattern generation case study. This talk was followed by Arthur Reutenauer presenting on “Hyphenation patterns in TeX Live and beyond”. David Fuchs, a student who worked with Prof. Donald Knuth on the TeX project in 1978, then presented on “What six orders of magnitude of space-time buys you”, where he discussed the design trade-offs in TeX implementation between olden days and present day hardware.

After a short break, Tom Rokicki, who was also a student at Stanford and worked with Donald Knuth on TeX, gave an excellent presentation on searching and copying text in PDF documents generated by TeX for Type-3 bitmap fonts. This session was followed by Martin Ruckert’s talk on “The design of the HINT file format”, which is intended as a replacement of the DVI or PDF file format for on-screen reading of TeX output. He has also authored a book on the subject - “HINT: The File Format: Reflowable Output for TeX”. Doug McKenna had implemented an interactive iOS math book with his own TeX interpreter library. This allows you to dynamically interact with the typeset document in a PDF-free ebook format, and also export the same. We then took a group photo:

I then had to go to Stanford, so missed the post-lunch sessions, but, returned for the banquet dinner in the evening. I was able to meet and talk with Prof. Donald E. Knuth in person. Here is a memorable photo!

He was given a few gifts at the dinner, and he stood up and thanked everyone and said that “He stood on the shoulders of giants like Isaac Newton and Albert Einstein.”

< />

I had a chance to meet a number of other people who valued the beauty, precision and usefulness of TeX. Douglas Johnson had come to the conference from Savannah, Georgia and is involved in the publishing industry. Rohit Khare, from Google, who is active in the Representational State Transfer (ReST) community shared his experiences with typesetting. Nathaniel Stemen is a software developer at Overleaf, which is used by a number of university students as an online, collaborative LaTeX editor. Joseph Weening, who was also once a student to Prof. Donald Knuth, and is at present a Research Staff member at the Institute for Defense Analyses Center for Communications Research in La Jolla, California (IDA/CCR-L) shared his experiences in working with the TeX project.

Day III

The final day of the event began with Antoine Bossard talking on “A glance at CJK support with XeTeX and LuaTeX”. He is an Associate Professor of the Graduate School of Science, Kanagawa University, Japan. He has been conducting research regarding Japanese characters and their memorisation. This session was followed by a talk by Jaeyoung Choi on “FreeType MF Module 2: Integration of Metafont and TeX-oriented bitmap fonts inside FreeType”. Jennifer Claudio then presented the challenges in improving Hangul to English translation.

After a short break, Rishi T presented “TeXFolio - a framework to typeset XML documents using TeX”. Boris Veytsman then presented the findings on research done at the College of Information and Computer Science, University of Massachusetts, Amherst on “BibTeX-based dataset generation for training citation parsers”. The last talk before lunch was by Didier Verna on “Quickref: A stress test for Texinfo”. He teaches at École Pour l’Informatique et les Techniques Avancées, and is a maintainer of XEmacs, Gnus and BBDB. He also an avid Lisper and one of the organizers of the European Lisp Symposium!

After lunch, Uwe Ziegenhagen demonstrated on using LaTeX to prepare and automate exams. This was followed by a field report by Yusuke Terada, on how they use TeX to develop a digital exam grading system at large scale in Japan. Chris Rowley, from the LaTeX project, then spoke on “Accessibility in the LaTeX kernel - experiments in tagged PDF”. Ross Moore joined remotely for the final session of the day to present on “LaTeX 508 - creating accessible PDFs”. The videos of both of these last two talks are available online.

A number of TeX books were made available for free for the participants, and I grabbed quite a few, including a LaTeX manual written by Leslie Lamport. Overall, it was a wonderful event, and it was nice to meet so many like-minded Free Software people.

A special thanks to Karl Berry, who put in a lot of effort in organizing the conference, but, could not make it to due to a car accident.

The TeX User Group Conference in 2020 is scheduled to be held at my alma mater, Rochester Institute of Technology.

Happy Birthday Dear me!

I was still up at this unearthly hour, thinking about life for a while now - fumbled thoughts about where I had come, where I started, and quite expectedly, Omar Bhai, your name popped in.

The stream continued. I started thinking about everything I’ve learned from you and was surprised with merely the sheer volume of thoughts that followed. I felt nostalgic!

I made a mental note to type this out the next day.

I wanted to do this when we said our final goodbyes and you left for the States, but thank God, I didn’t - I knew that I would miss you, but never could I have guessed that it would be so overwhelming - I would’ve never written it as passionately as I do today.

For those of you who don’t already know him, here’s a picture:

I’m a little emotional right now, so please bear with me.

You have been warned - the words “thank you” and “thanks” appear irritatingly often below. I tried changing, but none other has quite the same essence.

How do I begin thanking you?

Well, let’s start with this - thank you for kicking me on my behind, albeit civilly, whenever I would speak nuisance (read chauvinism). I can’t thank you enough for that!

I still can’t quite get how you tolerated the bigot I was and managed to be calm and polite. Thank You for teaching me what tolerance is!

Another thing which I learnt from you was what it meant to be privileged. I can no longer see things the way I used to, and this has made a huge difference. Thank You!

I saw you through your bad times and your good. The way you tackled problems, and how easy you made it look. Well, it taught me [drum roll] how to think (before acting and not the other way round). Thank You for that too!

And, thank you for buying me books, and even more so, lending away so many of them! and even more so, educating me about why to read books and how to read them. I love your collection.

You showed all of us, young folks, how powerful effective communication is. Thank You again for that! I know, you never agree on this, but you are one hell of a speaker. I’ve always been a fan of you and your puns.

I wasn’t preparing for the GRE, but I sat in your sessions anyways, just to see you speak. The way you connect with the audience is just brilliant.

For all the advice you gave me on my relationships with people - telling me to back off when I was being toxic and dragging me off when I was on the receiving side - I owe you big time. Thank You!

Also, a hearty thank you for making me taste the best thing ever - yes, fried cheese it is. :D

Thank You for putting your trust and confidence in me!

Thank you for all of this, and much more!

Yours Truly, Rahul

September 11, 2019

It’s been a very long time since I wrote here for the last.

The reason is nothing big but mainly because:

1. Apparently, I was not able to finish some tasks in time that I used to write about.
2. I was not well for a long time that could be an another reason .
3. Besides, life happened in many ways which ultimately left me working on some other things first, because they seemed to be *important* for the time.

And, yes, there is no denying the fact that I was procastinating too because writing seems to be really hard at most times.

Though I had worked on many things throughout the time and I’ll try to write them here as short and quick logs below.

• Around the second last week of august, I worked on setting up a self-hosted OpenVPN server which supported client scalability. The infrastructure required two servers/VMs, each having a basic firewall setup and a non-root “sudo” priviliged user. One among them was to host the OpenVPN service and another one was to serve as a Certificate Authority (CA). You can refer the following links to check for the related process.
  • How To Set Up an OpenVPN Server on Ubuntu 18.04
  • Easy Way to Install and Configure OpenVPN Server on Ubuntu 18.04 / Ubuntu 16.04
  • How to use nmcli to connect to OpenVPN Server on Linux

• In the last week of August, I worked on another task i.e to read about Syslogs and figure out how each of the systems can email the root mails to a certain email address for log collection. Thus, I read about Syslogs, how it works, its format and various Syslogs message levels. The latter part of the task was accomplished using ssmtp as mail program & writing cron jobs to actually send them to the intended email addresses. Check the following links for resources.
  • Syslog Tutorial: How It Works, Examples, Best Practices, and More
  • How to Send Email via SMTP Server from Linux Command Line (with SSMTP)
  • Setup cron job to send email to a bunch of people on a regular basis

This one question always came up, many times, the students managed to destroy their systems by doing random things. rm -rf is always one of the various commands in this regard.

Kushal Das

• While I was doing the above task, at one time I ruined my local system’s mail server configs and actually ended up doing something which kushal writes about in one of his recent post (quoted above). I was using the command rm -rf to clean some of the left-over dependencies of some mail packages, but that eventually resulted into machine being crashed. It was not the end of the mess this time. I made an another extremely big mistake meanwhile. I was trying to back up the crashed system, into an external hard disk using dd. But because I had never used dd before, so again I did something wrong and this time, I ended up losing ~500 GBs of backed up data. This is “the biggest mistake” and “the biggest lesson” I have learnt so far. (now I know why one should have multiple backups) And as there was absolutely no way of getting that much data back, the last thing I did was, formatting the hard-disk into 2 partitions, one with ext4 file system for linux backup and the other one as ntfs for everything else.

Thank you so much jasonbraganza for all the help and extremely useful suggestions during the time.

• Okay, now after all the hassle bustle above, I got something really nice. This time, I received the “Raspberry Pi 4, 4GB, Complete Kit ” from kushal.

Thank you very much kushal for the RPi and an another huge thanks for providing me with all the guidance and support that made me reach to even what I am today.

• During the same time, I attended a dgplug guest session from utkarsh2102. This session gave me a “really” good beginner’s insight of how things actually work in Debian Project. I owe a big thanks to utkarsh2102 as well, for he so nicely voluteered me from there onwards, to actually start with Debian project. I have started with DPMT and have done packaging 4 python modules so far. And now, I am looking forward to start contributing to Debian Ruby Team as well.

• With the start of september, I spent some time solving some basic Python problems from kushal’s lymworkbook. Those issues were related to some really simply sys-admins work. But for me, working around and wrapping them in Python was a whole lot of learning. I hope I will continue to solve some more problems/issues from the lab.

• And lastly (and currently), I am back to reading and implementing concepts from Ops School curriculum.

Voila, finally, I finish compiling up the logs from some last 20 days of work and other stuffs. (and thus, I am eventually finishing my long pending task of writing this post here as well).

I will definitely try to be more consistent with my writing from now onwards.

That’s all for now. o/

This blog is more like a bookmark for me, the solution was scavenged from internet. Recently I have been working on an analytics project where I had to generate pivot transpose tables from the data. Now this is the first time I faced the limitations set on postgres database. Since its a pivot, one of my column would be transposed and used as column names here, this is where things started breaking. Writing to postgres failed with error stating column names are not unique. After some digging I realized Postgres has a column name limitation of 63 bytes and anything more than that will be truncated hence post truncate multiple keys became the same causing this issue.

Next step was to look at the data in my column, it ranged from 20-300 characters long. I checked with redshift and Bigquery they had similar limitations too, 128 bytes. After looking for sometime found a solution, downloaded the postgres source, changed NAMEDATALEN to 301(remember column name length is always NAMEDATALEN – 1) src/include/pg_config_manual.h, followed the steps from postgres docs to compile the source and install and run postgres. This has been tested on Postgres 9.6 as of now and it works.

Next up I faced issues with maximum number columns, my pivot table had 1968 columns and postgres has a limitation of 1600 total columns. According to this answer I looked into the source comments and that looked quite overwhelming . Also I do not have a control over how many columns will be there post pivot so no matter whatever value i set , in future i might need more columns, so instead I handled the scenario in my application code to split the data across multiple tables and store them.

References:

1. https://til.hashrocket.com/posts/8f87c65a0a-postgresqls-max-identifier-length-is-63-bytes
2. https://stackoverflow.com/questions/6307317/how-to-change-postgres-table-field-name-limit
3. https://www.postgresql.org/docs/9.6/install-short.html
4. https://dba.stackexchange.com/questions/40137/in-postgresql-is-it-possible-to-change-the-maximum-number-of-columns-a-table-ca