I've been told that python package management is bad. I have seen some really bad practices online, asking you to run commands here and there without an understanding of the bigger picture, what they do and sometimes with escalated privileges.
Along the years, I have compiled a list of practices I follow, and a list of tools I use. I hope to be able to share some of the knowledge I've acquired and show you a different way of doing things. You might learn about a new tool, or a new use for a tool. Come along for the ride !
Python
As most know, Python is an interpreted programming language. I am not going to go into the details of the language in this post, I will only talk about management.
If you want to develop in Python, you need to install libraries. You can find some in your package manager but let's face it pip is your way.
The majority of Linux distributions will have Python installed as a lot of system packages now rely on it, even some package managers.
Okay, this is the last time I actually use the system's Python. What ? Why ? You ask !
pyenv
I introduce you to pyenv. Pyenv is a Python version management tool, it allows you to install and manage different versions of Python as a user.
Beautiful, music to my ears.
Let's get it from the package manager, this is a great use of the package manager if it offers an up to date version of the package.
sudo pacman -S pyenv
If you're not using an Archlinux based distribution follow the instructions on their webpage.
Alright ! Now that we've got ourselves pyenv, let's configure it real quickly.
Following the docs, I created ~/.config/fish/config.d/pyenv.fish and in it I put the following.
# Add pyenv executable to PATH by running
# the following interactively:
set -Ux PYENV_ROOT $HOME/.pyenv
set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths
# Load pyenv automatically by appending
# the following to ~/.config/fish/config.fish:
status is-login; and pyenv init --path | source
Open a new shell and you're all ready to continue along, you're all locked, loaded and ready to go!
Setup the environment
This is the first building block of my environment. We first start by querying for Python versions available for us.
pyenv install --list
Then, we install the latest Python version. Yes, even if it's an upgrade, I'll handle the upgrade, as well, as we go along.
Set everything up to use the new installed version.
First, we set the global Python version for our user.
pyenv global 3.9.5
Then, we switch our current shell's Python version, instead of opening a new shell.
pyenv shell 3.9.5
That was easy. We test that everything works as expected by checking the version.
pyenv version
Now, if you do a which on the python executable, you will find that it is in the pyenv shims' directory.
Upgrade
In the future, the upgrade path is exactly the same as the setup path shown above. You query for the list of Python versions available, choose the latest and move on from there.
Very easy, very simple.
At this stage, you have to understand that you are using a Python version installed by pyenv as your user. The pip provided, if you do a which, is also in the same shims directory.
Using pip at this stage as a user is better than running it as root but it is also not touching your system; just your user. We can do one better. I'm going to use pip as a user once !
I know, you will have a lot of questions at this point as to why. You will see, patience is a virtue.
pipx
Meet pipx, this tool is the amazing companion for a DevOps, and developer alike. Why ? You would ask.
It, basically, creates Python virtual environments for packages you want to have access to globally. For example, I'd like to have access to a Python LSP server on the go.
This way my text editor has access to it too and, of course, can make use of it freely. Anyway, let's cut this short and show you. You will understand better.
Let's use the only pip command as a user to install pipx.
pip install --user pipx
warning
You are setting yourself up for a world of hurt if you use sudo with pip or run it as root. ONLY run commands as root or with escalated privileges when you know what you're doing.
LSP Server
As I gave the LSP server as an example, let's go ahead and install it with some other Python packages needed for global things like emacs.
pipx install black
pipx install ipython
pipx install isort
pipx install nose
pipx install pytest
pipx install python-lsp-server
Now each one is in it's own happy little virtual environment separated from any other dependency but its own. Isn't that lovely ?
If you try to run ipython, you will see that it will actually work. If you look deeper at it, you will see that it is pointing to ~/.local/bin/ipython which is a symlink to the actual package in a pipxvirtual environment.
Upgrade
After you set a new Python version with pyenv, you simply reinstall everything.
pipx reinstall-all
And like magic, everything get recreated using the new version of Python newly set.
virtualfish
Now that pipx is installed, let's go head and install something to manage our Python virtual environments on-demand, for use whenever we need to, for targeted projects.
Some popular choices people use are Pipenv, Poetry, virtualenv and plain and simple python with the venv module.
You're welcome to play with all of them. Considering I use fish as my default shell, I like to use virtualfish.
Let's install it.
pipx install virtualfish
This offers me a new command; vf. With vf, I can create Python virtual environments and they will all be saved in a directory of my choosing.
This should activate it. Then, we install Ansible.
pip install ansible molecule docker
At this stage, you will notice that you have ansible installed. You will also notice that all the pipx packages are also still available.
If you want to tie virtualfish to a specific directory, use vf connect.
Upgrade
To upgrade the Python version of all of our virtual environments, virtualfish makes it as easy as
vf upgrade
And we're done !
Workflow
At this stage, you have an idea about the tools I use and where their scope falls. I like them because they are limited to their own scope, each has its own little domain where it reigns.
I use pyenv to install and manage different versions of Python for testing purposes while I stay on the latest.
I use pipx for the commands that I need access to globally as a user.
I use virtualfish to create one or more virtual environment per project I work on.
With this setup, I can test with different versions of Python by creating different virtual environments with different version each, or two versions of the tool you're testing as you keep the Python version static.
It could also be different versions of a library, testing forward compatibility for example.
At each step, I have an upgrade path to keep all my environments running the latest versions. I also have a lot of flexibility by using requirements.txt files and others for development sometimes or even testing.
Conclusion
As you can see, with a little bit of knowledge and by standing on the shoulders of giants, you can easily manage a Python environment entirely as a user.
You have full access to a wide array of Python distributions to play with. Endless different versions of packages, globally and locally installed.
If you create virtual environments for each of your projects, you won't fall in the common pitfalls of versioning hell.
Keep your virtual environments numerous and dedicated to projects, small sets, and you won't face any major problems with keeping your system clean yet up to date.
Another day of doing only Django.
Not that I’m complaining. Finishing up this Django course is my priority.
Followed by PostgreSQL and Data Structures and Algorithms.
I just have to do them step by step, one after another.
Let’s get on with today’s log!
I studied only Django today.
Was plagued with rains and power cuts and family stuff.
Managed to work well and patiently though.
A few days back, I had a small, very useful conversation with a friend from the community. We were discussing, How to ask someone for a 1:1 (learning or mentoring) session?
This is what I do! ⬇️
(and so, I’m putting it here just that I feel this might help someone else too, who are like me. 🙂)
Although I have never been a book worm, and even I did not get the atmosphere of reading books a lot when I was growing but when I was in class 8 my library period played a major role for inculcating the magic of reading a book in me. I chose Nancy Drew from the shelf of books, and then it was one book after the other. I started enjoying and the most amazing part which I realized was I had my imagination playing in my mind. My grandpa is a bookworm and has his own library in our home. The question might pop up that if my grandpa had a room filled with books why did not I get the atmosphere of reading? Well, I did not lived with my grandparents back then, I grew mostly on Southern and Western part of India (Mysuru and Udaipur).
After my board exam I decided to reward myself with a book and bought my first book The Immortals of Meluha. and it just blew my mind. Hence, my love for reading books evolved; although from my class 11-12 I could not read or buy any book because of "pressure". Even when I entered my college, the schedule was quite tight to take out time and read book but still I managed to buy some more books and tried reading it. Up until now I have The Power of Subconcious Mind. and I have not been able to complete this book because it is bit of much grown-up book for me. I have to read it thrice to understand the concept, and Wuthering Heights, The Upside Down King and few more mini books (names I don’t remember).
The next books on my list is recommended list of books (from one of my inspired idol of k-pop group BTS) which I’ll buy one by one with my pocket money (as I have been doing) and I am very excited to read them. Those books are
After moving away from Google Fonts, Google Analytics, I also removed Disqus comments from my website. Instead of implementing an alternative commenting system, I decided to remove it altogether. I will be covering why I did it, and how I’m showing the old comments from Disqus statically on the website.
Why am I removing the comments? This was not a very easy decision to take. But finally I took the decision to remove the commenting system, and here are the reasons:
As a takeaway, I had left for myself a home-work exercise in the end of that discussion. Below is where I’m trying to take notes of what I learnt broadly after doing that homework exercise.
And yea, the homework was to explore what I learnt about the kubernetes testgrid (above), now through the source code.
The source code for the kubernetes CI test jobs are present in this repository here ~ kubernetes/test-infra
There are broadly the following three types of CI test jobs in the kubernetes project ~
Pre-submit jobs
Post-submit jobs
Periodic jobs
For the purpose of note taking, let’s start by taking the Pre-submit jobs as our example usecase.
( Here is something important to know (which is also the reason why I'm picking Pre-submit jobs for our example):
Only the Pre-submit CI jobs have this properly defined & maintained dashboard. You would not find a similar testgrid dashboard (something that we are going to discuss below in the notes) for the Post-submit jobs or Periodic jobs. Following being the major reasons:
The Pre-submit jobs are the kind of jobs that runs everytime before a PR is submitted or merged. So, it contains mostly blocking jobs. And because these are blocking jobs & run on almost all the PRs during a day, we want the number of these jobs to be very less. And thus, the effort of maintaining a dashboard for these limited number of pre-submit jobs sound reasonabe & quite doable.
On the other hand, the number of Periodic jobs & Post-submit jobs is very high, & currently most of these are frequently ending up failing only. So, the dashboard usually end up being all red. People argue that these jobs fail so often & so much, that maybe the evaluation/validation criteria of these jobs are not good enough & so, no use of maintaining an all time red board for this large number of jobs. )
That being said, now we could start exploring things in action.
Head onto the kubenetes CI jobs testgrid dashboard groups UI using this link ~ https://testgrid.k8s.io.
You will find something like following ~
All these tiles such as conformance, gardener, google, … , presubmits, … , wg-multi-tenancy corresponds to a set of CI jobs dashboards or a dashboard-group (as we call it in the source-code), for a specific project or a sig-group.
You’ll see in the top left corner, a grid-like icon. Clicking that grid-like icon would take you back to the same list of dashboard-groups home page UI as we discussed in the last point above.
Now, on this pre-submits dashboard-group page, we see all these tabs (or the similar row entries under dashboard column in the Dashboard Group presubmits :: Overview table) ~
These tabs corresponds to the dashboards that goes under our presubmits dashboard-group. And of course, these dashboard tabs will further divide into CI jobs (as we’ll see later in the notes).
Before we move further, it’s also interesting to look at the information provided in the Dashboard Group presubmits :: Overview table.
As we can see, there are three columns:
Status,
Dashboard (which is basically list of names of the dashboard such as presubmits-alibaba-cloud-csi-driver, … , etc.),
Health
This table gives a broader view of how many CI jobs under a certain dashboard are PASSING, or FAILING, or are there any FLAKY jobs. For ex, in case of dashboard presubmits-kubernetes-nonblocking (from the above screenshot),
The overall status of this dashboard is FAILING (because we see there is atleast one CI job failing)
There are 6 PASSING CI jobs, 15 FLAKY CI jobs, & 1 FAILING CI job.
Something to note is that although all the CI jobs under this dashbaord will be listed in same config file, but those CI jobs may or may not be defined here (we’ll discuss more about it, below in the notes).
Continuing with the same presubmits-kubernetes-blocking dashboard example above, let’s now explore the source-code for the various CI jobs listed under it:
The following lines of code under the dasboards –> name: presubmits-kubernetes-blocking –> dashboard_tab section, defines what CI jobs will fall under this particular dashboard.
Now, as we see, this section just lists what all CI jobs will go under this one specific dashboard tab, but the definition of these CI jobs are not given here.
And that’s our next step to explore below.
These CI jobs under the presubmits-kubernetes-blocking tab are defined at multiple places based on the category or sub-project these CI job(s) fall into.
(something to note here is, one CI job can fall under multiple dashboard tabs, further under multiple dashboard-groups. So, that is one reason why these general purpose CI jobs are not defined just under one dashboard-group config file, but at certain separate places so they could be referenced by multiple dashboards)
Let’s start by taking few of the CI jobs listed above under the presubmits-kubernetes-blocking dashboard tab for reference example:
-name:pull-kubernetes-e2e-kindcluster:k8s-infra-prow-buildoptional:falsealways_run:truedecorate:trueskip_branches:-release-\d+\.\d+# per-release settingslabels:preset-dind-enabled:"true"preset-kind-volume-mounts:"true"decoration_config:timeout:60mgrace_period:15mpath_alias:k8s.io/kubernetesspec:containers:-image:gcr.io/k8s-testimages/krte:v20210512-b8d1b30-mastercommand:-wrapper.sh-bash--c-curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && e2e-k8s.shenv:-name:FOCUSvalue:"."# TODO(bentheelder): reduce the skip list further-name:SKIPvalue:\[Slow\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]|PodSecurityPolicy|LoadBalancer|load.balancer|Simple.pod.should.support.exec.through.an.HTTP.proxy|subPath.should.support.existing|NFS|nfs|inline.execution.and.attach|should.be.rejected.when.no.endpoints.exist-name:PARALLELvalue:"true"# we need privileged mode in order to do docker in dockersecurityContext:privileged:trueresources:limits:cpu:7memory:9000Mirequests:cpu:7memory:9000Miannotations:testgrid-num-failures-to-alert:'10'testgrid-alert-stale-results-hours:'24'testgrid-create-test-group:'true'fork-per-release:"true"
Similarily, the CI jobs pull-kubernetes-node-e2e & pull-kubernetes-node-e2e-conatinerd are defined in this same one file, here & here respectively.
For finding where rest of the CI jobs are defined under the kubernetes/test-infra repository, you can utilize the following:
Moving one more step further in the testgrid dashboard, let’s try to exapand & explore one of the CI jobs under one of the dashboard tabs now.
In continuation to our above examples, let’s try to see what comes when you click presubmits (dashboard-group) –> presubmits-kubernetes-blocking (dashboard-tab) –> pull-kubernetes-e2e-kind (CI job):
We could see the following:
on the left hand side, we see this wide grey section ~ These are the logs generated when the CI job pull-kubernetes-e2e-kind ran (spanned across multiple rows). To look at the full logs in a plain text, look at the artificats created by the CI job.
on the right hand side, we see the small tiles in green, red, white & grey colors ~ These are statuses of this CI job run across a period of time (days, divided by hours, and so on). This tells us how this certain CI job is performing.
Based on how frequently, a certain CI job fails, you would also see that CI jobs are marked as FLAKY.
The kind of calculations required to decide if a certain job will be marked as flaky based on “this is x no of times when this job failed”, is done using the metrics calculation config files present here ~ kubernetes/test-infra/metrics/configs
And yea, finally we reached at the end. All these above steps can be repeated to understand different dashboard-groups –> dashboards –> CI jobs across the kubernetes prow testgrid platform.
It has been almost 2 months since I have been off from writing my blog. It has not been an easy time, and I thought of taking a short break; things always got overwhelming, and I could not manage them properly. It was perhaps because of that, that I felt guilty about not being able to stay up to the mark.. But my mentors always taught me "it’s okay to pause but you must resume" hence I am resuming my one habit which gives me peace and teaches me something every-time I write. I last wrote on 28th April so I’ll be more regular.
With that out of the way, let me tell you the reason for today’s title – it also happens to be one of the reasons for my inactivity. I applied to an open-source internship program which happens one in summer round and another in winter around. In this program, if your initial application is accepted you can contribute to any of their listed projects and then there will be final selection. My journey began when my initial application was selected; I was ecstatic and determined to try my best and contribute regardless of the final outcome.. I knew it that if I be a part of the community and contribute to it, I would step out of my comfort zone and push past my fear. So I chose a project named OpenStack and started off by reading about how to get connected, which channel, the other necessary steps and so on.
On the first day I started exploring the project, read documentation . To be honest it was not an easy project. There were lot of things happening and I had to grasp the basic concepts, but first I had to set up my project. My laptop did not met the required space, first few days went into these complications. Complications such as installing VirtualBox (read about it at the same time because this was my first time using or knowing about it) and then run the project so that it was easy to run the project, if by chance my project met with uncertain problems then I wouldn’t have to rebuild everything from scratch; but due to my low storage capacity I couldn’t successfully run the project. Next solution was to get a server/virtual machine and I am blessed being surrounded by mentors who will guide me in desperate times and I had my saviour; got a VM from him and again I started my set-up. What amazed me was the community members and the mentors constantly helping me, once I got the VM and I started again. But the problem this time was, because it was a VM it lacked a feature of a processor and the disadvantage this time was if things go South I had to rebuild the machine and start again from scratch. But by all luck things did not go South and it took me half-month to set-up the project, the mentor was super amazing and helped through the entire process. To run the project I had to run a script but somehow the script was not able to run and there was an error popping up. The mentor tried to run the entire script in her system and tried to see where are the things going wrong. Initially even in her system the script did not run properly and we both tried hard to find out the error together in the script. After few trials the script ran in her system properly but it wasn’t running in my system properly. But finally after few more days the script ran in my system successfully and I was able to set-up the project in my system after struggling for about half a month. I was panicking at the same time as there were only two weeks left before my contribution and final application submission. There were times where I lost hope but I got myself together and thought that I’ll still contribute even if I fail to submit my final application within the time limit . This project set-up not only tested my patience but taught a lot of other new things and concepts which I read and researched. Concepts such as VirtualBox, devsatck environment, kuryr-kubernetes, Finally, I made my first contributions and even worked on few other issues.
The D-Day – the day the results were to be announced; Unfortunately, I did not make it. It was an opportunity lost, but the experience of that one month taught me about community, people, how to start contributing, and how to proceed. I knew that this experience will break my shell and help me get out of my comfort zone which was very much required, and I longed for the day when I will be contributing to an open source project. I realized that, every community for any specific project is out there to help you but first I need to help myself to break the barrier of self-doubt and move ahead without worrying too much about the end result.
In the efforts of making my website privacy-friendly, I moved away from Google Analytics. This was the next step after self-hosting fonts for the website. I have been self-hosting Plausible Analytics since last 3 months. I will be covering why I did it, how I’m hosting it on my VPS, the email setup, and more.
Why do I need analytics? Before we get to any further details, why do I even want analytics in the first place?
I have been having some issues with my calendar. Recurring stuff have been going out of wack for some reason. In general, the setup I've had for the past few years have now become a problem I need to fix.
I decided to turn to my trusted emacs, like I usually do. Doom comes bundled with something. Let's figure out what it is and how to configure it together.
Calendar in Emacs
I dug deeper into Doom's Calendar module and I found out that it is using calfw.
I went to GitHub and checked the project out. It's another emacs package, I'm going to assume you know how to install it.
That looks like an extensive example. We don't need all of it, I only need the part pertaining to org.
Configuration
The example looks straight forward. I'm going to keep only the pieces I'm interested in. The configuration looks like the following.
(require'calfw-cal)(require'calfw-org)(defunmy-blog-calendar()(interactive)(cfw:open-calendar-buffer:contents-sources(list(cfw:org-create-file-source"Blog""~/blog.org""Orange"); our blog organizational calendar)))
That was easy. but before we jump to the next step, let's talk a bit about what we just did.
We, basically, created a new function which we can call later with M-x to open our calendar.
We configured the function to include the org files we want it to keep track of.
In this case, we only have one. We named it Blog and we gave it the color Orange.
Creating our org file
After we have configured calfw, we can create the blog.org file.
#+TITLE: Blog
#+AUTHOR: Who
#+DESCRIPTION: Travels of Doctor Who
#+TAGS: organizer organization calendar todo tasks
* Introduction
This is the /calendar/ of *Dr Who* for the next week.
* Travels
** DONE Travel to Earth 1504
CLOSED: <2021-07-03 za 09:18> SCHEDULED: <2021-07-02 vr>
- CLOSING NOTE <2021-07-03 za 09:18> \\
The doctor already traveled to earth /1504/ for his visit to the /Mayans/.
A quick visit to the /Mayan/ culture to save them from a deep lake monster stealing all their gold.
** TODO Travel back to Earth 2021
SCHEDULED: <2021-07-04 zo>
Traveling back to earth 2021 to drop the companion before running again.
** TODO Travel to the Library
SCHEDULED: <2021-07-04 zo>
The doctor visits the /Library/ to save it again from paper eating bacteria.
** TODO Travel to Midnight
SCHEDULED: <2021-07-08 do>
The doctor visits *Midnight* in the /Xion System/.
** TODO Travel to Earth 2021
SCHEDULED: <2021-07-09 vr>
Snatching back the companion for another travel advanture.
Let's get the party started
Now that we have everything set into place. We can either reloademacs or simply run the code snippet that declares our function.
Next step is checking out if it works. Let's run M-x then call our function my-blog-calendar.
I thought it was going to be more extensive to configure the calendaring feature in emacs.
I couldn't be further away from the truth.
Not only was it a breeze to configure, it was even easier to create the calendar and maintain it.
If you are already familiar with org, then you're already there.
Point the calendar to your org file, iCal file or even Google Calendar link and you're all set.
The bottom line of working with org is the ease of use, to me.
If you already use it to organize some aspects of your life, you can just as easily create calendars for all these events.
I came to see this warning a few days back when I was seeing the git diff of a file in reaction to this I open the file and hit enter at the last line to my surprise the warning remains still the same.
What is a new line?
The new line is usually \n, aka (CR or CRLF) at the end of the file. It's helpful to identify the last byte of the file.
Why it is good to have a new line at the end of the file?
A source file that is not empty shall end in a new-line character, which shall not be immediately preceded by a backslash character.
Since this is a “shall” clause, we must emit a diagnostic message for a violation of this rule.
So, it turns out that, according to POSIX, every text file should end with a \n, or “newline” (not “a new line”) character. This acts as the eol, or the “end of line” character. It is a line “terminator”.
It's helpful to identify the end of file.
How to automatically add one with your favorite editor?
In your favorite editor, you can add newline automatically like this.
Emacs : Add (setq require-final-newline t) to your .emacs or .emacs.d/init.el file.
Last week I did two rounds of my Creating Python modules in Rust workshop.
During the second session on Sunday, someone asked if we can create standard
dunder methods, say __str__ or __repr__. I never did that before, and
during the session I tried to read the docs and implement it. And I failed :)
Later I realized that I should have read the docs carefully. To add those
methods, we will have to implement PyObjectProtocol for the Rust structure.
#[pyproto]
impl PyObjectProtocol for Ros {
fn __repr__(&self) -> PyResult<String> {
let cpus = self.sys.get_processors().len();
let repr = format!("<Ros(CPUS: {})>", cpus);
Ok(repr)
}
fn __str__(&self) -> PyResult<String> {
let cpus = self.sys.get_processors().len();
let repr = format!("<Ros(CPUS: {})>", cpus);
Ok(repr)
}
}
>>> from randomos import Ros
>>> r = Ros()
>>> r
<Ros (CPUS: 8)>
>>> str(r)
'<Ros (CPUS: 8)>'
This code example is in the
ros-more branch of the
code.
Recently, Google announced their new ad-tracking and surveillance tool called Federated Learning of Cohorts (FLoC). This is a new alternative to the third-party cookie tracking that is otherwise widely used for advertising business.
EFF has written more about the issues with using Google FLoC and also created a website where you can test if you are already a victim of their FLoC tests.
Google will track any user visiting your website even if it doesn't have Google analytics or any other services related to Google. One easy way for users visiting websites to opt out of this is to not use Google Chrome and use browsers like Firefox, etc. However, website maintainers can also help against this new tracking technology by opting out of the FLoC network.
Permissions-Policy Header
So the main way of opting out of this surveillance technology is to add a HTTP response header to their websites.
The HTTP response header is
Permissions-Policy: interest-cohort()
The FLoC technology uses interest-cohort to check for an allowlist. By default, everything is allowed as long as the user is visiting from a browser which supports InterestCohort API. However, by mentioning interest-cohort() in the Permissions-Policy header, the website is opting out from allowing any origin (including the current page) from being tracked using FLoC. Hence the FLoC feature is turned off for the website, even if the user is visiting your website from a Google Chrome browser.
NGINX
To add the above header, go to your nginx configuration file and add the following inside the server block:
server {
...
add_header Permissions-Policy "interest-cohort=()";
...
}
If you have different nginx confs for multiple websites running via nginx, you have to do the above in all the server blocks or nginx configuration files.
Then run nginx -t to test that everything is correct in your updated nginx configuration.
Then, restart nginx by running the command service nginx restart (or any other command that you might use based on your OS to restart nginx)
I am conducting 2 repeat sessions for a workshop on "Writing Python modules in Rust".
The first session is on 16th April, 1500 UTC onwards, and the repeat session will be on
18th April 0900 UTC. More details can be found in this issue.
You don't have to have any prior Rust knowledge. I will be providing working code, and we will
go very slowly to have a working Python module with useful functions in it.
If you are planning to attend or know anyone else who may want to join, then please point them to the issue link.
Chatted with a few people about the state of Python and pip on Debian.
General notes
Looking back, I think I picked up a couple of new projects based on
random brain waves I had! That’s perfectly timed, because I’ve decided
to pivot away from my earlier approach of “yay, more responsibility!”.
What next?
This is getting increasingly harder to decide on, as my free time
chunks are becoming smaller and I’m picking up bigger projects. :)
Technical
Sphinx Theme PEP 517 stuff: Make the initial release.
sphinx-basic-ng: Make the first usable release.
pip: Clear some of the backlog on the pull request front.
pip: More progress on the documentation rewrite.
Communication
Spend more time looking into the Python lockfile standardisation effort.
Write a blog post, on automated code formatting.
Find more speaking opportunities, to talk about things that aren’t Python packaging!
Sure, it sounds like a click-bait headline, but I'm serious. I've bought my Android phone, a Google Nexus X5, second-hand from ebay. It's a great device, popular with developers but it's not known for its long lasting battery. In fact, I had to charge my phone at least once a day, if not more often.
But not anymore. During the last three weeks I charged it twice, no more. I didn't just turn it off. What I did is simple: I stopped using it. I turned off the WiFi, the mobile data connection and as a result I don't have much reason to look at it very often. Sometimes not even once a day. I do get phone calls from time to time, and I even play some puzzle games while I brush my teeth, but that's it.
Why no smart phone?
I would say I was an average smart phone user. I do some of that social networking thingies, message with friends and family and if I don't know the answer to a random question I ask duckduckgo. I've not been a poweruser, I don't have a zillion followers on any platform, and I don't take pictures of me or my food or my cats (don't have any) all the time. But I did use my phone regularly to check my email, my calendar or via termux work remotely on several Linux servers.
I cannot give any reliable numbers about the amount of time I've spent staring on my smart phone, but I can tell that I took it everywhere. I took it to the toilet to read my mastodon timeline and I took it with me when I went to bed to read my RSS feeds. I had it lying next to me on the table during meals or at work, so I could peek at incoming messages.
After a couple of weeks without using my smart phone now, I can tell that I miss none of that. I integrated some things into a more or less regular schedule, like checking feeds and mastodon on the laptop only once a day before I go to bed. It saves a lot of time and I never had the feeling I would miss anything. However, I find much more time to read books or play some guitar now, which I enjoy very much.
I spending a huge amount of my day in front of a screen anyway, due to my job and hobbies, but I feel that dropping the phone is absolutely beneficial for my health and also I could be the better role model for my children.
How no smart phone?
That leaves us with the question: What apps/services to drop and how to replace those that I'm not willing to drop?
The obvious
The easiest part was to port the most phone actions to the laptop. I did read mails on the phone a lot, though I didn't reply very often. Sometimes I just flagged some as important and left the work for later. Doing email on my laptop only resulted in more consistency and structure for me, and additionally saved quite some time and stress in my off-work time.
Randomly browsing the web or looking up stuff on Wikipedia, just like reading feeds and timelines was also easy to replace and led to more structure and much less distraction. Interestingly, I don't think that I'm spending more time on the laptop now, but I never took any measure to verify that ;)
I used my phone as second factor for some two-factor authentication logins. For now I also replaced that with the OTP module of my password manager pass. It still is a valid second factor, as I need the device additionally to the laptop, however, I think I will replace that with a security key in future.
I still enjoy Spotify sometimes. To make that work without my phone, I installed spotifyd on my raspberry pi, which I use as media center. The daemon just serves as the player, while I'm controlling the playlist from my laptop (using a beautiful TUI written in Rust, called spt).
The messaging
I'm using Signal as my main messaging app, but also joined some Matrix rooms on different servers and even follow some Mattermost conversations. Since I never seriously used IRC on the phone, I instead tried to integrate the messaging to my favorite IRC client weechat, with quite decent success.
I'm running signald on my weechat server and fixed some on the signal-weechat plugin and message happily ever since. Signald is a java app that subscribes to my phone number and provides a socket that can be used to interact with the API. Is it as secure as the official Signal app? Nope. But that's not my threat model ;) It's way better than the crappy electron app, in my opinion.
After I configured that, I realized that there are also plugins available for matrix and Mattermost (the latter of which is in pre-alpha, but hey). Using weechat for all my messaging feels great and to me is a big improvement over the phone (and over web- and electron clients). I know that there may be security drawbacks, not in transportation but in the way data is stored on my server. Luckily I can address these issues and I can do that independent of any app.
The rest
I thought that I could just drop everything that I cannot port to laptop. However, I hardly found anything I had to give up. I still use the phone to take pictures sometimes (never took many) and use the audio recorder occasionally. I play less random games.
The only thing I miss is the calendar functionality. I do have an excellent calendar on the laptop, or course (khal). But sometimes I would like to quickly glance at my upcoming events or schedule an appointment without being on the laptop. I will need to think how to do that without going fully analog on this one again. I'm having an eye on the Mudita Pure which I think would support me in exactly the way I chose to take.
I presented 2 talks at FOSDEM: in the Python devroom and
Open Source Design devroom. Shout-out to Bernard Tyers, for all the help and
the bazillion reminders to make sure I do all the things on time. :)
Collaborating on designing a lockfile format for Python, that can hopefully
be standardised for interoperability.
General notes
Onboarding in a new company, relocating internationally, settling into a new
space has been… well, it’s all been a very interesting learning experience.
Given the fairly strict lockdown and the percentage of people wearing masks
in my locality, I’ve spent a lots of time indoors. Looking forward to the
social weekends experiment I’m doing.
What next?
Technical
pip: Work on the documentation rewrite, hopefully to get it ready in time for
the next release.
pip: Clear some of the backlog on the pull request front.
pip: General discussions for new features and enhancements.
TOML: Work on writing that the compliance test suite.
TOML: Bring toml for Python back from the dead.
Furo: Make the first stable release.
Start work on the other Sphinx theme I have in mind.
Communication
Spend more time looking into the Python lockfile standardisation effort.
Catch up on the Python-on-Debian saga, and see how I can contribute
constructively.
A common question that we got during developing this feature is what's the use of an anonymous chat room since we already have end-to-end encrypted messaging apps. It leaves a lot fewer traces.
The way we achieve this is very simple. There is no form of storage whatsoever in OnionShare chat mode. The chat is not persistent. The chat server stores no information at all (not even the usernames of people chatting). So once the chat server is closed, and the Tor Browser tab with the chat client is closed, there is no data (or metadata) related to chat that remains, even in the person's system who started the server. Hence, it leaves much less trace compared to other chat applications.
A good example of the above as mentioned by Micah in his blog is:
If, for example, you send a message to a Signal group, a copy of your message ends up on each device (the devices, and computers if they set up Signal Desktop of each member of the group). Even if disappearing messages is turned on it’s hard to confirm all copies of the messages are actually deleted from all devices, and from any other places (like notifications databases) they may have been saved to. OnionShare chat rooms don’t store any messages anywhere, so the problem is reduced to a minimum.
Given that the OnionShare chat feature works over the onion network, so it also has the additional anonymity feature. Also, adding to the anonymity feature, OnionShare chat doesn't need any form of signing in. Hence, people chatting can stay anonymous, and everything happens inside the tor network. One can just start a chat server, share the link via some disposable way, and then wait for the other people to join while maintaining anonymity.
Because it's an onion service, there is no way for an attacker to eavesdrop on the messages. The closest they can get is if they run a malicious Tor rendezvous node that's chosen for the onion service, they'll be able to spy on encrypted onion traffic. So, there's no capturing ciphertext to decrypt later on.
So what happens under the hood?
The chat feature is dependent on flask-socketio and eventlet for the WebSocket server implementation, and socket.io client js for the frontend implementation of the chat client. So when a chat server is started, the WebSocket is started in a namespace "/chat". Whenever a new user joins the link, they are given a randomly generated username and they are added to the room "default". There is only one room, and the actual name of the room can be set from the OnionShare settings-related code, but it doesn't really impact anything in the implementation. Both the room name and the randomly generated username are stored in a flask session. But that information is also completely gone once the chat server is stopped. The room and username information are only there to emit the messages properly.
You can also change the randomly generated username to a username (or pseudo username) of your choice for that particular session.
There are two main types of messages:
status messages - these are sent from the client to the server only when a new user joins or someone updates their username. The status message is then broadcasted to all the other connected clients, who will then see it as a form of a status message in the chat window.
user messages - these are sent when a user sends a message. All messages are broadcasted, so in case you share the link to multiple users, there is no concept of private message and everyone connected to the room can view your messages. Hence, sharing the onion link securely is important.
All of these WebSocket communication happens over the Tor onion services. OnionShare in itself doesn't implement any encryption algorithm to the chat and heavily relies on the Tor onion service's encryptions for the same. The message from the client to the OnionShare server is E2EE as it goes via Tor's onion connection. Then the OnionShare server broadcasts the message to all the other clients connected to the chat room through their E2EE onion connection, over WebSockets.
So what now?
I feel, as of now, the OnionShare anonymous chat is great if you quickly want to have an anonymous, secure, non-persistent conversation with someone or a group of people. It is also great if a whistleblower wants to share some details over chat with a journalist and then remove all traces of that conversation completely. But I feel if someone needs to host a chat server for a long time where people can connect anonymously, this is probably not the best solution for that.
There are still some issues that we will hopefully improve in the next releases. Firstly, we need to try and make it a bit more asynchronous. Right now, if left inactive for a long time, the Tor connection over WebSocket is sometimes dropped which isn't a great user experience. We are working on improving that.
Also, we will improve the UI for the chat more to give a better user experience.
With the new tabs feature, one can have all different modes (even multiple servers of same mode) running at the same time. So you can have a chat server and share mode server running at the same time. All the modes are very independent of each other and hence don't affect one another in any way.
I hope you all enjoy the new chat feature and leave feedbacks/suggestions on how to improve it. You can also read more about this and other features at docs.onionshare.org
I recently get really pissed quite often when I browse the web; I'm nuked with ads and pop-ups or - should I dare to use an adblocker - will not be able to see the content I was looking for at all. With the GDPR in place, living in Europe also reveals the incredible amount of Cookies used for profiling and tracking my browsing behavior. I'm being tracked everywhere, always.
Sure, I can opt-out. That's what they say. And it's a bad joke.
I cannot opt-out of what is called "essential to make the site run", which is obviously a whole lot. Technically it's forbidden to make the opt-out process harder then the opt-in, but I have never seen a single of these modal dialogs that would have offered me an "opt-out of everything" button that was just as easy to find as the "I agree to everything" button. Now even if I opt-out of everything, sites hide their "legitimate interest" settings on another page where I often need to manually de-select zillion items.
That's not the web I learned to love and I want to use as a major source of daily information.
I also recently learned about a new internet protocol that aims to make things different than HTTP/S, called gemini. If you're familiar with the gopher protocol, which has been around for some time now, you can put gemini somewhere between gopher and HTTP/S. By intention it does not much. It will let you display text, basically, that you type in a markdown-like syntax called gemtext. The function set is very limited on purpose, so content counts. It does not support images or inline links, but three levels of headings and code blocks. gemini does also TLS by default and explicitly welcomes self-signed certificates.
Hosting a Gemlog
To serve a gemlog, a server is needed; since TLS is a requirement, it's not sufficient to just serve the file structure. There are a couple of lightweight servers available, I'm using gemserv here, which I found fairly easy to compile and configure.
Generating a Certificate
I use openssl to generate a private key and a certificate signing request (CSR):
This will generate an X.509 certificate signing request with a new RSA 4096 bit key. The nodes option will skip encryption of the key.
You can put more information in the subject name, it's important to have the CN right, however, to let it match your domain.
The private_key.txt and certificate.txt will need to be on our server.
Configure gemserv
If available in your packet manager, install gemserv from there, or clone the repository and follow the build instructions. The configuration can be as simple as the following config.toml file:
port = 1965
host = "0.0.0.0"
[[server]]
hostname = "gemini.schubisu.de"
key = "/path/to/private_key.txt"
cert = "/path/to/certificate.txt"
dir = "/path/to/content"
and simply run
/path/to/gemserv config.toml
For local testing you don't even need to have a registered domain, just add a line like this to your /etc/hosts file:
127.0.0.1 gemini.schubisu.de localhost
Converting my lektor blog to gemtext
I write my blog with a lightweight CMS for static websites written in python, called lektor. I always liked about lektor that it's file structure mirrors the resulting web page. Every page I write is represented by a folder that contains at least one content.lr file which consists of a bit yaml and markdown.
This is very much like what gemini expects, which will either show you the subfolders of a directory, or - if present - renders the index.gmi (or index.gemini) file.
I started by renaming all contents.lr files to index.gemini to get the quickest possible working gemlog and an initial idea of what work lies ahead of me. It works, of course, but there is still plenty of room for improvement. What needs to be adjusted are
Meta data: The contents.lr starts with a yaml block of meta data which could be anything. In my case it's the title, publication date and some tags.
Inline links and links in general: Gemini does not allow inline links, so my links need to be converted to the correct format.
Navigation: I want to allow quick navigation to previous and next blog posts, as well as returning to an overview.
I decided to write a small python script, to use lektor's API to access all meta information and contents. I also reviewed all my blog posts to bring the links into a standardized format:
# inline links:
[some text][identifyer:my_link_name]
# and at the bottom of the document:
[identifyer:my_link_name]: https://...
The script will convert that into valid gemini links that are numerically indexed inline when rendered:
# inline links:
some text[1]
# and just below the paragraph:
[1] my_link_name [some text]
I also edited a few minor things in my contents.lr to make the markdown comply with gemtext:
gemtext does not allow nested bullet lists, so I need to reduce lists to one level wherever I used more
Different than with when I convert markdown to HTML, gemtext will break a line wherever I inserted a line break. If I want something to show as a nice paragraph, it should not be interrupted by line breaks.
Inline text formatting to emphasize some words is not possible in gemtext. However, I find markdown with inline formatting still readable enough, so I left that as is.
I found a whole lot of typos :)
I've uploaded the script to convert the blog here. It's pretty much tailored to my needs. However, it may be useful if you happen to run into the same or similar idea :)
For what it's worth, I can now simply run my conversion script, rsync the files to my gemini server and that's it.
I was talking to Jason today, discussing aspects of life and building habits.
Last week was pretty rough, devoid of motivation to maintain my habits. To which he said to me, “to fall in love with boredom” and passed me a copule post from James Clear.
How to Fall in Love With Boredom and Unlock Your Mental Toughness How to Stay Focused When You Get Bored Working Toward Your Goals Lessons on Success and Deliberate Practice from Mozart, Picasso, and Kobe Bryant The first blog was one I could relate to in an instant.
Thunderbird is the free and open source email client by Mozilla Foundation. I have been using it for some years now. Till now the Thunderbird users had to use an extension Enigmail to use GnuPG. Thunderbird 78 now uses a different implementation of OpenPGP called RNP.
Since RNP library still does not support the use of secret key on smartcards, to use Yubikey or any other GnuPG enabled smartcards, we need manually configure Thunderbird with GnuPG. The steps as said are the following :
Install GPGME
dnf install GPGME
GPGME, GnuPG Made Easy library makes the GnuPG easily accessible by providing a high level crypto API for encrypt, decrypt, sign, verify and key management. I already have GnuPG installed in my Fedora 33 machine and my Yubikey ready.
Modify Thunderbird configuration
Go to the Preferences menu then click on the config editor button at the very end.
Click on the I accept the risk.
Search for mail.openpgp.allow_external_gnupg and switch to true.
Remember to restart the Thunderbird after that.
Configure the secret key usage form Yubikey
Now go to the Account Settings and then go to the End-To-End-Encryption at the sidebar. Select the Use your external key through GnuPG(e.g. from a smartcard) option and click on continue.
Type your Secret Key ID in the box and click on Save key ID.
Now open the OpenPGP Key Manager and import your public key and then verify.
Now you can start using your hardware token in Thunderbird.
In this case we have to use 2 keyrings - GnuPG and RNP’s keyring (internal in Thunderbird). This is an extra step, which I hope in future can be avoided.
Mailvelope is an extension on web browsers to send end to end encrypted emails. This is a good option available to the users to send end to end encrypted without changing the email service they use. It is licensed under AGPL v3, making it Free and Open Source software. The code is there in Github for the community to have a look. This can be added as an extension to the - Chrome, Firefox and Edge browsers to securely encrypt emails with PGP using your email providers.
Mailvelope does provide end to end encryption for the email content but does not protect the metadata (subject, IP address of the sender) from third parties. As most of the email encryption tools, it does not work on the mobile browser. There is a detailed user guide on Mailvelope from the Freedom of the Press Foundation, which is really helpful for the new users.
By default, Mailvelope uses its own keyring. To use my Yubikey along with GnuPG keyring, I had to take the following steps:
I recently read this paper titled, Understanding Real-World Concurrency Bugs in Go (PDF), that studies concurrency bugs in Golang and comments on the new primitives for messages passing that the language is often known for.
I am not a very good Go programmer, so this was an informative lesson in various ways to achieve concurrency and synchronization between different threads of execution. It is also a good read for experienced Go developers as it points out some important gotchas to look out for when writing Go code. The fact that it uses real world examples from well known projects like Docker, Kubernetes, gRPC-Go, CockroachDB, BoltDB etc. makes it even more fun to read!
The authors analyzed a total of 171 concurrency bugs from several prominent Go open source projects and categorized them in two orthogonal dimensions, one each for the cause of the bug and the behavior. The cause is split between two major schools of concurrency
Along the cause dimension, we categorize bugs into those that are caused by misuse of shared memory and those caused by misuse of message passing
and the behavior dimension is similarly split into
we separate bugs into those that involve (any number of ) goroutines that cannot proceed (we call themblocking bugs) and those that do not involve any blocking (non-blocking bugs)
Interestingly, they chose the behavior to be blocking instead of deadlock since the former implies that atleast one thread of execution is blocked due to some concurrency bug, but the rest of them might continue execution, so it is not a deadlock situation.
Go has primitive shared memory protection mechanisms like Mutex, RWMutex etc. with a caveat
Write lock requests in Go have ahigher privilege than read lock requests.
as compared to pthread in C. Go also has a new primitive called sync.Once that can be used to guarantee that a function is executed only once. This can be useful in situations where some callable is shared across multiple threads of execution but it shouldn't be called more than once. Go also hassync.WaitGroups , which is similar to pthread_join to wait for various threads of executioun to finish executing.
Go also uses channels for the message passing between different threads of executions called Goroutunes. Channels can be buffered on un-buffered (default), the difference between them being that in a buffered channel the sender and receiver don't block on each other (until the buffered channel is full).
The study of the usage patterns of these concurrency primitives in various code bases along with the occurence of bugs in the codebase concluded that even though message passing was used at fewer places, it accounted for a larger number of bugs(58%).
Implication 1:With heavier usages of goroutines and newtypes of concurrency primitives, Go programs may potentiallyintroduce more concurrency bugs
Also, interesting to note is this observation in tha paper
Observation 5:All blocking bugs caused by message passing are related to Go’s new message passing semantics like channel. They can be difficult to detect especially when message passing operations are used together with other synchronization mechanisms
The authors also talk about various ways in which Go runtime can detect some of these concurrency bugs. Go runtime includes a deadlock detector which can detect when there are no goroutunes running in a thread, although, it cannot detect all the blocking bugs that authors found by manual inspection.
For shared memory bugs, Go also includes a data race detector which can be enbaled by adding -race option when building the program. It can find races in memory/data shared between multiple threads of execution and uses happened-before algorithm underneath to track objects and their lifecycle. Although, it can only detect a part of the bugs discovered by the authors, the patterns and classification in the paper can be leveraged to improve the detection and build more sophisticated checkers.
TLDR; Trying to learn new things I tried writing a URL shortner called shorty. This is a first draft and I am trying to approach it from first principle basis. Trying to break down everything to the simplest component.
I decided to write my own URL shortner and the reason for doing that was to dive a little more into golang and to learn more about systems. I have planned to not only document my learning but also find and point our different ways in which this application can be made scalable, resilient and robust.
A high level idea is to write a server which takes the big url and return me a short url for the same. I have one more requirement where I do want to provide a slug i.e a custom short url path for the same. So for some links like https://play.google.com/store/apps/details?id=me.farhaan.bubblefeed, I want to have a url like url.farhaan.me/linktray which is easy to remember and distribute.
The way I am thinking to implement this is by having two components, I want a CLI interface which talks to my Server. I don’t want a fancy UI for now because I want it to be exclusively be used through terminal. A Client-Server architecture, where my CLI client sends a request to the server with a URL and an optional slug. If a slug is present URL will have that slug in it and if it doesn’t it generates a random string and make the URL small. If you see from a higher level it’s not just a URL shortner but also a URL tagger.
The way a simple url shortner works:
Flow Diagram
A client makes a request to make a given URL short, server takes the URL and stores it to the database, server then generates a random string and maps the URL to the string and returns a URL like url.farhaan.me/<randomstring>.
Now when a client requests to url.farhaan.me/<randomstring>, it goest to the same server, it searches the original URL and redirects the request to a different website.
The slug implementation part is very straightforward, where given a word, I might have to search the database and if it is already present we raise an error but if it isn’t we add it in the database and return back the URL.
One optimization, since it’s just me who is going to use this, I can optimize my database to see if the long URL already exists and if it does then no need to create a new entry. But this should only happen in case of random string and not in case of slugs. Also this is a trade off between reducing the redundancy and latency of a request.
But when it comes to generating a random string, things get a tiny bit complicated. This generation of random strings, decides how many URLs you can store. There are various hashing algorithms that I can use to generate a string I can use md5, base10 or base64. I also need to make sure that it gives a unique hash and not repeated ones.
Unique hash can be maintained using a counter, the count either can be supplied from a different service which can help us to scale the system better or it can be internally generated, I have used database record number for the same.
If you look at this on a system design front. We are using the same Server to take the request and generate the URL and to redirect the request. This can be separated into two services where one service is required to generate the URL and the other just to redirect the URL. This way we increase the availability of the system. If one of the service goes down the other will still function.
The next step is to write and integrate a CLI system to talk to the server and fetch the URL. A client that can be used for an end user. I am also planning to integrate a caching mechanism in this but not something out of the shelf rather write a simple caching system with some cache eviction policy and use it.
Till then I will be waiting for the feedback. Happy Hacking.
I now have a Patreon open so that you folks can support me to do this stuff for longer time and sustain myself too. So feel free to subscribe to me and help me keeping doing this with added benefits.
TLDR; Link Tray is a utility we recently wrote to curate links from different places and share it with your friends. The blogpost has technical details and probably some productivity tips.
Link Bubble got my total attention when I got to know about it, I felt it’s a very novel idea, it helps to save time and helps you to curate the websites you visited. So on the whole, and believe me I am downplaying it when I say Link Bubble does two things:
Saves time by pre-opening the pages
Helps you to keep a track of pages you want to visit
It’s a better tab management system, what I felt weird was building a whole browser to do that. Obviously, I am being extremely naive when I am saying it because I don’t know what it takes to build a utility like that.
Now, since they discontinued it for a while and I never got a chance to use it. I thought let me try building something very similar, but my use case was totally different. Generally when I go through blogs or articles, I open the links mentioned in a different tab to come back to them later. This has back bitten me a lot of time because I just get lost in so many links.
I thought if there is a utility which could just capture the links on the fly and then I could quickly go through them looking at the title, it might ease out my job. I bounced off the same idea across to Abhishek and we ended up prototyping LinkTray.
Our first design was highly inspired by facebook messenger but instead of chatheads we have links opened. If you think about it the idea feels very beautiful but the design is “highly” not scalable. For example if you have as many as 10 links opened we had trouble in finding our links of interest which was a beautiful design problems we faced.
We quickly went to the whiteboard and put up a list of requirements, first principles; The ask was simple:
To share multiple links with multiple people with least transitions
To be able to see what you are sharing
To be able to curate links (add/remove/open links)
We took inspiration from an actual Drawer where we flick out a bunch of links and go through them. In a serendipitous moment the design came to us and that’s how link tray looks like the way it looks now.
Link Tray
Link Tray was a technical challenge as well. There is a plethora of things I learnt about the Android ecosystem and application development that I knew existed but never ventured into exploring it.
Link Tray is written in Java, and I was using a very loosely maintained library to get the overlay activity to work. Yes, the floating activity or application that we see is called an overlay activity, this allows the application to be opened over an already running application.
The library that I was using doesn’t have support for Android O and above. To figure that out it took me a few nights , also because I was hacking on the project during nights . After reading a lot of GitHub issues I figured out the problem and put in the support for the required operating system.
One of the really exciting features that I explored about Android is Services. I think I might have read most of the blogs out there and all the documentation available and I know that I still don't know enough. I was able to pick enough pointers to make my utility to work.
Just like Uncle Bob says make it work and then make it better. There was a persistent problem, the service needs to keep running in the background for it to work. This was not a functional issue but it was a performance issue for sure and our user of version 1.0 did have a problem with it. People got mislead because there was constant notification that LinkTray is running and it was annoying. This looked like a simple problem on the face but was a monster in the depth.
Architecture of Link Tray
The solution to the problem was simple stop the service when the tray is closed, and start the service when the link is shared back to link tray. Tried, the service did stop but when a new link was shared the application kept crashing. Later I figured out the bound service that is started by the library I am using is setting a bound flag to True but when they are trying to reset this flag , they were doing at the wrong place, this prompted me to write this StackOverflow answer to help people understand the lifecycle of service. Finally after a lot of logs and debugging session I found the issue and fixed it. It was one of the most exciting moment and it help me learn a lot of key concepts.
The other key learning, I got while developing Link Tray was about multi threading, what we are doing here is when a link is shared to link tray, we need the title of the page if it has and favicon for the website. Initially I was doing this on the main UI thread which is not only an anti-pattern but also a usability hazard. It was a network call which blocks the application till it was completed, I learnt how to make a network call on a different thread, and keep the application smooth.
Initially approach was to get a webview to work and we were literally opening the links in a browser and getting the title and favicon out, this was a very heavy process. Because we were literally spawning a browser to get information about links, in the initial design it made sense because we were giving an option to consume the links. Over time our design improved and we came to a point where we don’t give the option to consume but to curate. Hence we opted for web scraping, I used custom headers so that we don’t get caught by robot.txt. And after so much of effort it got to a place where it is stable and it is performing great.
It did take quite some time to reach a point where it is right now, it is full functional and stable. Do give it a go if you haven’t, you can shoot any queries to me.
So, recently I started using windows for work. Why? There are a couple of reasons, one that I needed to use MSVC, that is the Microsoft Visual C++ toolchain and the other being, I wasn’t quite comfortable to ifdef stuff for making it work on GCC aka, the GNU counterpart of MSVC.
After an anxious month, I am writing a Krita Weekly again and probably this would be my last one too, though I hope not. Let’s start by talking about bugs. Unlike the trend going about the last couple of months, the numbers have taken a serious dip.
[Published in Open Source For You (OSFY) magazine, October 2017 edition.]
This article is the eighth in the DevOps series. In this issue, we shall learn to set up Docker in the host system and use it with Ansible.
Introduction
Docker provides operating system level virtualisation in the form of containers. These containers allow you to run standalone applications in an isolated environment. The three important features of Docker containers are isolation, portability and repeatability. All along we have used Parabola GNU/Linux-libre as the host system, and executed Ansible scripts on target Virtual Machines (VM) such as CentOS and Ubuntu.
Docker containers are extremely lightweight and fast to launch. You can also specify the amount of resources that you need such as CPU, memory and network. The Docker technology was launched in 2013, and released under the Apache 2.0 license. It is implemented using the Go programming language. A number of frameworks have been built on top of Docker for managing these cluster of servers. The Apache Mesos project, Google’s Kubernetes, and the Docker Swarm project are popular examples. These are ideal for running stateless applications and help you to easily scale them horizontally.
Setup
The Ansible version used on the host system (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. Internet access should be available on the host system. The ansible/ folder contains the following file:
ansible/playbooks/configuration/docker.yml
Installation
The following playbook is used to install Docker on the host system:
The Parabola package repository is updated before proceeding to install the dependencies. The python2-docker package is required for use with Ansible. Hence, it is installed along with the docker package. The Docker daemon service is then started and the library/hello-world container is fetched and executed. A sample invocation and execution of the above playbook is shown below:
With verbose ’-v’ option to ansible-playbook, you will see an entry for LogPath, such as /var/lib/docker/containers//-json.log. In this log file you will see the output of the execution of the hello-world container. This output is the same when you run the container manually as shown below:
$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://cloud.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/
Example
A Deep Learning (DL) Docker project is available (https://github.com/floydhub/dl-docker) with support for frameworks, libraries and software tools. We can use Ansible to build the entire DL container from the source code of the tools. The base OS of the container is Ubuntu 14.04, and will include the following software packages:
Tensorflow
Caffe
Theano
Keras
Lasagne
Torch
iPython/Jupyter Notebook
Numpy
SciPy
Pandas
Scikit Learn
Matplotlib
OpenCV
The playbook to build the DL Docker image is given below:
We first clone the Deep Learning docker project sources. The docker_image module in Ansible helps us to build, load and pull images. We then use the Dockerfile.cpu file to build a Docker image targeting the CPU. If you have a GPU in your system, you can use the Dockerfile.gpu file. The above playbook can be invoked using the following command:
Depending on the CPU and RAM you have, it will take considerable amount of time to build the image with all the software. So be patient!
Jupyter Notebook
The built dl-docker image contains Jupyter notebook which can be launched when you start the container. An Ansible playbook for the same is provided below:
- name: Start Jupyter notebook
hosts: localhost
gather_facts: true
become: true
tags: [notebook]
vars:
DL_DOCKER_NAME: "floydhub/dl-docker"
tasks:
- name: Run container for Jupyter notebook
docker_container:
name: "dl-docker-notebook"
image: "{{ DL_DOCKER_NAME }}:cpu"
state: started
command: sh run_jupyter.sh
You can invoke the playbook using the following command:
The Dockerfile already exposes the port 8888, and hence you do not need to specify the same in the above docker_container configuration. After you run the playbook, using the ‘docker ps’ command on the host system, you can obtain the container ID as indicated below:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a876ad5af751 floydhub/dl-docker:cpu "sh run_jupyter.sh" 11 minutes ago Up 4 minutes 6006/tcp, 8888/tcp dl-docker-notebook
You can now login to the running container using the following command:
$ sudo docker exec -it a876 /bin/bash
You can then run an ‘ifconfig’ command to find the local IP address (“172.17.0.2” in this case), and then open http://172.17.0.2:8888 in a browser on your host system to see the Jupyter Notebook. A screenshot is shown in Figure 1:
TensorBoard
TensorBoard consists of a suite of visualization tools to understand the TensorFlow programs. It is installed and available inside the Docker container. After you login to the Docker container, at the root prompt, you can start Tensorboard by passing it a log directory as shown below:
# tensorboard --logdir=./log
You can then open http://172.17.0.2:6006/ in a browser on your host system to see the Tensorboard dashboard as shown in Figure 2:
Docker Image Facts
The docker_image_facts Ansible module provides useful information about a Docker image. We can use it to obtain the image facts for our dl-docker container as shown below:
The ANSIBLE_STDOUT_CALLBACK environment variable is set to ‘json’ to produce a JSON output for readability. Some important image facts from the invocation of the above playbook are shown below:
[NOTE: This post originally appeared on
deepsource.io, and has been
posted here with due permission.]
In the early part of the last century, when David Hilbert was working on
stricter formalization of geometry than Euclid, Georg Cantor had worked out a
theory of different types of infinities, the theory of sets. This
theory would soon unveil a series of confusing paradoxes, leading to
a crisis in the Mathematics community regarding the stability of the
foundational principles of the math of that time.
Central to these paradoxes was the Russell’s paradox (or more generally, as
we’d talk about later, the Epimenides Paradox). Let’s see what it is.
In those simpler times, you were allowed to define a set if you could describe
it in English. And, owing to mathematicians’ predilection for self-reference,
sets could contain other sets.
Russell then, came up with this:
\(R\) is a set of all the sets which do not contain themselves.
The question was "Does \(R \) contain itself?" If it doesn’t, then according to
the second half of the definition it should. But if it does, then it no longer
meets the definition.
The same can symbolically be represented as:
Let \(R = \{ x \mid x \not \in x \} \), then \(R \in R \iff R \not \in R \)
Cue mind exploding.
“Grelling’s paradox” is a startling variant which uses adjectives instead of
sets. If adjectives are divided into two classes, autological
(self-descriptive) and heterological (non-self-descriptive), then, is
‘heterological’ heterological? Try it!
Epimenides Paradox
Or, the so-called Liar Paradox was another such paradox which shred apart
whatever concept of ‘computability’ was, at that time - the notion that things
could either be true or false.
Epimenides was a Cretan, who made one immortal statement:
“All Cretans are liars.”
If all Cretans are liars, and Epimenides was a Cretan, then he was lying when
he said that “All Cretans are liars”. But wait, if he was lying then, how can
we ‘prove’ that he wasn’t lying about lying? Ein?
This is what makes it a paradox: A statement so rudely violating the assumed
dichotomy of statements into true and false, because if you tentatively think
it’s true, it backfires on you and make you think that it is false. And a
similar backfire occurs if you assume that the statement is false. Go ahead,
try it!
If you look closely, there is one common culprit in all of these paradoxes,
namely ‘self-reference’. Let’s look at it more closely.
Strange Loopiness
If self-reference, or what Douglas Hofstadter - whose prolific work on the
subject matter has inspired this blog post - calls ‘Strange Loopiness’ was the
source of all these paradoxes, it made perfect sense to just banish
self-reference, or anything which allowed it to occur. Russell and Whitehead,
two rebel mathematicians of the time, who subscribed to this point of view, set
forward and undertook the mammoth exercise, namely “Principia Mathematica”,
which we as we will see in a little while, was utterly demolished by Gödel’s
findings.
The main thing which made it difficult to ban self-reference was that it was
hard to pin point where exactly did the self-reference occur. It may as well be
spread out over several steps, as in this ‘expanded’ version of Epimenides:
The next statement is a lie.
The previous statement is true.
Russell and Whitehead, in P.M. then, came up with a multi-hierarchy set
theory to deal with this. The basic idea was that a set of the lowest ‘type’
could only contain ‘objects’ as members (not sets). A set of the next type
could then only either contain objects, or sets of lower types. This,
implicitly banished self-reference.
Since, all sets must have a type, a set ‘which contains all sets which are not
members of themselves’ is not a set at all, and thus you can say that Russell’s
paradox was dealt with.
Similarly, if an attempt is made towards applying the expanded Epimenides to
this theory, it must fail as well, for the first sentence to make a reference
to the second one, it has to be hierarchically above it - in which case, the
second one can’t loop back to the first one.
Thirty one years after David Hilbert set before the academia to rigorously
demonstrate that the system defined in Principia Mathematica was both
consistent (contradiction-free) and complete (i.e. every true statement
could be evaluated to true within the methods provided by P.M.), Gödel
published his famous Incompleteness Theorem. By importing the Epimenides
Paradox right into the heart of P.M., he proved that not just the
axiomatic system developed by Russell and Whitehead, but none of the
axiomatic systems whatsoever were complete without being inconsistent.
Clear enough, P.M. lost it’s charm in the realm of academics.
Before Gödel’s work too, P.M. wasn’t particularly loved as well.
Why?
It isn’t just limited to this blog post, but we humans, in general, have a diet
for self-reference - and this quirky theory severely limits our ability to
abstract away details - something which we love, not only as programmers, but
as linguists too - so much so, that the preceding paragraph, “It isn’t …
this blog … we humans …” would be doubly forbidden because the ‘right’
to mention ‘this blog post’ is limited only to something which is
hierarchically above blog posts, ‘metablog-posts’. Secondly, me (presumably a
human) belonging to the class ‘we’ can’t mention ‘we’ either.
Since, we humans, love self-reference so much, let’s discuss some ways in which
it can be expressed in written form.
One way of making such a strange loop, and perhaps the ‘simplest’ is using the
word ‘this’. Here:
This sentence is made up of eight words.
This sentence refers to itself, and is therefore useless.
This blog post is so good.
This sentence conveys you the meaning of ‘this’.
This sentence is a lie. (Epimenides Paradox)
Another amusing trick for creating a self-reference without using the word
‘this sentence’ is to quote the sentence inside itself.
Someone may come up with:
The sentence ‘The sentence contains five words’ contains five words.
But, such an attempt must fail, for to quote a finite sentence inside itself
would mean that the sentence is smaller than itself. However, infinite
sentences can be self-referenced this way.
The sentence
"The sentence
"The sentence
...etc
...etc
is infinitely long"
is infinitely long"
is infinitely long"
There’s a third method as well, which you already saw in the title - the Quine
method. The term ‘Quine’ was coined by Douglas Hofstadter in his book “Gödel
Escher, Bach” (which heavily inspires this blog post). When using this, the
self-reference is ‘generated’ by describing a typographical entity, isomorphic
to the quine sentence itself. This description is carried in two parts - one is
a set of ‘instructions’ about how to ‘build’ the sentence, and the other, the
‘template’ contains information about the construction materials required.
The Quine version of Epimenides would be:
“yields falsehood when preceded by it’s quotation” yields falsehood when preceded by it’s quotation
Before going on with ‘quining’, let’s take a moment and realize how awfully
powerful our cognitive capacities are, and what goes in our head when a
cognitive payload full of self-references is delivered - in order to decipher
it, we not only need to know the language, but also need to work out the
referent of the phrase analogous to ‘this sentence’ in that language. This
parsing depends on our complex, yet totally assimilated ability to handle the
language.
The idea of referring to itself is quite mind-blowing, and we keep doing it all
the time — perhaps, why it feels so ‘easy’ for us to do so. But, we aren’t born
that way, we grow that way. This could better be realized by telling someone
much younger “This sentence is wrong.”. They’d probably be confused - What
sentence is wrong?. The reason why it’s so simple for self-reference to occur,
and hence allow paradoxes, in our language, is well, our language. It allows
our brain to do the heavy lifting of what the author is trying to get through
us, without being verbose.
Back to Quines.
Reproducing itself
Now, that we are aware of how ‘quines’ can manifest as self-reference, it would
be interesting to see how the same technique can be used by a computer program
to ‘reproduce’ itself.
To make it further interesting, we shall choose the language most apt for the
purpose - brainfuck:
Running that program above produces itself as the output. I agree, it isn’t the
most descriptive program in the world, so written in Python below, is the
nearest we can go to describe what’s happening inside those horrible chains of
+’s and >’s:
The first line generates """ on the fly, which marks multiline strings in
Python.
Next two lines define the eniuq function, which prints the argument template
twice - once, plain and then surrounded with triple quotes.
The last 4 lines cleverly call this function so that the output of the program
is the source code itself.
Since we are printing in an order opposite of quining, the name of the function
is ‘quine’ reversed -> eniuq (name stolen from Hofstadter again)
Remember the discussion about how self-reference capitalizes on the processor?
What if ‘quining’ was a built-in feature of the language, providing what we in
programmer lingo call ‘syntactic sugar’?
Let’s assume that an asterisk, * in the brainfuck interpreter would copy the
instructions before executing them, what would then be the output of the
following program?
*
It’d be an asterisk again. You could make an argument that this is silly, and
should be counted as ‘cheating’. But, it’s the same as relying on the
processor, like using “this sentence” to refer to this sentence - you rely on
your brain to do the inference for you.
What if eniuq was a builtin keyword in Python? A perfect self-rep was then
just be a call away:
eniuq('eniuq')
What if quine was a verb in the English language? We could reduce a lot of
explicit cognitive processes required for inference. The Epimenides paradox
would then be:
“yields falsehood if quined” yields falsehood if quined
Now, that we are talking about self-rep, here’s one last piece of entertainment
for you.
If you take that absurd thing above, and move around in the cartesian plane for
the coordinates \(0 \le x \le 106, k \le y \le k + 17\), where \(k\) is a
544 digit integer (just hold on with me here), color every pixel black for
True, and white otherwise, you'd get:
I wanted to attend TUG 2019 for two main reasons - to present my work on the “XeTeX Book Template”, and also to meet my favourite computer scientist, Prof. Donald Knuth. He does not travel much, so, it was one of those rare opportunities for me to meet him in person. His creation of the TeX computer typesetting system, where you can represent any character mathematically, and also be able to program and transform it is beautiful, powerful and the best typesetting software in the world. I have been using TeX extensively for my documentation and presentations over the years.
Day I
I reached the hotel venue only in the afternoon of Friday, August 9, 2019, as I was also visiting Mountain View/San Jose on official work. I quickly checked into the hotel and completed my conference registration formalities. When I entered the hall, Rishi T from STM Document Engineering Private Limited, Thiruvananthapuram was presenting a talk on “Neptune - a proofing framework for LaTeX authors”. His talk was followed by an excellent poetic narration by Pavneet Arora, who happened to be a Vim user, but, also mentioned that he was eager to listen to my talk on XeTeX and GNU Emacs.
After a short break, Shreevatsa R, shared his experiences on trying to understand the TeX source code, and the lessons learnt in the process. It was a very informative, user experience report on the challenges he faced in navigating and learning the TeX code. Petr Sojka, from Masaryk University, Czech Republic, shared his students’ experience in using TeX with a detailed field report. I then proceeded to give my talk on the “XeTeX Book Template” on creating multi-lingual books using GNU Emacs and XeTeX. It was well received by the audience. The final talk of the day was by Jim Hefferon, who analysed different LaTeX group questions from newbies and in StackExchange, and gave a wonderful summary of what newbies want. He is a professor of Mathematics at Saint Michael’s College, and is well-known for his book on Linear Algebra, prepared using LaTeX. It was good to meet him, as he is also a Free Software contributor.
The TUG Annual General Meeting followed with discussions on how to grow the TeX community, the challenges faced, membership fees, financial reports, and plan for the next TeX user group conference.
Day II
The second day of the conference began with Petr Sojka and Ondřej Sojka presenting on “The unreasonable effectiveness of pattern generation”. They discussed the Czech hyphenation patterns along with a pattern generation case study. This talk was followed by Arthur Reutenauer presenting on “Hyphenation patterns in TeX Live and beyond”. David Fuchs, a student who worked with Prof. Donald Knuth on the TeX project in 1978, then presented on “What six orders of magnitude of space-time buys you”, where he discussed the design trade-offs in TeX implementation between olden days and present day hardware.
After a short break, Tom Rokicki, who was also a student at Stanford and worked with Donald Knuth on TeX, gave an excellent presentation on searching and copying text in PDF documents generated by TeX for Type-3 bitmap fonts. This session was followed by Martin Ruckert’s talk on “The design of the HINT file format”, which is intended as a replacement of the DVI or PDF file format for on-screen reading of TeX output. He has also authored a book on the subject - “HINT: The File Format: Reflowable Output for TeX”. Doug McKenna had implemented an interactive iOS math book with his own TeX interpreter library. This allows you to dynamically interact with the typeset document in a PDF-free ebook format, and also export the same. We then took a group photo:
I then had to go to Stanford, so missed the post-lunch sessions, but, returned for the banquet dinner in the evening. I was able to meet and talk with Prof. Donald E. Knuth in person. Here is a memorable photo!
He was given a few gifts at the dinner, and he stood up and thanked everyone and said that “He stood on the shoulders of giants like Isaac Newton and Albert Einstein.”
< />
I had a chance to meet a number of other people who valued the beauty, precision and usefulness of TeX. Douglas Johnson had come to the conference from Savannah, Georgia and is involved in the publishing industry. Rohit Khare, from Google, who is active in the Representational State Transfer (ReST) community shared his experiences with typesetting. Nathaniel Stemen is a software developer at Overleaf, which is used by a number of university students as an online, collaborative LaTeX editor. Joseph Weening, who was also once a student to Prof. Donald Knuth, and is at present a Research Staff member at the Institute for Defense Analyses Center for Communications Research in La Jolla, California (IDA/CCR-L) shared his experiences in working with the TeX project.
Day III
The final day of the event began with Antoine Bossard talking on “A glance at CJK support with XeTeX and LuaTeX”. He is an Associate Professor of the Graduate School of Science, Kanagawa University, Japan. He has been conducting research regarding Japanese characters and their memorisation. This session was followed by a talk by Jaeyoung Choi on “FreeType MF Module 2: Integration of Metafont and TeX-oriented bitmap fonts inside FreeType”. Jennifer Claudio then presented the challenges in improving Hangul to English translation.
After lunch, Uwe Ziegenhagen demonstrated on using LaTeX to prepare and automate exams. This was followed by a field report by Yusuke Terada, on how they use TeX to develop a digital exam grading system at large scale in Japan. Chris Rowley, from the LaTeX project, then spoke on “Accessibility in the LaTeX kernel - experiments in tagged PDF”. Ross Moore joined remotely for the final session of the day to present on “LaTeX 508 - creating accessible PDFs”. The videos of both of these last two talks are available online.
A number of TeX books were made available for free for the participants, and I grabbed quite a few, including a LaTeX manual written by Leslie Lamport. Overall, it was a wonderful event, and it was nice to meet so many like-minded Free Software people.
A special thanks to Karl Berry, who put in a lot of effort in organizing the conference, but, could not make it to due to a car accident.
Flow Diagram
Link Tray
, also because I was hacking on the project during nights 
. After reading a lot of GitHub issues I figured out the problem and put in the support for the required operating system.
Architecture of Link Tray
< />
