Tor provides a SOCKS proxy so that you can have any application using the same
to connect the Onion network. The default port is 9050. The Tor Browser also
provides the same service on port 9150. In this post, we will see how can we
use the same SOCKS proxy to access the Internet using Rust.
You can read my previous post to do the same using Python.
[package]
name = "usetor"
version = "0.1.0"
authors = ["Kushal Das <mail@kushaldas.in>"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
tokio = { version = "0.2", features = ["macros"] }
reqwest = { version = "0.10.4", features = ["socks", "json"] }
serde_json = "1.0.53"
The source code:
use reqwest;
use tokio;
use serde_json;
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
let proxy = reqwest::Proxy::all("socks5://127.0.0.1:9050").unwrap();
let client = reqwest::Client::builder()
.proxy(proxy)
.build().unwrap();
let res = client.get("https://httpbin.org/get").send().await?;
println!("Status: {}", res.status());
let text: serde_json::Value = res.json().await?;
println!("{:#?}", text);
Ok(())
}
Here we are converting the response data into JSON using serde_json. The output looks like this.
Many of us live on the terminal. We use tools which allows us to do things
faster and let us stay productive. Most of these tools are old. Sometimes we do
pick up a few new generation command line tools. Here is a small list of tools
I am using daily. All of these are written in Rust .
ripgrep
ripgrep was the first Rust tool I
started using daily as a replacement for grep. It is easy to use. The output
looks nice, and also works with my vim.
exa
exa is the replacement for ls. It includes many useful flags.
bat
bat is the one stop replacement for cat and
less. It also provides syntax highlighting with nice colours. I do have an
alias cat=/usr/bin/bat -p.
zoxide
zoxide allows to move around directories super fast.
starship
starship is the shell prompt you can see in all of the GIFs above. It
allows a lot of customization.
All of these tools are packaged in Fedora 32 by the amazing fedora-rust SIG.
Prometheus is a monitoring system and a time series database. It can collect metrics from different places and store it as series of values over time. It uses pull based mechanism to collect the metrics. Applications can expose the metrics in a plain text format using HTTP server, which is then fetched by Prometheus. Fetching of metrics is called scraping.
For other systems which don’t expose the metrics in Prometheus exposition format, we can use exporters.
The Gnome Foundation settled the dispute with Rothschild Patent Imaging (RPIL), RPIL agrees not to sue Gnome further for any intellectual property infringement. RPIL also signed to an undertaking to that effect.
In the last week of September 2019, Rothschild Patent Imaging (RPIL) filed a lawsuit against the Gnome Foundation. The case was filed under Title 35 of the United States Code for the infringement of the patent and violation of the intellectual property rights of the RPIL. They claimed that Gnome's Shotwell Photo manager, infringed the patent titled “Wireless Image Distribution System and Method,” being number 086.
What is Shotwell photo manager?
Shotwell photo manager is a personal image organizer in a Gnome environment. It imports photos using the libgphoto2 library, written in Vala. The library provides the bindings in Python, Java, C, Go, Rust, Ruby, and various other languages.
Shotwell can access and import photos and videos from a digital camera. It provides several editing features such as eliminating red-eye, crop, adjust the color balance. It can group the images, videos by date, supports tagging and allows the user to upload them on Facebook, Flickr, Piwigo, and YouTube.
The method for image capturing device work comprises receiving several photographic images. The images can be filtered based on a theme, topic. Then transmitting them to another device through wireless. Thus violates the patent of RPIL.
Let us pause for a second; we have been using these features for long in other platforms. Like receiving a bunch of photos on the phone, filtering them based on a topic or theme, and transmitting them wirelessly to someone else’s mobile device, say lightroom in Macbook.
RPIL is a perfect example of a patent troll. And the patent is an ideal specimen of a patent that should not have been granted. The patent intends to protect the intellectual property that did not satisfy two criteria for granting a patent - “novelty” and “inventive step.”
The settlement
On 20th May 2020, RPIL and Gnome Foundation have come to a settlement. In this walk-away settlement, the plaintiff and defendant both agreed not to carry on the legal proceedings. The plaintiff further agreed to an undertaking to not to sue the defendant for any patent held by Rothschild Patent Imaging. Both Leigh Rothschild and RPIL agreed to grant release to any software licensed under Open Source Initiative Licenses. This includes the entire patent portfolio of Rothschild and any such software that forms a foundational portion of the infringement allegation.
Neil McGovern, Executive Director for the GNOME Foundation (The defendant), said, “I’m exceptionally pleased that we have concluded this case. This will allow us to refocus our attention on creating a free software desktop, and will ensure certainty for all free and open source software in future.”
Leigh Rothschild said that he is happy to settle the matter amicably.
But this poses a grave question should the patent 083 was rightly granted in the first place? Should not the patent office be more careful while granting patents? It is high time for the patent office for introspection to save an organization from loss and put a halt to innovation.
I fell sick, and now I seem to have personal issues because of quarantine.
So a bit of hiatus. But I’m not abandoning it this time.
Will add all the days I missed and keep going, once I am back.
Which I really hope, is soon.
Lately I have been trying to improve my system administration skills. I needed to monitor some of the running processes. To do that I used psutil module in Python. It is a cross platform module to parse the information about running process. It also provides information on how the system is being utilized. In the area of - CPU, memory, disks, network, sensors. The name psutil is the abbreviation for - process and system utilities. It supports the following platforms:
Linux
Windows
macOS
FreeBSD, OpenBSD, NetBSD
Sun Solaris
AIX
…both 32-bit and 64-bit architectures. Supported Python versions are 2.6, 2.7 and 3.4+. PyPy3 is also known to work.
Process class
Process is an running program. Wikipedia defines process as “In computing, a process is the instance of a computer program that is being executed by one or many threads. It contains the program code and its activity. “
I use Fedora as my primary Operating System. The Linux filesystem contains the process related information in /proc directory. It does not stay inside the disk instead kernel creates it in its memory. /proc contains information about the system such as :
/proc/cpuinfo contains the information about the cpu like number of cores etc.
message output by kernel found in /proc/kmsg routed to syslog
/proc/meminfo has the information about the available system memory
/proc/ioports describes which input and output ports are being used.
Also there other details available in the /proc directory. One can find more information about this at
The psutil module helps us to find all these information through program. If we know the pid of a running process, we can get an instance of the [Process] class.
p = psutil.Process(7134)
Here I am using the pid value of 7134.
Now, we can check for the process name, and command line arguments. We
can also find the details about the init process or mother process or parent process.
>>> print(f"Name of the process : {p.name()}")
Name of the process : hexchat
>>> print(f"Parent process : {p.ppid()}")
Parent process : 2116
>>> print(f"Command Line : {p.cmdline()}")
Command Line : ['hexchat', '--existing']
Next we are getting executable, username, uid and gid
>>> print(f"The process executable as an absolute path : {p.exe()}")
The process executable as an absolute path : /usr/bin/hexchat
>>> print(f"Username : {p.username()}")
Username : adas
>>> print(f"uid : {p.uids()}")
uid : puids(real=1000, effective=1000, saved=1000)
>>> print(f"uid : {p.gids()}")
gid : pgids(real=1000, effective=1000, saved=1000)
>>> print(f"The environment variables of the process as a dictionary : {p.environ()}")
It returns a dictionary in key, value pair. I did not print the full output because the length :)
The environment variables of the process as a dictionary : {'SHELL': '/bin/bash', 'HISTCONTROL': 'ignoredups', 'HOSTNAME': ‘localhost.localdomain',
You can get network connection details in and all the open files by the process.
>>> for conn in p.connections():
... print(f" Socket file descriptor: {conn.fd}\n Address family : {conn.family}\n Local address : {conn.laddr}\n Remote address : {conn.raddr}\n Status of a TCP connection : {conn.status}")
Socket file descriptor: 15
Address family : 2
Local address : addr(ip='192.168.1.3', port=56080)
Remote address : addr(ip='1xx.xx.xxx.xxx', port=8080)
Status of a TCP connection : ESTABLISHED
JavaScript Object Notation, often referred as JSON is a data representation format which is human readable and easy to parse for machines. Personally, I find it hard to comprehend huge JSON files. In this blog post, I will be talking about the tool called jq. It’s a CLI tool to parse and manipulate JSON objects/files.
How I started using jq In the past one year, I have been writing a lot of shell scripts (mostly for bash).
After an anxious month, I am writing a Krita Weekly again and probably this would be my last one too, though I hope not. Let’s start by talking about bugs. Unlike the trend going about the last couple of months, the numbers have taken a serious dip.
Yesterday, I was trying to debug why my server running Plex rebooted and wasn't able to figure it out. There are several new commands that I discovered, the best of which is called last.
who command prints which users are currently logged in:
$ who
maxking tty2 2020-04-30 22:17 (tty2)
But, you can use it with -b to figure out the last reboot time:
$ who -b
system boot 2020-04-30 22:16
last
last command prints a bunch of useful information, including
Time of last reboot, last reboot
time of last shutdown last shutdown
All the system restart/shutdown events and run level changes last -x
It also has a bunch of useful flags like, --present which also prints which users were present at the time (I really liked this one!)
$ last -p 14:40 1 ↵
maxking tty2 tty2 Thu Apr 30 22:17 still logged in
reboot system boot 5.6.7-300.fc32.x Thu Apr 30 22:16 still running
useful logs
I also found this stackexchange answer with a lot of useful log files that have information related to reboots like Kernel messages, systemd-journal etc.
A lot of things have been going on with Qt these days. It all started with The Qt Company trying to get an increase in their revenues, specifically this blog post.
[Published in Open Source For You (OSFY) magazine, October 2017 edition.]
This article is the eighth in the DevOps series. In this issue, we shall learn to set up Docker in the host system and use it with Ansible.
Introduction
Docker provides operating system level virtualisation in the form of containers. These containers allow you to run standalone applications in an isolated environment. The three important features of Docker containers are isolation, portability and repeatability. All along we have used Parabola GNU/Linux-libre as the host system, and executed Ansible scripts on target Virtual Machines (VM) such as CentOS and Ubuntu.
Docker containers are extremely lightweight and fast to launch. You can also specify the amount of resources that you need such as CPU, memory and network. The Docker technology was launched in 2013, and released under the Apache 2.0 license. It is implemented using the Go programming language. A number of frameworks have been built on top of Docker for managing these cluster of servers. The Apache Mesos project, Google’s Kubernetes, and the Docker Swarm project are popular examples. These are ideal for running stateless applications and help you to easily scale them horizontally.
Setup
The Ansible version used on the host system (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. Internet access should be available on the host system. The ansible/ folder contains the following file:
ansible/playbooks/configuration/docker.yml
Installation
The following playbook is used to install Docker on the host system:
The Parabola package repository is updated before proceeding to install the dependencies. The python2-docker package is required for use with Ansible. Hence, it is installed along with the docker package. The Docker daemon service is then started and the library/hello-world container is fetched and executed. A sample invocation and execution of the above playbook is shown below:
With verbose ’-v’ option to ansible-playbook, you will see an entry for LogPath, such as /var/lib/docker/containers//-json.log. In this log file you will see the output of the execution of the hello-world container. This output is the same when you run the container manually as shown below:
$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://cloud.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/
Example
A Deep Learning (DL) Docker project is available (https://github.com/floydhub/dl-docker) with support for frameworks, libraries and software tools. We can use Ansible to build the entire DL container from the source code of the tools. The base OS of the container is Ubuntu 14.04, and will include the following software packages:
Tensorflow
Caffe
Theano
Keras
Lasagne
Torch
iPython/Jupyter Notebook
Numpy
SciPy
Pandas
Scikit Learn
Matplotlib
OpenCV
The playbook to build the DL Docker image is given below:
We first clone the Deep Learning docker project sources. The docker_image module in Ansible helps us to build, load and pull images. We then use the Dockerfile.cpu file to build a Docker image targeting the CPU. If you have a GPU in your system, you can use the Dockerfile.gpu file. The above playbook can be invoked using the following command:
Depending on the CPU and RAM you have, it will take considerable amount of time to build the image with all the software. So be patient!
Jupyter Notebook
The built dl-docker image contains Jupyter notebook which can be launched when you start the container. An Ansible playbook for the same is provided below:
- name: Start Jupyter notebook
hosts: localhost
gather_facts: true
become: true
tags: [notebook]
vars:
DL_DOCKER_NAME: "floydhub/dl-docker"
tasks:
- name: Run container for Jupyter notebook
docker_container:
name: "dl-docker-notebook"
image: "{{ DL_DOCKER_NAME }}:cpu"
state: started
command: sh run_jupyter.sh
You can invoke the playbook using the following command:
The Dockerfile already exposes the port 8888, and hence you do not need to specify the same in the above docker_container configuration. After you run the playbook, using the ‘docker ps’ command on the host system, you can obtain the container ID as indicated below:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a876ad5af751 floydhub/dl-docker:cpu "sh run_jupyter.sh" 11 minutes ago Up 4 minutes 6006/tcp, 8888/tcp dl-docker-notebook
You can now login to the running container using the following command:
$ sudo docker exec -it a876 /bin/bash
You can then run an ‘ifconfig’ command to find the local IP address (“172.17.0.2” in this case), and then open http://172.17.0.2:8888 in a browser on your host system to see the Jupyter Notebook. A screenshot is shown in Figure 1:
TensorBoard
TensorBoard consists of a suite of visualization tools to understand the TensorFlow programs. It is installed and available inside the Docker container. After you login to the Docker container, at the root prompt, you can start Tensorboard by passing it a log directory as shown below:
# tensorboard --logdir=./log
You can then open http://172.17.0.2:6006/ in a browser on your host system to see the Tensorboard dashboard as shown in Figure 2:
Docker Image Facts
The docker_image_facts Ansible module provides useful information about a Docker image. We can use it to obtain the image facts for our dl-docker container as shown below:
The ANSIBLE_STDOUT_CALLBACK environment variable is set to ‘json’ to produce a JSON output for readability. Some important image facts from the invocation of the above playbook are shown below:
From past few days I have been dwelling in android to make a utility, an application that I can be used when I am reading and article or when I am researching about something.
The premise lies around on the fact that the application itself doesn’t have a screen but what it plays around is on background activity. So it silently keeps on running and when an interrupt comes it performs an action.
Since I am not very well versed with how to make an android of such kind, I searched and found out about the component which does this and it’s called a Service. This is very similar to the concept of linux services or daemons.
The application I am designing is basically a combination of overlay activity and background services. Hence I wouldn’t say that there will be no user interaction at all but it will be really minimal and user doesn’t have a inherent knowledge about the service.
So it was time to do some more reading on android services.
A Service is an application component that can perform long-running operations in the background, and it doesn’t provide a user interface. Another application component can start a service, and it continues to run in the background even if the user switches to another application.
This was something that I really wanted, now there are a few caveats to this blogpost that is my understanding and knowledge about android application development. So I would tell you take things with a pinch of salt and let me know if there is anything wrong with my understanding.
When I read more about services I got to know there are 3 kinds of them:
Foreground
This is how you see spotify music work, even when the application is not in the display you can change songs through notification and there is one level of user interaction involved with this.
Background
Services with which user don’t want to know about or interact, like updating a database, fetching some resources etc.
Bound
Bound services are the one which are attach to the user activity, the quickest example I can give is music player, you don’t want the music to stop when you switch application and in the mean time you want to control the music when you switch back to the application.
Mostly people use services so that all the heavy lifting is done in the background. I had a unique case what I wanted is a service that keeps running and observing, when something is changed or when it is poked then react to it.
If you have seen the design of facebook messenger, the chat heads comes to life only when you have a message, this was somewhat the use case.
The biggest thing that I learnt is android doesn’t allow you run a background service without notifying the user. This is a new addition the happened after Android Oreo.
Implementation
There are two kinds of implementation that android provides,
The former as it’s name suggest is used to spawn service and is attached to the main thread. While IntentSrvice is something more peculiar where you can divide the work and do it without actually make your application wait for something. For example suppose you are playing a game and you are in middle of level 1, now an IntentService can be used to spawn to download and keep all the data required for level 2 without affecting your game play.
Another amazing thing about services is that, it is a singleton, that means however time you are going to start a service, you are not going to interact with too many objects, it’s the same class which you are going to talk to.
Conclusion
These are few of the learning that I got about services in android, I didn’t put much code here because most of them is available in the references. I enjoyed my time learning about how services are designed and how they are manged internally in android. Let me know what you think about it.
Recently I have been dwelling with a lot of NLP problems and jargons. The more I read about it the more I find it intriguing and beautiful of how we humans try to transfer this knowledge of a language to machines.
How much ever we try because of our laid back nature we try to use already existing knowledge or existing materials to be used to make machines understand a given language.
But machines as we know it can only understand digits or lets be more precise binary(0s and 1s). When I first laid my hands on NLP this was my first question, how does a machine understand that something is a word or sentence or a character.
I am still a learner in this field(and life ) but what I could understand information that we are going to use has to be converted into binary or some kind of a numerical representation for a machine to understand.
There are various ways to “encode” this information into numerical form and that is what is called word embeddings.
What are word embeddings?
Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with many dimensions per word to a continuous vector space with a much lower dimension.
In short word embedding is a way to convert a textual information into numerical form so that it can help us analyse it.
Analysis like similarity between words or sentences, understand the context in which a phrase or word is being spoken etc.
How are they formed?
Lets try to convert a given sentence into a numerical form:
A quick brown fox jumps over the lazy dog
How do we convert the above sentence into a numerical form such that our machine or even we can perform operations on it. And its hard to figure out the mathematics of language but we can always try.
So lets try, what we can do is, get all unique words and sort the words in the sentences and then makes a list of them. But then how do we get a numerical representation for it. It’s time for us to visit our long lost friend – Matrix.
Let’s get the words in proper order i.e unique and sorted
Now we will try to convert these words into numerical form using some matrix concepts(mostly representation) so that we can make a word look different from another word.
If you see there are totally 10 words and so we took 10 blocks to represent it. In a more mathematical term each representation is called a vector and the dimension of this vector is 1 x 10. So each word in this universe can be represented by a vector of that dimension and we can now carry operations on it to get our desired result.
Few prominent operations are how similar are two vectors or how different are two vectors. We can dive into that later.
Now the method that we just followed is a very brute force way of doing this and is officially called as One-Hot Encoding or Count Vectorizing.
Why we do this?
Now the way we encoded above words can be really useless because it’s just a representation and it doesn’t have any other idea so we don’t know how two words are related or are they morphologically similar etc.
The prime reason we want to have encoding is to find similar words, gauge the context of the topics etc.
There are various other techniques which actually produce intelligent embeddings that has an idea about what is going on.
When constructing a word embedding space, typically the goal is to capture some sort of relationship in that space, be it meaning, morphology, context, or some other kind of relationship
and a lot of other embeddings like Elmo, USE etc. does a good job at that.
As we go ahead and explore more embeddings you will see it goes on becoming more complex. There are layers of training models introduced etc.
We even have sentence embeddings which are way different from just word embeddings.
Conclusion
This was just a tip of the iceberg or may be not even that but I thought it will be helpful for someone who is starting their exploration because it took time for me to get around this concept. Thanks a lot for reading.
In a previous post, I showed you how you can automate your borg backups with borgmatic.
After I started using borgmatic for my backups and hooked it to a cron running every 2 hours, I got interested into knowing what's happening to my backups at all times.
My experience comes handy in here, I know I need a monitoring system.
I also know that traditional monitoring systems are too complex for my use case.
I need something simple. I need something I can deploy myself.
[NOTE: This post originally appeared on
deepsource.io, and has been
posted here with due permission.]
In the early part of the last century, when David Hilbert was working on
stricter formalization of geometry than Euclid, Georg Cantor had worked out a
theory of different types of infinities, the theory of sets. This
theory would soon unveil a series of confusing paradoxes, leading to
a crisis in the Mathematics community regarding the stability of the
foundational principles of the math of that time.
Central to these paradoxes was the Russell’s paradox (or more generally, as
we’d talk about later, the Epimenides Paradox). Let’s see what it is.
In those simpler times, you were allowed to define a set if you could describe
it in English. And, owing to mathematicians’ predilection for self-reference,
sets could contain other sets.
Russell then, came up with this:
\(R\) is a set of all the sets which do not contain themselves.
The question was "Does \(R \) contain itself?" If it doesn’t, then according to
the second half of the definition it should. But if it does, then it no longer
meets the definition.
The same can symbolically be represented as:
Let \(R = \{ x \mid x \not \in x \} \), then \(R \in R \iff R \not \in R \)
Cue mind exploding.
“Grelling’s paradox” is a startling variant which uses adjectives instead of
sets. If adjectives are divided into two classes, autological
(self-descriptive) and heterological (non-self-descriptive), then, is
‘heterological’ heterological? Try it!
Epimenides Paradox
Or, the so-called Liar Paradox was another such paradox which shred apart
whatever concept of ‘computability’ was, at that time - the notion that things
could either be true or false.
Epimenides was a Cretan, who made one immortal statement:
“All Cretans are liars.”
If all Cretans are liars, and Epimenides was a Cretan, then he was lying when
he said that “All Cretans are liars”. But wait, if he was lying then, how can
we ‘prove’ that he wasn’t lying about lying? Ein?
This is what makes it a paradox: A statement so rudely violating the assumed
dichotomy of statements into true and false, because if you tentatively think
it’s true, it backfires on you and make you think that it is false. And a
similar backfire occurs if you assume that the statement is false. Go ahead,
try it!
If you look closely, there is one common culprit in all of these paradoxes,
namely ‘self-reference’. Let’s look at it more closely.
Strange Loopiness
If self-reference, or what Douglas Hofstadter - whose prolific work on the
subject matter has inspired this blog post - calls ‘Strange Loopiness’ was the
source of all these paradoxes, it made perfect sense to just banish
self-reference, or anything which allowed it to occur. Russell and Whitehead,
two rebel mathematicians of the time, who subscribed to this point of view, set
forward and undertook the mammoth exercise, namely “Principia Mathematica”,
which we as we will see in a little while, was utterly demolished by Gödel’s
findings.
The main thing which made it difficult to ban self-reference was that it was
hard to pin point where exactly did the self-reference occur. It may as well be
spread out over several steps, as in this ‘expanded’ version of Epimenides:
The next statement is a lie.
The previous statement is true.
Russell and Whitehead, in P.M. then, came up with a multi-hierarchy set
theory to deal with this. The basic idea was that a set of the lowest ‘type’
could only contain ‘objects’ as members (not sets). A set of the next type
could then only either contain objects, or sets of lower types. This,
implicitly banished self-reference.
Since, all sets must have a type, a set ‘which contains all sets which are not
members of themselves’ is not a set at all, and thus you can say that Russell’s
paradox was dealt with.
Similarly, if an attempt is made towards applying the expanded Epimenides to
this theory, it must fail as well, for the first sentence to make a reference
to the second one, it has to be hierarchically above it - in which case, the
second one can’t loop back to the first one.
Thirty one years after David Hilbert set before the academia to rigorously
demonstrate that the system defined in Principia Mathematica was both
consistent (contradiction-free) and complete (i.e. every true statement
could be evaluated to true within the methods provided by P.M.), Gödel
published his famous Incompleteness Theorem. By importing the Epimenides
Paradox right into the heart of P.M., he proved that not just the
axiomatic system developed by Russell and Whitehead, but none of the
axiomatic systems whatsoever were complete without being inconsistent.
Clear enough, P.M. lost it’s charm in the realm of academics.
Before Gödel’s work too, P.M. wasn’t particularly loved as well.
Why?
It isn’t just limited to this blog post, but we humans, in general, have a diet
for self-reference - and this quirky theory severely limits our ability to
abstract away details - something which we love, not only as programmers, but
as linguists too - so much so, that the preceding paragraph, “It isn’t …
this blog … we humans …” would be doubly forbidden because the ‘right’
to mention ‘this blog post’ is limited only to something which is
hierarchically above blog posts, ‘metablog-posts’. Secondly, me (presumably a
human) belonging to the class ‘we’ can’t mention ‘we’ either.
Since, we humans, love self-reference so much, let’s discuss some ways in which
it can be expressed in written form.
One way of making such a strange loop, and perhaps the ‘simplest’ is using the
word ‘this’. Here:
This sentence is made up of eight words.
This sentence refers to itself, and is therefore useless.
This blog post is so good.
This sentence conveys you the meaning of ‘this’.
This sentence is a lie. (Epimenides Paradox)
Another amusing trick for creating a self-reference without using the word
‘this sentence’ is to quote the sentence inside itself.
Someone may come up with:
The sentence ‘The sentence contains five words’ contains five words.
But, such an attempt must fail, for to quote a finite sentence inside itself
would mean that the sentence is smaller than itself. However, infinite
sentences can be self-referenced this way.
The sentence
"The sentence
"The sentence
...etc
...etc
is infinitely long"
is infinitely long"
is infinitely long"
There’s a third method as well, which you already saw in the title - the Quine
method. The term ‘Quine’ was coined by Douglas Hofstadter in his book “Gödel
Escher, Bach” (which heavily inspires this blog post). When using this, the
self-reference is ‘generated’ by describing a typographical entity, isomorphic
to the quine sentence itself. This description is carried in two parts - one is
a set of ‘instructions’ about how to ‘build’ the sentence, and the other, the
‘template’ contains information about the construction materials required.
The Quine version of Epimenides would be:
“yields falsehood when preceded by it’s quotation” yields falsehood when preceded by it’s quotation
Before going on with ‘quining’, let’s take a moment and realize how awfully
powerful our cognitive capacities are, and what goes in our head when a
cognitive payload full of self-references is delivered - in order to decipher
it, we not only need to know the language, but also need to work out the
referent of the phrase analogous to ‘this sentence’ in that language. This
parsing depends on our complex, yet totally assimilated ability to handle the
language.
The idea of referring to itself is quite mind-blowing, and we keep doing it all
the time — perhaps, why it feels so ‘easy’ for us to do so. But, we aren’t born
that way, we grow that way. This could better be realized by telling someone
much younger “This sentence is wrong.”. They’d probably be confused - What
sentence is wrong?. The reason why it’s so simple for self-reference to occur,
and hence allow paradoxes, in our language, is well, our language. It allows
our brain to do the heavy lifting of what the author is trying to get through
us, without being verbose.
Back to Quines.
Reproducing itself
Now, that we are aware of how ‘quines’ can manifest as self-reference, it would
be interesting to see how the same technique can be used by a computer program
to ‘reproduce’ itself.
To make it further interesting, we shall choose the language most apt for the
purpose - brainfuck:
Running that program above produces itself as the output. I agree, it isn’t the
most descriptive program in the world, so written in Python below, is the
nearest we can go to describe what’s happening inside those horrible chains of
+’s and >’s:
The first line generates """ on the fly, which marks multiline strings in
Python.
Next two lines define the eniuq function, which prints the argument template
twice - once, plain and then surrounded with triple quotes.
The last 4 lines cleverly call this function so that the output of the program
is the source code itself.
Since we are printing in an order opposite of quining, the name of the function
is ‘quine’ reversed -> eniuq (name stolen from Hofstadter again)
Remember the discussion about how self-reference capitalizes on the processor?
What if ‘quining’ was a built-in feature of the language, providing what we in
programmer lingo call ‘syntactic sugar’?
Let’s assume that an asterisk, * in the brainfuck interpreter would copy the
instructions before executing them, what would then be the output of the
following program?
*
It’d be an asterisk again. You could make an argument that this is silly, and
should be counted as ‘cheating’. But, it’s the same as relying on the
processor, like using “this sentence” to refer to this sentence - you rely on
your brain to do the inference for you.
What if eniuq was a builtin keyword in Python? A perfect self-rep was then
just be a call away:
eniuq('eniuq')
What if quine was a verb in the English language? We could reduce a lot of
explicit cognitive processes required for inference. The Epimenides paradox
would then be:
“yields falsehood if quined” yields falsehood if quined
Now, that we are talking about self-rep, here’s one last piece of entertainment
for you.
If you take that absurd thing above, and move around in the cartesian plane for
the coordinates \(0 \le x \le 106, k \le y \le k + 17\), where \(k\) is a
544 digit integer (just hold on with me here), color every pixel black for
True, and white otherwise, you'd get:
I've started to draw regularly (or as regularly as possible) with my new graphic display tablet. Unfortunately sitting down and taking the pen seems to be a very attracting thing to children. Letting three children draw something for 30 minutes significantly reduces the time to learn for myself.
I also noticed, that usually I rely strongly on written documentation to learn something new. The advantage of reading is, I can do it everywhere and whenever I find the time. Documentation for creating artwork seems to be quite different; the richest source of information I could find are videos. There are plenty of video tutorials for each and every little thing, but it's quite time consuming to go through and way less convenient than just reading text.
And I need to learn a lot, like workflows for different software and projects, tools and handling of krita and blender (which feels like a life's work at this stage) and of my tablet's driver and hardware, as well as actually build drawing skills:
sketching
inking
coloring
Or blender specific:
modeling
animating
lighting
textures and shaders
and so much more.
But considering the hours that (according to my steam account) I have spend in Skyrim or Redania, I should as well find some spare hours to draw all kinds of different eyes, noses and mouths.
Here are two of my experimental scribbles that I have been practicing on this week, some carrot family I drew for my daughter and a wall of my living room, the first thing I stared on when looking up from the monitor (we finally got rid of our Christmas decoration).
So it's year 2020 now. Although it is kind of random, to me New Year still feels like the beginning of something actually new. A good time to remind myself of some habits I wanted to adopt; a good timepoint to relate my progress to, and to measure my consistency in keeping these habits.
Speaking of habits: I want to
...write more blog posts
Well, yes. I'm starting year 2020 with writing that kind of blog post that everyone loves to read *cough* about what I intend to do, giving spurious excuses about why I wasn't able to write more recently, and making maybe unrealistic promises about my future writing frequency. But hey, I'm writing. Means: I'm learning and eventually improving.
...be creative
I have an exciting job, that allows me to be creative every day, and I'm very thankful for that. Every day I can be creative in writing software, doing data analysis, communicate with smart people all over the world.
However, when I received my Master's degree in physics, I was considering to never do science again. Actually I considered becoming a singer and songwriter and produce myself; playing guitar, piano and every other instrument I could get my hands on. No need to explain, that my back-then-pregnant wife was not very happy with these plans, and I know that she was right.
But I miss making music. I miss writing fictional stories and to draw and paint ('though I'm not a big artist, as of yet).
My favorite incarnation of Santa (aka my wonderful wife) gifted me with an XP-Pen Artist 12 this Christmas, which I found to run smoothly on my Linux machines with Krita and Blender, so at this point I really have no excuse for not unleashing my creative super powers (which develop over time, I guess).
Combining my interests for music, writing and drawing (AND coding AND computer games), and given the lack of the most precious resource - time, I might as well start to develop my own indie game. I'm not sure if I will ever be able to spend enough time to become proficient in any of these disciplines, but I will try and time will show.
...endure, be continuous, be consistent
My #NewYearsResolutions will be worthless if I cannot keep it up. There will be periods when it's hard to be productive, when my family and my job will consume all the resource I have to give. I might be forced to bend my habits while not breaking them. In German there is a saying
Aufgeschoben ist nicht aufgehoben (postponed is not cancelled)
So I will need to pick up and continue my work for myself, whenever I get distracted.
On this note, happy New Year everyone! Let's have a great year 2020!
I wanted to attend TUG 2019 for two main reasons - to present my work on the “XeTeX Book Template”, and also to meet my favourite computer scientist, Prof. Donald Knuth. He does not travel much, so, it was one of those rare opportunities for me to meet him in person. His creation of the TeX computer typesetting system, where you can represent any character mathematically, and also be able to program and transform it is beautiful, powerful and the best typesetting software in the world. I have been using TeX extensively for my documentation and presentations over the years.
Day I
I reached the hotel venue only in the afternoon of Friday, August 9, 2019, as I was also visiting Mountain View/San Jose on official work. I quickly checked into the hotel and completed my conference registration formalities. When I entered the hall, Rishi T from STM Document Engineering Private Limited, Thiruvananthapuram was presenting a talk on “Neptune - a proofing framework for LaTeX authors”. His talk was followed by an excellent poetic narration by Pavneet Arora, who happened to be a Vim user, but, also mentioned that he was eager to listen to my talk on XeTeX and GNU Emacs.
After a short break, Shreevatsa R, shared his experiences on trying to understand the TeX source code, and the lessons learnt in the process. It was a very informative, user experience report on the challenges he faced in navigating and learning the TeX code. Petr Sojka, from Masaryk University, Czech Republic, shared his students’ experience in using TeX with a detailed field report. I then proceeded to give my talk on the “XeTeX Book Template” on creating multi-lingual books using GNU Emacs and XeTeX. It was well received by the audience. The final talk of the day was by Jim Hefferon, who analysed different LaTeX group questions from newbies and in StackExchange, and gave a wonderful summary of what newbies want. He is a professor of Mathematics at Saint Michael’s College, and is well-known for his book on Linear Algebra, prepared using LaTeX. It was good to meet him, as he is also a Free Software contributor.
The TUG Annual General Meeting followed with discussions on how to grow the TeX community, the challenges faced, membership fees, financial reports, and plan for the next TeX user group conference.
Day II
The second day of the conference began with Petr Sojka and Ondřej Sojka presenting on “The unreasonable effectiveness of pattern generation”. They discussed the Czech hyphenation patterns along with a pattern generation case study. This talk was followed by Arthur Reutenauer presenting on “Hyphenation patterns in TeX Live and beyond”. David Fuchs, a student who worked with Prof. Donald Knuth on the TeX project in 1978, then presented on “What six orders of magnitude of space-time buys you”, where he discussed the design trade-offs in TeX implementation between olden days and present day hardware.
After a short break, Tom Rokicki, who was also a student at Stanford and worked with Donald Knuth on TeX, gave an excellent presentation on searching and copying text in PDF documents generated by TeX for Type-3 bitmap fonts. This session was followed by Martin Ruckert’s talk on “The design of the HINT file format”, which is intended as a replacement of the DVI or PDF file format for on-screen reading of TeX output. He has also authored a book on the subject - “HINT: The File Format: Reflowable Output for TeX”. Doug McKenna had implemented an interactive iOS math book with his own TeX interpreter library. This allows you to dynamically interact with the typeset document in a PDF-free ebook format, and also export the same. We then took a group photo:
I then had to go to Stanford, so missed the post-lunch sessions, but, returned for the banquet dinner in the evening. I was able to meet and talk with Prof. Donald E. Knuth in person. Here is a memorable photo!
He was given a few gifts at the dinner, and he stood up and thanked everyone and said that “He stood on the shoulders of giants like Isaac Newton and Albert Einstein.”
< />
I had a chance to meet a number of other people who valued the beauty, precision and usefulness of TeX. Douglas Johnson had come to the conference from Savannah, Georgia and is involved in the publishing industry. Rohit Khare, from Google, who is active in the Representational State Transfer (ReST) community shared his experiences with typesetting. Nathaniel Stemen is a software developer at Overleaf, which is used by a number of university students as an online, collaborative LaTeX editor. Joseph Weening, who was also once a student to Prof. Donald Knuth, and is at present a Research Staff member at the Institute for Defense Analyses Center for Communications Research in La Jolla, California (IDA/CCR-L) shared his experiences in working with the TeX project.
Day III
The final day of the event began with Antoine Bossard talking on “A glance at CJK support with XeTeX and LuaTeX”. He is an Associate Professor of the Graduate School of Science, Kanagawa University, Japan. He has been conducting research regarding Japanese characters and their memorisation. This session was followed by a talk by Jaeyoung Choi on “FreeType MF Module 2: Integration of Metafont and TeX-oriented bitmap fonts inside FreeType”. Jennifer Claudio then presented the challenges in improving Hangul to English translation.
After lunch, Uwe Ziegenhagen demonstrated on using LaTeX to prepare and automate exams. This was followed by a field report by Yusuke Terada, on how they use TeX to develop a digital exam grading system at large scale in Japan. Chris Rowley, from the LaTeX project, then spoke on “Accessibility in the LaTeX kernel - experiments in tagged PDF”. Ross Moore joined remotely for the final session of the day to present on “LaTeX 508 - creating accessible PDFs”. The videos of both of these last two talks are available online.
A number of TeX books were made available for free for the participants, and I grabbed quite a few, including a LaTeX manual written by Leslie Lamport. Overall, it was a wonderful event, and it was nice to meet so many like-minded Free Software people.
A special thanks to Karl Berry, who put in a lot of effort in organizing the conference, but, could not make it to due to a car accident.
I was still up at this unearthly hour, thinking about life for a while now - fumbled thoughts about where I had come, where I started, and quite expectedly, Omar Bhai, your name popped in.
The stream continued. I started thinking about everything I’ve learned from you and was surprised with merely the sheer volume of thoughts that followed. I felt nostalgic!
I made a mental note to type this out the next day.
I wanted to do this when we said our final goodbyes and you left for the States, but thank God, I didn’t - I knew that I would miss you, but never could I have guessed that it would be so overwhelming - I would’ve never written it as passionately as I do today.
For those of you who don’t already know him, here’s a picture:
I’m a little emotional right now, so please bear with me.
You have been warned - the words “thank you” and “thanks” appear irritatingly often below. I tried changing, but none other has quite the same essence.
How do I begin thanking you?
Well, let’s start with this - thank you for kicking me on my behind, albeit civilly, whenever I would speak nuisance (read chauvinism). I can’t thank you enough for that!
I still can’t quite get how you tolerated the bigot I was and managed to be calm and polite. Thank You for teaching me what tolerance is!
Another thing which I learnt from you was what it meant to be privileged. I can no longer see things the way I used to, and this has made a huge difference. Thank You!
I saw you through your bad times and your good. The way you tackled problems, and how easy you made it look. Well, it taught me [drum roll] how to think (before acting and not the other way round). Thank You for that too!
And, thank you for buying me books, and even more so, lending away so many of them! and even more so, educating me about why to read books and how to read them. I love your collection.
You showed all of us, young folks, how powerful effective communication is. Thank You again for that! I know, you never agree on this, but you are one hell of a speaker. I’ve always been a fan of you and your puns.
I wasn’t preparing for the GRE, but I sat in your sessions anyways, just to see you speak. The way you connect with the audience is just brilliant.
For all the advice you gave me on my relationships with people - telling me to back off when I was being toxic and dragging me off when I was on the receiving side - I owe you big time. Thank You!
Also, a hearty thank you for making me taste the best thing ever - yes, fried cheese it is. :D
Thank You for putting your trust and confidence in me!
Ultralearning, Scott H Young (must read.
if you are looking to tackle something foundationally important, this book gives you one solid approach.
it’s mostly common sense.
but common sense that is laid out in a really logical manner.
i learnt to plan my project, that hard learning is normal, that failure is normal, and that persistence is a prerequisite.
all critical things, since learning no longer “comes naturally” to me.)
Memories, Lang Leav (must read)
The Universe of Us, Lang Leav (must read. Leav writes beautifully haunting poetry)
Dissent on Aadhaar, compilation, Reetika Khera (editor) (must read. this insightful, erudite read, tackles the various issues of Aadhaar on multiple levels, with multiple experts from various fields, voicing their concern.
if you want to know, why Big Brother is Bad Business, why Aadhaar is a bad idea and what its fallout c(w)ould be, this is the book to read)
Learning Python, Mark Lutz (this was a text book I needed to read to learn programming. loved it.)
It’s been a very long time since I wrote here for the last.
The reason is nothing big but mainly because:
Apparently, I was not able to finish some tasks in time that I used to write about.
I was not well for a long time that could be an another reason .
Besides, life happened in many ways which ultimately left me working on some other things first, because they seemed to be *important* for the time.
And, yes, there is no denying the fact that I was procastinating too because writing seems to be really hard at most times.
Though I had worked on many things throughout the time and I’ll try to write them here as short and quick logs below.
Around the second last week of august, I worked on setting up a self-hosted OpenVPN server which supported client scalability. The infrastructure required two servers/VMs, each having a basic firewall setup and a non-root “sudo” priviliged user. One among them was to host the OpenVPN service and another one was to serve as a Certificate Authority (CA). You can refer the following links to check for the related process.
In the last week of August, I worked on another task i.e to read about Syslogs and figure out how each of the systems can email the root mails to a certain email address for log collection. Thus, I read about Syslogs, how it works, its format and various Syslogs message levels. The latter part of the task was accomplished using ssmtp as mail program & writing cron jobs to actually send them to the intended email addresses. Check the following links for resources.
This one question always came up, many times, the students managed to destroy their systems by doing random things. rm -rf is always one of the various commands in this regard.
While I was doing the above task, at one time I ruined my local system’s mail server configs and actually ended up doing something which kushal writes about in one of his recent post (quoted above). I was using the command rm -rf to clean some of the left-over dependencies of some mail packages, but that eventually resulted into machine being crashed. It was not the end of the mess this time. I made an another extremely big mistake meanwhile. I was trying to back up the crashed system, into an external hard disk using dd. But because I had never used dd before, so again I did something wrong and this time, I ended up losing ~500 GBs of backed up data. This is “the biggest mistake” and “the biggest lesson” I have learnt so far. (now I know why one should have multiple backups) And as there was absolutely no way of getting that much data back, the last thing I did was, formatting the hard-disk into 2 partitions, one with ext4 file system for linux backup and the other one as ntfs for everything else.
Thank you so much jasonbraganza for all the help and extremely useful suggestions during the time.
Okay, now after all the hassle bustle above, I got something really nice. This time, I received the “Raspberry Pi 4, 4GB, Complete Kit ” from kushal.
Thank you very much kushal for the RPi and an another huge thanks for providing me with all the guidance and support that made me reach to even what I am today.
During the same time, I attended a dgplug guest session from utkarsh2102. This session gave me a “really” good beginner’s insight of how things actually work in Debian Project. I owe a big thanks to utkarsh2102 as well, for he so nicely voluteered me from there onwards, to actually start with Debian project. I have started with DPMT and have done packaging 4 python modules so far. And now, I am looking forward to start contributing to Debian Ruby Team as well.
With the start of september, I spent some time solving some basic Python problems from kushal’s lymworkbook. Those issues were related to some really simply sys-admins work. But for me, working around and wrapping them in Python was a whole lot of learning. I hope I will continue to solve some more problems/issues from the lab.
And lastly (and currently), I am back to reading and implementing concepts from Ops School curriculum.
Voila, finally, I finish compiling up the logs from some last 20 days of work and other stuffs. (and thus, I am eventually finishing my long pending task of writing this post here as well).
I will definitely try to be more consistent with my writing from now onwards.
I'm an advocate of using SSH authentication and connecting to services like Github, Gitlab, and many others.
On this, I received a bunch of messages over IRC asking why do I prefer SSH for Git over HTTPS.
I find the Github documentation quite helpful when it comes down to learning the basic operation of using Git and Github. So, what has Github to say about "SSH v/s HTTPS"?
Github earlier used to recommend using SSH, but they later changed it to HTTPS. The reason for the Github's current recommendation could be:
Ease to start with: HTTPS is very easy to start with, as you don't have to set up your SSH keys separately. Once the account is created, you can just go over and start working with repositories. Though, the first issue that you hit is that you need to enter your username/password for every operation that you need to perform with git. This can be overcome by caching or storing the password using Git's credential storage. If you cache, then it is cached in memory for a limited period after which it is flushed so you need to enter your credentials again. I would not advise storing the password, as it is stored as plain-text on disk.
Easily accessible: HTTPS in comparison to SSH is easily accessible. Why? You may ask. The reason is a lot of times SSH ports are blocked behind a firewall and the only option left for you might be HTTPS. This is a very common scenario I've seen in the Indian colleges and a few IT companies.
Why do I recommend SSH-way?
SSH keys provide Github with a way to trust a computer. For every machine that I have, I maintain a separate set of keys. I upload the public keys to Github or whichever Git-forge I'm using. I also maintain a separate set of keys for the websites. So, for example, if I have 2 machines and I use Github and Pagure then I end up maintaining 4 keys. This is like a 1-to-1 connection of the website and the machine.
SSH is secure until you end up losing your private key. If you do end up losing your key, even then you can just login using your username/password and delete the particular key from Github. I agree, that the attacker can do nasty things but that would be limited to repositories and you would have control of your account to quickly mitigate the problem.
On the other side, if you end up losing your Github username/password to an attacker, you lose everything.
I also once benefitted from using SSH with Github, but IMO, exposing that also exposes a vulnerability so I'll just keep it a secret :)
Also, if you are on a network that has SSH blocked, you can always tunnel it over HTTPS.
Jest is a really good frontend testing framework and works great with React and Babel out of the box, along with Enzyme for component testing. But, imports with React and Babel can often be filled with nasty imports. I wrote in a previous blog about how to make better more cleaner imports using some webpack tweaks.
But the problem appears when we try to write Jest and Enzyme tests with them. Because Babel can now longer understand and parse the imports. And without Babel parsing them and converting to ES5, jest cannot test the components. So we actually need a mix of Babel configuration and Jest configuration.
Note: This post assumes you already have jest, babel-jest and babel/plugin-transform-modules-commonjs packages installed using your favorite javascript package manager.
Basically, the workaround is first we need to resolve the cleaner imports into the absolute paths for the import using Jest configurations, and then use the Babel configurations to parse the rest code (without the modules) to ES5.
The configuration files look something like these: babel.config.js
In babel.config.js, we make a check to see if the code is right now in test environment. This is helpful because
Jest sets the environment to "test" when running a test so it is easily identifiable
It ensures that the test configuration don't mess up with the non test configurations in our Webpack config (or any other configuration you are using)
So in this case, I am returning the exact same Babel configuration that I need in my Webpack config in non-test environment.
In the test configuration for Babel, we are using a plugin "@babel/plugin-transform-modules-commonjs". This is needed to parse all the non component imports like React, etc. along with parsing the components from ES6 to ES5 after jest does the path resolution. So it helps to convert the modules from ES6 to ES5.
Now, let's see the jest.config.js. The jest configuration allows us to do something called moduleNameMapper. This is a very useful configuration in many different usecases. It basically allows us to convert the module names or paths we use for module import to something that jest understands (or in our case, something that the Babel plugin can parse).
So, the left hand part of the attribute contains a regular expression which matches the pattern we are using for imports. Since our imports look something like '~/path/from/jsRoot/Component', so the regular expression to capture all such imports is '^~/(.*)$'. Now, to convert them to absolute paths, we need to append '<rootDir>/path/to/jsRoot/' in front of the component path.
And, voila! That should allow Jest to properly parse, convert to ES5 and then test.
The best part? We can use the cleaner imports even in the .test.js files and this configuration will work perfectly with that too.
You can bring in modules from different javascript file using require based javascript code, or normal Babel parse-able imports. But the code with these imports often become a little bad because of relative imports like:
importComponentfrom'../../path/to/Component'
But a better, more cleaner way of writing ES6 imports is
importComponentfrom'~/path/from/jsRoot/Component'
This hugely avoids the bad relative paths for importing depending on where the component files are. Now, this is not parse-able by babel itself. But you can parse this by webpack itself using it's resolve attribute. So your webpack should have these two segments of code:
The {modules: false} ensures that babel-preset-env doesn't handle the parsing of the module imports. You can check the following comment in a webpack issue to know more about this.
I'm an advocate of using SSH authentication and connecting to services like Github, Gitlab, and many others. I do make sure that the use the git:// URL while cloning the repo but sometimes I do make mistake of using the https:// instead. Only to later realise when git prompts me to enter my username to authenticate the SSH connection. This is when I have to manually reset my git remote URL.
Today, I found a cleaner solution to this problem. I can use insteadOf to enforce the connection via SSH.
This blog is more like a bookmark for me, the solution was scavenged from internet. Recently I have been working on an analytics project where I had to generate pivot transpose tables from the data. Now this is the first time I faced the limitations set on postgres database. Since its a pivot, one of my column would be transposed and used as column names here, this is where things started breaking. Writing to postgres failed with error stating column names are not unique. After some digging I realized Postgres has a column name limitation of 63 bytes and anything more than that will be truncated hence post truncate multiple keys became the same causing this issue.
Next step was to look at the data in my column, it ranged from 20-300 characters long. I checked with redshift and Bigquery they had similar limitations too, 128 bytes. After looking for sometime found a solution, downloaded the postgres source, changed NAMEDATALEN to 301(remember column name length is always NAMEDATALEN – 1) src/include/pg_config_manual.h, followed the steps from postgres docs to compile the source and install and run postgres. This has been tested on Postgres 9.6 as of now and it works.
Next up I faced issues with maximum number columns, my pivot table had 1968 columns and postgres has a limitation of 1600 total columns. According to this answer I looked into the source comments and that looked quite overwhelming . Also I do not have a control over how many columns will be there post pivot so no matter whatever value i set , in future i might need more columns, so instead I handled the scenario in my application code to split the data across multiple tables and store them.
I felt compelled to write this after my personal experience trying to get help with my code on IRC.
We all love to make the computer do things exactly the way we want, so some of us choose to take the bold step of learning to communicate with the machine. And it is not uncommon to find many of our burgeoning kind go from location to location on the web space trying to get help along the way. We are prompt to ask questions when we sight help.
When you learn to program, you are often encouraged to learn by doing.
The domain of computer programming or software development is a very practical one. Before now, I had carried this very principle everywhere with me -- in fact, preached it -- but hadn't really put it to use.
The thing about learning languages (or technologies) by reading big manuals is that, often times, beginners will approach the process like they would any other literature book. But that is clearly a wrong approach as empirical evidence has shown. You don't read these things to simply stomach them. Instead, you swallow and then post-process. In essence, you ruminate over stuff.
In truth, the only way you can really process what you read is to try things out and see results for yourself.
Weeks ago, while building an app, I visited IRC frequently to ask questions on just about everything that was unclear to me. While this mode of communication and seeking help is encouraged, abuse of it is strongly discouraged. The good guys over on the IRC channels get pissed off when it appears you're boycotting available resources like documentation, preferring to be spoonfed the whole time. (Remember this is not Quora, where the philosophy is for you to ask more and more questions).
This was the sort of thing that happened to me when I began flooding the channels with my persistent querying. Most of the time the IRC folks kept pointing me to the documentation, as workarounds for most of the issues I had were already documented. A lot of things became much clearer when I decided to finally read-the-docs.
What did I learn from that? "Do your own research!" It's so easy to skip this part, but if you make efforts at finding things out for yourself, you'll be surprised at how much you can dig out without having to bug people. However, this does not guarantee that even the few important questions you'll ask may not be met with hostility, but do not let that discourage you. The people who appear to be unwelcoming are doing so only as a way to discourage you from being over-dependent on the channel.
Another advantage of finding things for yourself is that, you learn the why and not just the how.
"And it's not just asking questions; questioning other people, like what other people do or what you do yourself.
By far, the worst parts in all of my libraries are where I took the design from somewhere else. And it's not because I know better, it's because pretty much everything everybody does at any point in time has some sort of design decision behind it ... that major decision process.
So someone came up with a solution for a particular problem, and he thought about it and then wrote it down. But if you look at someone else's design, you might no longer understand why the decision was made in the first place. And ... in fact, if the original implementation is ten years old, who knows if the design ideas behind it are still entirely correct."
Personally, I like to answer questions and help put people on track. Nonetheless, if the queries got too overwhelming -- especially coming from the same person -- I would eventually lose interest in answering questions.
Let me remind you of some tidbits or etiquettes of IRC:
Construct your questions well (concise, well written and straight-to-the-point questions are more likely to attract help)
Don't EVER post code in a channel! Pastebin[1] it and share the link in the channel instead. While at it, don't post your entire code (unless you specifically need to). Post only the relevant portion -- the one you have an issue with. The only exception to this is if the snippet of code is considerably short, say one or two lines.
Don't be overly respectful. Yes, dont be too respectful -- cut all the 'Sirs'. Only be moderately polite.
Ensure you have and use a registered nick. This gives you an identity.
This last one is entirely my opinion but it's also based on what I have observed. Don't just be a leech, try to contribute to the community. Answer questions when you can.
So where do you look to before looking to IRC? There are three sources you may read from before turning to internet-relay-chat for help:
Read the documentation. * Documentation is the manual the creator or experts of a software product or tool provide their users with. So you want to know the ins and outs of a technology? That's the right place to look.
Read blog posts related to your topic-area. Blog posts are often based on people's experiences, so you're likely to find help from there, especially if the writer has faced the same issue. Remember to bookmark the really helpful ones as you go ;).
Last and very important. Read the source code!. This is two-fold: First is actually looking into your own code carefully and seeing what syntax or semantic errors you might have made. Secondly, you look into the original code of libraries/frameworks you are using if they are open source, otherwise revert to documentation. With this, you have it all stripped to its bare bones. Point blank! The source code reveals everything you need to know, once you know how to read it.
So why not arm yourself properly before going to post that question. That way, you would not only make it easier to get help [for yourself], but you would be better informed.
Github Gists -- A Github-owned pastebin platform with support for version-controlled pastes.
Note: Because Hastebin heavily depends on Javascript, some people have complained of text-rendering issues possibly arising from browser-compatibility issues with it. So take caution using it. That said, I love its ease-of-use. It supports the use of keyboard shortcuts such as [Ctrl]+[S] to Save.
I know what you are going to do this October. Scratching your head already? No, don’t do it because I will be explaining you in details all that you can do to make this october a remarkable one, by participating in Hacktoberfest.
Guessing what is the buzz of Hacktoberfest all around?
Hacktoberfest is like a festival celebrated by people of open source community, that runs throughout the month. It is the celebration of open source software, and welcomes everyone irrespective of the knowledge they have of open source to participate and make their contribution.
Hacktoberfest is open to everyone in our global community!
Five quality pull requests must be submitted to public GitHub repositories.
You can sign up anytime between October 1 and October 31.
<<<<Oh NO! STOP! Hacktoberfest site defines it all. Enough! Get me to the point.>>>>
Already had enough of the rules and regulations and still wondering what is it all about, why to do and how to get started? Welcome to the right place. This hacktoberfest is centering a lot around open source. What is it? Get your answer.
What is open source?
If you are stuck in the name of open source itself, don’t worry, it’s nothing other than the phrase ‘open source’ mean. Open source refers to the availability of source code of a project, work, software, etc to everyone so that others can see, modify changes to it that can be beneficial to the project, share it, download it for use. The main aim of doing so is to maintain transparency, collaborative participation, the overall development and maintenance of the work and it is highly used for its re-distributive nature. With open source, you can organize events and schedule your plans and host it onto an open source platform as well. And the changes that you make into other’s work is termed as contribution. The contribution do not necessarily have to be the core code. It can be anything you like- designing, organizing, documentation, projects of your liking, etc.
Why should I participate?
The reason you should is you get to learn, grow, and eventually develop skills. When you make your work public, it becomes helpful to you because others analyze your work and give you valuable feedback through comments and letting you know through issues. The kind of work you do makes you recognized among others. By participating in an active contribution, you also find mentors who can guide you through the project, that helps you in the long run.
And did I tell you, you get T-shirts for contributing? Hacktoberfest allows you to win a T-shirt by making at least 5 contributions. Maybe this is motivating enough to start, right? Time to enter into Open Source World.
How to enter into the open source world?
All you need is “Git” and understanding of how to use it. If you are a beginner and don’t know how to start or have difficulty in starting off, refer this “Hello Git” before moving further. The article shows the basic understanding of Git and how to push your code through Git to make it available to everyone. Understanding is much more essential, so take your time in going through it and understanding the concept. If you are good to go, you are now ready to make contribution to other’s work.
Steps to contribute:
Step 1; You should have a github account.
Refer to the post “Hello Git“, if you have not already. The idea there is the basic understanding of git workflow and creating your first repository (your own piece of work).
Step 2: Choose a project.
I know choosing a project is a bit confusing. It seems overwhelming at first, but trust me once you get the insights of working, you will feel proud of yourself. If you are a beginner, I would recommend you to first understand the process by making small changes like correcting mistakes in a README file or adding your name to the contributors list. As I already mention, not every contributions are into coding. Select whatever you like and you feel that you can make changes, which will improve the current piece of work.
There are numerous beginner friendly as well as cool projects that you will see labelled as hacktoberfest. Pick one of your choice. Once you are done with selecting a project, get into the project and follow the rest.
Step 3: Fork the project.
You will come across several similar posts where they will give instructions to you and what you need to perform to get to the objective, but most important is that you understand what you are doing and why you are doing. Here am I, to explain you, why exactly you need to perform these commands and what does these terms mean.
Fork means to create a copy of someone else’s repository and add it to your own github account. By forking, you are making a copy of the forked project for yourself to make changes into it. The reason why we are doing so, is that you would not might like to make changes to the main repository. The changes you make has to be with you until you finalize it to commit and let the owner of the project know about it.
You must be able to see the fork option somewhere at the top right.
Do you see the number beside it. These are the number of forks done to this repository. Click on the fork option and you see it forking as:
Notice the change in the URL. You will see it is added in your account. Now you have the copy of the project.
Step 4: Clone the repository
What cloning is? It is actually downloading the repository so that you make it available in your desktop to make changes. Now that you have the project in hand, you are ready to amend changes that you feel necessary. It is now on your desktop and you know how to edit with the help of necessary tools and application on your desktop.
“clone or download” written in green button shows you a link and another option to directly download.
If you have git installed on your machine, you can perform commands to clone it as:
git clone "copied url"
copied url is the url shown available to you for copying it.
Step 5: Create a branch.
Branching is like the several directory you have in your computer. Each branch has the different version of the changes you make. It is essential because you will be able to track the changes you made by creating branches.
To perform operation in your machine, all you need is change to the repository directory on your computer.
cd <project name>
Now create a branch using the git checkout command:
git checkout -b
Branch name is the name given by you. It can be any name of your choice, but relatable.
Step 6: Make changes and commit
If you list all the files and subdirectories with the help of ls command, your next step is to find the file or directory in which you have to make the changes and do the necessary changes. For example. if you have to update the README file, you will need an editor to open the file and write onto it. After you are done updating, you are ready for the next step.
Step 7: Push changes
Now you would want these changes to be uploaded to the place from where it came. So, the phrase that is used is that you “push changes”. It is done because after the work i.e., the improvements to the project, you will be willing to let it be known to the owner or the creator of the project.
so to push changes, you perform as follows:
git push origin
You can reference the URL easily (by default its origin). You can alternatively use any shortname in place of origin, but you have to use the same in the next step as well.
Step 8: Create a pull request
If you go to the repository on Github, you will see information about your updates and beside that you will see “Compare and pull request” option. This is the request made to the creator of the main project to look into your changes and merge it into the main project, if that is something the owner allows and wants to have. The owner of the project sees the changes you make and do the necessary patches as he/she feels right.
And you are done. Congratulations!
Not only this, you are always welcome to go through the issues list of a project and try to solve the problem, first by commenting and letting everyone know whatever idea you have to solve the issue and once you are approved of the idea, you make contributions as above. You can make a pull request and reference it to the issue that you solved.
But, But, But… Why don’t you make your own issues on a working project and add a label of Hacktoberfest for others to solve? You will amazed by the participation. You are the admin of your project. People will create issues and pull requests and you have to review them and merge them to your main project. Try it out!
I hope you find it useful and you enjoyed doing it.
If you've already read about my journey to PyCon AU, you're aware that I was working on a Chinese app. I got one more month to work on the Chinese app after PyCon AU, which meant improving my talk to have more things such as passing the locale info in async tasks, switching language in templates, supporting multiple languages in templates etc.
I presented the second version of the talk at DjangoCon US. The very first people I got to see again, as soon as I entered DjangoCon US venue were Russell and Katie from Australia. I was pretty much jet-lagged as my International flight got delayed by 10 hours, but I tried my best to deliver the talk.
Here is the recording of the talk:
You can see the slides of my talk below or by clicking here:
Carol, Jeff, and Carlton encouraged me to start contributing to Django, so I was waiting eagerly for the sprints.
Unfortunately, Carlton wasn't there during the sprints, but Andrew Pinkham was kind enough to help me with setting up the codebase. We were unable to run the test suite successfully and tried to debug that, later we agreed to use django-box for setting things up. I contributed few PRs to Django and was also able to address reviews on my CPython patches.
During the sprints, I also had a discussion with Rebecca and we listed down some points on how we can lower the barrier for new contributions in Django and bring in more contributors.
I also published a report of my two days sprinting on Twitter:
I also met Andrew Godwin & James Bennett. If you haven't yet seen the Django in-depth talk by James I highly recommend you to watch that. It gave me a lot of understanding on how things are happening under the hood in Django.
It was a great experience altogether being an attendee, speaker, and volunteer at DjangoCon. It was really a very rewarding journey for me.
There are tons of things we can improve in PyCon India, taking inspiration from conferences like DjangoCon US which I hope to help implement in further editions of the conference.
Here is a group picture of everyone at DjangoCon US. Credits to Bartek for the amazing click.
I want to thank all the volunteers, speakers and attendees for an awesome experience and making DjangoCon a lot of fun!
I was working on a Chinese app for almost 4 months and developing a backend that supports multiple languages. I spent almost daily reading documentation in Chinese and converting it through Google Translate app to integrate third-party APIs. It was painful, yet rewarding in terms of the knowledge that I gained from the Django documentation and several other resources about how to support multiple languages in Django based backends and best practices around it.
While providing multilingual support through Django backend, I realized that every now and then I was hitting a block and then had to read through the documentation and research the web. There were certain caveats that I got around while researching stuff whenever I was stuck and noted them as "gotcha moments" that I decided to cover later in a talk.
I got an opportunity to be at Sydney, Australia for DjangoCon AU and PyCon AU. This was very special, because it was my first International trip, and the first ever time when I was attending a Python conference outside India.
I was excited and anxious at the same time. Excited to meet new people, excited to be at a new place, excited to see how other PyCon takes place and preferably get some good parts about organizing a conference back to India for PyCon India :) I was anxious as it was a solo trip, I was alone with that impostor-syndrome kicking in. "Will I be able to speak?" -- But then I decided that I will share whatever I've learned.
Even before the conference began, I got an opportunity to spend some time with Markus Holtermann (Django core-dev). We roamed around Sydney Opera House and met Ian, Dom, Lilly and later went to Dinner.
I'm bad at remembering names! And when I say this, I mean super-bad. But to my astonishment, I was able to remember names for almost everyone whom I had an opportunity to interact with.
I registered as a volunteer for PyConAU which in-turn gave me a lot of perspective on how PyCon AU manages different aspects of logistics, food, video recording, speaker management, volunteer management, etc. There were certain moments when I was like "Oh, we could've done like this in PyCon India! We never thought about this!" and Jack Skinner was really helpful in discussing how they organize different things at PyCon AU.
My talk was on August 24, 2018, and it went pretty well.
You can see the slides of my talk below or by clicking here
I never had an opportunity to try my hands on hardware in my life and seeing so many hardware sprinters, I was curious to start playing with some of the hardware.
During the two days of sprint, I was able to fix my CPython patches, land a few PRs to Hypothesis which is a testing tool and play with Tomu to use it as a 2FA device.
Throughout the sprints, I met many people and yet got so much work done which left me with an astonishment. (I really wish I could be that productive daily :) )
Overall, it was a really pleasant experience and I prepared a list of notes of my PyCon AU takeaways which I shared with PyCon India team.
We had a grand 10th-anniversary celebration and first-time ever we had a Jobs board in PyCon India, along with various other things :)
I want to thank all the organizers, volunteers and attendees of PyCon AU for all the efforts to make the conference so welcoming and inclusive for everyone.
) but what I could understand information that we are going to use has to be converted into binary or some kind of a numerical representation for a machine to understand.
< />
(now I know why one should have multiple backups) And as there was absolutely no way of getting that much data back, the last thing I did was, formatting the hard-disk into 2 partitions, one with 
. Also I do not have a control over how many columns will be there post pivot so no matter whatever value i set , in future i might need more columns, so instead I handled the scenario in my application code to split the data across multiple tables and store them.
