Planet dgplug

June 17, 2019

Kushal Das

DMARC, mailing list, yahoo and gmail

Last Friday late night, I suddenly started getting a lot of bounced emails from the dgplug mailing list. Within a few minutes, I received more than a thousand emails. A quick look inside of the bounce emails showed the following error:

Unauthenticated email from yahoo.in is not accepted due to     domain's 550-5.7.1 DMARC policy.

Gmail was blocking one person’s email via our list (he sent that using Yahoo and from his iPhone client), and caused more than 1700 gmail users in our list in the nomail block unless they check for the mailman’s email and click to reenable their membership.

I panicked for a couple of minutes and then started manually clicking on the mailman2 UI for each user to unblock them. However, that was too many clicks. Suddenly I remembered the suggestion from Saptak about using JavaScript to do this kind of work. Even though I tried to learn JavaScript 4 times and failed happily, I thought a bit searching on Duckduckgo and search/replace within example code can help me out.

$checkboxes = document.querySelectorAll("[name$=nomail]");
for (var i=0; i<$checkboxes.length; i++)  {
      $checkboxes[i].checked = false;
}

The above small script helped me to uncheck 50 email addresses at a time, and I managed to unblock the email addresses without spending too many hours clicking.

I have also modified the mailing list DMARC settings as suggested. Now, have to wait and see if this happens again.

by Kushal Das at June 17, 2019 05:02 AM

Jason Braganza (Work)

The Best Writing Advice I Could Give You

Sometimes Seth Godin makes it easy for me to do the newsletter.
There’s a pithy post that says everything I want to say.
So, to the kids I coach, this is the best writing advice I could give you!
(Everything below the break is Seth, (emphases mine))


Decorating a car with bling, mudflaps and an airhorn is a form of signalling. You can show your peers that you have the resources to waste on superfluous adornments.

(Did you see what I just did there? I could have said, “You can show your friends that you have money to burn,” but I didn’t.)

Overwriting has a long tradition, particularly among academics. Make it a bit more complex and wordy than it needs to be. Write run-on sentences. Apparently, complicated writing must be more true.

One reason for this commitment to overwriting is that it keeps the hordes away. It’s difficult to read and hard to imagine writing. And so scarcity is created.

And yet, the articles and books that stand the test of time are straightforward. They’re memorable. They can be understood by the reader you seek to serve.

Simply write.

Write simply.

As few words as you need, but no fewer.

But simply write.


Subscribe to read more every week!

by Mario Jason Braganza at June 17, 2019 12:30 AM

June 14, 2019

Bhavin Gandhi

Triaging bugs of GNU Emacs

I have been using Emacs for more than 6 months. mbuf and me were discussing about some way to package Emacs as container image so that we can run that image to test things, triage bugs for a particular version etc. Before getting started with that, he wanted me to have idea about existing bugs of Emacs and how to triage them, so that I get better idea of whole work flow.

by @_bhavin192 (Bhavin Gandhi) at June 14, 2019 12:23 PM

June 10, 2019

Jason Braganza (Work)

Absolutely Make Time for Reading

From the Writing Routines1 pdf that is available on signing up to their mailing list…

“If I had a nickel for every person who ever told me he/she wanted to become a writer but “didn’t have time to read,” I could buy myself a pretty good steak dinner. Can I be blunt on this subject? If you don’t have time to read, you don’t have the time (or the tools) to write. Simple as that.”
Stephen King, On Writing

“I write two pages. And then I read and read and read.”
José Saramago, recipient of the 1998 Nobel Prize in Literature

“I can’t begin to tell you the things I discovered while I was looking for something else.”
Shelby Foote, The Civil War: A Narrative

“The greatest part of a writer’s time is spent in reading; a man will turn over half a library to make one book.”
Samuel Johnson, A Dictionary of the English Language

“...EVENINGS: See friends. Read in cafés....”
Henry Miller author of Tropic of Cancer, Black Spring, Tropic of Capricorn, and more.

P.S. Subscribe to the newsletter!


  1. I love the site and its interviews. Go visit and see if you like their stuff. 

by Mario Jason Braganza at June 10, 2019 12:30 AM

June 05, 2019

Kushal Das

Indian news websites over Tor and HTTP vs HTTPS

Following the idea of Secure The News, I wanted to verify if the Indian news websites have proper certificates or can they be viewed over Tor network.

The major problem was to get the list of urls, and I managed to create that from the Wikipedia list of Indian news organizations. Next, I had to write a straightforward Python script to verify the sites over Tor.

I have 181 site URLs and out of those, 5 are down. Among the rest 176 sites, surprisingly all but 3 sites could not be open from Tor network. The following 3 sites are blocking the users from Tor, thus, compromising the privacy and security Tor Browser provides to their readers.

On the other hand, when it comes to enabling HTTPS by default and redirecting people to that, most of these sites failed miserably.

117 sites do not provide their sites over HTTPS. If you do not know why it is important to provide the same, please read this. If you don’t know how to enable HTTPS on your website, you can read this guide.

This is the list of the URLs who are only on HTTP.

by Kushal Das at June 05, 2019 08:37 AM

June 03, 2019

Jason Braganza (Personal)

Books I’ve Read, May Edition

If you are looking for something to read, you might find something interesting here.

May

  • The Great Mental Models, Shane Parrish
    (the first of a soon to be multivolume work.
    must read times a hundred.
    this book teaches you how to think.
    and how to do it well.
    have i said it’s a must read? you must read it.)

  • Coraline, Neil Gaiman
    (must read. scarily charming.)

April

  • Keep Going, Austin Kleon
    (must read. new annual read. timely. beautiful quotes. hugely inspirational)

  • Chocolate Wars, Deborah Cadbury
    (must read. as a child growing up in the shadow of the large Cadbury factory, near home, Cadbury has always fascinated me. i still remember their school tours where we could go see how the chocolate was made and come home with a couple of bars of Dairy Milk. the factory is now shutdown, and the tempting aroma of chocolate no longer fills the air. this book delves into nearly 200 years of Cadbury’s (as well as its contemporaries) history. a lovely nostalgic throwback to a more innocent, more generous age.)

March

  • Titan’s Wrath, Rhett C Bruno

  • Never Grow Up, Jackie Chan
    (charmingly mistitled, because it is all about Jackie growing up, albeit a little too late. beautiful notes of apology and gratitude to the people in his life and of course, being Jackie, loads of hilarious stories)

  • Company of One, Paul Jarvis

  • Titan’s Son, Rhett C Bruno
    (something to get my mind off studies. the series is still fun)

  • Digital Minimalism, Cal Newport
    (must read. short. an in-depth practical treatise on getting out of the clicky, clicky, swipe, swipe circle of digital death i was in. found it really helpful)

  • Never Eat Alone, Keith Ferrazzi & Tal Raz
    (good read. if you’re an introvert like me, this is a good stepping stone to help you get out there.)

  • Thinking in Bets, Annie Duke
    (must read. short treatise on how you need to think probabilistically and divorce your efforts and your work, Arjun-like, from their results)

  • Titanborn, Rhett C Bruno
    (short fun read. in the vein of Asimov’s detective stories)

P.S. Keep yourself updated with the books I read. Subscribe to the mailing list!


by Mario Jason Braganza at June 03, 2019 12:30 AM

June 02, 2019

Shakthi Kannan

Building Erlang/OTP sources with Ansible

[Published in Open Source For You (OSFY) magazine, September 2017 edition.]

Introduction

Erlang is a programming language designed by Ericsson primarily for soft real-time systems. The Open Telecom Platform (OTP) consists of libraries, applications and tools to be used with Erlang to implement services that require high availability. In this article, we will create a test Virtual Machine (VM) to compile, build, and test Erlang/OTP from its source code. This allows you to create VMs with different Erlang release versions for testing.

The Erlang programming language was developed by Joe Armstrong, Robert Virding and Mike Williams in 1986 and released as free and open source software in 1998. It was initially designed to work with telecom switches, but is widely used today in large scale, distributed systems. Erlang is a concurrent and functional programming language, and is released under the Apache License 2.0.

Setup

A CentOS 6.8 Virtual Machine (VM) running on KVM will be used for the installation. Internet access should be available from the guest machine. The VM should have at least 2 GB of RAM alloted to build the Erlang/OTP documentation. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. The ansible/ folder contains the following files:

ansible/inventory/kvm/inventory
ansible/playbooks/configuration/erlang.yml

The IP address of the guest CentOS 6.8 VM is added to the inventory file as shown below:

erlang ansible_host=192.168.122.150 ansible_connection=ssh ansible_user=bravo ansible_password=password

An entry for the erlang host is also added to the /etc/hosts file as indicated below:

192.168.122.150 erlang

A ‘bravo’ user account is created on the test VM, and is added to the ‘wheel’ group. The /etc/sudoers file also has the following line uncommented, so that the ‘bravo’ user will be able to execute sudo commands:

## Allows people in group wheel to run all commands
%wheel	ALL=(ALL)	ALL

We can obtain the Erlang/OTP sources from a stable tarball, or clone the Git repository. The steps involved in both these cases are discussed below:

Building from the source tarball

The Erlang/OTP stable releases are available at http://www.erlang.org/downloads. The build process is divided into many steps, and we shall go through each one of them. The version of Erlang/OTP can be passed as an argument to the playbook. Its default value is the release 19.0, and is defined in the variable section of the playbook as shown below:

vars:
  ERL_VERSION: "otp_src_{{ version | default('19.0') }}"
  ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
  ERL_TOP: "{{ ERL_DIR }}/{{ ERL_VERSION }}"
  TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

The ERL_DIR variable represents the directory where the tarball will be downloaded, and the ERL_TOP variable refers to the top-level directory location containing the source code. The path to the test directory from where the tests will be invoked is given by the TEST_SERVER_DIR variable.

Erlang/OTP has mandatory and optional package dependencies. Let’s first update the software package repository, and then install the required dependencies as indicated below:

tasks:
  - name: Update the software package repository
    become: true
    yum:
      name: '*'
      update_cache: yes

  - name: Install dependencies
    become: true
    package:
      name: "{{ item }}"
      state: latest
    with_items:
      - wget
      - make
      - gcc
      - perl
      - m4
      - ncurses-devel
      - sed
      - libxslt
      - fop

The Erlang/OTP sources are written using the ‘C’ programming language. The GNU C Compiler (GCC) and GNU Make are used to compile the source code. The ‘libxslt’ and ‘fop’ packages are required to generate the documentation. The build directory is then created, the source tarball is downloaded and it is extracted to the directory mentioned in ERL_DIR.

- name: Create destination directory
  file: path="{{ ERL_DIR }}" state=directory

- name: Download and extract Erlang source tarball
  unarchive:
    src: "http://erlang.org/download/{{ ERL_VERSION }}.tar.gz"
    dest: "{{ ERL_DIR }}"
    remote_src: yes

The ‘configure’ script is available in the sources, and it is used to generate the Makefile based on the installed software. The ‘make’ command will build the binaries from the source code.

- name: Build the project
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - ./configure
    - make
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

After the ‘make’ command finishes, the ‘bin’ folder in the top-level sources directory will contain the Erlang ‘erl’ interpreter. The Makefile also has targets to run tests to verify the built binaries. We are remotely invoking the test execution from Ansible and hence -noshell -noinput are passed as arguments to the Erlang interpreter, as show in the .yaml file.

- name: Prepare tests
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - make release_tests
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

- name: Execute tests
  shell: "cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/erl -noshell -noinput -s ts install -s ts smoke_test batch -s init stop"

You need to verify that the tests have passed successfully by checking the $ERL_TOP/release/tests/test_server/index.html page in a browser. A screenshot of the test results is shown in Figure 1:

Erlang test results

The built executables, libraries can then be installed on the system using the make install command. By default, the install directory is /usr/local.

- name: Install
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - make install
  become: true
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

The documentation can also be generated and installed as shown below:

- name: Make docs
  shell: "cd {{ ERL_TOP }} && make docs"
  environment:
    ERL_TOP: "{{ ERL_TOP }}"
    FOP_HOME: "{{ ERL_TOP }}/fop"
    FOP_OPTS: "-Xmx2048m"

- name: Install docs
  become: true
  shell: "cd {{ ERL_TOP }} && make install-docs"
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

The total available RAM (2 GB) is specified in the FOP_OPTS environment variable. The complete playbook to download, compile, execute the tests, and also generate the documentation is given below:

---
- name: Setup Erlang build
  hosts: erlang
  gather_facts: true
  tags: [release]

  vars:
    ERL_VERSION: "otp_src_{{ version | default('19.0') }}"
    ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
    ERL_TOP: "{{ ERL_DIR }}/{{ ERL_VERSION }}"
    TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

  tasks:
    - name: Update the software package repository
      become: true
      yum:
        name: '*'
        update_cache: yes

    - name: Install dependencies
      become: true
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - wget
        - make
        - gcc
        - perl
        - m4
        - ncurses-devel
        - sed
        - libxslt
        - fop

    - name: Create destination directory
      file: path="{{ ERL_DIR }}" state=directory

    - name: Download and extract Erlang source tarball
      unarchive:
        src: "http://erlang.org/download/{{ ERL_VERSION }}.tar.gz"
        dest: "{{ ERL_DIR }}"
        remote_src: yes

    - name: Build the project
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - ./configure
        - make
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Prepare tests
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - make release_tests
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Execute tests
      shell: "cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/erl -noshell -noinput -s ts install -s ts smoke_test batch -s init stop"

    - name: Install
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - make install
      become: true
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Make docs
      shell: "cd {{ ERL_TOP }} && make docs"
      environment:
        ERL_TOP: "{{ ERL_TOP }}"
        FOP_HOME: "{{ ERL_TOP }}/fop"
        FOP_OPTS: "-Xmx2048m"

    - name: Install docs
      become: true
      shell: "cd {{ ERL_TOP }} && make install-docs"
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

The playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/erlang.yml -e "version=19.0" --tags "release" -K

Build from Git repository

We can build the Erlang/OTP sources from the Git repository. The complete playbook is given below for reference:

- name: Setup Erlang Git build
  hosts: erlang
  gather_facts: true
  tags: [git]

  vars:
    GIT_VERSION: "otp"
    ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
    ERL_TOP: "{{ ERL_DIR }}/{{ GIT_VERSION }}"
    TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

  tasks:
    - name: Update the software package repository
      become: true
      yum:
        name: '*'
        update_cache: yes

    - name: Install dependencies
      become: true
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - wget
        - make
        - gcc
        - perl
        - m4
        - ncurses-devel
        - sed
        - libxslt
        - fop
        - git
        - autoconf

    - name: Create destination directory
      file: path="{{ ERL_DIR }}" state=directory

    - name: Clone the repository
      git:
        repo: "https://github.com/erlang/otp.git"
        dest: "{{ ERL_DIR }}/otp"

    - name: Build the project
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - ./otp_build autoconf
        - ./configure
        - make
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

The ‘git’ and ‘autoconf’ software packages are required for downloading and building the sources from the Git repository. The Ansible Git module is used to clone the remote repository. The source directory provides an otp_build script to create the configure script. You can invoke the above playbook as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/erlang.yml --tags "git" -K

You are encouraged to read the complete installation documentation at: https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md.

June 02, 2019 07:45 PM

May 28, 2019

Saptak Sengupta

What's a ShadowDOM?

I first came to know about Shadow DOM from my curiosity about how browsers implement tags like <video> with all the controllers or <input> which changes based on the type attribute. We can never see any HTML code for these implementation and yet they are shown. How? That is when I stumbled upon this blog by Dimitri Glazkov which explains beautifully the concept of shadow DOM encapsulation used by browsers to implement such tags.

However, none of the browsers allowed developers to write their own custom shadow DOM (though google chrome had a v0 version implemented). I stumbled upon shadow DOM again while looking at an issue in jQuery to fix. So now, since 2018, most of the browsers have started supporting shadow DOM APIs and hence jQuery needed to implement support for that too.
https://caniuse.com/#feat=shadowdomv1
So, what is this shadow DOM and why do we even use it?

What's a DOM?

W3C specification describes it as "a method of combining multiple DOM trees into one hierarchy and how these trees interact with each other within a document, thus enabling better composition of the DOM".

Now, to understand that, we need to understand what a DOM is. DOM or Document Object Model is a tree-like structure containing the different elements (or tags) and strings of text that are shown by the markup language (like HTML, XML, etc.).

So, let's say we have a HTML code something like this:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Title</title>
</head>
<body>
<div>
<h1>This is header</h1>
<p>This is a
<a href="https://www.saptaks.blog">
link
</a>
</p>
</div>
</body>
</html>

So, visually you can show the DOM structure as something like:




Shadow DOM

Now, shadow DOM allows us to create separate hidden DOM trees that are attached to the elements of a regular DOM tree. This allows you to implement functional encapsulation, i.e. someone parsing the regular DOM tree and applying styling to the regular DOM tree doesn't know or affect the properties and functionalities of the shadow DOM tree. Hence you use the shadow DOM without knowing the intricate details of how the DOM is implemented. This is important, because this follows the basic ideas of Object Oriented Programming.

The shadow DOM tree starts with a shadow root and can then have any regular DOM element attached underneath it.

Let's see an example:

HTML
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Title</title>
</head>
<body>
<div id="shadowHost">
</div>
</body>
</html>

JavaScript
const shadowHost = document.getElementById('shadowHost');
const shadowRoot = shadowHost.attachShadow({mode: 'open'});
shadowRoot.innerHTML = '<h1>Hello Shadow DOM</h1>';

So, this will create a shadow DOM. Visually you can represent this as:


So, as you can see, there are few different parts in a shadow DOM apart from it being just another DOM.
  • Shadow tree: The DOM tree inside the shadow DOM.
  • Shadow boundary: the place where the shadow DOM ends, and the regular DOM begins.
  • Shadow root: The root node of the shadow tree.
  • Shadow host: The regular DOM node that the shadow DOM is attached to.
  • Shadow child: The tree below the shadow root node.
The shadow DOM cannot be attached to few different elements as mentioned in the spec. Some of the reasons are:
  • The different form tags such as <input>, <textarea>, etc or any other html tag for which the browser implements its own Shadow DOM
  • Elements like <img> or <br> or <hr> which are usually self enclosing tags and don't usually contain a child node.
Also if you see in the code there is a "{mode: open}". The mode determines whether you can edit the style of the shadow DOM from outside the shadow DOM or not.

Why do we need Shadow DOM anyways?

There are few different scenarios where you might want to use shadow DOM. The most important functionality of shadow DOM is the implementation of the concept of encapsulation. So basically, someone using the shadow DOM host doesn't need to care the style and implementation of the DOM inside. So few use-cases would be following:
  • The browser implements Shadow DOM for various different tags such as <video>, <audio>, <input>, <select>, etc.
  • You can make your own custom Shadow DOM when you need to create an element that you want to not be modified by the styling of remaining DOM.
  • You can also use Shadow DOM, when you want to create a separation of a particular DOM element from the main DOM element.
So is Shadow DOM very shadow-ey? Well maybe, since it stays hidden from the main DOM traversal. But at the same time it is often very useful because of its encapsulation properties.

by SaptakS (noreply@blogger.com) at May 28, 2019 01:55 PM

May 25, 2019

Robin Schubert

First experience with Scrum

Scrum is an agile software development approach, that allows small teams to quickly adapt to changes in planning to reach a common goal. Long-term goals are broken down to smaller tasks which are handled in sprints of usually 2-4 weeks. This allows to reach goals in a flexible but target-oriented manner in small steps.

While this concept is not exactly new, I - working in a quite small and research oriented business - just learned about it a few months ago: dgplug is the Durgapur Linux User Group with a great IRC channel on freenode.net (#dgplug). Shakthi Kannan (alias @mbuf) is scrum master of the Operation Blue Moon (OBM), a project that uses scrum to organize participants' schedules with tasks related to learning and open source contribution.

In February this year I started to work along the scrum guidelines on my own and all by myself at work. Since this concept is designed for small teams of 3-9, I couldn't adapt everything from the start. As an emacs user, these templates and macros from github were very helpful to organise my sprints and track my progress. I also implemented the enhancements suggested by Shakthi on the OBM project.

Even if I could not follow the scrum pattern very strictly, working as a one-man-team, I quickly noticed some positive effects on my work. Particularly I learned that I do not do a very good job, estimating the time I need to finish a task. I frequently underestimate the effort it takes. At the same time, I found myself working much more focussed and less often distracted from my actual goals. The costs caused by task-switching are staggering. Before following these patterns, I was often interrupted by colleagues asking for favors, and often I did not return to my initial task very quickly.

Since April I have a new colleague who actually has some scrum experience from an earlier job in a software company. Having him on board, we also managed to convince two further colleagues to work together in scrum mode. Working in a 4-men team showed me another flavor of this technique - we're now planning our time together, harmonize our goals and discuss the order and priorities of tasks we need to accomplish to get there. We were working on projects together before, but it's never been very transparently reviewed what each of us was actually working on, or what problems and obstacles we were facing. With our regular meetings and planning sessions, it feels like we were reborn as a team that we've never really been before.

When we started our meetings, it was an unfamiliar feeling at first, to report about the last sprint and especially about failures or unforseen delays. It took hours to plan the next steps and sprints. But we have improved quickly and got used to the procedures and the insight about the hube benefit of reporting openly to the colleagues and to plan next steps in collaboration came even quicker.

There are other popular agile frameworks, as of yet I did not try any of them. However, what I can say today is that - for me - working according to scrum is much better than working without guidance at all. Maybe many aspects of the concept come naturally for some good organised people, but I very much benefit from the framework to put me back on track and focus my energy in the right direction.

by Robin Schubert at May 25, 2019 12:00 AM

May 20, 2019

Jason Braganza (Personal)

Test your vocabulary

Go see how large your vocabulary is!
And don’t cheat! At least the first time :)

Click here to take the test.

This is me1.

Subscribe to the newsletter already, will ya?


  1. Ok, I cheated. The original one was 40,000 so i took it again :P 

by Mario Jason Braganza at May 20, 2019 12:30 AM

April 08, 2019

Subho

Increasing Postgres column name length

This blog is more like a bookmark for me, the solution was scavenged from internet. Recently I have been working on an analytics project where I had to generate pivot transpose tables from the data. Now this is the first time I faced the limitations set on postgres database. Since its a pivot, one of my column would be transposed and used as column names here, this is where things started breaking. Writing to postgres failed with error stating column names are not unique. After some digging I realized Postgres has a column name limitation of 63 bytes and anything more than that will be truncated hence post truncate multiple keys became the same causing this issue.

Next step was to look at the data in my column, it ranged from 20-300 characters long. I checked with redshift and Bigquery they had similar limitations too, 128 bytes. After looking for sometime found a solution, downloaded the postgres source, changed NAMEDATALEN to 301(remember column name length is always NAMEDATALEN – 1) src/include/pg_config_manual.h, followed the steps from postgres docs to compile the source and install and run postgres. This has been tested on Postgres 9.6 as of now and it works.

Next up I faced issues with maximum number columns, my pivot table had 1968 columns and postgres has a limitation of 1600 total columns. According to this answer I looked into the source comments and that looked quite overwhelming 😛 . Also I do not have a control over how many columns will be there post pivot so no matter whatever value i set , in future i might need more columns, so instead I handled the scenario in my application code to split the data across multiple tables and store them.

References:

  1. https://til.hashrocket.com/posts/8f87c65a0a-postgresqls-max-identifier-length-is-63-bytes
  2. https://stackoverflow.com/questions/6307317/how-to-change-postgres-table-field-name-limit
  3. https://www.postgresql.org/docs/9.6/install-short.html
  4. https://dba.stackexchange.com/questions/40137/in-postgresql-is-it-possible-to-change-the-maximum-number-of-columns-a-table-ca

by subho at April 08, 2019 09:25 AM

April 06, 2019

Tosin Damilare James Animashaun

To be Forearmed is to be Help-ready

I felt compelled to write this after my personal experience trying to get help with my code on IRC.

We all love to make the computer do things exactly the way we want, so some of us choose to take the bold step of learning to communicate with the machine. And it is not uncommon to find many of our burgeoning kind go from location to location on the web space trying to get help along the way. We are prompt to ask questions when we sight help.

When you learn to program, you are often encouraged to learn by doing.

The domain of computer programming or software development is a very practical one. Before now, I had carried this very principle everywhere with me -- in fact, preached it -- but hadn't really put it to use.

The thing about learning languages (or technologies) by reading big manuals is that, often times, beginners will approach the process like they would any other literature book. But that is clearly a wrong approach as empirical evidence has shown. You don't read these things to simply stomach them. Instead, you swallow and then post-process. In essence, you ruminate over stuff.


In truth, the only way you can really process what you read is to try things out and see results for yourself.

Weeks ago, while building an app, I visited IRC frequently to ask questions on just about everything that was unclear to me. While this mode of communication and seeking help is encouraged, abuse of it is strongly discouraged. The good guys over on the IRC channels get pissed off when it appears you're boycotting available resources like documentation, preferring to be spoonfed the whole time. (Remember this is not Quora, where the philosophy is for you to ask more and more questions).

This was the sort of thing that happened to me when I began flooding the channels with my persistent querying. Most of the time the IRC folks kept pointing me to the documentation, as workarounds for most of the issues I had were already documented. A lot of things became much clearer when I decided to finally read-the-docs.

What did I learn from that? "Do your own research!" It's so easy to skip this part, but if you make efforts at finding things out for yourself, you'll be surprised at how much you can dig out without having to bug people. However, this does not guarantee that even the few important questions you'll ask may not be met with hostility, but do not let that discourage you. The people who appear to be unwelcoming are doing so only as a way to discourage you from being over-dependent on the channel. Another advantage of finding things for yourself is that, you learn the why and not just the how.

I think it's fair to quote Armin Ronacher here,

"And it's not just asking questions; questioning other people, like what other people do or what you do yourself.

By far, the worst parts in all of my libraries are where I took the design from somewhere else. And it's not because I know better, it's because pretty much everything everybody does at any point in time has some sort of design decision behind it ... that major decision process.

So someone came up with a solution for a particular problem, and he thought about it and then wrote it down. But if you look at someone else's design, you might no longer understand why the decision was made in the first place. And ... in fact, if the original implementation is ten years old, who knows if the design ideas behind it are still entirely correct."


Personally, I like to answer questions and help put people on track. Nonetheless, if the queries got too overwhelming -- especially coming from the same person -- I would eventually lose interest in answering questions.


Let me remind you of some tidbits or etiquettes of IRC:

  • Construct your questions well (concise, well written and straight-to-the-point questions are more likely to attract help)

  • Don't EVER post code in a channel! Pastebin[1] it and share the link in the channel instead. While at it, don't post your entire code (unless you specifically need to). Post only the relevant portion -- the one you have an issue with. The only exception to this is if the snippet of code is considerably short, say one or two lines.

  • Don't be overly respectful. Yes, dont be too respectful -- cut all the 'Sirs'. Only be moderately polite.

  • Ensure you have and use a registered nick. This gives you an identity.

  • This last one is entirely my opinion but it's also based on what I have observed. Don't just be a leech, try to contribute to the community. Answer questions when you can.


So where do you look to before looking to IRC? There are three sources you may read from before turning to internet-relay-chat for help:

  • Read the documentation. * Documentation is the manual the creator or experts of a software product or tool provide their users with. So you want to know the ins and outs of a technology? That's the right place to look.

  • Read blog posts related to your topic-area. Blog posts are often based on people's experiences, so you're likely to find help from there, especially if the writer has faced the same issue. Remember to bookmark the really helpful ones as you go ;).

  • Last and very important. Read the source code!. This is two-fold: First is actually looking into your own code carefully and seeing what syntax or semantic errors you might have made. Secondly, you look into the original code of libraries/frameworks you are using if they are open source, otherwise revert to documentation. With this, you have it all stripped to its bare bones. Point blank! The source code reveals everything you need to know, once you know how to read it.


So why not arm yourself properly before going to post that question. That way, you would not only make it easier to get help [for yourself], but you would be better informed.



  1. Some Pastebin platforms I use:

Note: Because Hastebin heavily depends on Javascript, some people have complained of text-rendering issues possibly arising from browser-compatibility issues with it. So take caution using it. That said, I love its ease-of-use. It supports the use of keyboard shortcuts such as [Ctrl]+[S] to Save.

by Tosin Damilare James Animashaun at April 06, 2019 09:40 PM

April 02, 2019

Bhavin Gandhi

Using Gadgetbridge and openScale Amazfit Bip

Around 6 months ago the wrist watch that I was using from last 11 years broke. It was not possible to get it repaired as company does not manufacture any parts of it now. I was looking for an alternative but didn’t like any other normal watches available. So I decided to buy Amazfit Bit by Huami. Huami is brand by Xiaomi. While I’m not really interested in the steps count, sleep count, I liked the design of the watch.

by @_bhavin192 (Bhavin Gandhi) at April 02, 2019 02:23 PM

March 28, 2019

Shakthi Kannan

"Opportunities in FLOSS" at Computer Society of India, Madras

I had given a talk on “Opportunities in Free (Libre) and Open Source Software”(FLOSS) on Saturday, March 2, 2019 at the Computer Society of India, Madras Chapter, jointly organized by the IEEE Computer Society, Madras Chapter and ACM India Chennai Professional Chapter. The Computer Society of India, Education Directorate is located opposite to the Institute of Mathematical Sciences in Taramani, close to the Tidel Park. Students, IT professionals and professors from IIT Madras, Anna University and engineering colleges in and around Chennai attended the event.

Session in progress

At around 6:00 p.m. people had assembled for networking, snacks and beverage. I started the “Building Careers with FLOSS” presentation at 6:30 p.m. Prof. Dr. D. Janakiram, CSE, IIT Madras also shared his insights on the benefits of learning from source code available under a FLOSS license. The session was very interactive and the audience asked a lot of good questions. Dr. B. Govindarajulu, author of the famous “IBM PC and Clones: Hardware, Troubleshooting and Maintenance” book then presented his views on creating a Computer History Museum in Chennai. Dinner was served around 8:00 p.m. at the venue.

Group photo

A review of my book has been published in Volume 14: No. 1, January-March 2019 IEEE India Council Newsletter. The excerpts of Chapter 4 of my book on “Project Guidelines” is also available. I had also written an article on “Seasons of Code” which is published in this edition of the IEEE newsletter.

Special thanks to Mr. H. R. Mohan, Editor, IEEE India and Chairman, ACM Professional Chapter, Chennai for organizing the event and for the logistics support.

March 28, 2019 01:30 PM

March 27, 2019

Sayan Chowdhury

Kubernetes Day India 2019, Bangalore

Kubernetes Day India 2019, Bangalore

I returned to another conference with the jetlag of another conference, FOSSASIA but this conference was something I was very eagerly waiting to attend. Why? Well there are a couple of reasons, but the main reason for which I bought the ticket was to listen to Liz Rice in person.

The other reason was to see and meet the growing community of Kubernetes in Bangalore.

23rd March 2019, I woke up early morning and left along with Sayani, and Saptak to this place quite outside of Bangalore. Going to the venue almost felt like a weekend getaway trip.

Anyways we reached the venue at 8AM, and the Bangalore sun was already killing. CNCF had announced people would be eligible for KubeCon tickets if they came and collected their badges before 8:30AM. This was a nice trick because I could see a huge crowd in-front of me standing and collecting their badges. After collecting the badges, we headed back to the auditorium.

We grabbed some breakfast as soon as we reached the venue. The first talk/keynote did not delay much and started just after the breakfast.

Dan kicked off the event with an introduction to the conference, CNCF and a brief overview of the projects within CNCF.

Liz Rice, took the stage after Dan and talked on permission in Kubernetes. She gave a very nice ELI5 type analogy for permission/rbac in Kubernetes with file permission in Linux.

I moved outside after the talk and was mostly talking with people at the booths. Red Hat had it's booth also, so I spend some time talking to people about Fedora CoreOS and Silverblue

It was very nice to find the growing audience of Kubernetes, and I also happen to know about a couple of more interesting projects the community is building.


Photo by Cameron Venti on Unsplash

by Sayan Chowdhury at March 27, 2019 04:32 PM