Planet dgplug

October 13, 2018

Sayan Chowdhury

-d in go get

Saturday, I am sitting at a Starbucks in Bangalore, trying my hands on a Golang project. I come across this argument -d in go get:

The go cli help says:

The -d flag instructs get to stop after downloading the packages; that is,
it instructs get not to install the packages.

Wonderful! So, if you just want to download the golang project for the sake of contributing, you can use:

go get -d k8s.io/kubernetes

... and it will download the package for you, after which you can start working on the project.

by Sayan Chowdhury at October 13, 2018 03:20 PM

October 11, 2018

Saptak Sengupta

iTerm + zsh + tmux for your MacOS

Being a linux user for most part of my developer life, the most important thing of my life is the terminal (or shell or command line or console or whatever you want to call it) followed by text editor. Here I am going to talk mainly about the terminal (yes, I am just gonna call that throughout). So, when I started using a Macbook, the first question (apart from when to use CMD and CTRL) was which terminal and how to set it up with zsh and tmux. So I am gonna talk a little about that.

iTerm 2 for the win

The very first question was "Is the default terminal of Mac good?" Well, it's good enough, but most of the digging into led me to the same answer - Use iTerm2. So the very first step is to forget using the default terminal, and use iTerm2. The main reason is it comes with more customising options and better color support more than anything else.

You can download and unzip iTerm2 from here. All you need to do is unzip and put it into Applications folder and you are ready to use it.

ZSH

I have been using Z Shell for quite some time and have been used to the commands and ways so much, it is a little difficult to use bash. I feel Z shell increases productivity a lot, but personal opinion. So, to install ZSH in your Mac, all you do is:


brew install zsh

Once you have it installed, you can run the command `zsh` to start the Z Shell.

Oh My Zsh

A big shoutout to Robby Russell for creating this gem. If you felt Zsh increases your productivity and takes your command line experience to  a whole new level, wait till you have used Oh My Zsh. It makes your life much more easier by showing a lot more information in your shell especially when you are using git. It can be used to customize and show a lot of information according to your will. The amount of already present templates is just huge.

Execute this in your terminal to get Oh-my-zsh:


sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

Once you have installed, you can open the `~/.zshrc` file to add plugins like git, change themes and modify stuff. Here is a complete documentation and available template links.

Tmux

Tmux is basically a terminal multiplexer allowing you to create and access different terminal sessions within a single window. It also allows you to split your window into different panes, hence allowing you to do a lot more in a single window. Just like zsh, tmux is also something I have been using for some time and helps me maximise my productivity. 

You can install tmux, just by doing:


brew install tmux

After that, you need to configure your tmux to make it comfortable for your key bindings. One good link to get started I guess is this.

Integrate ZSH and Tmux

Now, after all configuration, the biggest pain is you have to open the terminal, change to tmux and then start zsh and then do work. But what we want is iTerm default opens in Zsh. Now, I prefer that it doesn't open with tmux by default, because there are scenarios where I want to work outside of tmux sessions, so I leave tmux to be started by me. But then make tmux start with Zsh instead of bash.

So here is what you do.

1. Make zsh your default shell.
chsh -s `which zsh`

2. Make tmux open with zsh. To do this, put the below line in ~/.tmux.conf:
# set shell
set -g default-shell /bin/zsh

3. Kill all previously running tmux instances. 
killall tmux

4. Close iTerm2 and restart it.

Now, your terminal should start with zsh. When you enter into tmux, it should go into tmux session along with zsh instead of bash.

And Voila! There you have a nice productive terminal. Or so I feel. 

by SaptakS (noreply@blogger.com) at October 11, 2018 03:04 PM

Sayan Chowdhury

Introducing SkipTheLine

2013

Introducing SkipTheLine

The year I graduated from a third-tier college in Durgapur. I did not expect much from the college, mostly because of the poor placement record in the college.

To get myself a job, I was primarily going through hasjob.co and any job opportunities that came through Bangpypers mailing list.

Finally, I landed myself into an internship and later which turned into a full-time at HackerEarth was through the hasjob.co.


2018

5 years later, finding a job in a suitable company is still tough. In the last 5 years, I have come to know about more options, be it AngelList, Hasjob, StackOverflow, Facebook Job Groups etc.

But the problem still I think is the same, even if you apply through these websites a lot of times you don't get a reply from the company you applied to. On other hand, referrals work. I've referred good people to companies I know. This way, there is a chance of getting an interview in the company. Cracking the interview is a different story altogether which I believe depends a lot on the candidate.

Recently, I got to know that Prashant, one of my school senior and a close friend, who famously known in the tech community for his "Bitcoin Wedding" started an effort called "SkipTheLine".

SkipTheLine is a newsletter where Prashant publishes profiles of three developers. These developers are from the community, who are active in open source, or competitive programming, or just good at technologies they work on. He goes on to introduce the developers through email with the companies point of contact via email and then the candidate and the POC take the discussion forward.

I personally loved the initiative that he took as at the end of the day if people whom I know come asking for a job referral I would just direct them to SkipTheLine. Prashant has quite a stronghold in the startup community and does great work in connecting the folks with some really good startup across the country.

I know people personally who got hired a within few days of their newsletter published so if you are looking out for a job, do fill out the SkipTheLine form.

If you are looking to hire, do drop me an email at gmail AT yudocaa DOT in.

by Sayan Chowdhury at October 11, 2018 11:51 AM

September 29, 2018

Jaysinh Shukla

Analyzing the behaviour of Python function slice

Title Image

Last Friday, I was sitting in one of the good coffee shops in Bangalore with my friend. Coffee and discussion is the best combination to release stress. It was looking like a perfect Friday evening until my friend was struck by an idea of asking me a question.

“Let me conduct a quiz.” he said, interrupting our conversation.

“A quiz? Quiz on what?”, I asked.

“On programming”, he said

“What is the level of difficulty then?” I said.

Asking the level of difficulty is important. I never invest my efforts in solving something easy. If he had said easy, I would have ignored to answer, but he said “It is a bit difficult, but not that difficult. I gave a wrong answer to this question in my last interview.”

There was no reason to go back from here. Taking some deep breaths I said, “Please go ahead.”

He stood up, took a tissue paper from a nearby counter and scratched below code on it. 1 And he asked me by pointing towards that code, “What will be the output of this code?”

def my_function(): l = [0, 1, 2] print(l[30:]) my_function() 

Now it was my turn to give the answer. I looked at the code and tried parsing it in my head — line by line. In my mind, I observed the first line. It was defining a function which seems to be correct. I moved my eyes to the next line. It was defining a variable l of type list and assigning values ranging from 0 to 2. Even this wasn’t looking problematic. So I forwarded to the next line where it was trying to print that variable l by slicing it from starting value 30 to the infinity.

“Well, the start value is 30 which is greater than the length of the list. This should raise an IndexError” I said in my mind. I was about to speak an answer, but suddenly Devil of me flashed.

“It is less than a banana job my dear,” the Devil said to me, “You should take a little advantage of this opportunity my boy.”

Because things were looking in my control, I shook my hands with the Devil.

I said to my friend, “How about betting for some real values?”

Going closer I spoke, “If I answer correctly, You will pay the bill and If I am wrong, This will be a treat from my side.”

He thought for a while and nodded. Now it was my turn to unveil the cards.

I said in a strong voice, “It will raise an IndexError.” And shifted my focus towards the chocolate.

He starred my face for a second and spoke, “Okay. Are you sure about this?”.

This was the hint he gave. I should have taken another shot here. What happens next become a lesson for me.

I said with a flat face, “Yes I am.”

With my answer, he instantly opened his backpack, took his Laptop out and typed the code which he wrote on that tissue.

When I stopped hearing a sound of typing I yelled, “So did I win?”

He turned his laptop towards me and exclaimed, “Not at all!”

When I focused on the screen, the interpreter was printing []. Damn! I lost the bet. Why the hell slice is returning an empty list even when we are trying to slice it with a value which is greater than the length of it! It was surely looking unpythonic behavior. I paid whatever the bill amount was. Entire evening this question was all roaming my mind. After coming home, I decided to justify reasons for returning an empty list instead of raising an IndexError from a slice.

Below are a few reasons justifying such behavior of slice function. I am sharing this with you so that you don’t lose a bet with your friend :) For those who haven’t used slice anytime in their life, I advise to read this tutorial. Reading this guide for understanding how a slice function converts the input values. Especially rule number 3 and 4 referenced there.

  • Reason number one:

    Python lists are more commonly used in iterations. Consider below example:

    numbers = [0, 1, 2, 3] for number in numbers[30:]: print(number) 

    If slice was raising an IndexError, then the above code would have to written like this

    written like like this

    numbers = [0, 1, 2, 3] try: for number in numbers[30:]: print(numbers) except IndexError: pass 

    Or in another way below is also looking reasonable

    numbers = [0, 1, 2, 3] start = 30 if start < len(numbers): for number in numbers[start:]: print(number) 

    Both the approaches are looking little lengthy by an obvious reason. And that reason is to prevent executing loop if there are no elements in it. When we observe the behavior of slice called at for in, it makes sense to return an empty list instead of raising an IndexError.

I am not able to find further reasons to return an empty list instead of raising the IndexError. But I am sure, there will be. If you know any other potential reasons for such behavior of slice, please drop me a mail at jaysinhp at gmail dot com or contact me over Twitter @jaysinhp. I will update the reasons at this post and give credits to you. Thanks for reading this post.

Proofreaders: Geoffrey Sneddon, Elijah, Mahendra Yadav, Dhavan Vaidya,
  1. I am using the word “Scratch” because that tissue paper was such a thin that writing by a ballpen torn it. 

by Jaysinh Shukla at September 29, 2018 04:47 AM

September 07, 2018

Farhaan Bukhsh

6 Bags and A Carton

This is not a technical post; this is something that I have been going through, in life right now. A few months ago, when I left my first job (another time, another post 😉 ), I had a plan. I wanted to take few months off and work on my technical knowledge and write amazing software and get a lot of learning out of my little sabbatical.

But I was not able to do that for a few reasons, primo being I had to move homes in Bangalore because my brother got transferred, so the savings that I had set aside wouldn’t be enough. This was not the end. When it rains, it pours apparently. My dad got super sick, he had a growth near his kidney which the doctors diagnosed as cancer. I got really scared with the situation I was going through. The thing about your parents is that no matter how much you fight with them or how much they “control” you; at the end of the day the thought of losing them can scare the hell out of you. For me, they are my biggest support system so I was not scared, I was terrified.

I gave it a really deep thought and took a call. I needed to find a job. The sabbatical could wait. I started applying to companies and talking to people if they needed extra hand at work. One piece of advice – never leave a job unless you have another in hand. Luckily, I had my small pot of gold, savings, so even in this phase I was sustaining myself. Yes, savings are real and you should have a sufficient amount at any given point of your life. This helps you to take the hard decisions and also to think independently (what Jason calls F*ck you money).

It still feels like a nightmare to me. I use to feel that I will wake up and it will all be over. Reality check; it wasn’t a dream so I have to live with it and make efforts to overcome this situation.

Taking up a job for me was important for two reasons,

  1. I have to sustain myself
  2. I need to have a back up in case my dad needs something (I also have super amazing siblings who were doing the same)

I realised one thing about prayer and God; yes, I believe in God, and I don’t know if prayer works but you definitely get the strength to face your problems and the unknown. I use to call my dad regularly asking how he was doing and some days he could not speak all that much and he use to talk in his weak tone. I use to cry. I was in so much pain although it was not physical or visible. And then, I would cry again.

But tough times teach you a lot, it shows you real friends, it shows you the people you care for and as Calvin’s dad would have said, “It build character!”. I have been through bad times before and the thing about time is , “It changes!”. I knew someday this bad time I am going through will change. Either the agony I am going through will reduce or I will get used to it.

So as I was giving interviews within a month of me moving on from my old job, I was offered one at Clootrack. I like the people who interviewed me and I like that ideas they have been working on. But I have seen people change and I have gone through a bad experiences and at no point of time did I want to repeat past mistakes, so I did a thorough background check before I said yes to them. I got a really good response so here I am working with them.

The accommodation problem that I had was my brother was shifting out of his quarters  and I used to live with him. Well, I helped him pack and I still remember the time when I was bidding farewell to him and my sister-in-law. I had tears in my eyes and after my goodbyes, the moment I stepped in the house I could feel the emptiness and I cried the whole night.  I  could stay at the old place for a week, not more. At this point I can’t thank Abhinav enough that he came as  a support I needed. He graciously let me live with him as long as  I wanted to. Apparently he needed help, paying his bills :P.  This bugger would never accept the fact, he helped me. When dad’s condition was getting bad he gave me really solid moral support. I had also shared my situation with Jason, Abraar, Kushal and Sayan. I received a good amount of moral support from each one of them, specially Jason. I use to tell him everything and he would just calm me down and talk me through it.

So when I shifted to Abhinav’s place all I had was 6 bags and a carton. My whole life was 6 bags and a carton. My office was a 2 hour bus ride one way and another 2 hours to come back. But I didn’t have any problems with this arrangement because this was the least of my problems. I literally use to live out of my bags and I wasn’t sure this arrangement would last long. I had some really amazing moments with Abhinav, I enjoyed our ups and downs and those little fights and leg pulling.

Well, my dad is still not in the best of his health, but he is doing better now. I visit my family more frequently now and yes call them regularly with a miss. I realised the value of health after seeing my dad. I went home after a month of joining Clootrack and stayed with him for a whole month and worked remotely, we visited few doctors and they said he is doing better. After coming back I realised I was not getting any time for myself so I shifted to a NestAway near my office. Although I feel I’ve gotten used to the agony, you never know what life has in store for you next.
It feels much better now, though.

I thank God for giving me strength and my friends and family for supporting me in a lot of different ways.

With Courage in my Heart,
And Faith over Head

 

 

by fardroid23 at September 07, 2018 04:12 AM

September 02, 2018

Sanjiban Bairagya

Akademy 2018 experience

This year’s Akademy, the annual world summit of KDE, was held in the beautiful city of Vienna, Austria, from 11th to 17th August, 2018. The 7-day event was divided in two parts, with the first 2 days being mostly keynote addresses and different talks by KDE contributors, followed by 5 more days BoFs, and workshops. Just like every other KDE event, this one was also as awesome as it could get.

Welcome party
The evening before the day the conference was scheduled to start, there was a nice welcome party with loads of food and drinks, where I got to catch up with most of the fellow KDE contributors who I hadn’t met for quite some time, and also got to see a lot of new faces as well, talking to whom felt like a breath of fresh air. Overall, it was a warm welcome, and raised everyone’s spirits to get ready for Akademy for the next day.

Conference Day 1
Day 1 was opened by Lydia, our beloved President of KDE. Dan Bielefeld gave the keynote speech, where it was interesting to learn about how free software helps in tackling human rights issues in North Korea. Numerous insightful talks followed it throughout the day. Bhushan’s update on Plasma on mobile devices was interesting, along with David Faure’s talk on how to run KDE softwares without installing them.

Conference Day 2
Day 2 began with the keynote by Claudia Garad where she spoke on how KDE could learn from how Wikimedia faces its hurdles. Aditya Mehra spoke on the visionary Mycroft AI on Plasma. Bhavisha spoke about her contributions to OpenQA. Andreas’ talk on building Automotive ECUs with Yocto was absolutely inspiring. In the Akademy awards, it was great to see deserving individuals being recognised for their amazing contributions.

Social Event
At the end of day 2 of conference, we headed off and met at the nearby Cafe Derwisch – Partycellar to party. Even though there was a looong wait in a queue, it was worth the wait. With loads of food and drinks, and a dance floor, it was the perfect recipe for having fun and socialising. And boy did we make legit use of the dance floor. The party was quite eventful and went way late into the night. It was an evening to remember.

BoFs and Trainings

  • In the KDE-India BoF, we discussed the journey of India in KDE so far, the obstacles we faced previously, the best steps we can take for the next conf.kde.in event, among other topics.
    kde-india-bof.jpeg
  • The Mycroft BoF taught us how to use the AI, how to add new skills, its progress with Plasma mobile, and the obstacles it faces regarding communicating with 3rd party apps.
  • The BoF on the VVAVE Project was particularly interesting to me as it was centred around a music-player app that is similar to the line of software I work on in my current company. We discussed a number of issues, including how to prioritise between online-streaming and playing local files, and how to overcome the technical challenges.
  • The training on documentation provided tips and tricks on writing short, informative and comprehensible documents, followed by a hands-on assignment. This was very helpful.

Also, this year is the first time I was able to be a part of the Annual General Meeting, and it was an interesting experience especially for being able to influence the decisions made inside KDE in such a direct and important way.

At the end of the final day, I (among others) got to draw (read scribble) a colorful message about the principles of KDE on a piece of paper, thanks to Lydia. I’m eagerly waiting to see it uploaded somewhere soon! 🙂

Hikes, trips and picnics
There was a short walking trip organized on Tuesday evening, August 14, where we walked around interesting parts of the city, with magnificent monuments, churches, libraries, museums, palaces, and sculptures flooded all around. The guide was kind enough to explain the historical significance of each building. Clearly a treat for the eyes and knowledgeable as well. The following day, August 15, we went to Kahlenberg, where we enjoyed the amazing view of the beautiful city of Vienna from above. We also went to the top of one of the towers to take a look at it from even higher. On the final evening of the conference, August 17, we went to have a short picnic somewhere overlooking the Danube river. That was fun as well. Or rather sad, as that was the final day of the conference. My phone is filled with amazing pictures, thanks to all these fun initiatives.

Thank you, KDE, for letting me be a part of this amazing event. Keep rocking!

akademy2018-groupphoto

 

by sanjibanbairagya at September 02, 2018 01:50 AM

August 19, 2018

Kushal Das

Aadhaar, the mass surveillance system

If you are following me on Twitter, you have already seen a lot of (re)tweets related to Aadhaar. For the people first time hearing this term, it is a 12 digit unique identification number provided by the Unique Identification Authority of India (UIDAI). It is also the world’s largest bio-metric ID system. It is supposed to be a voluntary service.

From the very beginning, this project tried to hide the details from the Indian citizens. Let it be privacy advocates or security researchers or human rights activists, everyone predicted that this will become a monster, a mass surveillance system, a tool of choice of the power hungry dictators.

Like any other complex system, the majority of the people only see the advertisements from the government and completely miss all the problems and horror stories this project is creating. Here are a few links below for the interested people to read.

Neither my wife, nor our daughter has an Aadhaar (I also don’t have one), that means Py (our daughter) did not get admission to any school last year.

Whenever security researchers or journalists tried to report on the project, the UIDAI tried to hide behind denials and police complaints against the journalists or researchers. There are various reports on how one can get access (both read/write) to the actual production database with as little as $10-30. We now have examples of terrorist organizations having access to the same database. The UIDAI kept telling how this is an unhackable technology and for security they have a 13 feet wall outside of the data center which in turn will keep all hackers away.

They have already build 360 degree databases on top of Aadhaar, and now they are trying to link DNA to the same system.

The current government of India tried their level best to argue in the Supreme Court of India to tell that Indians don’t have any rights to privacy. But, thankfully they failed in this effort, and the Supreme Court ruled privacy as a fundamental right. We are now waiting for the judgment on the Aadhaar (which will hopefully come out in the next few weeks).

Meanwhile, the evil nexus is pushing down Aadhaar to the throats of the Indian citizens and Pakistani spies and gods.

A few days ago, in an event in Jaipur, they asked Edward Snowden the following question.

How big of an issue is privacy?

The answer started with from where that argument comes from.

The answer is that Nazi Germany. The nazi minister of propaganda Joseph Goebbels did this. Because he was trying to change the conversation away from “What are your rights?” and “What evidences must the government show?” to violet them, to intrude into your private life and instead said “Why do you need your rights?”, “How can you justify your rights?”, “Isn’t strange that you are invoking your rights? Isn’t that unusual?”. But, in a free society this is the opposite of the way it is supposed to work. We don’t need to explain why you have a right. You don’t need to explain why it is valuable, why you need it. It is for the government to explain why you don’t deserve it. They go to a court, they show that you are a criminal. This is increasingly falling out of favor, because the governments and companies think that it is inefficient. It is too much work. Life would be easier, life would be more convenient for them, life would be more profitable for them if we didn’t have any rights at all.

But, privacy isn’t about something to hide, privacy is about something to protect. And that is the very concept of liberty. It is the idea that there can be some part of you, of your life, of your ideas that belong to you, not to society. And you get to make the decision about who you share that with. -- Edward Snowden

Why are we reading this in your blog?

This might a question for many of you. Why are reading this in a blog post or in a planet? Because we, the people with the knowledge of technology are also part of these evil plans. We now know about many private companies taking part with their local government to build 360 degree profiles, to track the citizens and to run the mass surveillance systems. For example, related to Aadhaar, for the last 4 years, Google silently pushed the Aadhaar support phone number (which now UIDAI is trying to stay away from) to every Google Android phone in India. When they got caught red handed, they claimed that they did it inadvertently. Finacle software by Infosys denies creation of bank accounts without Aadhaar. Microsoft is working to link Skype with Aadhaar. Bill Gates is trying to push the idea that Aadhaar is all good, and does not have any issues.

What can you do?

You can start by educating yourself first. Read more about the technologies which controls our lives. Have doubt about the things and try to understand how they actually work. Write about them, ask questions to the people in power. Talk about the issues to your friends and family.

This is not gong to be an easy task, but, we all should keep fighting back to make sure of a better future for our next generation.

by Kushal Das at August 19, 2018 06:33 AM

August 16, 2018

Robin Schubert

Sync data from mobile phone with rsync

I'm quite fond about having a google-free phone. I run a lineage OS without g-apps, use my own caldav and carddav server to sync my contacts and calendars, can find every software I need in the F-Droid app and live happy ever since.

However, what has bugged me was synchronization of personal photos and videos. I did several attempts to solve that using webdav (owncloud/nextcloud), but my phone was not happy with the available clients and the battery would not last very long.

I now have a setup that allows me secure synchronization that I'm quite happy with, using rsync from my phone to server. It's simple and slim and feels just too familiar to be a bad idea.

Termux - Terminal Emulator and Linux Environment

Termux is what I've been looking for on Android for a long time. Native Linux on my phone would be priceless, but the ease of use and installation makes Termux a more than good compromise. It comes with a nifty packet manager and allows me to run small Python and R scripts including web-scraping etc. on the fly.

You can get Termux for free in the Google Playstore, the extensions however will cost around 2\$ (which is okay, you'll support some really good work!). Using the F-Droid app you can install the extensions for free.

I also installed the extension Termux:Widget that allows to start scripts via single tap from home screen - a very handy addition, I use it to reboot my raspberry pi or wake-on-lan my computer that's in the basement.

Set up rsync

To set up Termux for this task, install rsync first:

pkg install rsync

Generate a key-pair (I just use the defaults):

ssh-keygen

Per default this will create the files id_rsa and id_rsa.pub in your ~/.ssh directory. Make sure to put the contents of id_rsa.pub into the ~/.ssh/authorized_keys file on your target server.

And set up the Termux storage:

termux-setup-storage

Create a folder for the Termux:Widget shortcuts and create the script:

mkdir .shortcuts
echo "rsync -h -r --info=progress2 ~/storage/dcim/* robin@<myserver>:/<path_to_my_rsync_folder>" > .shortcuts/sync

Conclusion

With the Termux widget this simple setup allows stable and quick sync of my data that leaves me with a few worries less. In fact I set this up on my wife's phone as well, since it's working so nicely.

by Robin Schubert at August 16, 2018 12:00 AM

August 15, 2018

Farhaan Bukhsh

File Indexing In Golang

I have been working on a pet project to write a File Indexer, which is a utility that helps me to search a directory for a given word or phrase.

The motivation behind to build this utility was so that we could search the chat log files for dgplug. We have a lot of online classes and guest sessions and at times, we just remember the name or a phrase used in the class, backtracking the files using these phrases aren’t possible as of now. I thought I will give a stab at this issue and since I am trying to learn golang I used it to implement my solution. It took me a span of two weeks where I spent time to upskill certain aspects and also to come up with a clean solution.

Exploration

This started with me exploring similar solutions, because why not? It is always better to improve an existing solution than to write your own. I didn’t find any which suited our need though so I ended up writing my own. The exploration led me to discover a few  libraries that proved useful. I found fulltext and Bleve.

I found bleve to have better documentation and some really beautiful thought behind the library. Really minimal yet effective. At the end of it all, I was sure I was going to use it.

Working On the Solution

After all the exploration I tried to break the problem into smaller pieces and then go about solving each one of them. So the first one was to understand how bleve worked. I found out that bleve creates an index first; for which we need to give it the list of files. The index is basically a map structure behind the scenes, where you give it the id and content to be indexed. So what could be a unique constraint for a file in a filesystem? The path of the file! I used it as the id to my structure and the content of my file as the value.

After figuring this out, I wrote a function which takes the directory as the argument and gives back the path of each file as well as its contents. After a few iterative. improvements it diverged into two functions; one responsible to get the path of all the files and the other to just read the file and get the content out.

func fileNameContentMap() []FileIndexer {
	var ROOTPATH = config.RootDirectory
	var files []string
	var filesIndex FileIndexer
	var fileIndexer []FileIndexer

	err := filepath.Walk(ROOTPATH, func(path string, info os.FileInfo, err error) error {
		if !info.IsDir() {
			files = append(files, path)
		}
		return nil
	})
	checkerr(err)
	for _, filename := range files {
		content := getContent(filename)
		filesIndex = FileIndexer{Filename: filename, FileContent: content}
		fileIndexer = append(fileIndexer, filesIndex)
	}
	return fileIndexer
}

This forms a struct which stores the name of the file and the content of the file. And since I can have many files I need to have a array of said struct. This is how a simple data structure evolves into a complex one.

Now I have the utility of getting all files, getting content of the file and making an index.

This leads us to the next crucial step.

How Do I Search?

Now that I’ve prepped my data the next logical step was to retrieve the searched results. The way we search something is by passing a query so I duck-typed a function which accepts a string and then went on a spree of documentation look up to find out how do I search in bleve. I found a simple implementation which returns the id of the file which is the path and match score.

&nbsp;func searchResults(indexFilename string, searchWord string) *bleve.SearchResult {
	index, _ := bleve.Open(indexFilename)
	defer index.Close()
	query := bleve.NewQueryStringQuery(searchWord)
	searchRequest := bleve.NewSearchRequest(query)
	searchResult, _ := index.Search(searchRequest)
	return searchResult
}

This function opens the index and search for the term and returns back the information.

Let’s Serve It

After all that is done I need to have a service which does this on demand so I wrote a simple API server which has two endpoints index and search.  The way mux works is you give the endpoint to the handler and the function to be mapped with it. I had to restructure the code in order to make this work. I faced a really crazy bug which when I narrowed it down, came to a point of a memory leak and yes, it was because I left the file read stream open, so remember when you Open always defer Close.

I used Postman to heavily test it and it was returning good responses. A dummy response looks like this:

&nbsp;[{"index":"irclogs.bleve","id":"logs/some/hey.txt","score":0.6912244671221862,"sort":["_score"]}]

Missing Parts?

The missing part was I didn’t use any dependency manager which Kushal pointed out to me, so I landed up using dep to do this for me. The next one was one of my favourite  problems of the project and that was how to auto-index a file. Suppose my service is running and I added one more file to the directory, then this file’s content wouldn’t come up in the search because the indexer hasn’t run on it yet. This was a fascinating  problem and I tried to approach it from many different angles. First I thought I would re-run the service every time I add a file but that’s not a graceful solution. Then I thought I would write a cron job which would ping /index at regular intervals and yet again that struck me as inelegant. Finally I wondered if I could detect changes in a file. This led me to explore gin, modd and fresh.

Gin was not very compatible with mux so didn’t use it, modd was really nice but I needed to kill the server to restart it since two services cannot run on a single port and every time I kill that service I kill the modd daemon too so that possibility also got ruled out.

Finally the best solution was fresh although I had to write a custom config file to suit the requirement, this approach still has issues with nested repository indexing which I am thinking how to figure out.

What’s Next?

This project is yet to be containerised and there are missing test cases so I would be working on them, as and when I get time.

I have learnt a lot of new things about the filesystem and how it works, because of this project. This little project also helped me appreciate a lot of golang concepts and made me realise the power of static typing.

If you are interested you are welcome to contribute to file-indexer. Feel free to ping me.

Till then, Happy Hacking!

 

by fardroid23 at August 15, 2018 02:49 PM

Jason Braganza (Work)

Book Review – i want 2 do project. tell me wat 2 do

Click me to buy!


TL;DR? It’s awesome. Buy it right now.

I was looking to dip my toes into some sort of structured help with the summer training and open source in general, because while I knew what I wanted, I just didn’t know how to go about it.

And then I realised that one of our mentors had actually gone and written a whole book on the how to. So, I bought the paperback. The binding is really good, the paper really nice (unlike other tech books I’ve read) and the words large enough to read. I expect to get a lot of use, out of the book.

And lot of use is right. While it’s a slim volume and a pretty quick read, the book is pretty dense when it comes to the wisdom it imparts.

The book has a simple (yet substantial to execute) premise. You’ve just tipped your toe into programming, or you’ve learnt a new language, or you’ve probably written a few programs or maybe you’re just brand new. You want to explore the vast thrilling world that is Open Source Software. What now?

“i want 2 do project. tell me wat 2 do.” answers the “what now” in painstaking detail.

From communication (Mailing List Guidelines) to the importance of focus (Attention to Detail) to working with mentors (the Project chapters) to the tools (Methodology & tools) to the importance of sharpening the saw (Reading …) and finally the importance of your environment (Sustenance), the book covers the entire gamut that a student or a novice programmer with open source would go through.

Shakthi writes like he speaks; pithily, concisely with the weight of his experience behind his words.

The book is chockfull of quotes (from the Lady Lovelace to Menaechmus to Taleb) that lend heft to the chapters. The references at the end of each chapter will probably keep me busy for the next few months.

The book’ll save you enormous amounts of time and heartache, in your journey, were you to heed its advice. It’s that good.

by Mario Jason Braganza at August 15, 2018 10:55 AM

August 13, 2018

Anwesha Das

Twitter from command line

Since the time I have started writing code, the toughest job for me is to be in peace with the black and green screen, the terminal. As it is being “The Thing” which keeps my lovely husband (ah ha, really?) from me. So as an initiative of my “peace making process” I have started doing my day mostly on this boring screen. A part of that is me trying to do twitter from the command line. Thus, let us make the (boring) terminal interesting.

To reach the aim I needed a Python module to access the Twitter api. I used a module called python-twitter. Click is a Python package to create command line applications. I used it to have a better command line interface. I used Microsoft Visual Studio Code as my primary editor. Like all previous projects, I am leaned on Jupyter Notebook to try out code snippets. I used Pipenv for the first time here.

import sys
import twitter
import json
import click

After importing the modules (as mentioned above) required, the job was to create the boolean command line flags through click, so,

@click.command()
@click.option("--tweet", "-t", is_flag=True, help="Does tweet.")
@click.option("--timeline", "-n", is_flag=True, help="Shows user's timeline.")
@click.option(
   "--directmessage", "-m", is_flag=True, help="Shows user's direct messages."
)

I learnt click from this blogpost, it was really helpful.

I wrote a config.json file where I have the required authentication details, such as consumer key, consumer secret, access token key, access token secret and user id. I got them from my Twitter developer account. In the account you have to set your access level as “Read, Write and Direct Messages”. I am creating an object of the twitter.Api class. I am passing different arguments, tweet, timeline and direct message to subsequently do tweet, see my timeline and get to see my direct messages, in the command line.

I used Black to format my code. Formatting of the code makes the code more readable and easy to review.

The next and final job was to upload it on [PyPI] using twine. For this I followed a blogpost I did earlier I The source code of the project is available on my github.

If you notice, I have used many things for the first time in this small learning effort. Projects of this size are really helpful to learn new things.

Happy tweeting (from the command line).

by Anwesha Das at August 13, 2018 06:07 PM

August 12, 2018

Jason Braganza (Work)

Programming, Day 55

Updates:

  • Much better at gtypist lesson Q1
  • Also automated resizing and compressing my gtypist screenshots via an Automator folder action
  • Done with Chapter 9 of the Lutz Book. I now know of tuples and strings

Automator Screenshot

gtypist screenshot gt16

by Mario Jason Braganza at August 12, 2018 06:48 AM

August 07, 2018

Kushal Das

August 06, 2018

Praveen Kumar

[Event Report] DevConf India-2018

This week I got a chance to attend Devconf India which held at Christ University Bangalore. As per stats, there were around ~1323 attendee and 110 speakers. There were around 14 parallel tracks (Agile, Blockchain, Cloud and Container, Community, Design, Developer Tools, DevOps, IOT, Machine Learning, Middleware, Platform, QE, Security, Storage) and BOFs, workshops so pretty much completely packed schedule.

Day 1 started with a dance performance by university students and after that the keynote by Ric Wheeler about "Open source is better for companies/businesses, communities and developers" and he talked about back in the day how things used to happen and how far we have come now in terms of software businesses. He also talked about the why nowadays most of the organization are moving toward Open Source.

After the keynote, I went to booth area where most of the communities booth were present (Fedora, Foreman, OpenShift, Sliverblue, Women who code, RDO, Mozilla, ElasticSearch, devopedia, Ansible) and spent some time answering some OpenShift related queries by participants.

After the tea break, I went for my workshop, which was about “Getting start with OpenShift using MiniShift”. I shared the stage with Budhram and we had Anjan Nath, Jatan dedicated volunteer for this workshop (Thanks guys.). We got around 38 participants and everything went smoothly except some of the issues with the windows users due to expected reasons (no admin permission on system and C:\ drive was not used). We explained the basic navigation of OpenShift web UI, how to use Openshift client (oc) tool to connect to a cluster, how to deploy an application. We also covered some of the basic of kuberentes resources.

After Lunch I spent most of the time roaming around the booth area, talking to different people. Met with a lot of old friends as usual.

Day 1 end with the keynote by Karanbir Singh (kbsingh) about “Open Source won”, he talked about the phase when Indian Linux group was started and now it’s very hard to find any organization which is not consuming an opensource project.

Some of us went outside for dinner and back early to hotel to get some rest  :)

Day 2 started with the keynote by Christian Heimes about “Lessons about security”, he talked about a bit of Roman culture and how information saved in old days. He talked about recent hardware vulnerabilities and why you should always keep your software up to date especially the security fixes.

After the tea break, I went to volunteer Baiju workshop, which was about “ RESTful API Development using Go”.  There were around 40 participants and most of them were familiar with the golang. Baiju started with a simple http server and then how you can build up your route on top of it without using any kind of framework to teach the logic what happens behind the scene. He then introduced a module `mux` which is used to filter out the request type (GET, DELETE, POST ...etc.) so that if a route only serves one request type then other should be ignored for it. He also talked about `negroni` for middleware use case.


 Then I attended a talk by Graham about “Data science in the cloud using Python”, he talked about how you can deploy Jupyter notebook on Openshift for personal use or even share it to other colleagues/friends in the organization.



Day 2 ended with thank you note for all the organizers, volunteers, college faculty members, housekeeping staff and whoever involved making this conference success.


Kudos to organizers and volunteers to pull off such an amazing conference.






by Praveen Kumar (noreply@blogger.com) at August 06, 2018 06:28 PM

July 31, 2018

Jason Braganza (Personal)

Daily Writing, 76 – Mother and Son

_MG_3872


Mothers yielding Bibles, contemplating smearing the blood of lamb chops over her doorway.
Anything to keep her son alive another day.

Antonia Perdu

by Mario Jason Braganza at July 31, 2018 02:51 AM