Let me introduce rpm-macros-virtualenv 0.0.1 to you all.

This is a small set of RPM macros, which can be used by the spec files to build and package any Python application along with a virtualenv. Thus, removing the need of installing all dependencies via dnf/rpm repository. One of the biggest usecase will be to help to install latest application code and all the latest dependencies into a virtualenv and also package the whole virtualenv into the RPM package.

This will be useful for any third part vendor/ISV, who would want to package their Python application for Fedora/RHEL/CentOS along with the dependencies. But, remember not to use this for any package inside of Fedora land as this does not follow the Fedora packaging guidelines.

This is the very initial release, and it will get a lot of updates in the coming months. The project idea is also not new, Debian already has dh-virtualenv doing this for a long time.

How to install?

I will be building an rpm package, for now download the source code and the detached signature to verify it against my GPG key.

wget https://kushaldas.in/packages/rpm-macros-virtualenv-0.0.1.tar.gz
wget https://kushaldas.in/packages/rpm-macros-virtualenv-0.0.1.tar.gz.asc
gpg2 --verify rpm-macros-virtualenv-0.0.1.tar.gz.asc rpm-macros-virtualenv-0.0.1.tar.gz

Untar the directory, and then copy the macros.python-virtualenv file to the RPM macros directory in your system.

tar -xvf rpm-macros-virtualenv-0.0.1.tar.gz
cd rpm-macros-virtualenv-0.0.1/
sudo cp macros.python-virtualenv /usr/lib/rpm/macros.d/

How to use?

Here is a minimal example.

# Fedora 27 and newer, no need to build the debug package
%if 0%{?fedora} >= 27 || 0%{?rhel} >= 8
%global debug_package %{nil}
%endif
# Use our interpreter for brp-python-bytecompile script
%global __python /opt/venvs/%{name}/bin/python3


%prep
%setup -q

%build
%pyvenv_create
%{__pyvenvpip3} install --upgrade pip
%pyvenv_build

%install
%pyvenv_create
%{__pyvenvpip3} install --upgrade pip
%pyvenv_install
ln -s /opt/venvs/%{name}/bin/examplecommand $RPM_BUILD_ROOT%{_bindir}/examplecommand

%files
%doc README.md LICENSE
/opt/venvs/%{name}/*

As you can see, in both %build and in %install, first we have to call %pyvenv_install, that will create our virtualenv. Then we are installing the latest pip in that environment.

Then in the %build, we are calling %pyvenv_build to create the wheel.

In the %install section, we are calling %pyvenv_install macro to install the project, this command will also install all the required dependencies (from the requirements.txt of the project) by downloading them from https://pypi.org.

If you have any command/executable which gets installed in the virtualenv, you should create a symlink to that from $RPM_BUILD_ROOT/usr/bin/ directory in the %install section.

Now, I have an example in the git repository, where I have taken the Ansible 2.7.1 spec file from Fedora, and converted it to these macros. I have build the package for Fedora 25 to verify that this works.

The dumbest thing you can do is to think you're smart.

We often tend to think we know a lot of things. Things we read, hear or see on whatever source of information may be perceived as just true. However, I think that it is very important to question even the most trivial of best known things. The believe in knowledge does not just kill creativity but can also be dangerous.

I've studied Physics and there is one take-home message that I would like to share. People often hear that physicists have discovered this or that. In most cases this leads to the believe that we know how the world around us works and what it is made of. Actually we don't. The way Physics works is different: It won't tell you how things work or what they are made of, instead it will provide you with a set of tools, models, and theories, derived from observations and previous models and theories, that will often result in pretty good approximations, and predictions of what we're observing in the world around us. This is not better or worse than the truth would be, in fact it's a very pure and straight forward approach that allows us to go far beyond of what seems possible sometimes.

It would in fact be quite optimistic to think that we could understand truth, with the limitations of our nature. We perceive the world in three dimensions, are heavily dependent on language (could write a whole book on that) and have a limited set of senses - but what is worse: we're not even using them. We rely on science and studies instead, loosing more and more the ability to perceive and interpret (and believe in) the signals of our own body. You cannot convince someone that might call himself scientist who knows how thinks work of the efficacy of some compound when you just feel that it is good and right for you. Instead, the compound has to go through several stages of clinical trials, that try to measure safety, tolerability and efficacy in vitro, in animals, in humans. While I understand and appreciate this approach, I often feel like the available tools to assess these domains are not even close to be suitable for that task. As a result, a negative trial will let us know that there is no effect.

It's neither easy nor fun to discuss with someone who is fiercely convinced by something just read in an article. While it's a very good thing to read (or to gather information through other channels), that information should not just be taken for granted because it has been printed in a journal. To question that information at least every once in a while should be a habit.

I spent most of my life using Fedora as my primary operating system on my desktop/laptops. I use CentOS on my servers, sometimes even Fedora, and a few special cases, I use *BSD systems.

But, for the last one year I am running Qubes OS as my primary operating system on my laptop. That enables me to still keep using Fedora in the AppVMs as I want, and I can also have different work VMs in Debian/Ubuntu or even Windows as required. Moving to a newer version of Fedora is just about installing the new template and rebooting any AppVM with the newest template.

Fedora 29 will release on 30th October, and Qubes team already built a template for the same and pushed to the testing repository. You can install it by the following command.

$ sudo qubes-dom0-update qubes-template-fedora-29 --enablerepo=qubes-templates-itl-testing

After this, I just installed all the required packages and setup the template as I want using my Qubes Ansible project. It took only a few minutes to move all of my development related VMs into Fedora 29 and this still keeps the option open to go back to Fedora 28 the moment I want. This is one of the beauty of Qubes OS and of course there are the regular security aspects too.

If you are a software developer using Linux, and also care about security practices, give Qubes OS a try. It has also a very active and helpful user community. I am sure it will not disappoint you.

Saturday, I am sitting at a Starbucks in Bangalore, trying my hands on a Golang project. I come across this argument -d in go get:

The go cli help says:

The -d flag instructs get to stop after downloading the packages; that is,
it instructs get not to install the packages.

Wonderful! So, if you just want to download the golang project for the sake of contributing, you can use:

go get -d k8s.io/kubernetes

... and it will download the package for you, after which you can start working on the project.

2013

The year I graduated from a third-tier college in Durgapur. I did not expect much from the college, mostly because of the poor placement record in the college.

To get myself a job, I was primarily going through hasjob.co and any job opportunities that came through Bangpypers mailing list.

Finally, I landed myself into an internship and later which turned into a full-time at HackerEarth was through the hasjob.co.

2018

5 years later, finding a job in a suitable company is still tough. In the last 5 years, I have come to know about more options, be it AngelList, Hasjob, StackOverflow, Facebook Job Groups etc.

But the problem still I think is the same, even if you apply through these websites a lot of times you don't get a reply from the company you applied to. On other hand, referrals work. I've referred good people to companies I know. This way, there is a chance of getting an interview in the company. Cracking the interview is a different story altogether which I believe depends a lot on the candidate.

Recently, I got to know that Prashant, one of my school senior and a close friend, who famously known in the tech community for his "Bitcoin Wedding" started an effort called "SkipTheLine".

SkipTheLine is a newsletter where Prashant publishes profiles of three developers. These developers are from the community, who are active in open source, or competitive programming, or just good at technologies they work on. He goes on to introduce the developers through email with the companies point of contact via email and then the candidate and the POC take the discussion forward.

I personally loved the initiative that he took as at the end of the day if people whom I know come asking for a job referral I would just direct them to SkipTheLine. Prashant has quite a stronghold in the startup community and does great work in connecting the folks with some really good startup across the country.

I know people personally who got hired a within few days of their newsletter published so if you are looking out for a job, do fill out the SkipTheLine form.

If you are looking to hire, do drop me an email at gmail AT yudocaa DOT in.

Last Friday, I was sitting in one of the good coffee shops in Bangalore with my friend. Coffee and discussion is the best combination to release stress. It was looking like a perfect Friday evening until my friend was struck by an idea of asking me a question.

“Let me conduct a quiz.” he said, interrupting our conversation.

“A quiz? Quiz on what?”, I asked.

“On programming”, he said

“What is the level of difficulty then?” I said.

Asking the level of difficulty is important. I never invest my efforts in solving something easy. If he had said easy, I would have ignored to answer, but he said “It is a bit difficult, but not that difficult. I gave a wrong answer to this question in my last interview.”

There was no reason to go back from here. Taking some deep breaths I said, “Please go ahead.”

He stood up, took a tissue paper from a nearby counter and scratched below code on it. ¹ And he asked me by pointing towards that code, “What will be the output of this code?”

def my_function(): l = [0, 1, 2] print(l[30:]) my_function() 

Now it was my turn to give the answer. I looked at the code and tried parsing it in my head — line by line. In my mind, I observed the first line. It was defining a function which seems to be correct. I moved my eyes to the next line. It was defining a variable l of type list and assigning values ranging from 0 to 2. Even this wasn’t looking problematic. So I forwarded to the next line where it was trying to print that variable l by slicing it from starting value 30 to the infinity.

“Well, the start value is 30 which is greater than the length of the list. This should raise an IndexError” I said in my mind. I was about to speak an answer, but suddenly Devil of me flashed.

“It is less than a banana job my dear,” the Devil said to me, “You should take a little advantage of this opportunity my boy.”

Because things were looking in my control, I shook my hands with the Devil.

I said to my friend, “How about betting for some real values?”

Going closer I spoke, “If I answer correctly, You will pay the bill and If I am wrong, This will be a treat from my side.”

He thought for a while and nodded. Now it was my turn to unveil the cards.

I said in a strong voice, “It will raise an IndexError.” And shifted my focus towards the chocolate.

He starred my face for a second and spoke, “Okay. Are you sure about this?”.

This was the hint he gave. I should have taken another shot here. What happens next become a lesson for me.

I said with a flat face, “Yes I am.”

With my answer, he instantly opened his backpack, took his Laptop out and typed the code which he wrote on that tissue.

When I stopped hearing a sound of typing I yelled, “So did I win?”

He turned his laptop towards me and exclaimed, “Not at all!”

When I focused on the screen, the interpreter was printing []. Damn! I lost the bet. Why the hell slice is returning an empty list even when we are trying to slice it with a value which is greater than the length of it! It was surely looking unpythonic behavior. I paid whatever the bill amount was. Entire evening this question was all roaming my mind. After coming home, I decided to justify reasons for returning an empty list instead of raising an IndexError from a slice.

Below are a few reasons justifying such behavior of slice function. I am sharing this with you so that you don’t lose a bet with your friend :) For those who haven’t used slice anytime in their life, I advise to read this tutorial. Reading this guide for understanding how a slice function converts the input values. Especially rule number 3 and 4 referenced there.

• Reason number one:

  Python lists are more commonly used in iterations. Consider below example:

  numbers = [0, 1, 2, 3] for number in numbers[30:]: print(number) 

  If slice was raising an IndexError, then the above code would have to written like this

  written like like this

  numbers = [0, 1, 2, 3] try: for number in numbers[30:]: print(numbers) except IndexError: pass 

  Or in another way below is also looking reasonable

  numbers = [0, 1, 2, 3] start = 30 if start < len(numbers): for number in numbers[start:]: print(number) 

  Both the approaches are looking little lengthy by an obvious reason. And that reason is to prevent executing loop if there are no elements in it. When we observe the behavior of slice called at for in, it makes sense to return an empty list instead of raising an IndexError.

I am not able to find further reasons to return an empty list instead of raising the IndexError. But I am sure, there will be. If you know any other potential reasons for such behavior of slice, please drop me a mail at jaysinhp at gmail dot com or contact me over Twitter @jaysinhp. I will update the reasons at this post and give credits to you. Thanks for reading this post.

Proofreaders: Geoffrey Sneddon, Elijah, Mahendra Yadav, Dhavan Vaidya,

This is not a technical post; this is something that I have been going through, in life right now. A few months ago, when I left my first job (another time, another post ), I had a plan. I wanted to take few months off and work on my technical knowledge and write amazing software and get a lot of learning out of my little sabbatical.

But I was not able to do that for a few reasons, primo being I had to move homes in Bangalore because my brother got transferred, so the savings that I had set aside wouldn’t be enough. This was not the end. When it rains, it pours apparently. My dad got super sick, he had a growth near his kidney which the doctors diagnosed as cancer. I got really scared with the situation I was going through. The thing about your parents is that no matter how much you fight with them or how much they “control” you; at the end of the day the thought of losing them can scare the hell out of you. For me, they are my biggest support system so I was not scared, I was terrified.

I gave it a really deep thought and took a call. I needed to find a job. The sabbatical could wait. I started applying to companies and talking to people if they needed extra hand at work. One piece of advice – never leave a job unless you have another in hand. Luckily, I had my small pot of gold, savings, so even in this phase I was sustaining myself. Yes, savings are real and you should have a sufficient amount at any given point of your life. This helps you to take the hard decisions and also to think independently (what Jason calls F*ck you money).

It still feels like a nightmare to me. I use to feel that I will wake up and it will all be over. Reality check; it wasn’t a dream so I have to live with it and make efforts to overcome this situation.

Taking up a job for me was important for two reasons,

1. I have to sustain myself
2. I need to have a back up in case my dad needs something (I also have super amazing siblings who were doing the same)

I realised one thing about prayer and God; yes, I believe in God, and I don’t know if prayer works but you definitely get the strength to face your problems and the unknown. I use to call my dad regularly asking how he was doing and some days he could not speak all that much and he use to talk in his weak tone. I use to cry. I was in so much pain although it was not physical or visible. And then, I would cry again.

But tough times teach you a lot, it shows you real friends, it shows you the people you care for and as Calvin’s dad would have said, “It build character!”. I have been through bad times before and the thing about time is , “It changes!”. I knew someday this bad time I am going through will change. Either the agony I am going through will reduce or I will get used to it.

So as I was giving interviews within a month of me moving on from my old job, I was offered one at Clootrack. I like the people who interviewed me and I like that ideas they have been working on. But I have seen people change and I have gone through a bad experiences and at no point of time did I want to repeat past mistakes, so I did a thorough background check before I said yes to them. I got a really good response so here I am working with them.

The accommodation problem that I had was my brother was shifting out of his quarters  and I used to live with him. Well, I helped him pack and I still remember the time when I was bidding farewell to him and my sister-in-law. I had tears in my eyes and after my goodbyes, the moment I stepped in the house I could feel the emptiness and I cried the whole night.  I  could stay at the old place for a week, not more. At this point I can’t thank Abhinav enough that he came as  a support I needed. He graciously let me live with him as long as  I wanted to. Apparently he needed help, paying his bills :P.  This bugger would never accept the fact, he helped me. When dad’s condition was getting bad he gave me really solid moral support. I had also shared my situation with Jason, Abraar, Kushal and Sayan. I received a good amount of moral support from each one of them, specially Jason. I use to tell him everything and he would just calm me down and talk me through it.

So when I shifted to Abhinav’s place all I had was 6 bags and a carton. My whole life was 6 bags and a carton. My office was a 2 hour bus ride one way and another 2 hours to come back. But I didn’t have any problems with this arrangement because this was the least of my problems. I literally use to live out of my bags and I wasn’t sure this arrangement would last long. I had some really amazing moments with Abhinav, I enjoyed our ups and downs and those little fights and leg pulling.

Well, my dad is still not in the best of his health, but he is doing better now. I visit my family more frequently now and yes call them regularly with a miss. I realised the value of health after seeing my dad. I went home after a month of joining Clootrack and stayed with him for a whole month and worked remotely, we visited few doctors and they said he is doing better. After coming back I realised I was not getting any time for myself so I shifted to a NestAway near my office. Although I feel I’ve gotten used to the agony, you never know what life has in store for you next.
It feels much better now, though.

I thank God for giving me strength and my friends and family for supporting me in a lot of different ways.

With Courage in my Heart,
And Faith over Head

This year’s Akademy, the annual world summit of KDE, was held in the beautiful city of Vienna, Austria, from 11th to 17th August, 2018. The 7-day event was divided in two parts, with the first 2 days being mostly keynote addresses and different talks by KDE contributors, followed by 5 more days BoFs, and workshops. Just like every other KDE event, this one was also as awesome as it could get.

Welcome party
The evening before the day the conference was scheduled to start, there was a nice welcome party with loads of food and drinks, where I got to catch up with most of the fellow KDE contributors who I hadn’t met for quite some time, and also got to see a lot of new faces as well, talking to whom felt like a breath of fresh air. Overall, it was a warm welcome, and raised everyone’s spirits to get ready for Akademy for the next day.

Conference Day 1
Day 1 was opened by Lydia, our beloved President of KDE. Dan Bielefeld gave the keynote speech, where it was interesting to learn about how free software helps in tackling human rights issues in North Korea. Numerous insightful talks followed it throughout the day. Bhushan’s update on Plasma on mobile devices was interesting, along with David Faure’s talk on how to run KDE softwares without installing them.

Conference Day 2
Day 2 began with the keynote by Claudia Garad where she spoke on how KDE could learn from how Wikimedia faces its hurdles. Aditya Mehra spoke on the visionary Mycroft AI on Plasma. Bhavisha spoke about her contributions to OpenQA. Andreas’ talk on building Automotive ECUs with Yocto was absolutely inspiring. In the Akademy awards, it was great to see deserving individuals being recognised for their amazing contributions.

Social Event
At the end of day 2 of conference, we headed off and met at the nearby Cafe Derwisch – Partycellar to party. Even though there was a looong wait in a queue, it was worth the wait. With loads of food and drinks, and a dance floor, it was the perfect recipe for having fun and socialising. And boy did we make legit use of the dance floor. The party was quite eventful and went way late into the night. It was an evening to remember.

BoFs and Trainings

• In the KDE-India BoF, we discussed the journey of India in KDE so far, the obstacles we faced previously, the best steps we can take for the next conf.kde.in event, among other topics.
  
• The Mycroft BoF taught us how to use the AI, how to add new skills, its progress with Plasma mobile, and the obstacles it faces regarding communicating with 3rd party apps.
• The BoF on the VVAVE Project was particularly interesting to me as it was centred around a music-player app that is similar to the line of software I work on in my current company. We discussed a number of issues, including how to prioritise between online-streaming and playing local files, and how to overcome the technical challenges.
• The training on documentation provided tips and tricks on writing short, informative and comprehensible documents, followed by a hands-on assignment. This was very helpful.

Also, this year is the first time I was able to be a part of the Annual General Meeting, and it was an interesting experience especially for being able to influence the decisions made inside KDE in such a direct and important way.

At the end of the final day, I (among others) got to draw (read scribble) a colorful message about the principles of KDE on a piece of paper, thanks to Lydia. I’m eagerly waiting to see it uploaded somewhere soon!

Hikes, trips and picnics
There was a short walking trip organized on Tuesday evening, August 14, where we walked around interesting parts of the city, with magnificent monuments, churches, libraries, museums, palaces, and sculptures flooded all around. The guide was kind enough to explain the historical significance of each building. Clearly a treat for the eyes and knowledgeable as well. The following day, August 15, we went to Kahlenberg, where we enjoyed the amazing view of the beautiful city of Vienna from above. We also went to the top of one of the towers to take a look at it from even higher. On the final evening of the conference, August 17, we went to have a short picnic somewhere overlooking the Danube river. That was fun as well. Or rather sad, as that was the final day of the conference. My phone is filled with amazing pictures, thanks to all these fun initiatives.

Thank you, KDE, for letting me be a part of this amazing event. Keep rocking!

I'm quite fond about having a google-free phone. I run a lineage OS without g-apps, use my own caldav and carddav server to sync my contacts and calendars, can find every software I need in the F-Droid app and live happy ever since.

However, what has bugged me was synchronization of personal photos and videos. I did several attempts to solve that using webdav (owncloud/nextcloud), but my phone was not happy with the available clients and the battery would not last very long.

I now have a setup that allows me secure synchronization that I'm quite happy with, using rsync from my phone to server. It's simple and slim and feels just too familiar to be a bad idea.

Termux - Terminal Emulator and Linux Environment

Termux is what I've been looking for on Android for a long time. Native Linux on my phone would be priceless, but the ease of use and installation makes Termux a more than good compromise. It comes with a nifty packet manager and allows me to run small Python and R scripts including web-scraping etc. on the fly.

You can get Termux for free in the Google Playstore, the extensions however will cost around 2\$ (which is okay, you'll support some really good work!). Using the F-Droid app you can install the extensions for free.

I also installed the extension Termux:Widget that allows to start scripts via single tap from home screen - a very handy addition, I use it to reboot my raspberry pi or wake-on-lan my computer that's in the basement.

Set up rsync

To set up Termux for this task, install rsync first:

pkg install rsync

Generate a key-pair (I just use the defaults):

ssh-keygen

Per default this will create the files id_rsa and id_rsa.pub in your ~/.ssh directory. Make sure to put the contents of id_rsa.pub into the ~/.ssh/authorized_keys file on your target server.

And set up the Termux storage:

termux-setup-storage

Create a folder for the Termux:Widget shortcuts and create the script:

mkdir .shortcuts
echo "rsync -h -r --info=progress2 ~/storage/dcim/* robin@<myserver>:/<path_to_my_rsync_folder>" > .shortcuts/sync

Conclusion

With the Termux widget this simple setup allows stable and quick sync of my data that leaves me with a few worries less. In fact I set this up on my wife's phone as well, since it's working so nicely.

I have been working on a pet project to write a File Indexer, which is a utility that helps me to search a directory for a given word or phrase.

The motivation behind to build this utility was so that we could search the chat log files for dgplug. We have a lot of online classes and guest sessions and at times, we just remember the name or a phrase used in the class, backtracking the files using these phrases aren’t possible as of now. I thought I will give a stab at this issue and since I am trying to learn golang I used it to implement my solution. It took me a span of two weeks where I spent time to upskill certain aspects and also to come up with a clean solution.

Exploration

This started with me exploring similar solutions, because why not? It is always better to improve an existing solution than to write your own. I didn’t find any which suited our need though so I ended up writing my own. The exploration led me to discover a few  libraries that proved useful. I found fulltext and Bleve.

I found bleve to have better documentation and some really beautiful thought behind the library. Really minimal yet effective. At the end of it all, I was sure I was going to use it.

Working On the Solution

After all the exploration I tried to break the problem into smaller pieces and then go about solving each one of them. So the first one was to understand how bleve worked. I found out that bleve creates an index first; for which we need to give it the list of files. The index is basically a map structure behind the scenes, where you give it the id and content to be indexed. So what could be a unique constraint for a file in a filesystem? The path of the file! I used it as the id to my structure and the content of my file as the value.

After figuring this out, I wrote a function which takes the directory as the argument and gives back the path of each file as well as its contents. After a few iterative. improvements it diverged into two functions; one responsible to get the path of all the files and the other to just read the file and get the content out.

func fileNameContentMap() []FileIndexer {
	var ROOTPATH = config.RootDirectory
	var files []string
	var filesIndex FileIndexer
	var fileIndexer []FileIndexer

	err := filepath.Walk(ROOTPATH, func(path string, info os.FileInfo, err error) error {
		if !info.IsDir() {
			files = append(files, path)
		}
		return nil
	})
	checkerr(err)
	for _, filename := range files {
		content := getContent(filename)
		filesIndex = FileIndexer{Filename: filename, FileContent: content}
		fileIndexer = append(fileIndexer, filesIndex)
	}
	return fileIndexer
}

This forms a struct which stores the name of the file and the content of the file. And since I can have many files I need to have a array of said struct. This is how a simple data structure evolves into a complex one.

Now I have the utility of getting all files, getting content of the file and making an index.

This leads us to the next crucial step.

How Do I Search?

Now that I’ve prepped my data the next logical step was to retrieve the searched results. The way we search something is by passing a query so I duck-typed a function which accepts a string and then went on a spree of documentation look up to find out how do I search in bleve. I found a simple implementation which returns the id of the file which is the path and match score.

 func searchResults(indexFilename string, searchWord string) *bleve.SearchResult {
	index, _ := bleve.Open(indexFilename)
	defer index.Close()
	query := bleve.NewQueryStringQuery(searchWord)
	searchRequest := bleve.NewSearchRequest(query)
	searchResult, _ := index.Search(searchRequest)
	return searchResult
}

This function opens the index and search for the term and returns back the information.

Let’s Serve It

After all that is done I need to have a service which does this on demand so I wrote a simple API server which has two endpoints index and search.  The way mux works is you give the endpoint to the handler and the function to be mapped with it. I had to restructure the code in order to make this work. I faced a really crazy bug which when I narrowed it down, came to a point of a memory leak and yes, it was because I left the file read stream open, so remember when you Open always defer Close.

I used Postman to heavily test it and it was returning good responses. A dummy response looks like this:

 [{"index":"irclogs.bleve","id":"logs/some/hey.txt","score":0.6912244671221862,"sort":["_score"]}]

Missing Parts?

The missing part was I didn’t use any dependency manager which Kushal pointed out to me, so I landed up using dep to do this for me. The next one was one of my favourite  problems of the project and that was how to auto-index a file. Suppose my service is running and I added one more file to the directory, then this file’s content wouldn’t come up in the search because the indexer hasn’t run on it yet. This was a fascinating  problem and I tried to approach it from many different angles. First I thought I would re-run the service every time I add a file but that’s not a graceful solution. Then I thought I would write a cron job which would ping /index at regular intervals and yet again that struck me as inelegant. Finally I wondered if I could detect changes in a file. This led me to explore gin, modd and fresh.

Gin was not very compatible with mux so didn’t use it, modd was really nice but I needed to kill the server to restart it since two services cannot run on a single port and every time I kill that service I kill the modd daemon too so that possibility also got ruled out.

Finally the best solution was fresh although I had to write a custom config file to suit the requirement, this approach still has issues with nested repository indexing which I am thinking how to figure out.

What’s Next?

This project is yet to be containerised and there are missing test cases so I would be working on them, as and when I get time.

I have learnt a lot of new things about the filesystem and how it works, because of this project. This little project also helped me appreciate a lot of golang concepts and made me realise the power of static typing.

If you are interested you are welcome to contribute to file-indexer. Feel free to ping me.

Till then, Happy Hacking!