Planet dgplug

June 27, 2016

Dhriti Shikhar

Working on Jira and Bugzilla issue in Project Almighty

I joined Red Hat on 15/06/2016.

I was assigned two issues on project Almighty to work on:

  1. Create simple example of fetching Issues from Jira in Go
  2. Create simple example of fetching Issues from Bugzilla in Go

I started with the Jira Issue. This was my first issue in golang. While solving this issue, i learnt the usage of “go get” command in golang. It downloads + installs an upstream package. For example, this is an upstream package which I used in the Jira Issue –> github.com/andygrunwald/go-jira

So, I installed+downloaded the package using following command:

go get github.com/andygrunwald/go-jira

The package got installed:

Screenshot from 2016-06-27 12-39-46

I learnt how to take input from command-line flags using “flag” package. To do this:

  • import flag package using:

import "flag"

  • Declare the variable in which you want to store the input

var username string
flag.StringVar(&username, "uname", "", "Username")

The first argument &uname is a pointer which stores the input value.
The second argument is the flag name which you use on the command line e.g. “-username”.
The third argument is the default value of the flag input.
The fourth argument is the description of the flag.

  • Parse the flag:

flag.Parse()

Next, I learnt to work with “net/http” package and how to parse http response in Go. Although I didnt use the package for this issue but I did experiment with it.

I used “reflect” package to find out the data type of http response. The two commands i used were:

reflect.TypeOf(result)
reflect.TypeOf(result).KindOf()

I found that http response have slice data type and each slice element had struct data type.

To access the value from struct data type, I had to find out its property name. For that, I used “reflect” package:
reflect.Indirect(result).FieldByName(field)

The next issue was bugzilla issue, The golang packages for bugzilla are listed here.

I used this package to get the results –> github.com/kolo/xmlrpc

XmlRpc is used to make remote procedure calls over HTTP.

The endpoint for xml-rpc interface is xmlrpc.cgi script in the bugzilla installation. So, for Red Hat Bugzilla, the endpoint is https://bugzilla.redhat.com/xmlrpc.cgi

You can find my code here.


by Dhriti Shikhar at June 27, 2016 08:04 AM

June 23, 2016

Sanjiban Bairagya

Marble Maps in KDE Randa Meetings 2016

One more year of fun and intense productivity in Randa came to an end just a few days back, and I feel so good to have been a part of it. Much progress was made by the Marble team this year by Dennis, Torsten, Friedrich, David and me. I mostly worked on the Marble Maps Android app’s navigation feature, and would like to mention the changes here very briefly:

  • First of all, cmake was picking up the wrong target for aprs plugin, so I made cmake to skip the aprs plugin for Android, so that it does not lead to an unnecessary crash during building it.
  • There was a bug in the Route Simulation position provider plugin, it was not working in the Navigation mode. Made a fix to that.
  • Replaced the older QImage based current-position pointer, with a QML based one in the Android app. So now we are able to make our custom animations on it.
  • The current-position pointer will be staying at points on the route itself, when the user is very close to the route (not too deviated from it). With the new QML based pointer, we made sure that the radius of accuracy remains the same with respect to the Earth irrespective of which height we are viewing the map from. Plus, we have got rid of this accuracy-indicator when the position-pointer is already on the route, and are showing it only otherwise.
  • There was a minor bug with this in the desktop version as well. Imagine that while you’re simulating a navigation through a route between New York and Boston, you zoom out till the whole Earth is visible, then rotate the Earth so that the other half of the globe, for example India, is visible now, instead of America. In that case the position marker was showing up on the top-left corner of map (somewhere in the sky), when it was supposed to not be visible at all since it is theoretically present on the other “not-visible” side of the globe now. So this bug was fixed as well.
  • Used some new maneuver direction pixmaps for navigation in the Android app, getting rid of old ones, and also made them look sharper in the Navigation info-bar while they were shown during turn-by-turn navigation, by making some tweaks in the QMLs that were using them.
  • Finally, the distance text that shows on the top while in Navigation mode, its border height has been changed to match the height of that of the maneuver pixmaps shown on its left as well, so that the view looks much more uniform. Plus, a similar panel has been added at the bottom as well, that shows the speed and distance information during the navigation, each at two different sides of the panel.

That’s all about work. And adding to the yummy food that turns out to be truly delicious every single year, this time we had some more fun activities as well. We were fortunate enough to board a cable-car to ride to as high as possible in the mountains, so much that we ended up finding ourselves in snowfall ( it’s my first time being in a snowfall:) ), which was amazing! The special red tea we were greeted at a stop at a restaurant during the hike was pretty unique. We then visited a museum, and that was a lot of fun as well.

Such a lively and eventful week it was, thanks a lot to Mario and Simon for organizing the Randa Meetings one more year. Let’s keep the Marble rolling, and I hope to be a part of this again in the subsequent years to come!  :)

banner-fundraising2016.png


by sanjibanbairagya at June 23, 2016 09:42 PM

Shakthi Kannan

GNU Emacs - rcirc IRC client

[Published in Open Source For You (OSFY) magazine, October 2015 edition.]

Internet Relay Chat (IRC) provides instant communication over the Web. It is generally used for one-to-one communication or group discussion forums, also known as channels. In this next article in the GNU Emacs series, we shall learn how to use GNU Emacs as an Internet Relay Chat (IRC) client.

What is IRC?

Internet Relay Chat (IRC) is a messaging protocol that was created by Jarkko Oikarinen in 1988. An IRC server has a number of channels (or rooms, as how they are called in other chat software) where both technical and non-technical discussions take place. Every user requires a nickname to chat. You will need to register your nickname with a password to identify yourself every time you log in to the IRC server. IRC can also be used for file sharing among users.

You can connect to any one of a number of IRC servers that are available worldwide, free. The most popular is Freenode (irc.freenode.net). The other IRC servers, to name a few, are IRCNet, OFTC, and EFNet. The Debian and Ubuntu projects have their own chat servers – irc.debian.org and irc.ubuntu.com. All IRC channel names begin with ‘#’. Some channels begin with ‘##’. Examples of channels are ##linux-india, #dgplug, #emacs, #ruby, #guile, #lisp, and #scheme. You can also create your own channels, or host your own IRC server. Some examples of free and open source IRC server software are IRCd, UnrealIRCd, and Synchronet. Each IRC channel has a topic that can include useful links relevant to the channel. Sometimes announcements, or news updates are also mentioned in the topic.

Basic commands

A number of commands can be given to the IRC server. A few of them are discussed below:

  1. /list is used to provide a list of all the available channels in a server. For example: /list in irc.freenode.net returned the following:

    ...
    #linod           1       
    ##welding        3       Welcome to ##Welding, We're a little bare at the moment, but will help if we can. Tutorials: https://www.youtube.com/channel/UCJAFY2kKKb5sg79yld7T3hA
    #drupal-ph       1       "Drupalista! Welcome to Philippine Drupal Users Group. Have a good time chatting. If you have a question, please don't ask to ask but fire up your question in very specific and constructive way! Please join #drupal or #drupal-support if no one is around"
    #orx-project     4       Orx: Portable Game Engine
    #tinkerforge     5       
    #osi             10      The Open Source Initiative
    #xampp           1       
    #guitar          8       
    #bitcoin-ar      3       Comunidad Bitcoin Argentina
    #LargeHadrosaurCollider 19      Welcome to the LHC, est. 2001 | http://www.largehadrosaurcollider.net | August Birthdays: Digby 08/21, Josh 08/31 | At night it is pitch black, often for months. | http://tinyurl.com/psgdagl
    * End of /LIST
  2. /msg NickServ REGISTER password e-mail is used to register your nickname to the IRC server. /msg NickServ IDENTIFY password is used to identify yourself to the server.

  3. /me message displays the message for a user. For example:

    /me says "Hello, World!"
    
     * mbuf says "Hello, World!"
  4. /whois nickname provides useful information for a user. For example:

     /whois mbuf
    
    * [mbuf] (~shakthi@123.123.123.123): Shakthi Kannan
    * [mbuf] #guile #scheme ##linux-india #stumpwm #guix #dgplug #lisp #emacs 
    * [mbuf] kornbluth.freenode.net :Frankfurt, Germany
    * [mbuf] is connecting from *@123.123.123.123 123.123.123.123
    * [mbuf] idle 00:41:52, signon: Thu Sep  3 20:36:52
    * [mbuf] is logged in as mbuf
    * [mbuf] End of WHOIS list.
  5. /msg nickname is used to send a private message to a nickname and to start a private conversation.

  6. /help provides useful help on basic IRC commands. These commands are:

      ADDBUTTON ALLCHAN   ALLCHANL  ALLSERV   AWAY
      BACK      BAN       CHANOPT   CHARSET   CLEAR
      CLOSE     COUNTRY   CTCP      CYCLE     DCC
      DEBUG     DEHOP     DELBUTTON DEOP      DEVOICE
      DISCON    DNS       ECHO      EXEC      EXECCONT
      EXECKILL  EXECSTOP  EXECWRITE FLUSHQ    GATE
      GETFILE   GETINT    GETSTR    GHOST     GUI
      HELP      HOP       ID        IGNORE    INVITE
      JOIN      KICK      KICKBAN   KILLALL   LAGCHECK
      LASTLOG   LIST      LOAD      MDEHOP    MDEOP
      ME        MENU      MKICK     MODE      MOP
      MSG       NAMES     NCTCP     NEWSERVER NICK
      NOTICE    NOTIFY    OP        PART      PING
      QUERY     QUIT      QUOTE     RECONNECT RECV
      SAY       SEND      SERVCHAN  SERVER    SET
      SETCURSOR SETTAB    SETTEXT   SPLAY     TOPIC
      TRAY      UNBAN     UNIGNORE  UNLOAD    URL
      USELECT   USERLIST  VOICE     WALLCHAN  WALLCHOP
    
    User defined commands:
    
      ACTION    AME       ANICK     AMSG      BANLIST
      CHAT      DIALOG    DMSG      EXIT      GREP
      J         KILL      LEAVE     M         ONOTICE
      RAW       SERVHELP  SPING     SQUERY    SSLSERVER
      SV        UMODE     UPTIME    VER       VERSION
      WALLOPS   WII       
    
    Plugin defined commands:
    
      UNLOAD    UNLOAD    LOAD      PY        LOAD
      RELOADALL SOURCE    TCL       RELOADALL UNLOADALL
      PL_RELOAD RELOAD    UNLOAD    LOAD      TIMER
    
    
    Type /HELP <command> for more information, or /HELP -l
  7. /quit is used to disconnect and exit from IRC.

  8. /join #channel - is used to join a channel. For example: /join #guix.

  9. /nick newnickname changes your nick to newnickname. Suppose, you wish to move away from the computer, you can change your nick to nick|away, or nick|phone.

  10. /part is used to leave a channel.

Using rcirc

If you are using a recent GNU/Linux distribution, you should already have rcirc as part of GNU Emacs. You can simply start it by typing M-x rcirc from inside Emacs. The ‘M’ key represents the Meta key, which is usually mapped to the ‘Alt’ key. After rcirc connects to the IRC server, you can use /nick to change your nickname, register (only the first time) your nick, identify yourself, join channels, and start chatting! Since everything is a buffer in GNU Emacs, each channel is a separate buffer. For example, #emacs@irc.freenode.net is the #emacs IRC channel. All your basic buffer navigation commands will work just like they would on a file!

Some basic rcirc commands

The rcirc commands for the above mentioned IRC commands are given in the following table:

IRC rcirc
/list rcirc-cmd-list
/msg NickServ rcirc-authenticate
/me rcirc-cmd-me
/whois rcirc-cmd-whois
/msg nick rcirc-cmd-msg
/help /help
/quit rcirc-cmd-quit
/join rcirc-cmd-join
/nick rcirc-cmd-nick
/part rcirc-cmd-part

~/.emacs

GNU Emacs is an extensible editor. There are a number of locations where Emacs checks for custom configurations before launching the editor. These are: ~/.emacs, /.emacs.el, and/.emacs.d/init.el. The start-up files can be customised, and their locations can also be changed. There are a number of ways to organize, and manage your Emacs configuration. Until we get to learn Emacs lisp, and customize Emacs as a project, we shall use ~/.emacs for all our user-specific customizations.

rcirc customization

Create a ~/.emacs.d/etc folder in your $HOME directory and an Elisp file called init-rcirc.el in it. It should contain the following (change nick, user-name and full-name to suit your needs):

;; Default user.
(setq rcirc-default-nick "shaks")
(setq rcirc-default-user-name "shakthimaan")
(setq rcirc-default-full-name "Shakthi Kannan")

;; Channels to join at startup.
(setq rcirc-server-alist
      '(("irc.freenode.net" :channels ("##linux-india" "#dgplug" "#rcirc" "#emacs"))))

The above is an example of Emacs Lisp code. Comments begin with two semi-colons. The setq construct sets the second argument value to the first argument, which is a quoted symbol. For example, the symbol ’rcirc-default-nick is set to “shaks”. The rcirc-server-alist defines the initial list of channels to login at startup.

You can now start GNU Emacs from the GNOME terminal using the following command:

$ emacs -q -l ~/.emacs.d/etc/init-rcirc.el

You will then automatically connect to the four IRC channels.

How to use IRC

People join IRC channels to have their doubts regarding free and open source software clarified. Sometimes, off-topic discussions also happen. It is like live technical support, but has a social context to it. Whenever you are connected online, you must be logged in to IRC. You can have discussions in the channel, or in private, if the other party agrees. It is a good place to learn a lot about free and open source software, and you are bound to make a lot of friends. Since people from all over the world participate, which means they are online in different time zones, some channels log the discussions for future reference. As always, before asking a question, it is important for you to do your homework first. Take sufficient time and put in an effort to debug and identify the problem to the best of your ability.

Some users in the channel may ask you for more information before being able to provide you with any assistance. So, be prepared to provide all the information necessary about the bug or error when you seek help. Sometimes, people might be logged in the channel, but, they might be away from the computer. So, even if you don’t get a response, be patient; come back later and ask again.

You should not paste more than four continuous lines of text in the channel, as it will ‘flood’ the screen for everyone else. Instead, use an external paste service like gist.github.com, or fpaste.org. These services will provide a shortened URL that you can pass around in the channel. Whoever is interested in helping you will view the contents from the link. If you enter text in the channel, it means that it is addressed to everyone in the channel. If you wish to say something to a specific user, mention their nickname first, and then type in the text.

Most IRC client software provide you with panels that list the channels that you are logged in, and show the list of users. If someone mentions your nickname in a channel, then the corresponding channel will change colour or representation to indicate that there is a message for you. A few users are channel operators (or moderators) and they have special privileges. They are similar to ‘root’ users in a system, and they exist to keep the signal-to-noise ratio to a minimum, and keep a vigil on the channel.

An IRC bot is a client software that connects to the IRC server as a user, but can respond to commands. It can thus be programmed to provide many services in a channel. You can customize existing bots, or write your own. Examples of IRC bots are Cerberus, Gambot and irccd. Cinch is an example of an IRC bot building framework written in Ruby. Bots can be used during an IRC meeting session to keep track of user questions. They can evaluate programming language constructs and return meaningful errors to newbies in the channel. They can be used to send a notification to the channel if a project test build fails, or when a new bug has been filed. The possibilities are endless.

IRC meeting protocol

A free and open source software project will have a dedicated IRC channel where the project members will meet to have discussions. Meetings can be scheduled, and can happen in different time zones depending on where the users are located. There is a protocol and etiquette to be followed during such meetings. The speaker or moderator should not be interrupted during the session.

If you wish to ask a question, type ’?’ and wait. When the speaker has finished and feels that you can type in your text, you will be asked to do so. After you have finished typing your content, end with ‘EOF’. Similarly, if you need to speak during the session, type ’!’, and wait. You can give your consent or dissent to statements made in the channel using +1 or -1, respectively.

You are encouraged to read the rcirc manual and customize rcirc to your needs. If you have made it this far, do connect to irc.freenode.net, and feel free to say ‘Hi’ to me. I am ‘mbuf’ on irc.freenode.net. A screenshot of an rcirc session is shown below:

19:35 <mbuf>    http://ecb.sourceforge.net/screenshots/index.html
19:35 ***       Arpita QUIT Client Quit
19:36 <rtnpro>  !
19:36 <mbuf>    rtnpro, shoot!
19:37 <rtnpro>  How do we get the emacs code browser?
19:37 <rtnpro>  <EOF>
19:37 <mbuf>    rtnpro, 1. Need to install "ecb" from your distro package manager
                2. you could have searched this on the Internet :)
19:38 <rtnpro>  It is not in my distro
19:38 <sumitc>  !
19:38 <mbuf>    rtnpro, and which distro are you using?
19:38 <rtnpro>  Its Fedora 9
19:39 <rtnpro>  I have got emacs but not emacs code browser
19:39 <mbuf>    rtnpro, you can always install from source
19:39 <techno_freak>    rtnpro, http://ecb.sourceforge.net/downloads.html
19:39 ***       khushbu QUIT Ping timeout: 244 seconds
19:39 <rtnpro>  ok
19:39 <mbuf>    sumitc, shoot!
19:39 <sumitc>  what is a tag-file?
19:40 <rtnpro>  What factors should decide the choice of our editor?
19:40 <mbuf>    rtnpro, wait!
19:40 ***       pushkal JOIN
19:40 <mbuf>    sumitc, the TAGS file contains the details of the reference count,
                and locations of variables/functions et. al.
19:41 <sumitc>  So, a tag file is always associated with a specific file?
19:41 <mbuf>    sumitc, no, it can have information of files in a directory
19:41 <sumitc>  ok
19:41 <sumitc>  <eof>
19:42 <mbuf>    sumitc, think of it as a database that answers all your queries
               regarding code references

References

  1. Freenode. https://freenode.net/

  2. rcirc manual. https://www.gnu.org/software/emacs/manual/html_mono/rcirc.html

  3. IRC Help. http://www.irchelp.org/

  4. Fedora project - How to use IRC. https://fedoraproject.org/wiki/How_to_use_IRC

June 23, 2016 04:00 PM

June 22, 2016

Anwesha Das

Software Licenses in Fedora Ecosystem

October,2011, I got to know a man who has Fedoralism as his religion. Before that I had no idea about that another kind of fedora, other than a curled brim hat exists. I considered the option 'giyf' and it showed me about "Fedora Project"(oh wow it is also connected with a hat but this time with a particular colour, red :)). I got to understand that Fedora is a Linux Operating System. The first among four foundations of Fedora is freedom. The freedom is upheld by free software and content. If a software is free or not depends upon the software licenses of the particular package. Therefore In this blog post we will check what and how the software licenses work in fedora land.

The Fedora Operating System is made of different software packages in RPM formats. Every package must have a 'License' entry. The developer chooses the license. And it is the maintainer's job to correctly mention it in the spec file. The license as mentioned in the spec file applies for that particular project.

Lets take a tour of software licenses in the Fedora land.

To give a shape to this project I have downloaded all the spec files using 'Python Request Module'(a thanks to Kenneth Reitz for writing it). I have downloaded only the approved packages for Fedora. There are total 19,221 approved packages in Fedora repository as found on 20.06.2016, among those 1124 were dead packages. Another note of gratitude must be given to the Fedora Infrastructure developers who wrote, and maintained such a nice documentation which a newbie like me can easily understand.

The primary aim of Fedora is to crate an operating system solely absolutely from Free and Open Source Software and to work with the Linux Community in general. There is list of licenses provided as "Fedora Licensing List". The same list has been approved, assented to by the Free Software Foundation, OSI and consultation with Red Hat Legal. To know in depth about Fedora Licensing Guidelines please visit this great documentation. This documentation is being maintained by Tom 'spot' Callaway.

This "Fedora Licensing List" actually divides the licenses into two categories :Good License and Bad License. Good and/or Approved Licenses are those licenses which are fine with Fedora. The total number of such licenses, used in different Fedora Packages, are 267.

It is important to mention here that I have not considered the licenses that has been used in less than 10 Fedora packages.

In the above bar-chart I have counted GPL and its different versions as one family, and I did the same with LGPL too. From this diagram it is very much clear that the MIT License is the most used license, with a total number of use case of 2706.Therefore comes GPL(i.e GNU General Public License) and its different versions, BSD, LGPL(i.e GNU Lesser General Public License) and its different versions, ASL (i.e Apache Software License) family, MPL (i.e Mozilla Public License).
Apart from these licenses there are projects who has submitted themselves in to Public Domain and that number is 137.

There are some licenses which are being used very less number of times.
Example of the licenses that has been used only once:

  1. Romio for openmpi package
  2. Saxpath for saxpath package

Example of the licenses that has been used only twice:

  1. Julius for julius package.

GPL Family

This is a pie-chart of the usage of GNU General Public License and its different versions in the several Fedora packages. As it is quite evident that
GPLv2+ is the most used license of this family.

LGPL Family

This is a pie-chart of the usage of GNU Lesser General Public License and its different versions in the several Fedora packages. So, here the chart shows that LGPLv2+ is the most used of this family.

Licensing Scenarios

There are different licensing scenarios which needs to be mentioned here that:

Multiple Licensing Scenarios

There are cases where the packages comprise of files which has different, distinct, multiple and independent licenses, these are the Multiple Licensing Scenarios. Example:

GPL+ and BSD for guitarix package.

Dual Licensing Scenarios

In these cases the packages are under dual, sometime triple licenses. Therefore the package may/can be licensed under either of the licenses. These cases are 2954 in total number. Example:

GPLv2 or BSD for opensm package

Combined Dual and Multiple Licensing Scenarios:

There are number of packages where some items/files have multiple Licenses i.e different, distinct, multiple and independent licenses and some items which have dual licenses. The actual number of these kinds of projects are 265.

(ASL 2.0 or EPL) and MIT for jetty package. The spec file of this example contains license as ASL 2.0 or EPL, Source6 is under *MIT License.

Single Licensing Scenario:

There are cases where the packages are being licensed under single license. Such cases are 17,425.

MIT License for python-retask

Apart from all above mentioned cases there is another scenario, which has not been shown in the pie-chart and that is Mixed Source Licensing Scenario.

In the future blog posts I will dig deep into each of these licenses and scenarios mentioned here.

Disclaimer

  • The numbers mentioned in the post may vary a bit from the actual, current numbers.
  • Whenever the word package has been mentioned, I meant a SRPM.

by Anwesha Das at June 22, 2016 01:45 PM

June 16, 2016

Trishna Guha

Another Way To Push Package Updates To Stable In Fedora-Bodhi

Screenshot from 2016-06-16 23-00-10

Bodhi is a web application that facilates the process of publishing package updates of Fedora. Once a package is submitted to Bodhi it goes through various stages: Pending, Testing, Stable, Obsolete. The details can be found here Package States.

There exist two types of policies in Bodhi, using any of them maintainers can publish their package updates (Pushing updates to Stable from Testing). Updates Policy documentation: https://fedoraproject.org/wiki/Updates_Policy

Updates Policy in Bodhi:

  • Manually push to stable based on time :
    • Auto-karma is disabled.
    • Update spends 14 days in testing.
    • Maintainer pushes the update to stable manually.
  • Automatic push  to stable based on karma :
    • Auto-karma is enabled.
    • Stable Karma threshold is reached.
    • The update is pushed to stable automatically.

The policies above are used to update packages, pushing from testing state to stable state. We could also have one more way to push package updates that Bodhi doesn’t have yet. It could be structured like this:

Manually push to stable based on karma:

  • Auto-karma is disabled.
  • Stable Karma-threshold is reached.
  • Maintainer pushes the update to stable manually.

The maintainers have always wanted this policy to be in Bodhi. There are already two tickets for the same, https://github.com/fedora-infra/bodhi/issues/796 and https://github.com/fedora-infra/bodhi/issues/772

I decided to work on the ticket. We had to remove some major lines of code from the model that implemented the already existing policies of Bodhi. I kept my fingers crossed until the test suite runs with success😛 And yes it showed  all the tests ran ok😉. Lmacken helped me with continually reviewing my patch and clearing my doubts. The Commit can be found here https://github.com/fedora-infra/bodhi/commit/45f288ae3992d29f3388c627554cc35286a958fb and The PR here https://github.com/fedora-infra/bodhi/pull/821.

Lmacken has always said, “Deleting huge chunks of code is way better than writing more code”. He has guided me to delete that chunks of code:).

I learned it is always good to remove lines of code from your codebase if that doesn’t break any feature. The tweet that always have inspired me:)

The code is already merged. Once Lmacken spins up a new release it will get in to staging for further testing. I hope that Fedora package maintainers will be happy soon:-).

 


by Trishna Guha at June 16, 2016 05:32 PM

June 02, 2016

Farhaan Bukhsh

Pagure CI

As my GSoC project one of the first goals is getting CI to Pagure. In my previous post I have been blogging about getting Fedmsg to work, Configuring Jenkins and my favorite Poor Man’s CI. Well, Poor Man’s CI evolved into Slightly Richer Man’s CI and now  Pagure CI.

This is how it started I was trying to test how Poor Man’s CI works and how can I actually integrate it, while doing that I hit a deadend when Poor Man’s CI would not respond to local Pagure instance because there is no GIT server running. So one cannot clone git repo with a normal URL, I landed up on a really amazing utility called Git Daemon, it gave me some hope but it didn’t last long, that also has disadvantage that the URL should be a git:// URL.

I somehow thought I needed to set my Pagure as an application and then I can make common clone commands. I tried using that and I am still in the process of doing it , I am still setting gitolite on that front.

While doing that and because of my lazy nature I found a neat hack and it was , my Poor Man’s CI was running on localhost and so was my Jenkins but the I made a repo on pagure.io.  The only thing to understand here is the Fedmsg should listen to all the msgs and Fedmsg plugin should be enabled in the repository on Pagure.

So, recap you have :

1. fedmsg-relay running and listening to all the production messages.

2.  fedmsg-tail  –really-pretty running to check the message is being read.

3. fedmsg-hub running in Poor man’s CI which is consuming the message and configure        to consume new pull request.

Voilla! it worked and there you go I got a build there and the flag is attached to it.

Selection_016

 

After talking to Pierre we arrived a on conclusion that we need a way to either merge the two application or make Pagure talk to Poor Man’s CI. I had a lot of doubts to achieve it, later Pierre gave me “Food for thought ”  which was

“If we are able to gather all information from Pagure and Poor Man’s CI could get that database access would it not do the deal?”

The way PMCI handles it is the consumer written for fedmsg-hubs parses the message received and according to the message received it triggers Jenkins build and does all the work and it just queries a single table.

Pierre wanted no individual UI for PMCI and so we made it a hook , now a person can enable the hook and add the useful information required. I brought the whole consumer as it is from PCMI. Consumers need to get registered in setup.py and dependencies needs to be build up using python setup.py develop which does a lot of magic behind the scenes.

One of the dependencies is python-jenkins library, which provides various functionality. The whole PMCI is reduced to three files in Pagure. The most important being consumer.py , I gathered a lot of functionalities into a lib/pagure_ci.py and made a model in hooks/jenkins_hook.py.  Since I got lot of functionalities in place the only thing then I struggled with was because of my poor understanding of Sqlalchemy. I read up a lot and was able to debug the code and make it work.

 

The most peculiar thing I came across was even the server needs to be run in the virtualenv, since Jenkins library is installed in that. Once that is done its like a train of dominoes , one thing will lead to another . I am trying it to show you diagrammatically.

hook


by fardroid23 at June 02, 2016 06:02 PM

June 01, 2016

Shakthi Kannan

GNU Emacs - Buffer navigation and Help

[Published in Open Source For You (OSFY) magazine, September 2015 edition.]

Continuing with our series on Emacs, in this article, let’s explore basic buffer navigation, formatting, and help commands.

Buffers

You can view a list of buffers in GNU Emacs using the C-x C-b command, and can switch to a buffer using the C-x b shortcut. The default list of buffers as seen in GNU Emacs 24.3.1 on Ubuntu 14.10 is shown below:

CRM Buffer                  Size Mode            File
.   *scratch*               191 Lisp Interaction 
*   *Messages*              485 Fundamental
%   *Completions*           264 Completion List
%   *Help*                 1855 Help

In order to kill (close) a buffer, you can use C-x k key combination. If you have multiple buffers that you would like to close, you can use M-x kill-some-buffers command, which will prompt you for a confirmation for every buffer. You can also rename a buffer using the M-x rename-buffer command. If you would like to save the buffer contents to a file, you can use C-x s, and it will prompt you with many options before proceeding to save the file. You can also save a modified buffer using C-x C-s.

Navigation

It is important to learn how to use the Emacs navigation commands instead of the arrow keys. When you learn to use macros, you will find it useful to automate steps. A summary of the basic navigation shortcuts are given in the following table:

Command Meaning
C-f Move forward one character
C-b Move backward one character
M-f Move forward one word
M-b Move back one word
C-p Move to previous line
C-n Move to next line
C-a Move to beginning of line
C-e Move to end of line

You can scroll down a buffer using C-v, and scroll up using M-v shortcut. If you would like to move to the beginning of a file, use M-< command. You can move to the end of a file using M-> shortcut.

In order to move the cursor to a specific line, use M-x goto-line or M-g g. If you would like to position the contents where the cursor is located to the center of the screen, you can use C-l. If you want to repeat a command ‘n’ (‘n’ is an integer) number of times, you can use M-n following by the command. You can also use C-u n, but, if you don’t specify ‘n’ here, the next command, by default, will be executed four times.

Clipboard

You can delete a character where the cursor is located using C-d. To remove a word, you can use M-d. You can remove text from the cursor position till the end of line using C-k. The cut text goes into a kill ring buffer. You can paste whatever you have removed using C-y. You can cycle through the text that has been removed by repeatedly using M-y after C-y. Consider the text:

When you have a question about Emacs, the Emacs manual is often the
best starting point.

If you place the cursor before “about” in the above sentence, and press ‘M-d’, it will remove " about“. You can now see the value of the kill-ring using C-h v (describe variable), and it will prompt you with the message”Describe variable:“, where you can enter kill-ring. You will see an entry for” about“. Now, place the cursor after”question" and press ‘M-d’ again, and “Emacs” will be removed. If you again check the kill-ring value, you will see an entry for " Emacs".

You can now move the cursor to any location in the buffer, and press ‘C-y’, and it will paste “Emacs”. If you then press M-y, it will paste “about” which was the text that was cut before “Emacs”. This way, you can cycle through the kill ring buffer.

To select a region of text, you first need to mark the starting position with the cursor using C-space. You can then use any of the navigation commands to highlight the text that you want. You can either cut the marked region using C-w, or copy it using M-w. This will get copied to the kill-ring. You can then paste the text to any buffer. You can also select the entire buffer using C-x h.

Formatting text

It is good to have text within 80 columns and 25 rows (80x25), not just for readability, but, to avoid making any assumptions about the users’ screen size. If you have a really long sentence or paragraph, you can reformat it using the M-q shortcut from anywhere in the sentence or paragraph.

You can transpose two characters using C-t shortcut. For example, if you have misspelt “emacs” as “emcas”, you can place the cursor on ‘a’ and type C-t to swap the letters ‘a’ and ‘c’ to get “emacs”. You can transpose two words using the M-t command. For example, if you have the phrase “morning good” in a buffer, and you type M-t anywhere on the word “morning”, it will swap the words to give you “good morning”. You can also swap lines and sentences using C-x C-t and M-x transpose-sentences commands respectively.

If you would like to capitalize the first character of a word in upper case, you need to place the cursor on it, and key in M-c to make the first letter capital. You can turn all those letters from the cursor location to the rest of the word into capitals using M-u command. For example, if you have the word “sentence”, and you place the cursor on the letter ’t’, and you key in M-u the result will be “senTENCE”. To make all the characters lower case, you can use M-l. You can turn the entire previous word into capital letters using the M - M-c shortcut. Similarily, to make the previous words either upper case and lower case, you can use M - M-u and M - M-l commands, respectively.

Info

You can read manuals written in Info format inside GNU Emacs. When you type M-x info, it opens up a menu with the built-in documentation available from ‘Info’ files. The Info mode is mostly used to read hypertext manuals. You can use Tab and Shift-tab to navigate through the available links. Since the contents are in a buffer, all the basic navigation commands are applicable. To exit from Info, you can use q (quit).

You can press the return key on a link to open its contents. For example, when you click on “Emacs FAQ” it opens “The GNU Emacs FAQ” contents. The following are the sections listed in its buffer:

* Menu
* FAQ notation
* General questions
* Getting help
* Status of Emacs
* Common requests
* Bugs and problems
* Compiling and installing Emacs
* Finding Emacs and related packages
* Key bindings
* Alternate character sets
* Mail and news
* Concept index

Suppose, you hit the return key on “FAQ notation”, it will open its contents with the title “1 FAQ notation”. If you want to cycle through the headings at this level, you can press n (next topic) and p (previous topic). If you want to move one level up, to the “The GNU Emacs FAQ”, you can use the u shortcut.

If you simply want to move to the topmost node (“The GNU Emacs FAQ”), you can use the t shortcut. For example, if you were in the section “3.1 I’m just starting Emacs; how do I do basic editing?” and you pressed ’t’, it will take you to the topmost section - “The GNU Emacs FAQ”.

You can move to the previous node that you visited using the l shortcut. You can also explicitly specify the node you want to move to using the g shortcut. For example, if you are in the “1 FAQ notation” page, and you pressed g, it will then prompt you with the message “Go to node:”. You can then type “Extended” and hit TAB to auto-complete to “Extended commands”, which is one of the links in the page.

You are encouraged to try the built-in info tutorial available by pressing h in the “info” buffer.

Help

You can read manual pages inside GNU Emacs using M-x man command followed by the return key. Suppose, you try this command from the scratch buffer, it will then prompt you with the message “Manual entry:”. You can then input ls (list directory contents), and it will open a new buffer and window titled “Man ls” with the contents of the manual entry for the ls command.

From the scratch buffer, you can scroll forward the other window (‘Man ls’) using C-M-v (Ctrl-Alt-v), and scroll back using C-M-shift-v (Ctrl-Alt-Shift-v).

You can list all the available, active key-bindings using the C-h b command.

The C-h f shortcut is used to provide documentation for a function available in your installed version of GNU Emacs. For example, if you use C-h f, it will prompt you with the message “Describe function:”. If you input scroll-other-window, it will open a new buffer and window titled “Help” with the description and usage of the function. The output as seen in GNU Emacs 24.3.1 is shown below:

scroll-other-window is an interactive built-in function in `C source
code'.

It is bound to <M-next>, C-M-v, ESC <next>.

(scroll-other-window &optional ARG)

Scroll next window upward ARG lines; or near full screen if no ARG.
A near full screen is `next-screen-context-lines' less than a full screen.
The next window is the one below the current one; or the one at the top
if the current one is at the bottom.  Negative ARG means scroll downward.
If ARG is the atom `-', scroll downward by nearly full screen.
When calling from a program, supply as argument a number, nil, or `-'.

If `other-window-scroll-buffer' is non-nil, scroll the window
showing that buffer, popping the buffer up if necessary.
If in the minibuffer, `minibuffer-scroll-window' if non-nil
specifies the window to scroll.  This takes precedence over
`other-window-scroll-buffer'.

[back]

On the other hand, if you know the command, and would like to get help related to it, you can use the C-h k command. This will prompt you with the message “Describe key (or click or menu item):”. You must key in the actual shortcut, for example, C-h b, and it will update the buffer “Help”, and open it in a new window, showing the description for the shortcut as shown below:

C-h b runs the command describe-bindings, which is an interactive
compiled Lisp function.

It is bound to C-h b, <f1> b, <help> b, <menu-bar> <help-menu>
<describe> <list-keybindings>.

(describe-bindings &optional PREFIX BUFFER)

Show a list of all defined keys, and their definitions.
We put that list in a buffer, and display the buffer.

The optional argument PREFIX, if non-nil, should be a key sequence;
then we display only bindings that start with that prefix.
The optional argument BUFFER specifies which buffer's bindings
to display (default, the current buffer).  BUFFER can be a buffer
or a buffer name.

[back]

You are encouraged to try out all the commands so that you understand what actions they perform. When you use GNU Emacs as your main text editor, you will be able to regularly practice these commands, and master them.

June 01, 2016 05:45 PM

Suraj Deshmukh

Running Kubernetes in container on Fedora/CentOS

Running Kubernetes cluster on docker container on fedora or CentOS can be problematic specially with SELinux. So you need to do small settings before you follow setup mentioned on Kubernetes docs.

m7S2-kubernetes

This is because the Kubernetes container you will run will be running in privileged mode. And when doing that this container will be doing reads and writes on host machine. And SELinux by default won’t allow that, to enable that you have to change SELinux Security context.

infinity-logo_400x400

And this is to be done specifically on folders that will be mounted as volume in Kubernetes container. So if you follow the instructions on Kubernetes docs, it mounts /var/lib/kubelet/ and /var/lib/docker/ from host system into Kubernetes container. Also the folder /var/lib/kubelet/ won’t be there by already so create it and change its context.

selinux-penguin-new_sized

So run below commands and then follow docs.

sudo mkdir -p /var/lib/kubelet/
sudo chcon -R -t svirt_sandbox_file_t /var/lib/kubelet/
sudo chcon -R -t svirt_sandbox_file_t /var/lib/docker/

Here is small script that you may follow, just for reference.


by surajssd009005 at June 01, 2016 12:57 PM

May 28, 2016

Kushal Das

Day -1 of PyCon US 2016

I reached Portland two days back, was happy to see that Remy helped us to find a hotel just opposite to the conference center. As I am typing this, I can see the empty roads, and the main doors of the venue. Yesterday also I woke up by 5AM, the virtue of being in a place 12:30 hours apart from your standard timezone :) After writing the article about Microbit support in Fedora (it is still under review) I moved to the conference venue. It was very easy to find the staff room. As usual Doug,Lvh,Eric were already there, later I met Ewa, and then met Betsy for the first time. While discussing security practices when I asked, Lvh pointed out that getting golang vendored sources in the source code repository and then not updating them later, is more of a software engineering problem than a security problem as he sees.

Later in the day few of us walked towards downtown to get some lunch, and then we decided to visit Powell’s again. Mike Pirnat gave some valuable insights about children books, and card games :) After coming back to the venue, we saw more people are coming in. Rami was the first person to meet, PyCon is really a great place to meet old friends. Later I met Glyph, Hynek, Guido, Harry, Mike Muller, Graham Dumpleton, and Yarko (I don’t think I will be able to write down people’s names like this from tomorrow, the list will be too long :D ). I also found PyDanny, and Audrey coming in for the registration. It was good see her walking around after the big surgery.

Guido pointed out the xkcd comics about the Python patch. The discussion then moved to regex, legal side of the software field. I also had a chat about how CentOS is helping CPython with CI effort. It turns out that Guido knew about CentOS, but never looked deep into what is that :) He was seems be happy about help CentOS is doing by providing the beefy boxes to do build+test super fast. I am actually going to talk about that today in the Language Summit.

Later at night a large group of us moved to a nearby Ethiopian place for dinner. The food was great, we had more people coming in for the next few hours. Ntoll also came in and joined our table. Came back to the hotel room around 22:30, and tried to sleep early so that I can wake up in time. Today I will be in the registration desk from 7AM just like other years, and then I will move into the Language Summit. See you all in PyCon :)

by Kushal Das at May 28, 2016 12:46 PM

May 23, 2016

Kushal Das

dgplug summer training student Tosin Damilare James Animashaun

Your name (blog/twitter) and what do you do

My name is Tosin Damilare James Animashaun (IRC: acetakwas; Twitter: @acetakwas; Github: github.com/takwas, Blog: takwas.github.io).

I am currently a part-time Software Engineering student at NIIT, Lagos. I am also a co-founder at a startup called Krohx. We are in the final stages of deploying a job/hiring web platform that targets seekers of short-term jobs while also easing the process of getting service delivery for the hirers.

With my involvements with the burgeoning Python community in Nigeria, I recently got elected as secretary for the organization.

How did you learn about the training? (++ My Experience) I fail to recall

exactly how I learnt about the community now, but I remember joining the mailing list and going on from there. This was in fact my first time of joining a mailing list. Upon learning about the #dgplug IRC channel, I promptly enlisted the channel to my IRC client’s favourites.

I was so enthusiastic about this, as I had no prior experience of an online tutoring session. The feeling was truly exhilarating.

Having come from a Java background, I was really interested in learning the Python programming language as soon as I caught on waves with. At the time, I had volunteered to write some scripts to help automate some of the manual processing of NIIT’s annual scholarship exam records/documents at my school for that year. I accomplished that using Java, but in hind-sight, I feel I could have saved a lot of time if I had utilised Python in the situation.

This was a blog post I wrote as part of tasks given to us. I have failed to maintain that blog though. I now write here

How this training changed (if at all) your life? Python is surely the more

beginner-friendly language and it’s making its rounds these days. Every programmer should have it in their arsenal. One of the things I love most about the language is it versatility in application.

No doubt, I have continued to improve with Python everyday, but more so in my programming journey. I have become more and more familiar with various mainstream concepts and tools. The idea of contributing to open-source can be overwhelming at first, but following the path laid out in this course, one would be surprised to learn how easy it could be.

I have volunteered to mentor attendees at a Django Girls workshop, which held in Lagos, Nigeria. I picked-up Django in a week prior to the event, because frankly, there wasn’t much of a learning curve to it since I already used Flask, a different Python-based web development framework.

Flask was a framework I first got introduced to at the summer training, but I did not revisit it until months later. As a backend web-developer, I now do most of my work in Flask.

In my experience, there is no better or faster way to grow, than to collaborate; and that is what open source preaches. Note that growth here is not only about the project, but the growth applies to each contributing individual. Heck, even if you do not contribute, you will grow by reading good open-source code – this I can attest to. And joining the dgplug community has surely made open-source a more approachable endeavour. # #

Have you contributed to any upstream project(s)? If yes, then details.

While (and a bit disappointingly) I have yet to contribute to an upstream project, I do have quite a number of them lined up. Most probably, the first would be to the [official Slack Client]() written in Python. Having recently developed a bot, which was used in conducting the election of committee members of the Python Users Nigeria Group, I do have a number of changes I plan to push to the aforementioned API library.

With this forked project, I have also tried my hands at revamping the architecture of the official dgplug bot ekanora, which is used in running the summer training sessions.

Some other projects I’d like to contribute to include:

Any tips for the next batch of participants. (++Final words)

I am not sure I am in the best position to answer this. However, I would advise that you endeavour to remain focused through this; simply keep at it. Even if your experience at the summer camp doesn’t go as you expect, you should definitely not count it loss. Instead, make it a reason to put in more effort and grow as you go.

Depending on your personal activity schedule, you might miss one/two classes. But that shouldn’t deter you from continuing the program. The logs are available for covering such lapses. Also, you could consult the summer training docs to get a hang of the agenda, so you can anticipate classes beforehand and be better prepared.

While I participated in the training, there were times when there was no power supply, and I had to use my phone. Thankfully, there are IRC clients for mobile phones (I recommend AndChat).

The class sessions have a few rules, and the community would love it if you played by them. Not to worry, they are no stringent rules; just there to keep everyone behaving. For instance, the use of SMS-speak is discouraged.

You should keep notes beside you when attending the sessions. Writing is a good way to memorize things in my opinion. Also, although most clients would have this by default, you should ensure your IRC client is configured to log conversations. Going back over what you’ve learnt would do you great good as you mightn’t be able to keep up with the speed of the class at times.

If you have some time before program begins, two things I’d advise you become familiar with are: - Your OS (Linux, preferably Fedora or Ubuntu) - An IRC client (I suggest HexChat)

I learned quite a lot from the summer training. I was really obsessed with attending whenever I could. Although coming from a different country, the timing wasn’t favourable, often meeting me in the middle of the day’s activities – school especially. I made efforts to invite a few friends to join on – one or two did, but the determination to keep on was obviously lacking, which reminds me of a statement I heard a while back that reads something like this:

"If 40 people begin a course like this together, only about 5 get to
finish."

That in my experience is very often the case. Be aware that a lot of people are on this bandwagon. The question to ask yourself, is “Do I want to be among the surviving commutants at the end of this journey?” Unless you plan to experiment with this experience, if your answer is yes, then I congratulate you as you begin the journey that could potentially kick-start your software engineering career/journey.

Know this, free and open source software is for the good of all. The open source community is wide and continues to grow. It welcomes all and sundry, but it poses a few hurdles to sieve the wheat from the chaff. In the end, the gains are worth it.

by Kushal Das at May 23, 2016 05:45 PM

May 22, 2016

Farhaan Bukhsh

Step 2: Configuring Jenkins

Jenkins is one of the major part in setting up Poor Man’s CI. Lets look into how Jenkins can be configured and how can we make it automate task.

Jenkins can be downloaded for the OS you are using from their website. After downloading the mistake that I did was using Jenkins with Global Credentials, as pointed by lsedlar on the channel, because of this I was not able to get the “Trigger by URL” option in the project.

Initial configuration is pointed by lsedlar in his blog. I will be covering extra configuration to have it working for local development. First and foremost being the authentication , this can be done by  Manage Jenkins –> Configure Global Security. Selection_013

 

Give Read, View and Discover to anonymous and add another user and give all the permission to that user. You need to restart Jenkins service.

sudo systemctl restart jenkins.service

On web ui jenkins will ask you to sign in ,  create a user with the username you gave all the permission and log in with the user. Now add New Item  and create a Freestyle Project. Now configure the project , click on  “This build is parameterized” and configure it according to Poor Man CI’s. Once that is done, select the option as shown below:

Selection_014

Once that is done you can use this token to trigger build using a POST request. The trick is you need to pass parameters too with the URL. Next thing is you need to tell Jenkins what to do and where to do. Since we are dealing with Jenkins and git we need a local repo or some URL to the git repo. For every operation carrying out in the repository the directory should have the group and user set to jenkins else you cat just put the repo in /var/lib/jenkins.

Download and install Git Plugin for Jenkins. Once that is done you need to point the git plugin to the repository you are going to test.

Selection_015

Once jenkins know where to perform action you need to tell what to perform this is done in the build section of the configuration and select Execute Shell.

if [ -n "$REPO" -a -n "$BRANCH" ]; then
git remote rm proposed || true
git remote add proposed "$REPO"
git fetch proposed
git checkout origin/master
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
git merge --no-ff "proposed/$BRANCH" -m "Merge PR"
fi

We are almost done, the last thing is we need an auth token for the user. Go to Manage Jenkins –> Manager User. Now get the api token for the user. Make sure that branch you are passing as parameter exists in the repository. Lets trigger the build using cuRL.

USER:fhackdroid

API Token: 728507950f65eec1d77bdc9c2b09e14b

Token: BEEFCAFE

BRANCH:checking

curl -X POST http://fhackdroid:728507950f65eec1d77bdc9c2b09e14b@localhost:8080/job/pagureExp/buildWithParameters\?token\=BEEFCAFE\&REPO\=file:///$\{JENKINS_HOME\}/new_one_three\&BRANCH\=checking\&cause\=200


by fardroid23 at May 22, 2016 07:20 AM

May 02, 2016

Dhriti Shikhar

Machine Learning

Learning

To gain knowledge or understanding or skill through:

  • study
  • instruction
  • experience

 

Machine Learning

The field of study that gives computers the ability to learn without the need of explicitly programming.

The goal is to device programs that learn and improve performance with experience without human intervention.

 

Training Data

  1. Set of examples (input -> output) for learning
  2. Used to build model

 

Test data

  1. Used to test:
    • how good your model can predict
    • estimate model properties
  2. It is always outside training data set but follows some probability distribution as training data

 

Feature

  1. also called predictor
  2. It is a meaningful attribute
  3. Internal representation of data
  4. quantity describing an instance
  5. property of an instance

 

Tuple

  1. A Record in data base
Screenshot from 2016-05-02 23-34-01Features are columns and Tuples are rows

 

If we increase the number of records, attributes in a data set, then Machine Learning problem also becomes a Big Data problem.

 

Supervised learning

  1. It uses training data set consisting of input -> correct output to train the model
  2. Example:
    • Page Ranking Algorithm
    • Next word recommendation in Instant Messaging Application/ Whatsapp/ SMS

 

Unsupervised learning

  1. No training data set exists
  2. Most difficult algorithms are unsupervised learning because there is no “fixed” objective.
  3. used in Explaratory Data Analysis (EDA)
  4. Example:
    • used in recommendation systems to determine users who are similar to me from existing database

 

tiff infomationMachine Learning types

 

 Classification  Clustering
We have a set of pre-defined classes and we want to know which class a new object belongs to. Group a set of objects and find whether there is some relationship between objects.
It is predictive modelling. We give          pre-defined groups and predict group of new data. It is descriptive modelling. We try to find groups which occur naturally in data .

 

Classification

There are 6 items categorised in 2 classes:

tiff infomationExample of Classification

 

Each category has a label e.g. Eatables and Non-Eatables. If we have to predict the class of a new item “strawberry”, then it will be assigned a label “Eatable”

 

Clustering

There are 6 items categorised in 2 groups:

tiff infomationExample of Clustering

 

Each category is unnamed i.e. there is no label attached to the group. If we have to predict the group of a new item “strawberry” then it will be in the first group.

 

Accuracy

  1. How often is the prediction correct?
  2. Accuracy is not reliable metric for real performance of model because it will yield misleading results if training data set is unbalanced (i.e.  number of samples in different classes vary greatly).
  3. Example:
    1. Let number of cats be 95 and number of dogs be 5
    2. Classifier can easily bias into classifying all samples as cats
    3. Overall acuuracy = 95%
    4. BUT 100% recognition rate for cats and 0% recognition rate for dogs
  4. One of the ways to improve accuracy is to provide more balanced data.

 

This is one of the interesting things explained by Satish Patil in Pune Python Meetup that:

There is no right or wrong model. There is no best or worst model. There is ONLY useful and non-useful model. 

Nobody knows how much percentage of accuracy is good. How much accuracy is needed depends on Business Context.

Consider a company which wants to launch a new product and they want the probability of success of the product using Machine Learning. So, it is the company which DECIDES that if they get probability below 60%, then they will not launch the product. So, this is not something that the developer decides. This totally depends on the business context.

 

Market Basket Analysis

  1. Also called affinity analysis
  2. Association Rule:
    • discovering interesting relation/connection/association between specific objects
  3. Sometimes, certain products are typically purchased together like:
    • beer and chips
    • beer and diapers
    • bread and eggs
    • shampoo and conditioner
  4. So, market basket analysis tells a retailer that promotion involving just one of the items from the set would likely drive sales of the other
  5. This technique is used by retailers to:
    • improve product placement
    • marketing
    • new product development
    • making discount plans

 

Titanic Data Set

The titanic data set was used in the machine learning talk in Pune Python Meetup. It can be downloaded here.

There are some features in the data set which can be ignored as they are not important like:

  • Passenger ID
  • Name
  • Ticket Number
  • Cabin

and there are some important features which help in classifying like:

  • Survived
  • Gender

 

Impurity Measure

  1. Measures how well are the classes separated
  2. Should be 0 when all data belong to one class

 

Entropy

  1. Entropy can be a measure of quality of model
  2. It is a measure of how distributed are the probabilities.
  3. The more equal is the share for the probability values in all the classes, the higher is the entropy.  The more skewed is the share among the classes, lesser is the entropy.
  4. The goal in machine learning is to get a very low entropy in order to make the most accurate decisions and classifications

 

Decision Tree

  1. A way of graphically representing an sequential decision process
  2. Non-leaf nodes are labelled with attribute/ question
  3. Leaf nodes are labelled with class
tiff infomationdecision tree based on titanic data set

 

Pruning

  1. Data can contain noise:
    • instance can contain error
    • wrong classification
    • wrong attribute value
  2. If a particular feature is not used by a tuple or if the feature is not influencing, then it is removed.

 

Data Preprocessing

  1. Converting data into interval form
  2. Machine learning algorithms learn from data so its important to feed it the right data
  3. Data preprocessing basically involves:
    • correcting mistakes
    • handle missing values
    • handle outliers
    • normalize values
    • nominal values

 

Missing Value

  1. The value of an attribute which is not known or does not exist
  2. Example:
    • value was not measured
    • instrument malfunction
    • attribute does not apply
  3. If a column contains “Not Available”, then it is NOT considered as a missing value.

 

Outliers

  1. samples which are far away from other samples
  2. They can be mistake/ noise or represent a special behaviour
  3. Outliers are generally removed

 

Questions that were asked in meetup

  1. Can data be extended to multiple dimension?
  2. Can distance be other than Euclidian?
    • Yes, Manhattan distance
  3. Are there online courses that teach ML intro?
    • Yes
  4. What is “k” in k-means?
    • k is no. of clusters
  5. Can we use ML for trading?
    • Yes
  6. Any daily life clustering example
  7. Any software product based on unsupervised learning?
    • Google Maps
    • Matrimony/ Dating websites
    • Red Coupon (real estate)
    • Amazon recommendation
    • Netflix
  8. Order in which features is given, is that important?
    • No
  9. Why do we say that one model is better than the other?
  10. What if accuracy is not the concern?
    • Accuracy is one way of looking at prediction
  11. Do you think that if model changes, something in feature has changed?
  12. We have tools like WEKA, so why would anyone prefer Python or R?
    • depends on the language available or language the company uses
  13. How do we know that a particular feature is important or not?
  14. What if some features are more influential than others? How will the decision tree be affected?
  15. How to handle outliers in a decision tree?
  16. Will the algorithm figure out the relationship between input and output?
    • This is possible through Regression

 


by Dhriti Shikhar at May 02, 2016 06:57 PM

April 27, 2016

Trishna Guha

Publish Docker image with Automated build

docker1

Docker is an open source tool that helps us to develop, ship and run applications.  Each and every application needs a Dockerfile. Instructions are written over Dockerfile in a layered/step-by-step structure to run the application. The Dockerfile is then built to create Docker image. We can then push the image either to Docker Hub or keep it in our Workstation. Finally to run the application we have to run container from the image which can be treated as a concept like an instance of the image since we can have more than one container of a single image. If the Docker image is not in our local Workstation we first have to pull the image from Docker hub and then run container from it. But when we need to make changes in Dockerfile(as needed to make changes in application) we have to build a new Docker image again from it. Or we can make change in the container we are running and commit it but that doesn’t reflect the Dockerfile which contains all the instructions to run an application say it for development or production purpose. Also building Docker image locally from Dockerfile makes us doing things like build, push and all manually. Automating build of Docker image from Dockerfile that is either on Github or Bitbucket is the only solution for this:) .

In Short:  We first create Dockerfile and then push it to Github or Bitbucket. After authenticating Github account with Dockerhub we choose repository on Dockerhub from Github that contains the Dockerfile. After that the Dockerfile triggers build which in result creates Docker image getting ready to pull.

I will share an example of making a CentOS image having Apache httpd server pre-installed.

First we need to create a Dockerfile which can be viewed here.

FROM centos:centos6
MAINTAINER trishnag <trishnaguha17@gmail.com>
RUN yum -y update; yum clean all
RUN yum -y install httpd
RUN echo "This is our new Apache Test Site" >> /var/www/html/index.html
EXPOSE 80
RUN echo "/sbin/service httpd start" >> /root/.bashrc 

Then push the Dockerfile to Github. I have created a repository named CentOS6-Apache and pushed the Dockerfile to it. The repository can be found here.

After doing so

  • Go to DockerHub and Settings —> Linked Account —> Link your Github account.
  • Create —> Create Automated Build —> Create Auto-build Github.
  • Select the repository that contains Dockerfile of your application.
  • Create followed by the Automating build process.

After the image is built successfully you will be able to see Success in Build Details which indicates that the image is successfully built. The Image is now live on docker hub https://hub.docker.com/r/trishnag/centos6-apache.

Now we have to test the image to make sure whether Apache httpd server is actually pre-installed or not.

docker pull trishnag/centos6-apache  #Pull the image from Dockerhub
docker run -t -i -d trishnag/centos-apache /bin/bash #Run the container as daemon
docker ps #Get the name of the container

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8bcd1199bb8f trishnag/centos6-apache "/bin/bash" 2 minutes ago Up 2 minutes 80/tcp jovial_cray

docker logs jovial_cray #To see What is happening on bash of container.Gives httpd status
docker inspect jovial_cray | grep IPAddress #Shows the IPAddress
curl 172.17.0.1 #curl the IPAddress we got
This is our new Apache Test Site

We just have got the text that we echoed out to index.html. Hence we can finally come to conclusion that Apache httpd server has already been pre-installed in the image.

Now even if we commit changes to the Dockerfile on Github we really don’t have to worry about the image. The build process of Docker image starts automatically once any changes are committed to Dockerfile. When we pull the newly built image and run container we will be able to find those changes.

Automating build process really makes our life easier.

Resources:

 


by Trishna Guha at April 27, 2016 06:53 PM

April 16, 2016

Suraj Deshmukh

All in one Kubernetes cluster on Fedora

This tutorial will walk through the installation steps of Kubernetes, which will run Kubernetes Master and Kubernetes Node on the same machine. Setup a Fedora VM and follow along

Install required packages:

sudo -i
dnf -y install --enablerepo=updates-testing kubernetes etcd iptables openssl

Create a entry for localhost, fed-master and fed-minion. These are all 127.0.0.1 because we are doing all-in-one setup. We are also adding localhost entry because sometime not having this has caused failure.

echo "127.0.0.1 localhost
127.0.0.1 fed-master
127.0.0.1 fed-node
" >> /etc/hosts

Disable and Stop iptables and firewalld

systemctl disable iptables-services firewalld
systemctl stop iptables-services firewalld

File /etc/kubernetes/apiserver should look like as follows. Make sure service-cluster-ip-range does not overlap with any of network attached to your machine

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# Add your own!
KUBE_API_ARGS=""

Make /var/run/kubernetes

mkdir /var/run/kubernetes
chown kube:kube /var/run/kubernetes
chmod 750 /var/run/kubernetes

Generate certificates to configure service account

mkdir -p /etc/pki/kube-apiserver
openssl genrsa -out /etc/pki/kube-apiserver/serviceaccount.key 2048
sed -i.back '/KUBE_API_ARGS=*/c\KUBE_API_ARGS="--service_account_key_file=/etc/pki/kube-apiserver/serviceaccount.key"' /etc/kubernetes/apiserver
sed -i.back '/KUBE_CONTROLLER_MANAGER_ARGS=*/c\KUBE_CONTROLLER_MANAGER_ARGS="--service_account_private_key_file=/etc/pki/kube-apiserver/serviceaccount.key"' /etc/kubernetes/controller-manager

Start all the services required for Kubernetes Master and Kubernetes Node both

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet docker; do
	systemctl restart $SERVICES
	systemctl enable $SERVICES
	systemctl status $SERVICES
done

If all the services started correctly that means that Kubernetes all-in-one cluster is set and you can use it. Here are some optional steps to verify if this cluster is setup.

Create a pod definition like this below

# cat pod.json 
{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "labels": {
            "app": "apache-centos7"
        },
        "name": "apache-centos7"
    },
    "spec": {
        "containers": [
            {
                "image": "centos/httpd",
                "name": "apache-centos7",
                "ports": [
                    {
                        "containerPort": 80,
                        "hostPort": 80,
                        "protocol": "TCP"
                    }
                ]
            }
        ]
    }
}

Create a pod using that pod definition

# kubectl create -f ./pod.json 
pod "apache-centos7" created

See if that pod is created

# kubectl get pod
NAME             READY     STATUS    RESTARTS   AGE
apache-centos7   0/1       Pending   0          48s

Wait until the pod’s status changes from Pending to Running

# kubectl get pod
NAME             READY     STATUS    RESTARTS   AGE
apache-centos7   1/1       Running   0          5m

Get the IP address of the Pod

# kubectl describe pod apache-centos7 | grep 'IP:'
IP:		172.17.0.1

See for yourself the output

# curl 172.17.0.1

Ref:


by surajssd009005 at April 16, 2016 12:58 PM

Tosin Damilare James Animashaun

After DjangoGirls, What Next?

These are my thoughts to the "girls" who stormed Lagos for Django. But I'd recommend same to any attendee of a DjangoGirls workshop anywhere.


My recent foray into the social circle of techies has proven to me that growth is often driven by community. Hence, the first advice I'll offer is that you take DjangoGirls back to your city.

The organizer of DjangoGirls Lagos, Aisha Bello herself only attended this event for the first time about a year ago. Isn't it wonderful that she's been able to put up two of such events within this period? Even though not everyone of you might be able to replicate what she has done, I encourage the few ones among you who can, to endeavour to.

Also endeavour to join communities; create them if they don't exist. For starters, the Nigeria Python Users Group has a mailing list where you can get updates on the Python community in Nigeria as well as a Slack community where you can interact with other programmers.


Did you get the contacts of your mentors? If you didn't, head over to the DjangoGirls Lagos page and get their Twitter handles. Bug them! Well, don't annoy them, but do not hesitate to ask them questions when you're stuck. They'd be glad to help, I'm sure.

And really, the learning goes both ways. Post-workshop, one of the participants of the workshop showed me an error she had encountered, and I was glad to learn that by tinkering with it, she found her way out of the maze. But that's not all. From seeing her approach, I picked up new knowledge from it too.


This post is meant to give you a quasi-roadmap of some sort, which you don't have to follow religiously but keep close enough to consult when you're lost.

So you learnt Python alongside one of its most popular frameworks: Django; and now you can code the next Facebook (Oh, were you told you could only build blogs with that new super power you picked up?) Now, I think the world has more blogging apps than we can keep track of, don't you think so?

So get creative; I assure you you've gotten past the first and most basic hurdle to creating just about any kind of web app: a social network or an e-commerce platform or something else that's interesting. Think of how important this is in an age when everything is moving to the web. You've definitely gotten a head start!

But you'd need to be prepared to learn new stuff as you go. Don't let that put you off as those things would be equally exciting. And you don't have to know it all before you begin; you can learn on-the-job.


Speaking of what you learnt at the workshop, I'm sure you know you were merely introduced to the technologies, not taught comprehensively. In my opinion, you may need to go over the whole tutorial again; then try to build something different (like a ToDo application) from the example shown. Afterwards, for some more comprehensive Python tutorial that is hands-on, try codecademy. It'll be fun, trust me.

There are several tutorials available on web platforms Youtube, Edx, Coursera, and Udacity among others. Who knows, you could build the next e-school for Africa. Pick any of these courses, and try to follow along. You don't have to be in a hurry, although you may find yourself skipping stuff you have already learnt.

I'll advise you pick up some Javascript as you go. I assure you that a mix of skills in these two languages will empower you so well.

I also recommend you put up blog posts as you learn new stuff as if to teach same to others (teaching is a proven way to better learn anything). Medium would be a good place to start, but there are several alternatives. You should also read blogs often. Let me mention that Django Girls has a blog you might be interested in checking out and so does PyLadies. See more interesting Python blogs here


During the workshop, I told my mentees that Python is a suitable language for such a workshop for three reasons:

  1. It's easy to learn.
  2. It's platform-independent.
  3. It has a lot of application domains.

Take caution not to misinterpret reason-1 to mean "easy to master" (See answer to "Is Python easy to master?"), especially because of reason-3: "Python has a lot of application domains". Be it scientific, systems administration, machine learning/artificial intelligence, statistical data analysis, game development or scripting, web scraping, and web development. For those with IoT in mind, especially with Raspberry Pi and or Arduino, Python is readily applicable to those platforms.

So you see there's quite a long way to go, yet with what little knowledge you've gained, you're able to achieve something worthwhile.


Moving on, you can share your work with others, and work collaboratively. You were introduced to version-controlling using Git and Github, which are well worth some learning time. You can learn Git through this course. Don't worry if you don't get it all at once, I struggled through this too at first. This video by Git evangelist, Scott Chacon also deals a good deal on the subject.


How much time it'd take to become quite comfortable with all of this would depend on three things:

  1. Your interest
  2. Your consistency
  3. Your learning speed

However, I'd say 6 months should be fine. Within this time you would have learnt enough to kick-start an exciting project. In fact, some people might have gotten halfway with some project in this time.


Good luck in your programming journey!

by Tosin Damilare James Animashaun at April 16, 2016 12:55 PM