How Python made it Better, Easier and Faster

In software engineering engineers come-across repetitive processes in their daily lives. These repetitive processes are more or less automated using scripting languages. This blog post is about a tool that we wrote to automate feature creation for components and etc, which is a repetitive processes we came-across. This blog post explains how we made the process better, easier and faster.

Introduction

This is a tool which is written with the focus of bringing all the Python script into one place. However, we thought of starting with automating feature creation for components as it is a burning problem in our organization. This document explains what are the existing problems of feature creation and its associated sub-tasks and how this tool overcome these problems.

The problem in hand

The process we use to release products is based on so-called chunking process in which we release products by releasing a chunk of the platform. The problem with this system is platform becomes a huge mix of released and un-released components and features. Not only that, it also allows the platform to have multiple sub-versions of components and features. This introduced some new problems on top of the existing problems. These are some of the problems we saw and came-across during ESB-4.8.0 release,

• It is hard to figure out if a component or a feature is released or not.
• Failing to propagate the changes to associated features.
• Failing to propagate the changes to associated features of features.
• Failing to find dependent projects.

As a result committing to released versions became more common, which in turn increased the number of build breaks. Making the problem to move from bad to worse. The tasks that we have in current release process is not only error-prone but also repetitive and time consuming. Therefore, to overcome these problems we thought of writing a tool.

Management Tool

In software engineering engineers come-across repetitive tasks in their daily lives. This is one good reason why scripting languages exist. Using scripting languages people can automate these repetitive tasks. Therefore, we thought of automating above mentioned tasks with the help of Python. One of the best scripting languages in the world. Following are the tasks we automated with our Python script.

• Validating the existing component.
• Creating a new component.
• Updating the component chunk pom.
• Finding dependent features.
• Validating the existing feature
• Finding dependent feature of features
• Creating new features or updating the existing ones.
• Updating the feature chunk pom.
• Revert the changes in case of failure.

Everything is done by looking at Nexus (with the help of Nexus REST API [1]). Therefore, it is literally impossible to commit to a released component or create an invalid component or feature version. Moreover, at the end of the execution it shows with a tree view how all the updated components are associated to each other, which helps you to have a better understanding about the updated components. Therefore, with that understanding you can either revert unwanted changes or proceed with the new changes. As a result of all these, now the process has become,

• Less error prone.
• Pretty fast.
• Consistent over the platform.
• How to run the tool
• To run the tool do the following,
• Get the code from link [2] or from the attachment (so2py.zip).

Run main.py with the component that you have done changes to (It has to be the latest version). For example: ./main.py /media/shafreen/source/public/turing-new/components/mediation/message-processor/org.wso2.carbon.message.processor/4.2.0

Note : when your are running the tool for the first time it will ask for the root of your SVN local copy. So, give the complete path to the root of your local SVN copy.
For instance: /media/shafreen/source/public/turing-new

That's it. Let the tool do the rest for you. Upon successful execution it will give something like in the below image and it will create log files in <ManagementTool_Home>/log directory.

Limitations and Future work

Following are some of the limitations that this tool have. However, These are not technical limitations. But we couldn’t find free time to implement it.

• Assumes that the svn is in buildable state.
• If the parent pom is not released it does not add the parent pom.
• Only works for components.

Following are some of the feature work we have in mind,

• Find dependent component of the updated component, purge those dependent. component in m2 and build the updated components. So that, all you need is a buildable svn and Nexus.
• Update feature chunk pom only with the features that are in the p2-profile pom of a particular product.
• Auto commit all the changes.
• Fix above mentioned limitations.
• Make it a proper command line tool that can take different commands and flags.

[1] https://repository.sonatype.org/nexus-restlet1x-plugin/default/docs/index.html
[2] https://github.com/raviu/so2py