Know what you deploy - a script using git log for listing all features that go live
Introduction
For me, even after having pushed the deploy button many times, deploying to production keeps being dreadful. Accidentally pushing an unwanted feature that sneaked in somehow is one of the many things I am afraid of. In this blog post I show you how we reduced our deployment anxiety with a script using git log for listing all the features that go live after pushing that deploy-to-pro button.
In this blog post I first introduce you to the way we work and provide you with the necessary information to understand the essence of the script. After that, I briefly show you how to use the script. Because your way-of-working might be different to ours, I explain the script in more detail so you can modify it to accommodate your specific situation. You don't need to be an expert in shell scripting or git to understand this blog post, but some basic knowledge on these matters definitely helps.
Our way of working
We keep track of our feature requests in jira - a web-based tool to plan, track, and release software features. We create a jira ticket for each request or user story. We develop each feature on a git feature branch having a name that starts with the jira ticket number and a short description of the feature, e.g. SEO-4734-report-diff-acc-prod. Any commits I push to that branch are prefixed with the jira ticket number followed by a sensible description of what the commit is about.
When I believe I have finished a feature and one of my team members approves my changes, I merge my feature branch into master. This triggers a Jenkins build that tags the application version and automatically pushes my changes to a test environment. On this environment we run automated integration and performance tests before going to production.
The script for listing all features that go live in action
Prerequisites
First off, your code should be stored in a git repository, but you probably already figured that out. In order to run the script on your local machine, you need a Linux/Unix like system with git and the following lightweight command line tools installed:
- jq - a lightweight and flexible command-line json processor
- the_silver_searcher - a code searching tool
Both tools can be installed with brew if you like:
brew install jq
brew install the_silver_searcherSo far for the boring part. The reason why you need these tools will become apparent in what follows. First, it is time to see the script in action.
Running the script for listing all features that go live
Before you run the script you need to be on master (or the production branch from which you deploy) and be sure to have fetched the latest changes:
git checkout master
git pullAfter that, you're ready to run the script from the directory where the script file resides in. Note that the script works without any command line arguments:
./diff-between-acc-and-prd-report.shThe script will then ask you for the version that is on production:
Fill in prod application version (see axle internal page) and press :You probably don't know what the 'axle internal' page is. No worries, I'll explain that later. After you entered the latest version, the script will list all the differences between the application version that is currently running on acceptance and the one that is running in production. More specifically, the script outputs the git log, links to jira tickets and feature branches that were merged into master, for example:
-------- Git Log ------------
67f9e8b5c SEO-4265 (tag: TRC-156.0.16.1106) Merge branch 'feature/SEO-4265-bug-fix' into 'master'
232135e1b SEO-4265 changed <= into >= that fixes it for sure
67f9e8b5c SEO-4265 changed < into <= does that fix it?
67f9e8b5c SEO-4265 changed > into < that should fix it
f408ed469 (tag: TRC-156.0.11.1105) Merge branch 'feature/SEO-4734-report-diff-acc-prod' into 'master'
6960985b9 SEO-4734 created script to see the difference between acc and prod using git tags
-------- Jira tickets ------------
https://jira.tools.bol.com/browse/SEO-4265
https://jira.tools.bol.com/browse/SEO-4734
-------- Features (possible incomplete list!) ------------
SEO-4265-very-important-bug-fix
SEO-4734-report-diff-acc-prodPretty cool, isn't it? You don't have to worry if all this seems magic to you - I'll explain everything step by step in the following section.
Breaking down the script for listing all features that go live
Ah, you are still here? Good, that probably means you're interested to know how the script works. So, let me break it down and explain it piece by piece.
Fetch latest version from acceptance
The script starts by fetching the latest tag from the application that runs on the acceptance environment. Here at bol.com we use an in-house framework for building REST applications: Axle. Each service built with Axle has an internal page showing useful information about the application, such as the git tag denoting the service's version. So in case you wondered what this 'Axle internal page' is while reading the previous section - now you know.
You can access the Axle's internal page by visiting a url that ends with /internal. This shows the application information in human readable format. Luckily, the Axle framework also provides an endpoint on /internal/info that returns the information as json. The script uses the curl command to obtain the application info:
acc_info_json=$(curl -s "$acc_url/internal/info")So, no need to parse html, we only have to parse json - and that is where the command line tool jq comes in handy. We use it to get the version field out of the raw json response:
acc_version=$(echo $acc_info_json | jq -r '.git.build.version')
The -r flag ensures that quotes surrounding the version string are stripped off. If you like to know more about the jq command, this might be a good read.
What if you don't have an endpoint that provides the version number?
In that case you could always provide the version number as input to the script, of course. However, having an endpoint that does provide this information is still something I would recommend. If you use Spring Boot 2, you can find an easy way to expose this information through an endpoint using a maven plugin here.
Determine git tags
To determine the difference between the versions that run on acceptance and production, the script also needs to know the version of the latter. Recall that we need to provide this as input to the script.
Did you just ask yourself why we do not simply fetch the production version in the exact same way as we do for acceptance? Great, that means that you have a great automate-all-the-things mindset. The reason is that the info endpoint on production is behind a login. Of course, there are still ways to read the version automatically. I leave that as an exercise to the reader. ;-) For now, the script asks its user to enter it manually:
echo -n "Fill in prod application version (see axle internal page) and press : "
read prd_versionThe script compares the application on production with the one on acceptance based on the tag. In our setup the Jenkins build tool tags our code automatically after each push to master. Tags consist of the application version prefixed with the service name. The script builds up the tag as follows:
prd_tag="$application_shortname-$prd_version"
acc_tag="$application_shortname-$acc_version"Show the git log
Comparison is done based on git tags. Now the script knows which tags to compare, showing it to the user is to run the following git command:
git --no-pager log $prd_tag..$acc_tag --onelineThe –no-pager argument makes git output just text which we can search on. The git log tag1..tag2 command outputs all commits being done between the two tags on the current branch - master in our particular case.
The script needs the output of this command later on. Therefore, it stores the command's output in a variable before executing and displaying its output to the user:
git_log_cmd="git --no-pager log $prd_tag..$acc_tag --oneline"
git_log=$(eval $git_log_cmd)
eval $git_log_cmdWhat if you don't use git tags?
No worries, the git log command also works with commit ids. You just have to replace git log tag1..tag2 with git log commit_id1..commit_id2.
Scan the git log for jira ticket numbers
Looking at the git log, you can see one thing that is a direct consequence of our way-of-working: we prefix each commit message with a jira ticket number starting with three up to nine alphanumerical characters, a dash and a digit code, for example: SEO-4734.
Recall that the script needs the silver search command line tool to run. The script uses this tool to search for ticket numbers in the git log:
jira_tickets=$(echo $git_log | ag "(\w){3,9}-(\d+)" -o | uniq)Note that a jira ticket can appear more than once in the git log, because multiple commits share the same jira ticket number. To ensure the script displays each ticket number only once, the last expression ends with the uniq command.
If you like to learn more about the ag command, you could start here. Also note that we could have used an alternative tool for the parsing, for instance awk.
Finally, the script echos each jira ticket number to the console:
for ticket in $jira_tickets
do
echo "$jira_url/$ticket"
doneScan the git log for feature branch names
As explained before, we develop features on a feature branch. Once reviewed and approved, we push it to master. As a convention, we name our branches starting with the jira ticket number followed by a short description of the feature in kebab case. That is, dashes separate the words of the description. So, extracting and displaying the feature branch names that were merged into master from the git log is very similar to extracting the jira ticket number:
features=$(echo $git_log | ag "(\w){3,9}-(\d+)-(<\w->+)" -o | uniq)
for feature in $features
do
echo "$feature"
donePutting it all together
That's it. I have explained all the steps that comprise this script for listing all features that will go live. It helps me deploy to production with great(er) confidence. Hope it does the same to you. Now it is up to you. Let me put it all together for you, so you only have to copy paste and modify:
#!/bin/bash
# Dependencies: jq, silversearcher
# Script to check what is the difference between prod and acc using git tags
# This can be handy to check before deploying to prod
# This script will get the git tag from traffic service internal page on acc and ask the user to manually enter the version from prod
# config
acc_url=<url to acceptance environment>
application_shortname=<your application name here>
jira_url=<url to jira>
# Get info from acc
acc_info_json=$(curl -s "$acc_url/internal/info")
acc_version=$(echo $acc_info_json | jq -r '.git.build.version')
# Determine git tags
echo "Create report with differences between acc and prod environment for a DC application"
echo -n "Fill in prod application version (see axle internal page) and press : "
read prd_version
prd_tag="$application_shortname-$prd_version"
acc_tag="$application_shortname-$acc_version"
# Show git log
echo "Going to be deployed to prod:"
echo ""
echo "-------- Git Log ------------"
echo ""
git_log_cmd="git --no-pager log $prd_tag..$acc_tag --oneline"
git_log=$(eval $git_log_cmd)
eval $git_log_cmd
# scan git log for jira ticket numbers
echo ""
echo "-------- Jira tickets ------------"
echo ""
jira_tickets=$(echo $git_log | ag "(\w){3,9}-(\d+)" -o | uniq)
for ticket in $jira_tickets
do
echo "$jira_url/$ticket"
done
# scan git log for feature branch names (for example: SEO-4570-shop-review-v3-endpoint)
echo ""
echo "-------- Features (possible incomplete list!) ------------"
echo ""
features=$(echo $git_log | ag "(\w){3,9}-(\d+)-(<\w->+)" -o | uniq)
for feature in $features
do
echo "$feature"
done