Monday, September 3, 2018

Ansible Test Driven Development with Molecule

Ansible Test Driven Development with Molecule

Molecule is a framework for doing TDD (Test Driven Development) for your Ansible roles. Using a variety of drivers, molecule lets you test your Ansible role on either Azure, Docker, Amazon EC2, Google Cloud, Linux Containers, Openstack or Vagrant. More information about how to use these drivers at https://molecule.readthedocs.io/en/latest/

In this post we will look at how we can use Docker locally for testing an Ansible role during development. If you want to test the commands given here, it is assumed that you have already installed Ansible, Docker, Python 2.7 and pip, and that you have a basic understanding on how to use them.

The source code for the role shown can be found at https://github.com/avnes/ansible-role-vscode and the role is used for installing Visual Studio Code on Linux.

Creating a Python virtual environment

Since both Ansible and molecule is written in Python, it is recommended to create a Python virtual environment where you can test your role. This also makes your tests more accessible for other developers if you document your PyPi packages in a requirements.txt file. I am a big fan of the virtualenvwrapper, which makes it a lot easier to work with multiple virtual environments:


mkvirtualenv moleculetdd
pip install docker-py==1.10.6
pip install molecule
molecule --version
ansible --version
# store it to a requirements.txt file:
pip freeze > requirements.txt

If you later need to return to this Python virtual environment, you can again use a feature from virtualennwrapper:

workon moleculetdd

Initializing a role

Normally you would use ansible-galaxy init <role name> to create a new Ansible role, but since we know we are going to use TDD with molecule, we can as well let molecule create the role structure for us too.

mkdir -p ~/git/roles
cd ~/git/roles
molecule init role --role-name ansible-role-vscode
# now you could do git init if this is a role you want
# to have under source control
cd ansible-role-vscode
ls -l


Personally I find myself using ansible galaxy init most of the time, and if you already have an Ansible role that just needs molecule tests, you would create the default test like this:

cd ~/git/roles/ansible-role-vscode
molecule init scenario --scenario-name default --role-name ansible-role-vscode
ls -l

The list of file and directories would look like this:

drwxrwxr-x. 2 <user> <user> 4096 <date and time> defaults
drwxrwxr-x. 2 <user> <user> 4096 <date and time> handlers
drwxrwxr-x. 2 <user> <user> 4096 <date and time> meta
drwxrwxr-x. 3 <user> <user> 4096 <date and time> molecule
-rw-r--r--. 1 <user> <user> 1330 <date and time> README.md
drwxrwxr-x. 2 <user> <user> 4096 <date and time> tasks
drwxrwxr-x. 2 <user> <user> 4096 <date and time> vars


We won't use handlers for this role, so I will remove that directory. It is also assumed that you write you own documentation, and places that in the README.md file using Markdown syntax.

In case it's been a while since you last created an Ansible role, or perhaps this is your first time, I will quickly go over what the purpose of the various directories are.

defaults is for storing variables with default values. These variables will be very easy to override, for instance through role inheritance, or if reading variables from an inventory. 

meta is, like the name suggest, for storing meta information about a role. This included the name of the author, a role description, supported platforms and dependencies on other roles.

molecule is for storing one to many test scenarios for the role, and this will be covered in great length later in this article. We have so far not specified which driver to use, but Docker is selected by default.

tasks is where your actual work is taking place. This of it as a place to keep one more related playbooks that together performs all the work the role is supposed to do.

vars is for (pretty static) variables that are harder to override. It might be good to study Ansible Variable Precedence to understand all the locations where Ansible looks for variables, and which precedence that takes effect if the same named variable is listed in several locations.

Inside the molecule's scenario directory, there is also a scenario specific documentation file, that is usually not used either, so I always delete it, and keep all my documentation in the README.md file as the role's root folder. So in this case, I would now run:

rm molecule/default/INSTALL.rst

For the sake of this article, we will only use molecule on a Fedora based testcase, so I have edited molecule/default/Dockerfile.j2 to install Python, Ansible and sudo inside a Fedora:26 docker image. Sudo is needed in the cases where you do become:yes in your role.

FROM {{ item.image }}

RUN dnf install -y python2 python2-dnf libselinux-python ansible sudo

CMD ["/bin/bash"]


So where is that {{ item.image }} defined? You will find it in molecule/default/molecule.yml:

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
  enabled: False
platforms:
  - name: ansible-role-vscode-default
    image: fedora:26
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8


You will there see that the image has been configured to be fedora:26.

You are now able to test your Ansible role by running molecule --debug test
which will launch a Docker container based on Fedora 26, install Python, Ansible and sudo and finally running your role inside it.

At the time of writing this, Fedora 26 has already been superseeded by Fedora 27 and Fedora 28. Fedora 29 is scheduled to be launched in about 2 months.

This is where the power of molecule comes in. Just by changing the image line line in molecule.yml to use fedora:28, you have quickly made it possible to test on a later release in an isolated Docker environment. Great, isn't it!


Aborting, target uses selinux but python bindings (libselinux-python) aren't installed

The problem

When running Ansible in a Python virtual environment. Or when running molecule --debug test, you encounter the following error:

Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!

The investigation

There are a couple of possible root causes for this:

1. Maybe the error is right, and that libselinux-python is not installed. 
2. Python libraries for selinux are not available in a Python virtual environment.


The solution

To make sure libselinux-python is installed on Fedora:
sudo dnf install -y libselinux-python

To make sure libselinux-python is installed on CentOS/Fedora:
sudo yum install -y libselinux-python

Imaging you have a virtual environment called ansible26, you first need to activate that virtual environment and then copy the Python libraries for selinux over:

workon ansible26
cp -r /usr/lib64/python2.7/site-packages/selinux $VIRTUAL_ENVv/lib/python2.7/site-packages
cp /usr/lib64/python2.7/site-packages/_selinux.so $VIRTUAL_ENVv/lib/python2.7/site-packages

Please note that workon is a command from https://virtualenvwrapper.readthedocs.io/ and if you haven't installed virtualenvwrapper, you would rather navigate to your virtual environment directory, and then run source bin/activate

Credits to https://github.com/metacloud/molecule/issues/1209 where I found this solution.

Monday, April 23, 2018

Using Azure Service Principal login with Ansible

The problem

When using the Azure REST API from Ansible using the uri module you need to ensure you are authenticated towards Azure. The easiest way to do that is to set a Bearer token based on a Service Principal user on Azure. 

This is the official Azure REST API documentation: https://docs.microsoft.com/en-us/rest/api/azure/

This is how you can create a service principal:

When you have created an Azure service principal you will have 4 necessary pieces of information:

  • Tenant id (the protected resource)
  • Client id (username)
  • Client secret (password)
  • Subscription id (you already had that from your user profile)
Make sure to save those values to an Ansible Vault. In my example I have called them:

  • vault_az_tenant_id
  • vault_az_client_id
  • vault_az_client_secret
  • vault_az_subscription_id

The solution

Create a playbook.yml with the following information:

---
- hosts: localhost
  vars:
    az_tenant_id: "{{ vault_az_tenant_id }}"
    az_client_id: "{{ vault_az_client_id }}"
    az_client_secret: "{{ vault_az_client_secret }}"
    az_subscription_id: "{{ vault_az_subscription_id }}"
    az_token_url: "https://login.microsoftonline.com/{{ az_tenant_id }}/oauth2/token"
    az_token_body: >
        resource=https://management.core.windows.net/
        &client_id={{ az_client_id }}
        &grant_type=client_credentials
        &client_secret={{ az_client_secret }}

  tasks:
    - name: Login to Azure
      uri:
        url: "{{ az_token_url }}"
        method: POST
        body: "{{ az_token_body }}"
        headers:
          Content-Type: "application/x-www-form-urlencoded"
        status_code: 200
      register: login

    - name: Setting Bearer token as fact
      set_fact:
        az_bearer: "{{ login.json.access_token }}"
 

That is how easy it is to authenticate with Azure from the Ansible uri module. Thanks to Vivek to helping out with the az_token_body.

Now every time you us the Ansible uri module in the same playbook, you just need to add the Bearer token to the request header, like this:

- uri:
        url: "{{ whatever-azure-url }}"
        method: POST
        headers:
          Authorization: "Bearer {{ az_bearer }}"

Wednesday, August 16, 2017

False positive: ansible-lint reports [ANSIBLE0002] Trailing whitespace when there are none

The problem

When running ansible-lint on a Ansible role, it reported [ANSIBLE0002] Trailing whitespace on every line in one of my task files, like shown in the example below:


(vansible23)[audun@hostname my-role]$ ansible-lint .
[ANSIBLE0002] Trailing whitespace
/home/audun/git/my-role/tasks/main.yml:1
---

[ANSIBLE0002] Trailing whitespace
/home/audun/git/my-role/tasks/main.yml:2


[ANSIBLE0002] Trailing whitespace
/home/audun/git/my-role/tasks/main.yml:3
- debug: msg="Hello World"



The investigation

I had a suspicion there might be hidden characters in the file, but before running it through a HEX editor, I tried to check it with the Linux file command:


(vansible23)[audun@hostname my-role]$ file /home/audun/git/my-role/tasks/main.yml
/home/audun/git/my-role/tasks/main.yml: ASCII text, with CRLF line terminators



As shown above, it turned out that the file had CRLF line terminators on every line, indicating that this file had been created on Windows. That was luckily easy to fix.


The solution

The dos2unix command was used to change the file to Unix format:


(vansible23)[audun@hostname my-role]$ dos2unix /home/audun/git/my-role/tasks/main.yml
dos2unix: converting file /home/audun/git/my-role/tasks/main.yml to Unix format ...



Afterwards ansible-lint reported no issues:


(vansible23)[audun@hostname my-role]$ ansible-lint .

Wednesday, March 15, 2017

Using atom-beautify package to beautify a Markdown file results in: Error: Cannot find module '../lib/language-code-rewrites'

The problem

I was using Atom Editor 1.14.4 with atom-beautify 0.29.17 package. When I tried to beautify a README.md file created with ansible-galaxy init <rolename> I got the following exception:

File 0Project 0No IssuesREADME.md9:77
LFUTF-8GitHub Markdowngit+
Cannot find module '../lib/language-code-rewrites'
Cannot find module '../lib/language-code-rewrites'
Hide Stack Trace
Error: Cannot find module '../lib/language-code-rewrites'
    at Module._resolveFilename (module.js:455:15)
    at Module._resolveFilename (/usr/share/atom/resources/electron.asar/common/reset-search-paths.js:35:12)
    at Function.Module._resolveFilename (/usr/share/atom/resources/app.asar/src/module-cache.js:383:52)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (/usr/share/atom/resources/app.asar/src/native-compile-cache.js:50:27)
    at Object.<anonymous> (/home/ane058/.atom/packages/atom-beautify/node_modules/tidy-markdown/lib/converters.js:12:23)
    at Module._compile (/usr/share/atom/resources/app.asar/src/native-compile-cache.js:109:30)
    at Object.value [as .js] (/usr/share/atom/resources/app.asar/src/compile-cache.js:216:21)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)
    at Module.require (module.js:483:17)
    at require (/usr/share/atom/resources/app.asar/src/native-compile-cache.js:50:27)
    at Object.<anonymous> (/home/ane058/.atom/packages/atom-beautify/node_modules/tidy-markdown/lib/index.js:16:14)
    at Module._compile (/usr/share/atom/resources/app.asar/src/native-compile-cache.js:109:30)
    at Object.value [as .js] (/usr/share/atom/resources/app.asar/src/compile-cache.js:216:21)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)
    at Module.require (module.js:483:17)
    at require (/usr/share/atom/resources/app.asar/src/native-compile-cache.js:50:27)
    at /home/ane058/.atom/packages/atom-beautify/src/beautifiers/tidy-markdown.coffee:13:22
    at Promise._execute (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/debuggability.js:300:9)
    at Promise._resolveFromExecutor (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:79:10)
    at TidyMarkdown.module.exports.TidyMarkdown.beautify (/home/ane058/.atom/packages/atom-beautify/src/beautifiers/tidy-markdown.coffee:12:16)
    at /home/ane058/.atom/packages/atom-beautify/src/beautifiers/index.coffee:318:24
    at Promise._execute (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/debuggability.js:300:9)
    at Promise._resolveFromExecutor (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:79:10)
    at /home/ane058/.atom/packages/atom-beautify/src/beautifiers/index.coffee:240:18
    at tryCatcher (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._resolveCallback (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:432:57)
    at Promise._settlePromiseFromHandler (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:524:17)
    at Promise._settlePromise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._resolveCallback (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:432:57)
    at Promise._settlePromiseFromHandler (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:524:17)
    at Promise._settlePromise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._resolveCallback (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:432:57)
    at ReductionPromiseArray._resolve (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/reduce.js:61:19)
    at Promise.completed [as _fulfillmentHandler0] (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/reduce.js:122:15)
    at Promise._settlePromise (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:566:21)
    at Promise._settlePromise0 (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:143:10)
    at Async.drainQueues (/home/ane058/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:17:14)

I logged it as an issue at https://github.com/Glavin001/atom-beautify/issues/1549, but after a few days of tinkering I finally found the likely root cause, and working solution.

The investigation

The exception suggest there is an issue with the tidy-markdown dependency, and it turned out that tidy-markdown 2.0.5 npm package had been released a few days earlier at https://www.npmjs.com/package/tidy-markdown 

ls ~/.atom/.apm/tidy-markdown/
showed that atom-beautify had pulled down the tidy-markdown 2.0.5 package.

The solution

cd ~/.atom/packages/atom-beautify
npm install tidy-markdown@2.0.4


Saturday, January 21, 2017

Released my second role to Ansible Galaxy today

The ansible-role-tint2 is an Ansible role for installing the tint2 panel for Linux. Currently the role supports the following platforms (in alphabetic order):


  • Arch Linux
  • Debian sid
  • Debian stretch
  • Fedora 24
  • Fedora 25 
  • Ubuntu 16.04
  • Ubuntu 16.10

The Ansible role can be found here: https://github.com/avnes/ansible-role-tint2
It has been released under the liberal MIT license.

While this Ansible role and this blog post of somewhat similar to my previous Ansible role, I used a much more flexible and portable test framework based on mulecule.

Saturday, January 14, 2017

Released my first role to Ansible Galaxy today

The ansible-role-conky is an Ansible role for installing the conky monitoring overlay tool. Currently the role supports the following platforms (in alphabetic order):


  • Arch Linux
  • Debian sid
  • Debian stretch
  • Fedora 24
  • Fedora 25 
  • Ubuntu 16.04
  • Ubuntu 16.10

The Ansible role can be found here: https://galaxy.ansible.com/avnes/ansible-role-conky/
It has been released under the liberal MIT license.