ansible

Complex data structures and the Ansible json_query filter

While Ansible is busy fighting its own internal battle not to become a fully fledged programming language, instead remaining as simple and purely declarative as possible, it is still often necessary to work with more complex data structures. This is where the Jinja2 and Ansible filters can really shine.

Recently I stumbled across the Ansible json_query filter as a very neat solution to a problem that would have been otherwise messy to solve in Ansible. The filter is already well-documented, but I thought I would share a few examples of how it came in handy for me.

Example 1 – Finding a specific value in a list of objects

In the first example, I needed to iterate over a list of objects returned from an API query, find an object based on a supplied name, and return the ID of the object. To illustrate what I mean, the API query result looks similar to this:

{
  "json": [
    {
      "id": "91af4658-ed14-4058-bf34-dae98435fbca",
      "name": "example-group-01",
      "path": "/example-group-01",
      "subGroups": []
    },
    ...
  ]
}

Using the Ansible json_query filter, this requirement is easily achieved:

- set_fact:
    group_id: "{{ groups | json_query(query) | first }}"
  vars:
    query: "json[?name=='{{ group_name }}'].id"

Firstly, the query string has been split out into a task variable for both readability and to avoid quote-escaping issues. As for what is actually going on, let’s break it down:

  1. The query is being run against our API results variable, groups, which contains the data structure illustrated above
  2. The query is iterating over all list items under the json attribute
  3. The query filters on objects in the list that have an attribute name that equals the variable group_name. In other words, does result.json[N].name match my requirement
  4. It returns the ID attribute, result.json[N].id, for any matching objects, and appends it to another list
  5. As the group name is unique in this situation, we are passing the whole result through another filter to retrieve the first and only value

Example 2 – Using a complex object in an Ansible loop

In the next example, I had yet another API query result that I wanted to utilise in a loop. While this could normally be achieved using a standard loop, I had added complexity due to the fact that the object was actually the collated result of many API queries run in a loop:

- name: Get user information
  uri:
    url: "{{ api_url }}/users?username={{ item }}"
  register: users
  with_items: "{{ username_list }}"

The returned object structure is somewhat similar to the group example above, but of course since the data is returned in a loop, Ansible appends each result to a results list.

So, to cap off this example, here is the Ansible loop I used to add these users to a group:

- name: Add users to group
  uri:
    url: "{{ api_url }}/users/{{ item }}/groups/{{ group_id }}"
    method: PUT
  with_items: "{{ users | json_query('results[*].json[*].id') }}"

Again, breaking this down:

  1. Taking the users variable above, we are iterating on all items in the results list, using a wildcard
  2. Similarly for each results item, we are iterating on all json items
  3. Finally, we are using the id attribute as our Ansible loop item

Example 3 – Filtering on a list item

Lastly, I ran into a situation where the conditional filter was based on an item existing in a list, rather than an attribute equaling a specific value. In real terms, I wanted to find a VM name based on a known IP address, where each VM may have one or many IP addresses. Here is the rough data structure:

{
  "vms": {
    "json": {
      "resources": [
        {
          "ips": [
            "192.168.1.10",
            "10.0.0.10"
          ],
          "name": "vm-001"
        },
        {
          "ips": [
            "192.168.1.11",
            "10.0.0.11"
          ],
          "name": "vm-002"
        },
        ...
      ]
    }
  }
}

Since the json_query filter is built on JMESPath, there are a large amount of built-in functions available. We are interested in the contains function, which can be used for both strings and lists. Putting this into action we get:

- set_fact:
    my_vm: "{{ vms | json_query(query) }}"
  vars:
    query: "json.resources[?ips.contains(@, '192.168.1.11')].name"

Given our data above, the my_vm variable will have the value [“vm-002”]

3 thoughts on “Complex data structures and the Ansible json_query filter

  1. After struggling to figure out JSON parsing in ansible playbooks (and refusing to give up and write a shell script “wrapper”) for some AWS tasks, I found your page in a google result.

    It works! It is now 2:30pm, and I’ve been fighting this beast since 6am. You saved me. Well done sir.

    Oh: I’m using the aws cli to get lists of resources but it returns fairly complex objects and I was trying to find the “ansible way” to address attributes based on a selector like this:
    get a list of kafka configs
    find the arn of the config, but with a NAME I know

    The most of the AWS cli has built in filters, some is half baked rubbish, like kafka.

    Liked by 1 person

  2. exactly.. what I was looking for, Thank you very much for breaking them down and explain.
    Being from a system admin background, this json query thing was frightening me 🙂

    Liked by 1 person

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.