It’s been a while since our last post, as our team have spent the last year busily working on a large project for the British Board of Film Classification (you can read more about it here).
Along with the World, we have been adjusting to a remote working life which has led us to look at ways of efficiency and improving our normal work flow.
What was the problem we are trying to solve?
One of the core technologies we use at NMR is the Vidispine API and the way we interact with this is using Python.
Vidispine offers an auto generated Python library to integrate with its API, currently this Python library is un-documented so over the years and projects we have worked on this leads to some of the following challenges and practices.
Any calls we make to the Vidispine API had to be bundled with each code repository that needed it. This caused a lot of code duplication for our repositories and sometimes duplication of code that was unnecessary for the repo it was in.
If there was a change to the Vidispine API then we would have to manually change this in each code repository we managed that consumed the newer API.
When interacting with API’s with Python, it’s preferable to work with JSON rather than XML which is the default for Vidispine. By making the Vidispine Adapter library, we are able to adjust the request headers accordingly so that we always work with JSON (this behaviour can be overridden).
Vidispine has many different API versions so another problem we wished to solve was having a library that could map to these versions so users can easily install the version they needed.
Not all of the repositories we work with have CircleCI which meant that automatic testing of code was not always possible. By having a single library we will be able to centralise all of our code and therefore tests that touch Vidispine.
Testing across multiple Python versions was not possible either, however, now that we have a dedicated library for Vidispine, we can now cover multiple versions of Python in our tests (currently 3.6-3.9).
We therefore thought it was a great opportunity to not only build something that would greatly improve our efficiency… but also to publish some Open Source work as “many hands make light work”.
What is it?
The Vidispine Adapter is designed to be a general-purpose lightweight wrapper around the Vidispine API written in Python (3.6+), containing NO business logic. It was designed to operate as a standardised way of using Vidispine and something that organically grows, starting with the implementation of the features we have historically used the most from Vidispine API. This has typically been requests made on:
Anything to do with Metadata or Searching.
For testing of the Vidispine Adapter, we wanted to be able to mock out the responses from Vidispine whilst being as accurate to the real thing as possible, fast and not reliant on an actual Vidispine instance when running tests in CircleCI. One approach we considered was to simply make a manual request to a real Vidispine instance and then take the response and use it in our tests as the mocked response using something like requests-mock.. This is the most common way of approaching this problem but it causes a few issues such as:
Maintenance of Vidispine responses
Manually update responses with different versions of Vidispine
We then came across vcrpy (https://pypi.org/project/vcrpy/), that handles mocking of the requests whilst it also stores responses and loads them on subsequent runs of the tests.. By doing this, we can write the test, run it which will attempt to make a real request the first time and store it as a cassette file (this is simply a YAML file with the request and response data that was made).
The next time the test is run, it will recognise that there is already a cassette file present, mock any and every request made during the test run, then playback each response when the correlating request is made.
This greatly sped up test creation and made the tests much cleaner and easier to read. Another bonus, is that the tests will fail if the request differed in the slightest from what it expects, removing the need for a lot of common assertions such as checking our code hit the correct endpoint, used correct query params, payload etc.
In the future we will improve these further by testing multiple versions of Vidispine API via a catalogue of cassettes.
We also decided to use Poetry (https://python-poetry.org) for the package management and publishing to PyPi. This again breaks away from the norm of installing packages from requirements.txt and publishing with Twine. By using Poetry we eliminated the need to create a setup.py to publish, instead, it uses a pyproject.toml file which is much simpler to create and manage.
What next (roadmap)?
We decided to create the Vidispine Adapter Open Source in order to showcase our work to the developer community. Additionally, given the large number of different Vidispine endpoints to implement, ultimately the more people who are able to contribute to the codebase the more useful the adapter will be for everyone who wishes to use it.
How do you get involved?
If you would like to contribute to the Vidispine Adapter please see the GitHub page here: