Automating network appliance testing

To illustrate the point we’re trying to make in the video above, let us tell you about a very interesting problem we’ve been asked to solve recently. Here in Tokyo, we at Axsh Co. LTD are known for our knowledge of Virtualization, SDN, CI/CD, and DevOps in general. Recently a customer approached us with the following problem. They had a particular firewall running in their production environment and making any changes to this device was a very scary process.

One simple mistake could result in huge security holes, network admins working weekends, and thousands of dollars lost. Our job was to make this process more reliable and less error prone.

Software Development

Software development used to face these same problems every time a new version was released. Today in 2018 however, these have been largely solved by Continuous Integration and Continuous Deployment. Let’s quickly run through some of these problems and their solutions.

  • Problem: Bugs aren’t discovered until they’re in production.
    Solution: Continuously deploy to a staging environment and run automated tests on for every change. These tests will find bugs early when they’re cheap to fix.

  • Problem: Code that works on dev/staging environments breaks on production.
    Solution: Use VPS clouds to provide identical copies of production for dev/staging use.

  • Problem: Deploying is a tedious multi-step process prone to human error.
    Solution: Automate the deployment using simple scripts. Machines are great at tedious repetitive tasks.

  • Problem: Only a few people in the office know how to deploy. When they leave, nobody knows.
    Solution: Automated scripts enable everyone to deploy. By reading the scripts, one has perfect documentation of the deployment process.

  • Problem: Changing one part of the code might unexpectedly break a completely different part.
    Solution: Run a test suite for the entire application for every change. Unlike manual tests, Automated tests are fast.

  • Problem: Despite all of the above, serious problems can still occur once deployed to production.
    Solution: Use version control to check out a known good version and re-deploy. Because we’re doing it continuously all the time, deploying has become quick and cheap.

Our Customer’s Problem

The problems our customer was facing were very similar to the problems listed above.

  • Very few people knew the exact state of the firewall.

  • Many engineers manually reviewed new settings before applying. A tedious time consuming process prone to human error.

  • Testing was done manually. Again very time consuming and error prone.

  • There was no identical copy of the production environment available.

Our Solution

Using SDN software like OpenVNet combined with a VPS cloud like OpenVDC, it becomes possible to create complete copies of the production network where everything is identical down to the exact IP and MAC addresses of each appliance.

Another problem is that networks appliances such as routers and firewalls often run proprietary operating systems that can’t easily be virtualized.

Attaching firewall to SDN Switch

For this purpose we present our SDN designated hardware switch. This box is basically just a regular Linux server with many network interfaces. We can now take any physical appliances we want, attach each of their ports to this switch and inside it we can use SDN to simulate any network topology we want. This effectively enables us to create that identical copy of production to run our test cases on.

By using orchestration tools such as Terraform we write the virtual network topology as code. Tools like Expect and Ansible help us write the physical network appliance configuration as code. OpenVDC allows us to easily log into any node on the virtual network and execute code. Therefore we can easily write automated test cases that send packets across the network and check for their arrival.

Now that our entire virtualized test environment consists of code, the exact same DevOps cycle we know can be applied here. For example the exact configuration of each physical network appliance is kept in a file. When a change to that file is made, that change can then be committed to the source code.

Once a commit occurs, the following steps occur automatically.

  • The virtualized copy of production is brought online

  • The configuration of the physical appliance (a firewall in this case) is updated

  • The test cases are run

  • Their result is reported back to the user

Once the tests have confirmed everything to be working according to expectations, we can now take the firewall configuration code and point it at the production environment. That way the tested configuration is guaranteed to be applied to production in the exact same way.

Ease of Use

Graphical User Interface

Network admins aren’t always programmers. That’s why we’re providing a graphical user interface to go along with this system. If coding isn’t your thing, you can use a convenient drag-and-drop based GUI to design your network, enter your appliance’s configuration and design test cases. The system will then generate all of the code required and commit this to version control. Of course editing the code directly is still an option.

How to Get Started

We are planning to release the first version to the public later this year. Until then we’ll just need to ask you to stay tuned for more information.

Friday March 23, 2018