Testing With IronPython

At my work, we do validation of system on a chip devices. We have our own internal hardware team that designs boards for us to use in testing. These boards are complicated pieces of equipment, containing multiple FPGAs, power supply components, and more. With this complication comes the possibility that something could go wrong with the board. I develop an application used by other engineers to use these boards to load and run their tests (which run on the embedded device). For a long time, in this application we had some simple board tests that would help debug issues as they came up. We started getting more and more products and the boards changed slightly and it was difficult to keep these tests up to par with what needed to be there for the different products. I had been playing with IronPython for some time on various projects at home and thought this might be a good place to use it. The idea being that changes are more easily made in scripts, and the hardware team could add new tests as problems crop up.

I decided that the unittest module for Python would do a great job of managing the tests that needed to be run and allowing grouping of the tests into test suites that could be run by different users.

I’m going to take you through the design and implementation of an application similar to that which I use at work to help our hardware team debug boards. It will not be the exact application I use at work, but will give you an idea of some of the simple, yet powerful things you can do with IronPython.

image</a>

The UI is very simple: a simple menu with a single entry (File > Exit), a toolbar with four buttons, a listview that shows the currently loaded set of tests and an output window for the test output. Here you can see I loaded the test_datetime.py file into the GUI to run the datetime tests (and sadly it looks like there are failures in IronPython’s datetime module!).

I’m not going to go into heavy detail about the GUI development, that is not the important part of this exercise. Let’s walk through the steps necessary to load in a Python file that contains unit tests.

We’ve got a couple issues to solve for this application:

  1. How do we get the output from the unit tests to go to the output text box?</li>
  2. How do we enumerate and show all the unit tests in the given file?</li>
    </ol>
    First things first, we need to get IronPython going inside of our application. Download the latest stable version from
    http://ironpython.codeplex.com</a> (as of this writing, the latest is 2.7.1 which means its roughly compatible with CPython 2.7). For embedding IronPython into an application, I prefer to get the zip file release that contains the necessary assemblies. Unzip the distribution to any place you would like and add the following as references to your project:

    1. IronPython.dll</li>
    2. IronPython.Modules.dll</li>
    3. Microsoft.Dynamic.dll</li>
    4. Microsoft.Scripting.dll</li>
    5. Microsoft.Scripting.MetaData.dll</li>
      </ol>
      These are the basic assemblies that will be used for embedding IronPython into the application.

      The creation of the GUI as I said, is not the important part of this tutorial, if you have questions about anything I did feel free to comment, or drop me an email.

      So, lets start getting our application ready to run Python code.

      We need an execution engine.For this application, I am only planning on supporting one engine for the entire application, I don’t have a need to support an engine per thread or anything like that. So, I’ll go ahead and create a ScriptEngine (Microsoft.Scripting.Hosting) at the class level of my main form.

      Most often when I am embedding IronPython into an application, I will have a method to initialize the engine and setup any paths I may need, pre-import some modules and do various other initialization steps. Here is the InitializeEngine method.

      [gist id=1765694 file=InitializeEngine.cs]

      A few things of note. You can see the creation of a ScriptScope object as well as the ScriptEngine object we already talked about. This ScriptScope can be thought of as the __main__ module for the execution of the Python code. With IronPython you can create multiple ScriptScopes per engine and execute your code in any of them and they are partially self-contained.

      The code is also creating and setting up the stdout and stderr handlers for the ScriptEngine. This was an easy way to run the unit tests and get their output (since the default test running just prints the results to the console using the Python print statement). The OutputStream implementation is fairly simple and is shown below.

      [gist id=1765694 file=OutputStream.cs]

      It uses events to send text to the main form when something is written to the stream. The InitializeEngine then sets the stdout and stderr for the ScriptEngine; now all text written to either stdout or stderr will be redirected and displayed in the output area.

      Now that an engine is setup, and the stdout and stderr output is redirected, the InitializeEngine method pre-imports a few modules so that the script writer doesn’t need to do so. The first three (sys, os, and unittest) are standard Python modules that come in the Lib directory of the IronPython distribution. The last one is a helper module used to help enumerate and execute the unit tests.

      [gist id=1765694 file=testermatic.py]

      The first item defined is a class which wraps around unittest tests that are found. It provides a mechanism to add listeners (callbacks) for the end of a test. This is how the GUI is updated when a test completes (color change for pass/fail status, etc.).

      The second item (getTests) is a method which uses the unittest module to find all of the unittest compatible tests in the loaded modules. This is one of the key things to remember about embedding IronPython: if you can do something much easier in Python code than trying to pull out variables and call Python methods from C#, then do it. Write a nice little wrapper method that you can easily call from C# and have it do most of the work. This will save you a lot of time with trying to get things to work. Python is great at interrogating Python code, use it to its full benefit.

      The last item (runTests) is the method that is called by the host application to actually run the tests. It receives, from C#, a list (Python list) of tests to run. If there is a completion callback passed in, it wraps up the test case with the TestermaticTestCase wrapper and adds it to the test suite. Then, to run the tests, its as simple as passing the list off to the TextTestRunner class from the unittest module.

      So, let’s look at how all of this gets pulled together.

      Testermatic Toolbar</a>

       

       

      The toolbar on the Testermatic application has four buttons. The first is used to load a new set of tests. It shows an OpenFileDialog and once the user selects a Python file (*.py) it will enumerate the tests in the module and populate the list of tests. It uses the method below to load the tests.

      [gist id=1765694 file=LoadTest.cs]

      The Import method is as follows

      [gist id=1765694 file=Import.cs]

      The second button on the toolbar is used to actually run the tests. It creates a List (Python list) using the items in the ListView that are checked (remember the test object -- the Python test object -- was assigned to the .Tag property of each ListViewItem so it is easy to pull out).

      [gist id=1765694 file=RunTests.cs]

      You can see that this is interacting with the Python code in a different way than was done to retrieve the list of tests. This retrieves a ScriptScope (think module) object for the testermatic Python module shown earlier. It then gets the runTests method from that module and since it is a dynamic, it can call it just like a function. The TestCompleteDelegate is the callback for when the test completes.

      [gist id=1765694 file=TestCompleteDelegate.cs]

      The TestComplete method which is called just updates the UI based on which tests passed, which tests failed and which tests had an error (green for pass, red for fail, purple for error conditions).

      The third item on the toolbar is just a reload button. This is helpful if you are editing the Python script containing the tests and you want to reload to get any new tests, or new changes you made to the file. This calls the same LoadTests method that was called when the file was selected. A new ScriptEngine instance and a new ScriptScope instance are created (and cleaned up if already created) every time the LoadTests method is called.

      The final button in the toolbar is a simple little email capability, because when something fails its always nice to notify people about it. The button displays an email form as shown below to allow the user to enter To and From addresses as well as a message. The output from the tests will be added after the message and sent to both the To and From addresses.

      Email Form</a>

      Now, do I use the application at work to run normal Python unit tests? No, as I mentioned before, this was mainly to allow the hardware team at my work to develop simple tests to quickly triage issues with the boards. They can quickly modify the test for special circumstances and run the whole suite of tests again. It also allows them to send the report to another person so they can review the test run.

      I hope this simple little application has shown you how easy it is to incorporate IronPython into your application. I attached the project for this application below so you can download it and play around with it.

      Testermatic Project</a>