Testing TUI applications in Python

When I decided to develop a curses-based TUI for my latest programming project, coBib, I knew from the beginning that I will need to test the features of the TUI. However, I came to realize that achieving this feat is not as straight forward as some of the other unittests which I had implemented for this project previously. Thus, in this post, I summarize how to test a TUI application written in Python.

What are we actually trying to do here?

I will start by addressing this very important question right in the beginning. Well, the answer is actually quite simple: we want to unit test as many features of an application as possible. Now, you might ask, what unit testing actually means. To put it in the words of Wikipedia:

<…> unit testing is a software testing method by which individual units of source code <…> are tested to determine whether they are fit for use.

Ah… Essentially this boils down to: “we want to test our source code in small logical pieces rather than as one huge black box”. In this way, we can ensure that the individual blocks which make up our code work as intended. Once that is the case, we can add some integration or system tests to ensure that all of our features combine and interact correctly, too.

Testing in Python

When programming in Python you have a variety of testing frameworks to choose from (e.g. unittest, pytest, doctest). For coBib I chose to go with pytest but what I present in this article should work with any of those testing frameworks.

I do not want to go into the details of how pytest works but will show you a very short example test case:

from cobib import zsh_helper
import cobib

def test_list_commands():
    """Test listing commands."""
    cmds = zsh_helper.list_commands()
    cmds = [c.split(':')[0] for c in cmds]
    expected = [cmd.replace('Command', '').lower() for
                cmd in cobib.commands.__all__]
    assert sorted(cmds) == sorted(expected)

The unittest above asserts that the zsh utility function list_commands() returns all commands that are available in coBib. If the assert statement on the last line fails, so does the test.

Testing a TUI application

Now, let us assume that we have unittests in place for all of the basic features of our application. However, we have written a new, fancy TUI which combines all of these features. If we want to avoid manually testing all possible workflows in the TUI when integrating new changes, it would be great to test the TUI itself, too. However, this is not as straight forward since we don’t have direct access to the terminal contents.

This is where pyte comes into play! What is pyte, you ask?

It’s an in memory VTXXX-compatible terminal emulator.

This means, pyte allows us to emulate a terminal within which we can run our TUI. This provides us with a programmatic access to the terminal contents, i.e. it allows us to scrape the screen for its contents and assert what is being displayed!

How to setup pyte

The tutorial page of pyte’s documentation shows how straight forward it is to initialize a virtual terminal window:

import pyte
screen = pyte.Screen(40, 12)
stream = pyte.Stream(screen)
stream.feed(b"Hello World!")
print(screen.display)

The last line will dump the contents of the terminal’s screen like below:

Hello World!                            
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        

This appears to be working just as expected!

Combining pytest and pyte

After a significant amount of research on the web, diving through many Stackoverflow pages and even reaching out to one of the pytest developers on IRC, I finally managed to come up with a solution that seems to be flexible enough for all the testing purposes which I have encountered so far.

def test_tui():
    """Test TUI."""
    # create pseudo-terminal
    pid, f_d = os.forkpty()
    if pid == 0:
        # child process spawns TUI
        curses.wrapper(TUI)
    else:
        # parent process sets up virtual screen of
        # identical size
        screen = pyte.Screen(80, 24)
        stream = pyte.ByteStream(screen)
        # scrape pseudo-terminal's screen
        while True:
            try:
                [f_d], _, _ = select.select(
                    [f_d], [], [], 1)
            except (KeyboardInterrupt, ValueError):
                # either test was interrupted or the
                # file descriptor of the child process
                # provides nothing to be read
                break
            else:
                try:
                    # scrape screen of child process
                    data = os.read(f_d, 1024)
                    stream.feed(data)
                except OSError:
                    # reading empty
                    break
        for line in screen.display:
            print(line)
        # now, do some assertions (see later)

That’s quite a test function… So let’s break it down:

1. On line 4 we are forking the process into a parent and child process. This is necessary because otherwise our TUI application will take full control of the process and disallow us from actually running any tests on it.
2. On lines 5 and 8 we differentiate between the parent and child process, starting our TUI application on the child process¹ (line 7).
3. In the parent process we can then use pyte to initialize a pseudo-terminal (as explained in How to setup pyte).
4. The endless loop starting on line 14 is then used to scrape the terminal content. We do so by means of the select.select() method which waits for the file descriptor, f_d, until it is ready for reading.
   1. If nothing is available for reading or the test interrupts, we break the loop on line 22 which will eventually lead to a failing test (that is, once we add some assertions as shown below).
   2. The else clause of the try block will execute if no exception was raised. Here, we simply scrape the screen of the child process and feed its contents into the pseudo-terminal. The reason for not directly operating on the data is to also allow easier processing of terminal attributes such as colors, etc.
5. Finally, on line 33 we can start adding some assert statements to check the contents of the terminal window. Note, that I print the contents of the screen prior to this because this will allow easy debugging when a test fails².

Asserting the pseudo-terminal contents

Now, it is finally time to assert the pseudo-terminal contents! I like to parametrize my test functions which allows me to run the same test with different inputs. In the case of this TUI test it makes sense to have function arguments for the key strokes which should be send to the TUI and for the assertion function which should be used to assert the outcome. This will look something like this:

@pytest.mark.parametrize(['keys', 'assertion'], [
        ['?', assert_help_screen],
    ])
def test_tui(keys, assertion):
    """Test TUI.

    Args:
        keys (str): keys to be send to the TUI.
        assertion (Callable): function to run the
                              assertions for the keys
                              to be tested.
    """
    pid, f_d = os.forkpty()
    if pid == 0:
        curses.wrapper(TUI)
    else:
        screen = pyte.Screen(80, 24)
        stream = pyte.ByteStream(screen)
        ### SEND KEYS
        # send keys char-wise to TUI
        for key in keys:
            os.write(f_d, str.encode(key))
        ### END
        while True:
            try:
                [f_d], _, _ = select.select(
                    [f_d], [], [], 1)
            except (KeyboardInterrupt, ValueError):
                break
            else:
                try:
                    data = os.read(f_d, 1024)
                    stream.feed(data)
                except OSError:
                    break
        for line in screen.display:
            print(line)
        ### ASSERT OUTCOME
        assertion(screen)
        ### END

As you can see, compared to before we now have two additional sections:

1. On lines 21-22 we iterate a string of key strokes and send them char-wise to the child process, triggering events in the TUI application.
2. And finally, on line 39 we have added a call to an assertion function which is also provided as an input argument. In this specific example the function looks something like this:

   def assert_help_screen(screen):
    """Asserts the contents of the Help screen."""
    assert "coBib TUI Help" in screen.display[2]
    for cmd, desc in TUI.HELP_DICT.items():
        assert any("{:<8} {}".format(cmd+':', desc) in
                   line for line in screen.display[4:21])
   

   This function is just meant as an example to showcase how you could go about asserting the contents of your terminal are correct.

Further extensions

That is all for now! I do have a few more features in my coBib testing suite which would have made this post even longer. So please, feel free to check them out! Some examples of what you will be able to find are:

• using pytest fixtures and executing teardown code
• passing additional kwargs to the assertion function (take a look at my test_tui())
• testing terminal color attributes (take a look at test_tui_config_color())
• testing resize events (take a look at test_tui_resize())

1. The os.forkpty() method assigns the pid 0 to the child process. ↩

2. pytest will only print to stdout when a test fails unless you run the tests verbosely. ↩