Testing

The CherryPy test suite has been solid for many years now. There are lots of tests for both basic and advanced functionality. There are command-line options to control which tests are run, and the output they produce. There are debugging tools specifically for web page tests. Finally, many of the test suite components are reusable by your applications, so you can start developing your own CherryPy applications using all of these benefits.

Running CherryPy tests

If you look in the cherrypy/test directory, you'll see a number of test files. You may certainly run any of those scripts on their own. However, the "test.py" script allows you to run any or all of them at once, and gives you some extra options.

Which tests to run

If you wish to run all of the tests in the suite, simply run test.py. If you wish to run a single test from the suite, provide the name of that test as a command-line argument. Multiple test names can be provided:

python cherrypy\test\test.py --test_core --test_baseurl_filter

Which servers to use

By default, test.py uses CherryPy's builtin WSGI server to run the test suite. You may optionally run the test suite using mod_python, mod_wsgi, or mod_fcgi (not all of these options are available in older versions of CherryPy). Example:

python cherrypy\test\test.py --server=modwsgi

In addition, many server modes can be run in HTTP/1.1 mode (the default), or in HTTP/1.0 mode by including the "--1.0" argument:

python cherrypy\test\test.py --1.0

Debugging tools

The CherryPy test suite has a module named "webtest.py", that does two things.

First, webtest helps when server errors occur. Web tests are hard to debug, because web test suites usually run in one process, sending HTTP requests to a separate server process. When errors occur on the server, they are often lost, because that information cannot be returned to the client. CherryPy, however, runs both the client and server sides of a test in the same process. Therefore, when an error is encountered on the server side of a test, we can print a nice traceback.

Second, once a response has been received by the client, it is tested against various assertions. Rather than simply fail the assertion and raise an error, webtest gives you an interactive prompt, with several options:

Show: [B]ody [H]eaders [S]tatus [U]RL; [I]gnore, [R]aise, or sys.e[X]it >>

Select Body, Header, or Status to show the content of the response. Large response bodies will be output in a Unix "more" style, 30 lines at a time (you can change this number via WebCase.console_height). Hit "q" to stop scrolling the body.

The "URL" option will show you which URL was requested most recently. Note that these webtest features are not threadsafe; you must run all such tests from a single thread (but that's normal).

The "Raise" option will proceed normally, and the usual AssertionError will be raised, stopping the test. If you want to continue the current test, choose "Ignore" and no error will be raised. Choose "eXit" to invoke sys.exit().

Code coverage

CherryPy includes a code-coverage tool. To include this output when running the test suite, use the "--cover" argument. Note that you cannot run the profiler (see below) at the same time. To use the coverage tool, you need to download "coverage.py" (either Gareth Rees' original implementation or Ned Batchelder's enhanced version), and place it in your PYTHONPATH.

python cherrypy\test\test.py --cover

When all tests have run, you'll get a report showing what percentage of the code was exercised, like this:

CODE COVERAGE (this might take a while).........................
Total: 3663 Covered: 2447 Percent: 66%

Statistics are collected for all Python modules, but reported only for the cherrypy package, by default. If you're using the coverage tool with your own CherryPy application, you'll want to report on your own package instead. Use the --basedir=path command-line argument:

python cherrypy\test\test.py --cover --basedir=myapp

If "path" is relative, it will be considered relative to the current working directory.

The coverage tool dumps its output into cherrypy/lib, and you can play with the data in an interactive session, if you like. A much easier way to see the results is to ask CherryPy to serve the data to you in your browser! Run cherrypy/lib/covercp.py, and browse the complete data at localhost:8080. This works just as well for your own applications (see below).

Profiling

CherryPy includes a profiling tool. To include this output when running the test suite, use the "--profile" argument. Note that you cannot run the coverage tool (see above) at the same time.

python cherrypy\test\test.py --profile

The profiler will dump its output into cherrypy/lib, and you can play with the data in an interactive session, if you like. A much easier way to see the results is to ask CherryPy to serve the data to you in your browser! Run cherrypy/lib/profiler.py, and browse the data at localhost:8080. This works just as well for your own applications (see below).

Using the test tools with your own applications

The webtest module doesn't reference CherryPy in any way. Feel free to use it with other web frameworks, or whatever web testing needs you have.

Here is an example test script for one of my own applications, "Mission Control" (minus some database-handling code):

import os, sys
localDir = os.path.dirname(__file__)
testConf = os.path.join(localDir, "test.conf")

from cherrypy.test import test


if __name__ == '__main__':
    # Place our current directory's parent (mcontrol/) at the beginning
    # of sys.path, so that all imports are from our current directory.
    curpath = os.path.normpath(os.path.join(os.getcwd(), localDir))
    sys.path.insert(0, os.path.normpath(os.path.join(curpath, '../../')))
    
    testList = ["test_root",
                "test_incident",
                "test_vehicle",
                "test_volunteer",
                "test_missiontrip",
                "test_project",
                # test_materialorder should come after project,
                # because it duplicates some calls (without asserting).
                "test_materialorder",
                # test_job should come after missiontrip and project,
                # because it duplicates some calls (without asserting).
                "test_job",
                ]
    test.TestHarness(testList).run(testConf)

I'm reusing the TestHarness? class from cherrypy/test/test.py, but with my own list of tests, and my own test config. I get all of the server options, plus coverage and profiling, for free.