Remote Testing with Dejagnu

DejaGnu is the testing framework used by the GCC, binutils, and gdb projects.

For local testing of native toolchains, you don't need to know a thing about how to configure DejaGnu; just type 'make check' in the gcc directory, and gcc's Makefile will run the gcc test suite.

Remote testing of cross-toolchains, on the other hand, has never been well-documented. The documentation even states in several places that you can use telnet to execute remote tests, when in fact only rsh and ssh can be used with DejaGnu to run gcc regression tests remotely. Like many others, I had trouble figuring out how remote testing works with DejaGnu. I wrote this document as a way of figuring it out myself, and hope this will save others some trouble.

What follows is a simple recipe to illustrate how to get remote testing of noninteractive programs working. Once you have this working, it should be fairly easy to do remote gcc testing. Remote testing of interactive tools like gdb is beyond the scope of this example.

When DejaGnu misbehaves:

Read through the log file, testrun.log, created by runtest
If you need to make the log file more verbose, add up to four -v's to the runtest command line
If that's not enough, approach the problem from a position of strength. Read up on the TCL programming language, then read the source code for DejaGnu (usually, /usr/share/dejagnu/*.exp).
To track down what code is executing, sprinkle 'puts' statements throughout the .exp scripts that make up DejaGnu and rerun, redirecting the output of runtest to a file; then look at the file for the output from those puts statements
If you've very lucky, you can start the built-in TCL debugger by giving runtest the -D1 flag. However, when I tried it, it was horribly broken.
The unix strace or truss utilities can be very handy for seeing what dejagnu is doing. To try this, run runtest via strace, e.g. 'strace -f -o foo.log runtest ...' Then use grep to pick out the events of interest, e.g. 'grep open foo.log' to display just the 'open' system calls, or 'grep exec foo.log' to display just what other programs it tries to run.

The example assumes you have already installed DejaGnu. DejaGnu consists largely of the shell script /usr/bin/runtest, which invokes the TCL script /usr/share/dejagnu/runtest.exp. If you give the command 'runtest --version', you should see something like this:

Expect version is       5.32.2
Tcl version is          8.3
Framework version is    1.4.3

(Don't worry if it complains that it can't find a global config file.)

If your framework version is lower than 1.4.3, you may not get the same results as I did. In particular, I've noticed that the DejaGnu installed on Debian 3.0, which reports 1.4.2.x, behaves differently (it uses the board name rather than the IP address or hostname set in the board file when communicating via rsh, which won't work).

It's probably a good idea to install the latest versions of tcl, expect, and dejagnu from source. See also gcc bug 12096 for a patch to expect that might be required to avoid occasional truncated results.

Files

The example consists of five files:

The C Program, simple.c

This is the program we're going to test. All it does it print out the hostname of the computer it's running on, and return the integer value of its first argument.

#include <stdio.h>
int main(int argc, char **argv)
{
    char buf[256];
    buf[0]=0;
    gethostname(buf, sizeof(buf));
    printf("hostname is %s\n", buf);
    return atoi(argv[1]);
}

The Test Script, simple.exp

This is the TCL script that will compile, run, and judge whether the C program managed to run and return the expected exit status.

# simple: compile and run a C program (possibly remotely) with various arguments

print "Compile simple.c"
target_compile simple.c simple executable -O2

print "simple.1: Run simple with '0' argument; it should exit with zero status, causing remote_load to return 'pass'."
set result [remote_load target simple 0]
if {[lindex $result 0] == "pass"} {
   pass "simple.1"
} else {
   fail "simple.1"
}

print "simple.2: Run simple with nonzero argument; it should exit with nonzero status, causing remote_load to return 'fail'."
set result [remote_load target simple 17]
if {[lindex $result 0] == "fail"} {
   pass "simple.2"
} else {
   fail "simple.2"
}

print "simple.3: Run simple without arguments; it should crash, causing remote_load to return 'fail'."
set result [remote_load target simple]
if {[lindex $result 0] == "fail"} {
   pass "simple.3"
} else {
   fail "simple.3"
}

The TCL command target_compile used in simple.exp is part of DejaGnu; it looks up how to compile a C program, and runs the compiler.

The TCL command remote_load used in simple.exp uploads the given file to the target machine and runs it, then returns whether it successfully ran.

The site configuration file, site.exp

This is read by runtest on startup. All we need to put in here for this example is a command to tell runtest our board files live in the current directory:

lappend boards_dir "."

The board definition files

When you tell runtest to run a test on a remote machine, you use the option "--target_board=foo", which makes runtest load the board definition file foo.exp to see how to compile, upload, and run programs for/to/on that board.

By default, dejagnu uses rsh for remote execution, and rcp for uploading. Here's an example for that case:

# Board definition file for board "myboard_rshrcp"
load_generic_config "unix";
set_board_info hostname dank2
set_board_info username dank
set_board_info compiler /usr/bin/gcc

If you want instead to use ssh for remote execution and scp for uploading, you need to change this a bit. Here's an example, myboard_sshscp.exp:

# Board definition file for board "myboard_sshscp"

# How to compile C programs for this board
set_board_info compiler /usr/bin/gcc

# Network address of board
set_board_info hostname dank2

# How to log into this board via ssh and copy files via scp.
# Ideally, you'll set it up to not need a password to log in via ssh
# (see e.g. http://www-csli.stanford.edu/semlab/muri/system/howto/ssh.html).
set_board_info username dank

set_board_info shell_prompt    "dank2>" 
# For DejaGnu 1.4.3 and above; DejaGnu 1.4.2.x (Debian 3.0) ignores these settings!
set_board_info rsh_prog /usr/bin/ssh
set_board_info rcp_prog /usr/bin/scp

If you want instead to use rsh for remote execution and ftp for uploading, you need to change this a bit. Here's an example, myboard_rshftp.exp:

# Board definition file for board "myboard_rshftp"

load_generic_config "unix";

# How to compile C programs for this board
set_board_info compiler /home3/dank/crosstool-0.7/result/ppc-750-linux-gnu/gcc-3.3-glibc-2.3.2/bin/ppc-750-linux-gnu-gcc

# Network address of board
# For DejaGnu 1.4.3 and above; DejaGnu 1.4.2.x (Debian 3.0) ignores this setting!
set_board_info hostname 11.3.1.1

# How to log into this board via rsh and transfer files via ftp.
set_board_info username root
set_board_info file_transfer   ftp
set_board_info ftp_username    root
set_board_info ftp_password    "DaisyMae"

You'll need to edit these files a bit; see below.

The Recipe

Run a local test

Run a local test with the command

runtest simple.exp

(Note that for local testing, the current directory should be in your path, else runtest will not find your executables. You can force this for just one run with PATH=.:$PATH runtest simple.exp if need be.)

All three tests in simple.exp should pass. You should see a bunch of output ending with

                ===  Summary ===

# of expected passes            3

Take a look at the log file testrun.log left behind by runtest. It should contain what you saw on stdout, plus a bit more detail. In particular, note the three runs of 'simple.c':

hostname is dank
Exiting with status 0
PASS: simple.1
hostname is dank
Exiting with status 17
PASS: simple.2
PASS: simple.3

Only the output from the first two is visible. (It should show the hostname of your workstation.) The third run crashed as expected, so its output was lost.

Run a remote test via ssh and scp

Pick a remote machine of the same type as your workstation; let's say its hostname is 'dank2'.

Set up the remote machine to let you do ssh and scp without entering a password (see one of the many documents on the net describing how to do this if you haven't done this before.)
Verify that 'ssh dank2 ls' shows you a list of files on the remote machine without prompting you for a password
Verify that 'scp /etc/hosts dank2:' copies that file to dank2 without prompting for a password.
edit myboard_sshscp.exp to reflect the hostname, username, and password for that machine.
Finally, run a remote test on that machine with the command
```
runtest --target_board=myboard_sshscp simple.exp
```

You should again see output ending with

===  Summary ===

# of expected passes            3

Take a look at the log file testrun.log left behind by runtest. On my system, it looks like this:

Executed simple, status 0
RSA host key for IP address '192.168.3.104' not in list of known hosts.
hostname is dank2
Exiting with status 0
PASS: simple.1
Executing on myboard_sshscp: /tmp/simple.25935 17   (timeout = 300)
Executing on myboard_sshscp: rm -f  /tmp/simple.25935    (timeout = 300)
Executed simple, status 1
RSA host key for IP address '192.168.3.104' not in list of known hosts.
hostname is dank2
Exiting with status 17
PASS: simple.2
Executing on myboard_sshscp: /tmp/simple.25935    (timeout = 300)
Executing on myboard_sshscp: rm -f  /tmp/simple.25935    (timeout = 300)
Executed simple, status 1
RSA host key for IP address '192.168.3.104' not in list of known hosts.
PASS: simple.3

It must have run on the remote machine, since the output from simple says 'hostname is dank2'.

Run a remote test via rsh and ftp

This is exactly the same as the previous step, but this time replacing ssh and scp with rsh and ftp.

Set up a remote machine of the same type as your workstation to allow executing commands via rsh, and saving files via ftp. Let's say it's at ip address 11.3.1.1. Verify that 'rsh 11.3.1.1 ls' gives you a list of files on the remote machine, and that 'ftp 11.3.1.1' lets you upload files to that machine. Then edit myboard_rshftp.exp to reflect the hostname, username, and password for that machine.

Finally, run a remote test on that machine with the command

runtest --target_board=myboard_rshftp simple.exp

What now?

Once you have the above working, go try to run remote gcc regression tests; I believe the way to coax gcc's Makefile into running the tests remotely is by setting the RUNTESTFLAGS environment variable, e.g.

RUNTESTFLAGS="--target_board=myboard_sshscp" make check

but you will need to put your board config files in a more standard place for runtest to find them. Consult the DejaGnu doc for details :-) or see my crosstool-howto.