Sometimes it’s handy to write Test Anything Protocol from a batch script.

The lessons I’ve learned are going to come…

File Format

The file format should pretty much follow this:

:: Test Description
@echo off
if defined DEBUG echo on
:: see Debugging
:INITIALISE
setlocal
set _me=%0
set _tmpfile=%_me:.bat=%.tmp
:: or
set _tmpfile=%tmp%\%_me:.bat=%.tmp
set /A _test_number=0
:TAP_PLAN
@echo TAP version13
@echo # Test {test-number}
@echo # {test-description}
@echo # Test run at %date% %time%
@echo 1 .. {n}
:TESTS
:: do some tests of the general form
set /A _test_number=_test_number+1
:: do some action and check the result
if ERRORLEVEL 1 (
echo not ok %_test_number% – {test-point-description}
set error={meaningful-test-message}
goto BAIL_OUT!
) else (
echo ok %_test_number% – {test-point-description}
)
…
:: more tests
…
:END_OF_TESTS
endlocal
exit /B 0
:BAIL_OUT!
echo Bail Out! %_error%. See %_tmpfile% for details.
:: restart any services that were stopped
endlocal
exit /B 1

Sample Template

It doesn’t stop services, but here’s a simple test-template.bat.

Test Logic

When checking ERRORLEVEL, e.g. in statements of the form “if ERRORLEVEL n” the test returns true if the ERRORLEVEL is equal to or greaterr than the number specified. As the de-facto standard is for scripts to return 0 on success and 1 or greater for failure then it isn”t possible to test for failure in the following manner:

if ERRORLEVEL 0 (
:: will always run as ERRORLEVEL is always 0 or greater
echo ok
) else (
:: will never reach here
echo not ok
)

It isn’t necessary to test for failure when checking for the existence of files or evnvironment variables. For instance the following will work:

if defined ENVIRONMENT_VARIABLE (
echo ok
) else (
echo not ok
)
if exist {filename} {
echo ok
} else {
echo not ok
}

Because of the inconsistency with the ERRORLEVEL tests it helps to test for failure first:

if not defined ENVIRONMENT_VARIABLE (
echo not ok
) else (
echo ok
)
if not exist {filename} (
echo not ok
) else (
echo ok
)

Capturing Command Output

Output of commands that may be of assistance in post-test analysis can be directed to a temporary file named for the script:

mkdir foo> %_tmpfile% 2>&1

Starting and Stopping Services

If any services are required to be stopped by the test then the service should be restarted before the test exist.

net stop “{service-name}” >%_tmpfile% 2>&1
:: ERRORLEVEL is stored in a temporary variable so
:: service can be restarted if necessary
set _restart_web=%ERRORLEVEL%
if %_restart_{service}% EQU 0 (
echo # {service} stopped – will restart once test complete
) else (
echo # {service} already stopped
)
:: do the tests
…
:: restart service if stopped
if %_restart_{service}% EQU 0 (
echo # restarting {service}
net start “{service-name}” >%_tmpfile% 2>&1
)
:: restart service at Bail Out if stopped
:BAIL_OUT!
echo Bail Out! %_error%. See %_tmpfile% for details.
:: restart any services that were stopped
if defined _restart_web (
if %_restart_{service}% EQU 0 (
echo # restarting {service}
net start “{service-name}” >%_tmpfile% 2>&1
)
)
endlocal
exit /B 1

Debugging

If the DEBUG environment variable is defined the script will intermingle TAP output and command flow, which may assist in debugging.

Invocation

Direct

$ test.bat
TAP version 13
# Test {n}
# {test-description}
# Test run at: {date} {time}
1..{n}
ok 1
…
ok {n}
$

Scheduled

Wrap the test(s) in a run-tests.bat script. This example assumes that the test scripts are located in \Testdata\Scripts and the results are going into \Testdata\Results:

:: Wrapper script for running tests
@echo off
:: Use full path to tests so that tests can be scheduled
:: and the results don’t disappear into the ether
if not exist \TestData\Results\ (
mkdir \TestData\Results\
)
cd \Testdata\Scripts
call \Testdata\Scripts\test-{x}.{y}.bat > \TestData\Results\test-{x}.{y}.tap
call \Testdata\Scripts\test-{x}.{z}.bat > \TestData\Results\test-{x}.{z}.tap
move /Y \Testdata\Scripts\*.tmp \TestData\Results\
@echo on

Note the use of call so that exit from a test script doesn’t exit from the wrapper.

Then to schedule the tests the run-tests.bat script is further wrapped by a schedule-tests.bat script that contains the schedule details:

:: Schedule tests to run in the future, e.g. overnight
net start “task scheduler”
at 20:00 /every:M,T,W,Th,F %cd%\run-tests.bat

Note that the task scheduler must be provided with the full path to the run-tests.bat script.

Analysis

TAP output can then be analysed as per TAP data from any other producer.