thrift/test/README.md

Apache Thrift - integration test suite

This is the cross everything integration test suite for Apache Thrift.

Run

A. Using Make

The test can be executed by:

make cross

This starts the test.py script which does the real cross test with different transports, protocols and languages.

Note that this skips any language that is not built locally. It also skips tests that are known to be failing. If you need more control over which tests to run, read following section.

B. Using test script directly

Alternatively, you can invoke test.py directly. You need to runmake precross once before executing it for the first time.

For example, if you changed something in nodejs library and need to verify the patch, you can skip everything except nodejs itself and some reference implementation (currently cpp and java are recommended) like this:

./configure --without-c_glib --without-erlang --without-lua ...
make precross -j8
test/test.py --server cpp,java --client nodejs
test/test.py --server nodejs --client cpp,java

Another useful flag is --regex. For example, to run all tests that involve Java TBinaryProtocol:

test/test.py --regex "java.*binary"

Test case definition file

The cross test cases are defined in tests.json. The root element is collection of test target definitions. Each test target definition looks like this:

{
  "name": "somelib",

  "client": {
    "command": ["somelib_client_executable"],
    "workdir": "somelib/bin",
    "protocols": ["binary"],
    "transports": ["buffered"],
    "sockets": ["ip"],
  },
  "server": {
    "command": ["somelib_server_executable"],
    "workdir": "somelib/bin",
    "protocols": ["binary"],
    "transports": ["buffered"],
    "sockets": ["ip", "ip-ssl"],
  }
}

Either client or server definition or both should be present.

Parameters that are common to both client and server can be put to target definition root:

{
  "name": "somelib",

  "workdir": "somelib/bin",
  "protocols": ["binary"],
  "transports": ["buffered"],
  "sockets": ["ip"],

  "client": { "command": ["somelib_client_executable"] },
  "server": {
    "command": ["somelib_server_executable"],
    "sockets": ["ip-ssl"]
  }
}

For the complete list of supported keys and their effect, see source code comment at the opt of crossrunner/collect.py.

List of known failures

Since many cross tests currently fail (mainly due to partial incompatibility around exception handling), the test script specifically report for "not known before" failures.

For this purpose, test cases known to (occasionally) fail are listed in known_failures_<platform>.json where <platform> matches with python platform.system() string.

Currently, only Linux version is included.

FYI, the file is initially generated by

test/test.py --update-expected-failures=overwrite

after a full test run, then repeatedly

test/test.py --skip-known-failures
test/test.py --update-expected-failures=merge

to update the known failures, run

make fail

Test executable specification

Command line parameters

Unit tests for languages are usually located under lib//test/ cross language tests according to ThriftTest.thrift shall be provided for every language including executables with the following command line interface:

Server command line interface:

$ ./TestServer -h
Allowed options:
  -h | --help                  produce help message
  --port=arg (9090)            Port number to listen
  --domain-socket=arg          Unix Domain Socket (e.g. /tmp/ThriftTest.thrift)
  --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
  --server-type=arg (simple)   type of server, "simple", "thread-pool",
                               "threaded", or "nonblocking"
  --transport=arg (buffered)   transport: buffered, framed, http, anonpipe, zlib
  --protocol=arg (binary)      protocol: binary, compact, header, json
  --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
  --abstract-namespace         Create the domain socket in the Abstract Namespace 
                               (no connection with filesystem pathnames)
  --ssl                        Encrypted Transport using SSL
  --zlib                       Wrapped Transport using Zlib
  --processor-events           processor-events
  -n=arg | --workers=arg (=4)  Number of thread pools workers. Only valid for
                               thread-pool server type

Client command line interface:

$ ./TestClient -h
Allowed options:
  -h | --help                  produce help message
  --host=arg (localhost)       Host to connect
  --port=arg (9090)            Port number to connect
  --domain-socket=arg          Domain Socket (e.g. /tmp/ThriftTest.thrift),
                               instead of host and port
  --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
  --anon-pipes hRead hWrite    Windows Anonymous Pipes pair (handles)
  --abstract-namespace         Create the domain socket in the Abstract Namespace
                               (no connection with filesystem pathnames)
  --transport=arg (buffered)   Transport: buffered, framed, http, evhttp, zlib
  --protocol=arg (binary)      Protocol: binary, compact, header, json
  --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
  --ssl                        Encrypted Transport using SSL
  --zlib                       Wrap Transport with Zlib
  -n=arg | --testloops=arg (1) Number of Tests
  -t=arg | --threads=arg (1)   Number of Test threads

If you have executed the make check or make cross then you will be able to browse gen-html/ThriftTest.html with the test documentation.

Return code

The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors. In order to signal failed tests, the return code shall be composed from these bits to indicate failing tests:

  #define TEST_BASETYPES     1  // 0000 0001
  #define TEST_STRUCTS       2  // 0000 0010
  #define TEST_CONTAINERS    4  // 0000 0100
  #define TEST_EXCEPTIONS    8  // 0000 1000
  #define TEST_UNKNOWN      64  // 0100 0000 (Failed to prepare environment etc.)
  #define TEST_TIMEOUT     128  // 1000 0000
  #define TEST_NOTUSED      48  // 0011 0000 (reserved bits)

Tests that have not been executed at all count as errors.

Example:

During tests, the test client notices that some of the Struct tests fail. Furthermore, due to some other problem none of the Exception tests is executed. Therefore, the test client returns the code 10 = 2 | 8, indicating the failure of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS).

SSL

Test Keys and Certificates are provided in multiple formats under the following directory test/keys