FATE Automated Testing Environment

1. Introduction

FATE provides a regression testsuite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

2. Running FATE

2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass `--samples=<path to the samples>' to configure or set the SAMPLES Make variable or the LIBAV_SAMPLES environment variable to a suitable value.

To use a custom wrapper to run the test, pass `--target-exec' to configure or set the TARGET_EXEC Make variable.

The dataset is available through rsync, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite

# make fate-rsync SAMPLES=fate-suite

3. Manual Run

FATE regression test can be run through make. Specific Makefile targets and Makefile variables are available:

3.1 FATE Makefile targets

`fate-list'

List all fate/regression test targets.

`fate-rsync'

Shortcut to download the fate test samples to the specified testsuite location.

`fate'

Run the FATE test suite (requires the fate-suite dataset).

3.2 FATE Makefile variables

`V'

Verbosity level, can be set to 0, 1 or 2.

`0'

show just the test arguments

`1'

show just the command used in the test

`2'

show everything

`SAMPLES'

Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.

`THREADS'

Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.

`THREAD_TYPE'

Specify which threading strategy test, either slice or frame, by default slice+frame

`CPUFLAGS'

Specify a mask to be applied to autodetected CPU flags.

`TARGET_EXEC'

Specify or override the wrapper used to run the tests.

`GEN'

Set to 1 to generate the missing or mismatched references.

    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate

4. Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, tests/fate.sh is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to http://fate.libav.org/ or directly examined locally.

4.1 Testing Profiles

The configuration file passed to fate.sh is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description
build_only=     # set to "yes" for a compile-only instance that skips tests

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'

4.2 Special Instances

The TARGET_EXEC option provides a way to run FATE wrapped in valgrind, qemu-user or wine or on remote targets through ssh.

4.3 Submitting Reports

In order to send reports you need to create an ssh key and send it to root@libav.org. The current server fingerprint is a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6

Table of Contents

About This Document

This document was generated on June, 29 2015 using texi2html 1.76.

The buttons in the navigation panels have the following meaning:

where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:

• 1. Section One
  • 1.1 Subsection One-One
    • ...
  • 1.2 Subsection One-Two
    • 1.2.1 Subsubsection One-Two-One
    • 1.2.2 Subsubsection One-Two-Two
    • 1.2.3 Subsubsection One-Two-Three     <== Current Position
    • 1.2.4 Subsubsection One-Two-Four
  • 1.3 Subsection One-Three
    • ...
  • 1.4 Subsection One-Four