Test definitions in version control¶
LAVA supports git version control for use with test definitions, and this is the recommended way to host and use test definitions for LAVA. When a repository is listed in a test definition, the entire repository is checked out. This allows YAML files in the repository to reliably access scripts and other files which are part of the repository, inside the test image.
- test: role: - server - client definitions: - repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/multi-node/multinode02.yaml name: multinode-intermediate
When this test starts, the entire repository will be available in the current
working directory of the test. Therefore,
include instructions to execute
Job definitions in version control¶
It is normally recommended to also store your test job YAML files in the repository. This helps others who may want to use your test definitions.:
There are numerous test repositories in use daily in Linaro that may be good examples for you, including:
Using specific branch of a test definition repository¶
If a public branch is specified as a parameter in the job submission YAML, that branch of the repository will be used instead of the default ‘master’ branch.
- test: timeout: minutes: 5 definitions: - repository: https://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/android/get-adb-serial-hikey.yaml name: get-hikey-serial branch: stylesen
Do not supply anything other than a branch name to this parameter, like tag or revision as this would create a clone in a ‘detached HEAD’ state.
Using specific revisions of a test definition¶
If a specific revision is specified as a parameter in the job submission YAML, that revision of the repository will be used instead of HEAD.
- test: failure_retry: 3 timeout: minutes: 10 name: kvm-basic-singlenode definitions: - repository: git://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/smoke-tests-basic.yaml name: smoke-tests - repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced revision: 441b61
Shallow clones in GIT¶
Some git repositories have a long history of commits and using a full clone takes up a lot of space. When a test job involves multiple test repositories, this can cause issues with adding the LAVA overlay to the test job. For example, ramdisks could become too large or there could be insufficient space in the partition used for the test shell or it could take longer than desired to transfer the overlay to the device.
git support is requested for a test shell definition, LAVA will
default to making a shallow clone using
--depth=1. The git history
will be truncated to the single most recent commit.
A full clone can be requested by passing the
shallow parameter with a
False. If the
revision option is used, shallow clone
support will need to be turned off or the change to specified revision
- test: failure_retry: 3 timeout: minutes: 10 name: kvm-basic-singlenode definitions: - repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced shallow: False
Removing git history¶
The size of the overlay can be an issue for jobs running on small devices.
By default, when cloning test definition from a git repository, LAVA will keep
the .git directory.
If needed, this directory can be removed from the overlay by setting
history to false.
- test: failure_retry: 3 timeout: minutes: 10 name: kvm-basic-singlenode definitions: - repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced history: False
Test repository for functional tests in LAVA¶
LAVA regularly runs a set of test definitions to check for regressions and the set is available for others to use as a template for their own tests:
Using kernel messages in a test shell¶
Simple test job flow for background on serial corruption and interleaving of kernel messages with test output. Also Isolating kernel messages from test output for information on using multiple serial connections to prevent such issues.
In situations where multiple serial ports are not available, there is a possible mitigation to the problem of serial corruption of test shell messages by kernel messages.
This is an example of the interleaving of kernel messages and test shell messages:
<LAVA_SIGNAL_ENDRUN 0_network[ 31.811978] ------------[ cut here ]------------ -inline 14658_3.[ 31.818261] WARNING: CPU: 4 PID: 2576 at /srv/oe/build/tmp-rpb-glibc/work-shared/hikey/kernel-source/include/net/sock.h:1703 af_alg_accept+0x1d8/0x208 [af_alg] 2.3.1> <LAVA_TE[ 31.834094] Modules linked in: algif_hash af_alg smsc75xx usbnet adv7511 kirin_drm drm_kms_helper dw_drm_dsi drm fuse Received signal: <ENDRUN> 0_network[ 31.811978] ------------[ cut here ]------------ -inline 14658_3.[ 31.818261] WARNING: CPU: 4 PID: 2576 at /srv/oe/build/tmp-rpb-glibc/work-shared/hikey/kernel-source/include/net/sock.h:1703 af_alg_accept+0x1d8/0x208 [af_alg] 2.3.1 Ending use of test pattern. Ending test lava.0_network[ (31.811978]), duration 1.92 case: 0_network[ case_id: 875068 definition: lava duration: 1.92 namespace: hikey-oe path: None repository: None result: pass revision: unspecified uuid: 31.811978] _on_endrun() takes exactly 3 arguments (20 given)
The expected signal would have been more like the another message of this type in the same test job:
<LAVA_SIGNAL_ENDRUN 1_kselftest 14658_220.127.116.11> Received signal: <ENDRUN> 1_kselftest 14658_18.104.22.168 Ending use of test pattern. Ending test lava.1_kselftest (14658_22.214.171.124), duration 0.00 case: 1_kselftest case_id: 876176 commit_id: fd1f392f983803e7189e264dda22790c54950c8f definition: lava duration: 0.0057 namespace: hikey-oe path: automated/linux/kselftest/kselftest.yaml repository: git://git.linaro.org/qa/test-definitions.git result: pass revision: unspecified uuid: 14658_126.96.36.199 <LAVA_TEST_RUNNER>: 1_kselftest exited with: 0
As described in isolating kernel messages in
the section on Multiple serial port support, this happens because the
test shell message
<LAVA_SIGNAL_ENDRUN 1_kselftest 14658_188.8.131.52>
is sent to
stdout. If the message was sent to
the kernel would take care of emitting the entire line before (or
just after) emitting the warning. The test shell message would
remain intact and processing could continue normally.
Test writers can decide which test shell definitions need to use this
support and which continue to use
1 2 3 4 5 6 7 8 9 10
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git lava-signal: kmsg path: lava-test-shell/smoke-tests-basic.yaml name: smoke-tests # this test definition will use stdout - repository: https://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced
lava-signal: kmsg for the first test shell definition
in the test job submission example above, LAVA can output the test
shell messages to
/dev/kmsg, resulting in output like:
[ 8.862986] <LAVA_SIGNAL_ENDRUN 0_smoke-tests 1998_184.108.40.206> + export TESTRUN_ID=1_singlenode-advanced + TESTRUN_ID=1_singlenode-advanced + cd /lava-1998/0/tests/1_singlenode-advanced ++ cat uuid + UUID=1998_220.127.116.11 + set +x <LAVA_SIGNAL_STARTRUN 1_singlenode-advanced 1998_18.104.22.168> + apt-get update -q Received signal: <TESTCASE> TEST_CASE_ID=linux-linaro-ubuntu-lsb_release RESULT=fail case: linux-linaro-ubuntu-lsb_release case_id: 49920 definition: 0_smoke-tests result: fail Received signal: <ENDRUN> 0_smoke-tests 1998_22.214.171.124
The LAVA test shell messages are now prefixed with the kernel message time stamp - this does not affect processing which is restricted to the content between the
STARTRUNmessage is not sent to
/dev/kmsg(see Limitations) and is still vulnerable to kernel messages being interleaved.
stderris implicitly included.
The ordering of messages in the output shows the inherent latency in the message processing.
If the limitations with this approach make it unsuitable, Multiple serial port support is preferred, where available.
The login shell is not part of the LAVA test shell, it is the final part of the boot sequence and the input/output of the login shell is not under the control of LAVA. If kernel messages appear when the device is attempting to read in the username or password or if a message appears when the login shell is attempting to output the prompt, then the parsing will still fail.
The problem of interleaving messages from multiple inputs onto a single
output has not gone away by sending the LAVA test shell messages to
/dev/kmsg. There is still only a single serial connection which
must be shared between all
stdout output and all
output. Test shell operations which produce a lot of
stderr output can still flood the serial connection and interrupt
kernel message lines.
Not all LAVA test shell operations can be sent to
/dev/kmsg - some
will go to
stdout even if all test definitions are marked to use
/dev/kmsg because the test shell runner script needs to be the same
for all definitions.