Status

Using autotest
- patch to autotest_kvm includes: (all under client/tests/kvm unless stated otherwise)
  - spice support (cmd line options, git building with spice-protocol and spice libs); doesn't give nice warnings about missing requirements (log4cpp, cel051, CEGUI)
- tests/display_whql.py: ratetest (from WHQL tests, not sure about distributability, may need a rewrite).
no parsing to test results (but return code might be enough)
no blue screen capture
- may need to make sure machine doesn't reboot on bluescreen first (bcdedit / something else in xp)

WindowsAutomatedTestNotes - tidbits about doing automated tests with windows guests

[edit]

Planned Work

push screendump patch to upstream
capture of tcpdump:
- options to turn on tcpdump capture for any test, for instance for display_whql test.
- specify which ports, interfaces (generally a tcpdump filter expression)
- where to log it.
- post process it
- where to log postprocessed results.
capture of bluescreen:
- how do I trigger a bluescreen (for testing of this feature)?
- capture based on screendump
- where to log captured image, make sure links created from autotest results.
starting spicec
- specify when to start it (timeout, retries maybe)
- on which host and how (linux - ssh, windows - some remoting option? can also run it under another guest for maximal ease of tracking)
  - actually starting another vm for this should be relatively easy.
automated building of drivers
- linux / windows
- windows
  - under another vm (standard) or dedicated windows machine.
  - build script should do it. (i.e. build.cfg)

Nitpicks

crash in autotest produces a dump (should) - with utrace it is possible to do suspend and attach a debugger instead.

[edit]

Requirements

Test parameters:

Images
test scripts
spice(server+qemu) versions (git)
qxl driver version (git)
spice client version (git)
client platform
network parameters (bandwidth, latency)
settings
- guest: bit depth, effects
- client: canvases (gdi/GL/cairo)
- spice server: qxl ram, compression types, ssl on/off
load at server
load at client

Measures:

passing whql tests
bandwidth utilization
number of packets, average packet size, max packet size
cpu usage (server, client)
server statistics (see below)
full command history (=> tree history)
features utilized (i.e. test running video, did spice notice it's a video)
rendering correctness (on server, on client)
- client and server fb identical
  - freeze server and client (see below)
  - snapshot from server (qemu already has this?)
  - snapshot on client (windows, linux)
  - compare
compare to rdp

Test scripts:

correctness
- driver installation
- whql, cairo testsuite, rendercheck
  - display test for qxl
  - general test for qxl
  - general test for spice-vmc
- video
  - s-cielab
- audio
performance
- guest scripted app loads
  - office
  - web browsing
  - arcgis
  - qcad
  - video
  - audio
- mouse and keyboard client side scripted
- 3d (when supported)
  - cad (autocad)

usage requirements:

being able to run on a user supplied git (to be used before push)
- running individual tests
matrix including web based reports
bugging someone when some test stops passing
capture backtraces and compile logs from broken compiles or crashed runs
- ulimit
being able to stop a test
progress report for running tests

[edit]

measurement implementation

compile spice with statistics support

dropped commands count.
...

tcpdump

[edit]

running a single test

client machine C, host H, image source, script runs on machine T (optionally T==H)

choose specific versions of client, server, driver. (optionally built from source)
choose image
inject driver to image
run guest G on H with image in snapshot mode
tell test agent running on T to run test t on G, H, C.
upload test results to test results database.

T ----- H, G

\----- C

[edit]

stage 1

whql test automated on latest git driver and server

does not require client at all
no need to capture anything, nor freeze
does require capture of whql results and a web frontend to show status
nightly (no hooks)
find some spare machines

X driver testing

cairo testsuite

After this

can record which operations were utilized
move to app testing (with client)

[edit]

automated build implementation

see windows builds
autobuild
tinderbox of mozilla
buildbot
other?

[edit]

freeze implementation

- client and server fb identical
  - freeze rendering support (server and client state)
    - server freeze
      - stop receiving guest commands
      - flush: send all non sent commands to the clients
      - wait for ack from the client
      - render on the server
    - client freeze
      - not required during server freeze

[edit]

client platforms

use actual hardware or vms?

windows, linux (future: mac)

[edit]

autotest kvm

using autotest client right now, cd autotest/client/tests/kvm, ../../bin/autotest_client control
needs to be root (creates some images with loopback mounting, annoyingly cannot be done unless root).
installation hints: (should be put back into the autotest-kvm wiki)
sudo yum install syslinux pxe-kexec qemu-kvm-tools
git clone git://github.com/ehabkost/autotest.git

[edit]

images

various windows versions, various linux versions

autotest does the whole image creation from scratch. That's not really needed for us - we are not testing kvm itself. Even though installation is a useful process to test from qxl pov because of multiple mode switches and reboots.

[edit]