Why have rules around disclosure?
Controlling the disclosure of test results generated by STAC Test Harnesses is essential to maintaining public confidence in the quality of STAC Benchmarks. If one or more vendors were to publish incomplete, biased, or otherwise inaccurate results, it could quickly undermine trust in STAC Benchmarks, which would undo much of the effort that the Council has invested to date.
What are the STAC Benchmark disclosure rules?
Users of STAC Test Harnesses agree to the following rules:
• Public disclosure of results requires the results to be audited. Only STAC and STAC-certified third parties are allowed to audit results. These parties charge fees for their audit services.
• Disclosure into the STAC Vault does NOT require the results to be audited. This eliminates the cash expense associated with disclosure, acting as an incentive for vendors to disclose results routinely for many product configurations. But Vault results are not without some checks and balances. While STAC does not conduct an audit, STAC does ensure that the vendor has supplied all the required test results and other information. Moreover, an unaudited Vault report can only be accessed by Council members who are also entitled to use the same STAC Test Harnesses and STAC Benchmark specifications that the vendor used to generate its results. That makes them educated and well-equipped consumers of test results. A vendor that cannot replicate its results for such a customer runs a significant reputational risk.
What does an audit consist of?
In an audit, STAC first verifies that the Test Harness configuration conforms to the applicable standards. Depending on the STAC Benchmark specification, this procedure may include inspecting the source code of the vendor’s STAC Pack (components developed by the vendor to integrate with the Test Harness), checking the timing skew between hardware devices used for timestamping, checking the configuration of STAC Libraries, etc. After verifying the Test Harness, STAC executes the required test runs and generates a STAC Report.
Does an audit render inaccurate results impossible?
No. The purpose of an audit is to catch errors in the Test Harness configuration and to eliminate bias in the execution or reporting of results. An ill-intentioned vendor prepared to invest considerable resources could still probably find ways to make even audited tests produce erroneous results. However, such a vendor would be taking huge risks (legal and reputational). A more likely source of error in unaudited results is confirmatory bias. This is where a vendor spends considerable effort correcting configuration errors in the test harness/procedure that yield unfavorable results but little time investigating the validity of favorable results. An audit is intended to prevent any such bias from affecting the results.
Ver 231009