Skip to content
Snippets Groups Projects
Commit c0a694b7 authored by Mark Seaborn's avatar Mark Seaborn Committed by CQ bot account: commit-bot@chromium.org
Browse files

[perfcompare] Use t-test CIs (confidence intervals) instead of z-test CIs

The t-test CIs are wider and more accurate, especially for
smaller sample sizes.  For our current sample size of 30 and setting
of alpha = 0.01, they are wider by 7%:

>>> scipy.stats.t.ppf(alpha/2, 30-1) / scipy.stats.norm.ppf(alpha/2)
1.0700964927577572

The blocker for using t-test CIs was making the SciPy library
available to perfcompare when run on the bots.  This is achieved by
adding a vpython spec file.  perfcompare.py is already run through
vpython by the fuchsia_perfcompare.py recipe.  vpython will download
the dependencies specified in .vpython and make them available to
perfcompare.py.

The dependencies used here are based on those used by Chromium's main
.vpython file:
https://chromium.googlesource.com/chromium/src/+/33da179d28beb0f03278a04b6ee2d97bbd8daf82/.vpython

The numpy and scipy packages come from here:
https://chrome-infra-packages.appspot.com/p/infra/python/wheels/

Bug: IN-646
Test: "python perfcompare_test.py" + perfcompare trybot
  + manually checked that this makes CIs wider by 7% for a given set of perf results
Change-Id: Ib3a6b1d733ea328180b804c3f2a080eb6330a263
parent 498d2228
No related branches found
No related tags found
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment