[perfcompare] Use t-test CIs (confidence intervals) instead of z-test CIs
The t-test CIs are wider and more accurate, especially for smaller sample sizes. For our current sample size of 30 and setting of alpha = 0.01, they are wider by 7%: >>> scipy.stats.t.ppf(alpha/2, 30-1) / scipy.stats.norm.ppf(alpha/2) 1.0700964927577572 The blocker for using t-test CIs was making the SciPy library available to perfcompare when run on the bots. This is achieved by adding a vpython spec file. perfcompare.py is already run through vpython by the fuchsia_perfcompare.py recipe. vpython will download the dependencies specified in .vpython and make them available to perfcompare.py. The dependencies used here are based on those used by Chromium's main .vpython file: https://chromium.googlesource.com/chromium/src/+/33da179d28beb0f03278a04b6ee2d97bbd8daf82/.vpython The numpy and scipy packages come from here: https://chrome-infra-packages.appspot.com/p/infra/python/wheels/ Bug: IN-646 Test: "python perfcompare_test.py" + perfcompare trybot + manually checked that this makes CIs wider by 7% for a given set of perf results Change-Id: Ib3a6b1d733ea328180b804c3f2a080eb6330a263
Showing
- garnet/bin/perfcompare/.vpython 15 additions, 0 deletionsgarnet/bin/perfcompare/.vpython
- garnet/bin/perfcompare/perfcompare.py 9 additions, 21 deletionsgarnet/bin/perfcompare/perfcompare.py
- garnet/bin/perfcompare/perfcompare_test_output.txt 12 additions, 12 deletionsgarnet/bin/perfcompare/perfcompare_test_output.txt
Loading
Please register or sign in to comment