Perfherder and regression alerts

In the dawn of arewefastyet only 2 machines were running and only three benchmarks were important. At that time it was quite easy to just iterate the different benchmarks once in a while and spot the regressions. Things have come a long way since. Due to the enormous increase in number of benchmarks and machines I created a regression detector, which wasn’t as easy as it sounds. False positives and false negative are the enemy of such a system. Bimodal benchmarks, noise, compiler perturbations, the amount of datapoints … all didn’t help. Also this had low priority, given I’m supposed to work onĀ  improving JIT performance and not recording JIT performance.

Perfherder came along, which aims to collect any performance data in a central place and to act on that information. Which has some dedicated people working on it. Since the beginning of 2016 AWFY has started to use that system more and more. We have been sending the performance data to Perfherder for a few months now and Perfherder has been improving to allow more and more functionality AWFY needs. Switching the regression detection from AWFY to Perfherder is coming closer, which will remove my largest time drain and allow me to focus even more on the JIT compiler!

The alerts perfherder creates are visible at:

Since last week we now can request alerts on the subscores of benchmark, which is quite important for JS benchmarks. Octane has a variance of about 2% and a subscore of the benchmark that regresses with 10% will only decrease the full benchmark score with 1%. As a result this is within the noise levels and won’t get detected. This week this feature was enabled on AWFY, which will increase the correctness and completeness of the alerts Perfherder creates.

I want to thank Joel Maher and William Lachance with the help of this.


AWFY now comparing across OS

At the end of last year (2015) we had performance numbers of Firefox compared to the other browser vendors in the shell and on windows 8 and Mac OSX in the browser on different hardware.

This opened requests for better information. We didn’t have any information if Firefox on windows was slower compared to other OS on the same hardware. Also we got requests to run Windows 10 and also to compare to the Edge browser.

Due to these request we decided to find a way to run all OS on the same specification of hardware. In order to do that we ordered 3 mac mini’s and installed the latest flavors of Windows, Mac OSX and Ubuntu on it. Afterwards we got our AWFY slave on each machine and started reporting to the same graph.

Compare OS

Finally we have Edge performance numbers. Their new engine isn’t sub-par with other JS engines. It is quite good! Though there are some side-notes I need to attach to the numbers above. We cannot state enough that Sunspider is actually obsolete. It was a good benchmark at the start of the JS performance race. Now it is mostly annoying, since it is testing wrong things. Not the things you want to/should improve. Next to that octane is still a great benchmarks, except for mandreel latency. That benchmark is totally wrong and we notified the v8 team about it, but we got no response so far. ( It is quite sad to see our hard work devalued due to other vendors gaming on that benchmark.

We also have performance across OS for the first time. This showed us our octane score was lower on windows. Over the last few months we have been rectifying this. We still see some slower cold performance due to MSVC 2015 being less eager in inlining fuctions. But our jit performance should be similar across platform now!