# Check tool - a small benchmark tool `eqaf` comes with a small benchmark tool to record time/tick spend by our functions: - `equal` - `compare` - `find_uint8` - `exists_uint8` This README.md wants to explain into details this tool. ## Some problems Try to record time spend is hard. Indeed, the operating system or, at least, the CPU can disturb this specific sample. Of course, into a virtualized operating-system, it's more difficult to rely on this sample. At least, `check.exe` should be executed into a bare-metal operating system. By this fact, when we try to record time/tick, we have some noise. It's easily understable by this simple code (assume a `Clock.now ()` function which gives you the current time). ```ocaml let bench f = let a = Clock.now () in f () ; let b = Clock.now () in Format.printf "%Ld ns.\n%!" (Int64.sub b a) ``` If you run `bench` two times, results are surely differents. The difference is small but enough to not be able to assert an equality. By this fact, try to check a predicate such as the _constant-time_ of our function is not so easy than that. **NOTE**: when we talk about _constant-time_, it's not an algorithmic _constant-time_ as we can believe. Currently, our equal function respects 2 predicates (and we will fold them into one term, _constant-time_): - for 2 strings where lengths differs, `equal` must spend `B` ns where `B` is a real constant - for 2 strings where lengths are equal but contents __can__ differs (or not), `equal` must spend `A * L + B` where `L` is a the length of input strings and `A` and `B` are real constants. Then, the first predicate should be a subset of the second where `L = 0`. ## Metrics Currently, `check` relies (when you compile it into a Linux operating-system) on ticks with the _Time Stamp Counter_ instead to use `clock_gettime`. For MacOS or Windows, we use `clock_gettime` or something equivalent. It's a special ASM instruction. So, we possibly not handle your architecture. In fact, `RDTSC` is more reliable than `clock_gettime`. ## Linear regresssion to infer `A * L + B` As we said, samples can be disturbed. So we are not able to just compare samples and see that they are equal or not. We must infer our equation `A * L + B` (eg. our _constant-time_ predicate). Our experience is done as follow: 1) we infer our equation `A1 * L + B1` when we compare 2 strictly equal strings 2) we infer our equation `A2 * L + B2` when we compare 2 strictly different strings Even if a computation of `Eqaf.equal` will return expected result (see _fuzzer_ and basic tests): 1) The function always returns `true` 2) The function always returns `false` We want to check that `A1 = A2` and `B1 = B2`, or, in other words, we infer the same equation independently than inputs (different or not) - if we check that time spent by `Eqaf.equal` does not depend on inputs. At the end, this check means that `Eqaf.equal` does not leak any information by the _time side channel_. To be able to infer our equation, we use a linear regression: 1) we follow a sequence to execute our function `R0 = 1`, `Rn = max (R(n-1) * 1.01, R(n-1) + 1)` when `R` is how many times we execute our function and `n` is our iteration Arbitrary, we do 750 iterations 2) For each iteration, we record our time metric as follow: `samples.(n).(0) <- tick` `samples.(n).(1) <- R` 3) At the end, from these samples, we can infer by a linear regression our equation. The resulted coefficient of determination should be higher than 0.95. It tells to us that the infered equation is good enough. ## A counter example However, we should show a counter example of our results and we can do that with `String.equal`. So we redo the experience with `String.equal` and we should have an inequality betwen our equations. **NOTE**: If 2 strings are physically equal, `String.equal` does not introspect contents and it returns directly `true`. So we ensure that strings can be equal (and they are) but they are not physically equal. ## Predictible results When we generate different strings, we do that randomly. The first diff character can be at the beginning of the string or at the end. Because of that, time spent by `String.equal` is not very predictable - it depends, as we want to solve, from contents. By this way, `check.exe` use a constant seed to generate strings to be more predictable about results. ## How to compare our equations Come back earlier, we want to check that our first equation when inputs are equal is the same than our equation when inputs are not equal. However, due to our problems (noise on our samples), we can not strictly assert that they are equal. However, we can infer the difference between them with 2 techniques: 1) CCEA technique 2) SPSS technique Both are explained into `check.ml` and they are not really formalized as we expect. However, they give to us a way to compare our results. From them, it comes a _coefficient_ which should be between `30.0` and `30.0`. Of course, this segment is arbitrary but it's far from what we can get when we do the same process with `String.equal`. So if we don't have a good _coefficient_ with the CCEA technique, we restart the process with the SPSS technique as the final result. ## Tries At the end, if we have a _good_ final coefficient, we can say that our function respects our predicate. If the coefficient of the determination is not good enough (`<= 0.95`), we restart the process. If we don't have a good final _coefficient_, we restart the process. At least, we give 20 chances to our process to check our predicate.