Failure to accurately measure the outcomes of an experiment can
lead to bias and incorrect conclusions. Online controlled experiments
(aka AB tests) are increasingly being used to make decisions
to improve websites as well as mobile and desktop applications.We
argue that loss of telemetry data (during upload or post-processing)
can skew the results of experiments, leading to loss of statistical
power and inaccurate or erroneous conclusions. By systematically
investigating the causes of telemetry loss, we argue that it is not
practical to entirely eliminate it. Consequently, experimentation
systems need to be robust to its effects. Furthermore, we note that it
is nontrivial to measure the absolute level of telemetry loss in an experimentation
system. In this paper, we take a top-down approach
towards solving this problem. We motivate the impact of loss qualitatively
using experiments in real applications deployed at scale,
and formalize the problem by presenting a theoretical breakdown
of the bias introduced by loss. Based on this foundation, we present
a general framework for quantitatively evaluating the impact of
telemetry loss, and present two solutions to measure the absolute
levels of loss. This framework is used by well-known applications
at Microsoft, with millions of users and billions of sessions. These
general principles can be adopted by any application to improve
the overall trustworthiness of experimentation and data-driven
decision making.
