Figuring out the minimal number of users one must expose to an experimental treatment to collect actionable data is not a trivial task. We explain how we approach this problem with Twitter’s A/B testing platform (DDG), and how we communicate issues of statistical power to experimenters.