Understandably the energy and water companies we work want to generate maximum impact from our customer engagement programs. To help with this, we conduct Randomized Control Trials (RCT) – which includes the better known A/B testing method. The main downside with RCTs is that the number of customers actively engaged in a program, or a new test, is reduced for a period of time, in order that we have a control group. We thought it pertinent to explain why a short term pause is key to delivering a long term gain, alongside the importance of RCTs.
What is an RCT?
An RCT is an experiment where a percentage of units, or in the case of our programs, a percentage of customers, are allocated to a control group that does not receive the nudge or communication being tested within the customer engagement program. It’s different to A/B testing that only proves the effect on one variable. RCTs can be multivariate, therefore more powerful and effective in delivering results. Having a comparable test group engaged against a control group is the only effective and statistically relevant way to accurately calculate the impact of an intervention as it establishes a reliable counterfactual scenario.
In an ideal world, a test is conducted using a control group for a sustained period of time in order to capture the impact of seasons on energy and water consumption. We recommend 12 months. This means that 50% of customers are initially left out of the treatment group, providing us with a solid benchmark against which we can reliably measure the water and energy savings made by customers who have been receiving our customer engagement treatments. Once the result has been reliably measured, the control group can be included in the program to generate additional savings.
RCTs are scientifically proven
In Elsevier’s Energy Research and Social Science Paper – ‘Evaluating energy behavior change programs using randomized controlled trials: Best practice guidelines for policymakers,’ Elsevier outlines practical guidelines for conducting RCTs and explains why they are generally the optimal approach for obtaining scientifically valid estimates of a behavioral program’s efficacy and effectiveness.
The paper has been written specifically for policymakers, practitioners and researchers who are grappling with the challenge of shifting energy consumer behavior in positive ways.
As the paper suggests – one very good reason for measuring the impact and success of such programs is to determine Return on Investment (ROI). It’s also a reliable way to understand whether the intervention should be rolled out more broadly to a wider spectrum of customers, and whether it is cost effective to scale the program to a larger proportion of the population.
Elsevier argues that it is imperative to use a robust experimental design in the program − in particular, an RCT where participants are randomly assigned to experimental groups (also known as ‘intervention’, ‘trial’ or ‘treatment’ groups) and control groups.
“The RCT approach offers the most scientifically robust and empirically defensible way of: (I) determining whether a cause-and-effect relationship exists between a particular intervention and its intended outcome, and (II) assessing the validity, utility and overall cost-effectiveness of an intervention, relative to business-as-usual or alternative interventions. In order to evaluate behavior change – and quantify the actual cost-effectiveness and return-on-investment of an energy-related intervention – a robust experimental design is required.” 1
Importance for policy makers
A good proportion of behavioral programs are designed and delivered in ways that constrain their ability to detect real behavior change. This limits policy implications that can be drawn from the experience, which in turn impacts the prospect of broader societal gains. It’s the reason that regulators mandate RCTs for customer engagement programs like ours in the USA.
Let us help you prove you’re not only meeting regulatory objectives, but also that you care about high standards of work by testing and measuring with a carefully designed RCT. We can help utilities to avoid or reduce potential regulatory penalties by reporting mandatory savings including residential per capita consumption reduction and reduction of water leakages. Get in touch today to find out more.
Source ref
1https://www.sciencedirect.com/science/article/pii/S2214629616302031