Causal Impact Analysis to Measure Campaign Effectivity
This is one of the use case example from Telco industry that specifically relate to campaign program. This article aim to give you another perspective in measuring campaign effectiveness.
Background & Problem Statement
During the pandemic, studying activities were done online meanwhile not all students can afford to buy data package frequently. Government subsidized student, teacher, and lecturer with free data package that can only be used for conference and learning apps (e.g: Google Meet, Zoom, Teams). Unfortunately, social media apps are not included (e.g: Instagram, TikTok, Facebook)
Although the package is free, it is only 86% customer who use it after 15th days injection. In comparison to previous injection, payload conversion reached 97% on the 15th day. Therefore, We run an emergency campaign for 7 days in order to achieve monthly target in payload conversion (having additional 11% conversion rate). We initiated to give free quota with limited validity to customer with no data consumption in order to create FOMO as well as boost conversion.
Caveat
- There was no control group as it will be considered as opportunity loss. Ideally, when we want to measure campaign effectivity we need to prepare control group that will not receive any treatment. By comparing the result between both groups using statistical test, we will know whether the campaign is statistically significant or not, if it does statistically significant (p-value<0.05) we can assume the program is effective
How can we measure the campaign effectiveness without control group?
Causal impact analysis
- Methodology to predict effectiveness of a treatment within certain period of time. Data before treatment is used to predict the impact during the treatment. The uplift is defined as the difference between the actual and predicted counterfactual behavior seen in a specific time period
- Ideally, causal impact analysis use control group as comparison aside from having data prior to intervention, thus the measurement can be more rigid. Although it is not ideal without control group, at least we have group comparison, in this case we can compare prior and during intervention data
Consideration :
- It’s important to make sure that only one experiment is carried out each time, as running multiple campaigns at the same time can affect the accuracy of the prediction
Method
- We will use R studio for analyzing.
- Observation date : 2 weeks (1 week before & 1 week after intervention)
- Macro Level Analysis : Converted customer in population level
- Micro Level Analysis: Payload consumption of targeted customer (those who are injected with free quota) and how many of them consume the quota
Causal Impact Analysis Step by Step
1. Download the library
library(CausalImpact)
library(dplyr)2. Import the dataset
df <- read.csv("[your file name].csv")
print (df)Prepare your data set according to analysis that you want to conduct. Data set I used for this analysis are: event date, total customer converted in population (macro level), target customer converted (micro level), and target customer’s payload conversion (micro level)
3. Set your pre (before treatment) and post period (during treatment)
pre.period <- as.Date(c("2021-10-14","2021-10-21"))
post.period <- as.Date(c("2021-10-22","2021-10-30"))4. Bind your covariate with time series
time.points <- seq.Date(as.Date("2021-10-14"),by = 1,length.out = 16)
#Declare Time Series
conv_cust <- ts(df$converted_cust_pop)
conv_target <- ts(df$target_cust_converted)
payload_target <- ts(df$target_cust_payload_consumption)pop_analysis <-zoo(cbind(conv_cust), time.points)
tg_analysis <- zoo(cbind(conv_target), time.points)
tg_payload <- zoo(cbind(payload_target), time.points)
5. Run the analysis
Macro Level: Converted customer in population level
Result: The intervention is not significantly impact on increasing payload conversion as we can’t see any peak during treatment (p-value = 0.086). From the graph we also can see that actual conversion below than prediction (below zero on “pointwise” & “cumulative” section)
impact <- CausalImpact(pop_analysis, pre.period, post.period)
plot(impact)
Posterior inference {CausalImpact}
Average Cumulative
Actual 43223 345787
Prediction (s.d.) 2.0e+06 (1.5e+06) 1.6e+07 (1.2e+07)
95% CI [-686836, 5e+06] [-5494690, 4e+07]
Absolute effect (s.d.) -2.0e+06 (1.5e+06) -1.6e+07 (1.2e+07)
95% CI [-5e+06, 730060] [-4e+07, 5840477]
Relative effect (s.d.) -98% (72%) -98% (72%)
95% CI [-247%, 36%] [-247%, 36%]
Posterior tail-area probability p: 0.086
Posterior prob. of a causal effect: 91%
For more details, type: summary(impact, "report")Micro Level: Payload Consumption
Result:
Align with the increased of active customer, payload/quota usage during treatment/intervention was also increased (actual above the prediction, see section “pointwise” and “cumulative” on the graph) and statistically significant (p-value= 0.001)
impact3 <- CausalImpact(tg_payload, pre.period, post.period)
plot(impact3)
Posterior inference {CausalImpact}
Average Cumulative
Actual 30010 240081
Prediction (s.d.) 1343 (867) 10748 (6939)
95% CI [-386, 2927] [-3085, 23412]
Absolute effect (s.d.) 28667 (867) 229333 (6939)
95% CI [27084, 30396] [216669, 243166]
Relative effect (s.d.) 2134% (65%) 2134% (65%)
95% CI [2016%, 2263%] [2016%, 2263%]
Posterior tail-area probability p: 0.001
Posterior prob. of a causal effect: 99.8997%
For more details, type: summary(impact, "report")Micro level : Converted customer from target customer
Result:
If we zoom in on target customer, it is seen that customer who use the quota was slightly increased (see section “pointwise” and “cumulative” on the graph) and statistically significant (p-value= 0.001). However, only around 250K customer out of 1.4Mn (17%) who are active using the quota.
impact2 <- CausalImpact(tg_analysis, pre.period, post.period)
Posterior inference {CausalImpact}
Average Cumulative
Actual 283602 2268812
Prediction (s.d.) 240188 (10175) 1921508 (81397)
95% CI [220574, 260577] [1764592, 2084619]
Absolute effect (s.d.) 43413 (10175) 347304 (81397)
95% CI [23024, 63027] [184193, 504220]
Relative effect (s.d.) 18% (4.2%) 18% (4.2%)
95% CI [9.6%, 26%] [9.6%, 26%]
Posterior tail-area probability p: 0.001
Posterior prob. of a causal effect: 99.8997%
For more details, type: summary(impact, "report")Conclusion
- In conclusion, giving free quota didn’t give significant impact on increasing payload conversion, because it is only converted 19% from the target, while the initiative was expected to convert 86% of them
- If we look closely to the target customer, those who were converted was actively consuming the quota during the intervention period. Therefore, the free quota was significantly able to trigger usage but limited to few customers
In doing a campaign, sometimes you might encounter unexpected result, but you can still get the silver lining, by means you gain learnings from it and it will encourage you to find another best way to make your campaign more successful :)
Hope you find this article helpful and I am open for any feedbacks!