Details of Observed Churn

The observed churn rate in this sample is 0.265 or 26.5%. Suppose that the company has a target churn rate of 25 percent.
⊕That’s an hypothesis regarding the probability that a random customer churns. It’s an informed company choice about an important organizational metric.

What do we know? A number of customers and a hypothetical probability – 0.25. And we also have some data – the 0.265. The observed data are 0.015 larger than the company thinks they should be.

⊕A binomial with size = 7043 and probability of churning for each individual that is equal to 0.25 is what the company thinks is going on. The binomial will show the number of Churns on the x axis and the probability of each value on the y axis.

Let’s plot it with the red line showing where our data locate.

data.frame(x=seq(1500,2000), # A sequence of x values 
           Prob = dbinom(seq(1500,2000), size=7043, 0.25))  %>% # Probability for each x
  ggplot() + # Plot
  aes(x=x, y=Prob) + # x as x and y as Prob defined above.
  geom_col() + # a bar plot with x and y
  geom_vline(aes(xintercept=1869), color="red") + # a vertical line at 1869
  geom_text(aes(x=1869, y=0.01, label="The Data"), color="red") + # An annotation for the data
  theme_minimal() + 
  labs(x="Churners", title="Probability of Churners given Binomial(7043, 0.25)")

We can also use radiant’s Probability Calculator with 1500 and 2000 as values.

result <- prob_binom(
  n = 7043, 
  p = 0.25, 
  lb = 1500, 
  ub = 2000, 
  dec = 5
)
plot(result) + labs(title="radiant binomial(n=7043, p=0.25)")

Mechanics

⊕Some Indeterminacy How will we decide? We want to turn this into a binary/logical decision; that is, which hypothesis is actually correct and now we need to be specific about how much confidence we wish to have. For a variety of reasons 95% confidence or 0.95 probability is the default. We will expect to be wrong about 1 time in 20. We expect to be in error with probability 0.05. We will adopt 95 percent confidence and 5 percent error.

The mechanics of hypothesis testing lay on top of this intuition. There are three potential hypotheses and we need to think in terms of evidence against them. Each is specified as an exhaustive set in the sense that any result is consistent with one or the other hypotheses – there is no indeterminacy. But there is some; we will adopt 95 percent confidence.

1. First, we could ask if the probability of churn, \(\pi = 0.25\) as opposed to not equal: \(\pi \neq 0.25\). The \(\pi=0.25\) is a null hypothesis; the claim to be evaluated. The alternative must then be anything that is not exactly \(\pi = 0.25\). That would mean that observed churn is either too big or too small once we look at the data to support the view that it is actually 0.25. In radiant, that is alternative two-sided – not equal to. Anything in red or grey is unacceptable

2. Second, we could make a directional claim about \(\pi\).
   

• We could ask, is \(\pi \geq 0.25\)? Do we have at least one-quarter churning? The only relevant evidence in this case that would lead us to deny that claim is evidence that \(\pi\) is actually less. The alternative, then, is that \(pi < 0.25\). In radiant, that is alternative less than; it appears in the figure in red.

• We could ask, is \(\pi \leq 0.25\)? Do we have at most one-quarter churning? The only relevant evidence in this case that would lead us to deny that claim is evidence that \(\pi\) is actually more than 0.25. The alternative, then, is that \(\pi > 0.25\). In radiant, that is alternative greater; it appears in the figure in grey.

Alternative: Two-sided [the default]

⊕We will compare the claim that \(\pi = 0.25\) against the alternative that \(\pi \neq 0.25\).

⊕Output Radiant is clear about the comparison, it tells us the claim being evaluated and the alternative. In this case, it is equal [the null hypothesis] to 0.25 or not equal to 0.25. Relevant Evidence We can deploy two approaches. One based on confidence intervals and the other based on p-values from hypothesis testing.

⊕Confidence intervals We can obtain a 95 percent confidence interval by solving for the values of \(\pi\) that could generate the data we observer with probability 0.95. In this case, it ranges from 0.255 to 0.276. If the claimed value of \(\pi = 0.25\) is in this range, then we cannot rule out that \(\pi=0.25\). If \(0.25\) is not in this range, this evidence that \(\pi \neq 0.25\). 0.25 is too small with 95% confidence so we must conclude that \(P(Churn) \neq 0.25\).

⊕p-value or probability value The probability of having seen the data that we observed or something even more extreme, whether bigger or smaller, if the null hypothesis was true is reflected in the p-value/p.value. In our case, what is the probability of observing something as extreme as 1869 if, in fact, the probability of churning is 0.25? It is quite unlikely: 0.003; this combines the probability above what we obtained and below the negative of what we obtained. As a result, P(Churn) is extremely unlikely to be 0.25. Moreover, this p-value is beneath the 0.05 level of acceptable error that we earlier resolved.

binom.test(1869, 7043, p = 0.25)

## 
##  Exact binomial test
## 
## data:  1869 and 7043
## number of successes = 1869, number of trials = 7043, p-value = 0.003091
## alternative hypothesis: true probability of success is not equal to 0.25
## 95 percent confidence interval:
##  0.2550860 0.2758483
## sample estimates:
## probability of success 
##              0.2653699

result <- single_prop(Churn, var = "Churn", lev = "Yes", comp_value = 0.25)
summary(result)

## Single proportion test (binomial exact)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn not equal to 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff    ns p.value  2.5% 97.5%   
##  0.015 1,869   0.003 0.255 0.276 **
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The various output parts. There is summary information that I will define below.

Alternative: Greater than

⊕We will compare the claim that \(\pi \leq 0.25\) against the alternative that \(\pi \gt 0.25\). Output The actual output is almost identical except for the different statements of the alternatives [note radiant does not note the less than in the null hypothesis]. Relevant Evidence The only relevant evidence to the claim that \(\pi \leq 0.25\) is evidence that it is actually greater. We have exactly this evidence. Our sample proportion is 0.265. The only remaining question is to consider the probability associated with this evidence.

⊕Confidence interval Given the data, what is the smallest that p(Churn) can be with 95% confidence? The evidence says that the 5th percentile of P(Churn) is 0.257. The company claim is 0.25 but this is far below what the evidence supports with 95 percent confidence. P(Churn) is not less than or equal to 0.25 but is instead greater; at a minimum, with 95 percent confidence, P(Churn) is 0.257.

⊕P-value What is the probability of 1869 or more if size is 7043 and p(Churn) is 0.25? This is the same probability 0.0016 [rounded to 0.002] that we obtained at the beginning. The evidence suggests that P(Churn) is not less than or equal to 0.25 but is instead greater because 0.002 is far below the standard of 0.05 that we set for making decisions. P(Churn) is not less than or equal to 0.25 with 95 percent confidence.

binom.test(1869, 7043, p = 0.25, alt="g")

## 
##  Exact binomial test
## 
## data:  1869 and 7043
## number of successes = 1869, number of trials = 7043, p-value = 0.0016
## alternative hypothesis: true probability of success is greater than 0.25
## 95 percent confidence interval:
##  0.2567179 1.0000000
## sample estimates:
## probability of success 
##              0.2653699

result <- single_prop(Churn, var = "Churn", lev = "Yes", comp_value = 0.25, alternative = "greater")
summary(result)

## Single proportion test (binomial exact)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn > 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff    ns p.value    5%  100%   
##  0.015 1,869   0.002 0.257 1.000 **
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Alternative: Less than

⊕We will compare the claim that \(\pi \geq 0.25\) against the alternative that \(\pi \lt 0.25\). Output Different statements of the alternatives [note radiant does not note the greater than in the null hypothesis]. There are two key differences in the evidence and interpretation. Relevant Evidence The only relevant evidence to the claim that \(\pi \geq 0.25\) is evidence that it is actually less, but we have a sample proportion of 0.265. It is greater than or equal to 0.25 instead of less.

⊕p-value The p-value shows the probability that \(\pi\) is greater than or equal to 0.25 and, with data so far above 0.25, that is nearly one. This p-value is far above the standard of 0.05 that we set so we are left with the claim that \(\pi \geq 0.25\) because there is little evidence against it.

⊕Confidence interval The confidence interval shows that, with 95 percent confidence, P(Churn) could be as high as 0.274. This does not really help to adjudicate the claim except to say that 0.25 or greater remains probable.

binom.test(1869, 7043, p = 0.25, alt="l")

## 
##  Exact binomial test
## 
## data:  1869 and 7043
## number of successes = 1869, number of trials = 7043, p-value = 0.9985
## alternative hypothesis: true probability of success is less than 0.25
## 95 percent confidence interval:
##  0.0000000 0.2741656
## sample estimates:
## probability of success 
##              0.2653699

result <- single_prop(Churn, var = "Churn", lev = "Yes", comp_value = 0.25, alternative = "less")
summary(result)

## Single proportion test (binomial exact)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn < 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff    ns p.value    0%   95%  
##  0.015 1,869   0.999 0.000 0.274  
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Alternative: Two-sided [not equal]

⊕Claim: \(\pi=0.25\) The alternative is then that \(\pi \neq 0.25\). Required Evidence To rule out that \(\pi = 0.25\), we need to see evidence that is sufficiently greater or smaller than would be expected with 95 percent confidence. The Normal for Reference The standard normal distribution has 0.95 probability between -1.96 and 1.96.

plot(prob_norm(0, 1, plb=0.025, pub=0.975), type="probs") + labs(title="95% of Normal(0,1): [-1.96,1.96]")

⊕A Decision Rule for z If the \(z\) we obtain is greater than 1.645, this is sufficient evidence to conclude, with 95 percent confidence, that \(\pi\) or the probability of Churn is not less than or equal to 0.25 and must be greater. Knowledge of \(\pi\) is enough to solve for \(z\). \[ z = \frac{\hat{p} - 0.25}{\sqrt{\frac{0.25(1-0.25)}{7043}}}\] Our data have 1869 of 7043 [0.2653699], so z is 2.98.

z <- ((1869/7043) - 0.25)/sqrt(0.25*0.75/7043)
z

## [1] 2.978849

plot(prob_norm(0, 1, plb=0.025, pub=0.975), type="probs") + labs(title="95% of Normal(0,1): [-1.96,1.96]") + geom_vline(aes(xintercept=c(-2.98,2.98)), color="blue") + theme_minimal()

⊕The data are 2.98 standard errors above 0.25. Recalling that evidence to decide it is simply not equal to 0.25 could come from far below or above 0.25. So how much probability is there in a normal distribution above 2.98 standard deviations/standard errors and below -2.98 standard deviations/standard errors? 0.003.

1-pnorm(((1869/7043) - 0.25)/sqrt(0.25*0.75/7043)) + # Above
pnorm(-((1869/7043) - 0.25)/sqrt(0.25*0.75/7043)) # Below

## [1] 0.002893337

What about the R functions? It is important to note that prop.test returns a \(\chi^2\). If we take the square root, we will get \(z\).

prop.test(1869, 7043, p = 0.25)

## 
##  1-sample proportions test with continuity correction
## 
## data:  1869 out of 7043, null probability 0.25
## X-squared = 8.7918, df = 1, p-value = 0.003026
## alternative hypothesis: true p is not equal to 0.25
## 95 percent confidence interval:
##  0.2551180 0.2758793
## sample estimates:
##         p 
## 0.2653699

result <- single_prop(
  Churn, 
  var = "Churn", 
  lev = "Yes", 
  comp_value = 0.25, 
  test = "z"
)
summary(result)

## Single proportion test (z-test)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn not equal to 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff z.value p.value  2.5% 97.5%   
##  0.015   2.979   0.003 0.255 0.276 **
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Alternative: Greater

⊕Claim We claim that \(\pi \leq 0.25\) and the complement/alternative must be that \(\pi \gt 0.25\). Required Evidence To rule out that \(\pi \leq 0.25\), we need to see evidence that is sufficiently greater than would be expected with 95 percent confidence. The Normal for Reference The standard normal distribution has 0.95 probability less than 1.645 and 0.05 greater than 1.645. A Decision Rule for z If the \(z\) we obtain is greater than 1.645, this is sufficient evidence to conclude, with 95 percent confidence, that \(\pi\) or the probability of Churn is not less than or equal to 0.25 and must be greater.

⊕Decision using z We have already solved for \(z\); it was 2.98. Our \(z\) of 2.98 standard errors is far more than the 1.645 standard errors above to decide that \(\pi \leq 0.25\) cannot be true with 95 percent confidence. The evidence favors the alternative \(\pi \gt 0.25\).

plot(prob_norm(0, 1, pub = 0.95), type="probs") + theme_minimal() + labs(title="Normal(0,1)", subtitle="95% of z is less than 1.645") + geom_vline(aes(xintercept=2.98), color="blue")

# 95th percentile of a normal 0 and 1.
qnorm(0.95, 0, 1)

## [1] 1.644854

summary(prob_norm(0, 1, pub = 0.95), type="probs")

## Probability calculator
## Distribution: Normal
## Mean        : 0 
## St. dev     : 1 
## Lower bound : 0 
## Upper bound : 0.95 
## 
## P(X < 1.645) = 0.95
## P(X > 1.645) = 0.05

⊕A Rule Using P(Churn) How big must P(Churn) be observed to be to render \(\pi \leq 0.25\) false? From the previous, 95 percent of the probability lies below 1.645 standard errors and our standard error is 0.005, thus \[ 0.25 + 1.645*0.005 = 0.258\] Any sample proportion greater than 0.258 is too large to believe that \(\pi \leq 0.25\) with 95% confidence,

⊕Decision with P(Churn) Our data show 0.265; this is very unlikely if \(P(Churn)=0.25\) or anything less. P(Churn) must be greater than 0.25 with 95 percent confidence because the observed proportion 0.265 is greater than 0.258. Decision with p.value If the claim was true and \(\pi \leq 0.25\) was true, with what probability might we obtain the observed proportion or more? In this case, this is the normal probability greater than 2.98 or 0.0015. 0.0015 is less than 0.05 and \(\pi \leq 0.25\) is judged false. \(\pi \gt 0.25\)

plot(prob_norm(0.25, sqrt(0.25*0.75/7043), pub = 0.95), type="probs") + theme_minimal() + labs(title="95% Upper Bound given 0.25") + geom_vline(aes(xintercept=0.265), color="blue")

qnorm(0.95, 0.25, sqrt(0.25*0.75/7043))

## [1] 0.2584869

summary(prob_norm(0.25, sqrt(0.25*0.75/7043), pub = 0.95), type="probs")

## Probability calculator
## Distribution: Normal
## Mean        : 0.25 
## St. dev     : 0.005 
## Lower bound : 0 
## Upper bound : 0.95 
## 
## P(X < 0.258) = 0.95
## P(X > 0.258) = 0.05

⊕Comment It is worth noting that the confidence intervals that are reported are the opposite bound of the alternative. For example, this case reports an upper bound on P(Churn) because of the - sign in the formula.

⊕Interpretation of the Confidence Interval Given the sample information, with 95 percent confidence, P(Churn) is at least 0.257.

# Using prop.test
prop.test(1869, 7043, p=.25, alt="g")

## 
##  1-sample proportions test with continuity correction
## 
## data:  1869 out of 7043, null probability 0.25
## X-squared = 8.7918, df = 1, p-value = 0.001513
## alternative hypothesis: true p is greater than 0.25
## 95 percent confidence interval:
##  0.2567371 1.0000000
## sample estimates:
##         p 
## 0.2653699

# The radiant result
result <- single_prop(Churn,
  var = "Churn", 
  lev = "Yes", 
  comp_value = 0.25, 
  alternative = "greater", 
  test = "z"
)
summary(result)

## Single proportion test (z-test)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn > 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff z.value p.value    5%  100%   
##  0.015   2.979   0.001 0.257 1.000 **
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Alternative: Less than 0.25

⊕Claim We claim that \(\pi \geq 0.25\) and the complement/alternative must be that \(\pi \lt 0.25\). Required Evidence To rule out that \(\pi \geq 0.25\), we need to see evidence that it is sufficiently less than would be expected with 95 percent confidence. The Normal for Reference The standard normal distribution has 0.05 probability less than -1.645.

plot(prob_norm(0, 1, pub = 0.05), type="probs") + theme_minimal() + labs(x="z") + geom_vline(aes(xintercept=2.98), color="blue")

⊕A Decision in z The obtained z of 2.98 [the blue vertical line] in the data is not less than -1.645. \(\pi \geq 0.25\) cannot be ruled out.

# 5th percentile of a normal 0 and 1.
qnorm(0.05, 0, 1)

## [1] -1.644854

# Radiant output
summary(prob_norm(0, 1, pub = 0.05), type="probs")

## Probability calculator
## Distribution: Normal
## Mean        : 0 
## St. dev     : 1 
## Lower bound : 0 
## Upper bound : 0.05 
## 
## P(X < -1.645) = 0.05
## P(X > -1.645) = 0.95

⊕Analysis of P(Churn) The standard error is 0.005. 1.645 of those [1.645*0.005=0.008] below the claim that \(\pi \geq 0.25\) will establish a lower bound on P(Churn). Put another way, the minimum value of \(\hat{p}\): 0.25 [the claim] - 0.008 = 0.242; we cannot see a sample proportion less than 0.242 and continue to believe that \(\pi \geq 0.25\) with 95% confidence. Decision with P(Churn) Anything below 0.242 with a sample of this size and P(Churn) must be less than 0.25 by the decision rules that we set. Ours is 0.265; we cannot rule out that \(\pi \geq 0.25\). Decision with p.value The p-value approximates the probability that the data might arise if the claim were true – if \(\pi \geq 0.25\). In this case, the value is very near one [0.999] because the evidence is consistent with the claim [and the alternative (\(\pi \lt 0.25\)) is certainly not true].

plot(prob_norm(0.25, sqrt(0.25*0.75/7043), pub = 0.05), type="probs")  + theme_minimal() + labs(x="P(Churn)") + geom_vline(aes(xintercept=0.265), color="blue")

⊕Comment It is worth noting that the confidence intervals that are reported are the opposite bound of the alternative. For example, this case reports an upper bound on P(Churn) because of the - sign in the formula.

⊕Interpretation of the Confidence Interval Given the sample information, with 95 percent confidence, P(Churn) is at most 0.274.

# Using prop.test
prop.test(1869, 7043, p=.25, alt="l")

## 
##  1-sample proportions test with continuity correction
## 
## data:  1869 out of 7043, null probability 0.25
## X-squared = 8.7918, df = 1, p-value = 0.9985
## alternative hypothesis: true p is less than 0.25
## 95 percent confidence interval:
##  0.0000000 0.2741843
## sample estimates:
##         p 
## 0.2653699

# Radiant result
result <- single_prop(Churn,
  var = "Churn", 
  lev = "Yes", 
  comp_value = 0.25, 
  alternative = "less", 
  test = "z"
)
summary(result)

## Single proportion test (z-test)
## Data      : Churn 
## Variable  : Churn 
## Level     : Yes in Churn 
## Confidence: 0.95 
## Null hyp. : the proportion of Yes in Churn = 0.25 
## Alt. hyp. : the proportion of Yes in Churn < 0.25 
## 
##      p    ns     n n_missing    sd    se    me
##  0.265 1,869 7,043         0 0.442 0.005 0.010
## 
##   diff z.value p.value    0%   95%  
##  0.015   2.979   0.999 0.000 0.274  
## 
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Contracts

There is an internal squabble about contracts. The churn rate does not impact the pay scale for front-line sales and this is argued to be inequitable because sales is incentivized to sign up high churn clients for bonuses while penalizing the divisions with compensation impacted by churn. Use the tools above to think about this question. To get us started, the easy way is to filter out the data in question. First, a look at it.

Churn <- read.csv(url("https://raw.githubusercontent.com/robertwwalker/DADMStuff/master/WA_Fn-UseC_-Telco-Customer-Churn.csv"))
library(janitor)
Churn %>% tabyl(Contract)

##        Contract    n   percent
##  Month-to-month 3875 0.5501917
##        One year 1473 0.2091438
##        Two year 1695 0.2406645

Next, create the data in question.

# Create Churn.OY or One Year
Churn.OY <- Churn %>% filter(Contract == "One year")
# Create Churn.TY or Two Year
Churn.TY <- Churn %>% filter(Contract == "Two year")
# Create Churn.MTM or Month to month
Churn.MTM <- Churn %>% filter(Contract == "Month-to-month")

Phone Service?

It has been suggested that the Churn rates are very different between types of PhoneService.

Churn%>% tabyl(PhoneService)

##  PhoneService    n    percent
##            No  682 0.09683374
##           Yes 6361 0.90316626

There are two datasets; one for each type.

# Those with Phone Service are Phone.Yes
Phone.Yes <- Churn %>% filter(PhoneService=="Yes")
# Those without Phone Service are Phone.No
Phone.No <- Churn %>% filter(PhoneService=="No")

More Generally

The base data file Churn is available here and you can use Data > View and the filter tick box to choose your own features to examine if you so choose. Radiant’s state file containing this part is on WISE. To save them to the markdown, you will have to copy the code and remember to only work with them after their code is issued. Knitting reads top to bottom.

You can, of course, work with the Code examples above on the filtered data or the tables.