# 6. Gap Analysis with Categorical Variables¶

## 6.1. Preliminaries¶¶

I include the data import and library import commands at the start of each lesson so that the lessons are self-contained.

```
import pandas as pd
from scipy import stats
bank = pd.read_csv('Data/Bank.csv')
# Recode JobGrade to Manager
grades = [1,2,3,4,5,6]
status = ["non-mgmt", "non-mgmt", "non-mgmt", "non-mgmt", "mgmt", "mgmt"]
bank['Manager'] = bank['JobGrade'].replace(grades, status)
```

## 6.2. Creating a contingency table¶

Pandas has a very simple contingency table feature. Below, I specify the two variables of interest (Gender and Manager) and set `margins=True`

so I get marginal totals (“All”).

```
contab_freq = pd.crosstab(
bank['Gender'],
bank['Manager'],
margins = True
)
contab_freq
```

Manager | mgmt | non-mgmt | All |
---|---|---|---|

Gender | |||

Female | 10 | 130 | 140 |

Male | 25 | 43 | 68 |

All | 35 | 173 | 208 |

## 6.3. Showing row percentages¶

Typically, showing frequencies is less useful than relative frequencies. Here, I am interested in the row percentages: what is the probability that a female is a manager versus the probability a male is a manager.

We can get relative frequencies using the `normalize`

argument. If `normalize = True`

, then we get the relative frequency in each cell relative to the total number of employees. This is not very useful. What we want instead is to normalize by row. The parameter for this is: `normalize = 'index'`

. Why “index” instead of “row”? Because each row has a row number (or index).

```
conttab_relfreq = pd.crosstab(
bank['Gender'],
bank['Manager'],
margins = True,
normalize='index'
)
conttab_relfreq
```

Manager | mgmt | non-mgmt |
---|---|---|

Gender | ||

Female | 0.071429 | 0.928571 |

Male | 0.367647 | 0.632353 |

All | 0.168269 | 0.831731 |

Here, each row sums to 100%. Thus, for the total set of female employees, 7% are managers and 94% are non-managers. For males, 37% are managers and 63% are non-managers. The advantage of this presentation is that these percentages are directly comparable even though the majority (140/208) employees of the bank are female.

## 6.4. Chi-squared test of independence¶

The row percentages leave us with the impression that managerial status *depends* on gender. We can test this more formally using the \(\chi^2\) (/ˈkaɪ skweə(r)) test of independence.

Scipy has a method called `chi2_contingency()`

that takes a contingency table of observed frequencies as input. Note that this table cannot include marginal totals or marginal frequencies. Instead, it must consist of *m* x *n* observations:

```
contab_obs = pd.crosstab(
bank['Gender'],
bank['Manager'],
margins = False)
chi = stats.chi2_contingency(contab_obs)
chi
```

```
(26.617776266575998,
2.479518719230249e-07,
1,
array([[ 23.55769231, 116.44230769],
[ 11.44230769, 56.55769231]]))
```

The output of the `chi2_contingency()`

method is not particularly attractive but it contains what we need:

The first line is the \(\chi^2\) statistic, which we can safely ignore

The second line is the probability of getting a \(\chi^2\) statistic that large

*if*the two variables are independent. This*p*-value is very small (\(10^{-7}\)) so we conclude there is almost zero chance that gender and managerial status are independent at this bank.The third line is the degrees of freedom, which we can safely ignore.

The remainder of the output is a matrix showing the

*expected*frequencies under the assumption in independence. These expected values are quite different from the observed values above.

You may notice that the \(\chi^2\) statistic and *p*-value are different from those provided by R. This is because scipy defaults to the “Pearson’s Chi-squared test with Yates’ continuity correction” version of the test.