Inference for numerical data¶
North Carolina births¶
In 2004, the state of North Carolina released a large data set containing information on births recorded in this state. This data set is useful to researchers studying the relation between habits and practices of expectant mothers and the birth of their children. We will work with a random sample of observations from this data set.
Exploratory analysis¶
Load the
nc
data set into our notebook.
In [1]:
# for Mac OS users only!
# if you run into any SSL certification issues,
# you may need to run the following command for a Mac OS installation.
# $/Applications/Python 3.x/Install Certificates.command
# if this does not fix the issue, run this command instead.
import os, ssl
if (not os.environ.get('PYTHONHTTPSVERIFY', '') and
getattr(ssl, '_create_unverified_context', None)):
ssl._create_default_https_context = ssl._create_unverified_context
import pandas as pd
nc = pd.read_csv('https://www.openintro.org/stat/data/nc.csv')
We have observations on 13 different variables, some categorical and some numerical. The meaning of each variable is as follows.
variable | description |
---|---|
fage |
father's age in years. |
mage |
mother's age in years. |
mature |
maturity status of mother. |
weeks |
length of pregnancy in weeks. |
premie |
whether the birth was classified as premature (premie) or full-term. |
visits |
number of hospital visits during pregnancy. |
marital |
whether mother is married or not married at birth. |
gained |
weight gained by mother during pregnancy in pounds. |
weight |
weight of the baby at birth in pounds. |
lowbirthweight |
whether baby was classified as low birthweight (low ) or not (not low ). |
gender |
gender of the baby, female or male . |
habit |
status of the mother as a nonsmoker or a smoker . |
whitemom |
whether mom is white or not white . |
Exercise 1
What are the cases in this data set? How many cases are there in our sample?
As a first step in the analysis, we should consider summaries of the data. This can be done using the
describe()
and info()
:
In [2]:
nc.describe()
Out[2]:
fage | mage | weeks | visits | gained | weight | |
---|---|---|---|---|---|---|
count | 829.000000 | 1000.000000 | 998.000000 | 991.000000 | 973.000000 | 1000.00000 |
mean | 30.255730 | 27.000000 | 38.334669 | 12.104945 | 30.325797 | 7.10100 |
std | 6.763766 | 6.213583 | 2.931553 | 3.954934 | 14.241297 | 1.50886 |
min | 14.000000 | 13.000000 | 20.000000 | 0.000000 | 0.000000 | 1.00000 |
25% | 25.000000 | 22.000000 | 37.000000 | 10.000000 | 20.000000 | 6.38000 |
50% | 30.000000 | 27.000000 | 39.000000 | 12.000000 | 30.000000 | 7.31000 |
75% | 35.000000 | 32.000000 | 40.000000 | 15.000000 | 38.000000 | 8.06000 |
max | 55.000000 | 50.000000 | 45.000000 | 30.000000 | 85.000000 | 11.75000 |
In [3]:
nc.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1000 entries, 0 to 999 Data columns (total 13 columns): fage 829 non-null float64 mage 1000 non-null int64 mature 1000 non-null object weeks 998 non-null float64 premie 998 non-null object visits 991 non-null float64 marital 999 non-null object gained 973 non-null float64 weight 1000 non-null float64 lowbirthweight 1000 non-null object gender 1000 non-null object habit 999 non-null object whitemom 998 non-null object dtypes: float64(5), int64(1), object(7) memory usage: 101.6+ KB
As you review the variable summaries, consider which variables are categorical and which are numerical. For numerical variables, are there outliers? If you aren't sure or want to take a closer look at the data, make a graph.
Consider the possible relationship between a mother's smoking habit and the weight of her baby. Plotting the data is a useful first step because it helps us quickly visualize trends, identify strong associations, and develop research questions.
Exercise 2
Make a side-by-side boxplot ofhabit
and weight
. What does the plot highlight about the relationship between these two variables?
The box plots show how the medians of the two distributions compare, but we can also compare the means of the distributions using the following function to split the
weight
variable into the habit
groups, then take the mean of each using mean()
.
In [4]:
nc.groupby(['habit'])['weight'].mean()
Out[4]:
habit nonsmoker 7.144273 smoker 6.828730 Name: weight, dtype: float64
There is an observed difference, but is this difference statistically significant? In order to answer this question we will conduct a hypothesis test .
Inference¶
Exercise 3
Check if the conditions necessary for inference are satisfied. Note that you will need to obtain sample sizes to check the conditions. You can compute the group size using the samegroupby
command above but replacing mean
with size
.
Exercise 4
Write the hypotheses for testing if the average weights of babies born to smoking and non-smoking mothers are different.
We will now conduct hypothesis tests for testing if the average weights of babies born to smoking and non-smoking mothers are different. For this task, we can use
statsmodels
, a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration.
In [5]:
import statsmodels.stats.weightstats as st
nc_weightANDsmoker = nc[nc['habit'] == 'smoker']['weight']
nc_weightANDnonsmoker = nc[nc['habit'] == 'nonsmoker']['weight']
dsw1 = st.DescrStatsW(nc_weightANDsmoker)
dsw2 = st.DescrStatsW(nc_weightANDnonsmoker)
cm = st.CompareMeans(dsw1, dsw2)
# calculate number of observations, mean and standard deviation for each group
n_smoker = dsw1.nobs
n_nonsmoker = dsw2.nobs
mean_smoker = dsw1.mean
mean_nonsmoker = dsw2.mean
sd_smoker = dsw1.std
sd_nonsmoker = dsw2.std
print(f'n_smoker = {n_smoker}')
print(f'mean_smoker = {mean_smoker}')
print(f'sd_smoker = {sd_smoker}')
print()
print(f'n_nonsmoker = {n_nonsmoker}')
print(f'mean_nonsmoker = {mean_nonsmoker}')
print(f'sd_nonsmoker = {sd_nonsmoker}')
print()
# conduct hypothesis test
ht = cm.ztest_ind(alternative = 'two-sided', usevar = 'unequal', value = 0)
# calculate and print the standard error, the Z-score, and p-value for the hypothesis test
se = cm.std_meandiff_separatevar
testZ = ht[0]
p_value = ht[1]
print(f'Standard error = {se}')
print(f'Test statistic: Z = {testZ}')
print(f'p-value = {p_value}')
# reject or accept null hypothesis
if(p_value) < 0.05:
print('reject null hypothesis')
else:
print('accept null hypothesis')
n_smoker = 126.0 mean_smoker = 6.828730158730159 sd_smoker = 1.3806681061171728 n_nonsmoker = 873.0 mean_nonsmoker = 7.1442726231386 sd_nonsmoker = 1.5178105512705897 Standard error = 0.13376049190705977 Test statistic: Z = -2.359010944933488 p-value = 0.018323715325166846 reject null hypothesis
Exercise 5
Construct a confidence interval for the difference between the weights of babies born to smoking and non-smoking mothers.On Your Own¶
- Calculate a 95% confidence interval for the average length of pregnancies (
weeks
) and interpret it in context. Note that since you're doing inference on a single population parameter, there is no explanatory variable, so you can omit thex
variable from the function. - Calculate a new confidence interval for the same parameter at the 90% confidence level.
- Conduct a hypothesis test evaluating whether the average weight gained by younger mothers is different than the average weight gained by mature mothers.
- Now, a non-inference task: Determine the age cutoff for younger and mature mothers. Use a method of your choice, and explain how your method works.
- Pick a pair of numerical and categorical variables and come up with a research question evaluating the relationship between these variables. Formulate the question in a way that it can be answered using a hypothesis test and/or a confidence interval. Answer your question with Python, report the statistical results, and also provide an explanation in plain language.
This lab was adapted by Vural Aksakalli and Imran Ture from OpenIntro by Andrew Bray and Mine Çetinkaya-Rundel.
www.featureranking.com