(Credit: A. C. & I. P.)
DISCLAIMER: Please do not assume this report is 100% accurate nor it fully satisfies all the project requirements for full credit.
Introduction ¶
Dataset Source¶
The Melbourne House Prices dataset used in study was sourced from Kaggle (Becker, 2021). This dataset contains prices of houses sold in Melbourne between January 2016 and September 2017.
Dataset Details¶
The dataset is about the housing market in Melbourne and contains information about the house sale price, location, and the brokering real estate agency. Additional features included in this dataset are the date of sale, year built, number of rooms & bathrooms, distance to city center, land size, suburb, and the number of properties within the suburb. These features seem to be sufficient for an attempt for predictive modeling of Melbourne house prices as a regression problem.
This dataset has a total of 20 features (excluding the street address of houses) and 13580 observations. Houses with no price information have already been removed from the dataset.
Dataset Retrieval
- We read in the dataset from our GitHub repository and load the modules we will use throughout this report.
- We display 10 randomly sampled rows from this dataset.
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import io
import requests
pd.set_option('display.max_columns', None)
###
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("seaborn")
###
# name of the dataset to be imported from our GitHub account
df_name = 'melbourne_house_prices.csv'
df_url = 'https://raw.githubusercontent.com/akmand/datasets/master/' + df_name
url_content = requests.get(df_url, verify=False).content
melb_df = pd.read_csv(io.StringIO(url_content.decode('utf-8')))
melb_df.sample(10, random_state=999)
Dataset Features¶
The features in our dataset are described in the table below. These descriptions are taken from the Kaggle data source.
from tabulate import tabulate
table = [['Name','Data Type','Units','Description'],
['Suburb','Nominal categorical','NA','Suburb of house sold'],
['Rooms','Numeric','NA','Number of rooms'],
['Type','Nominal categorical','NA','br - bedroom(s); h - house, cottage, villa, semi, terrace; \
\nu - unit, duplex; t - townhouse; dev site - development site; \no res - other residential.'],
['Price','Numeric','AUD','Price of house sold'],
['Method','Nominal categorical','NA','S - property sold; SP - property sold prior; \
\nPI - property passed in; PN - sold prior not disclosed; SN - \
\nsold not disclosed; NB - no bid; VB - vendor bid; W - withdrawn \
\nprior to auction; SA - sold after auction; SS - sold after auction \
\nprice not disclosed. N/A - price or highest bid not available.'],
['SellerG','Nominal categorical','NA','Real estate agent'],
['Date','Date','dd/mm/yyyy','Date sold'],
['Distance','Numeric','KM','Distance from Central Business District'],
['Bathroom','Numeric','NA','Number of bathrooms'],
['Car','Numeric','NA','Number of carspots'],
['Landsize','Numeric','Square Meter','Land size'],
['Building area','Numeric','Square Meter','Building size'],
['Year built','Date','year','Year the house was built'],
['Council area','Nominal categorical','NA','Governing council for the area'],
['Lattitude','Numeric','Degree','Lattitude'],
['Longtitude','Numeric','Degree','Longtitude'],
['Region name','Nominal categorical','NA','General Region (West, North West, etc)'],
['Property count','Numeric','NA','Number of properties that exist in the suburb']]
print(tabulate(table, headers='firstrow', tablefmt='fancy_grid'))
Target Feature¶
For this project, the target feature in this dataset will be the house price in Australian dollars. That is, the price of Melbourne houses will be predicted based on the explanatory/ descriptive variables.
Goals and Objectives¶
Melbourne has a very active housing market as demand for housing throughout the city is usually well above the supply. For this reason, a model that can accurately predict a house's selling price in Melbourne would have many real-world uses. For instance, real estate agents can provide better service to their customers using this predictive model. Likewise, banks lending out money to home buyers can better estimate the financial aspects of this home loan. Perhaps more importantly as potential home buyers, we as individuals can better figure out if we are being ripped off or we are getting a good deal provided that our predictive model is a reliable one.
Thus, the main objective of this project is two-fold: (1) predict the price of a house sold in Melbourne based on publicly available features of the house, and (2) which features seem to be the best predictors of the house sale price. A secondary objective is to perform some exploratory data analysis by basic descriptive statistics & data visualisation plots to gain some insight into the patterns and relationships existing in the data subsequent to some data cleaning & preprocessing, which is the subject of this Phase 1 report.
At this point, we make the important assumption that rows in our dataset are not correlated. That is, we assume that house price observations are independent of one another in this dataset. Of course, this is not a very realistic assumption, however, this assumption allows us to circumvent time series aspects of the underlying dynamics of house prices and also to resort to rather classical predictive models such as multiple linear regression.
Data Cleaning and Preprocessing¶
In this section, we describe the data cleaning and preprocessing steps undertaken for this project.
Data Cleaning Steps¶
- Drop irrelevant features in our dataset
- Check and rename/ modify some column names
- Check for missing values
- Remove all the rows with missing values
- Random sampling of the dataset for 5000 rows
Let's first display all the columns in our dataset.
melb_df.columns
Postcode is very similar to the Suburb feature, thus it is considered redundant for the model and removed. Bedroom2 is just the number of bedrooms based on other source, which is also the same as Rooms feature, thus it is also considered redundant and removed.
#drop irrelevant/repeated columns/features
melb_df = melb_df.drop(columns=["Postcode", "Bedroom2"])
As this is not a time series project, the Date feature is transformed into Months and Years features for the model. After transforming, the original Date feature is removed.
# Convert Date feature from string to Pandas' datatime dtype
melb_df['Date'] = pd.to_datetime(melb_df['Date'], dayfirst=True)
###
# Create Year and Month features
melb_df['year_sold'] = melb_df['Date'].dt.year
melb_df['month_sold'] = melb_df['Date'].dt.month
###
# Remove Date as we will not need it anymore
melb_df = melb_df.drop(columns=["Date"])
Some of the columns are not labelled properly, which will be problematic when modelling. We will make all column names lower case and replace any spaces with underscores for consistency. We will also remove any white spaces before & after column names.
# make column names lower case and also remove any white spaces
# before & after the column names using the strip() function
melb_df.columns = melb_df.columns.str.lower().str.strip()
columns_mapping = {
'sellerg': 'real_estate',
'landsize': 'land_size',
'buildingarea': 'building_area',
'yearbuilt': 'year_built',
'councilarea': 'council',
'regionname': 'region',
'propertycount': 'property_count'
}
# rename columns
melb_df = melb_df.rename(columns = columns_mapping)
melb_df.sample(5, random_state=999)
Next we check the data types and observe that they match the intended data types, thus no change is needed here.
# Check for data types
print(f"Shape of the dataset = {melb_df.shape} \n")
print(f"Data types are below where 'object' indicates a string type: ")
print(melb_df.dtypes)
The unique values for all columns with categorical data types are displayed to check for any white spaces and other data quality issues. It turns out that the data is already clean, and no futher data cleaning steps are necessary here.
from IPython.display import display, HTML
display(HTML('<b>Table 1: Summary of categorical features</b>'))
melb_df.describe(include='object').T
# To see all unique values for categorical data types
categoricalColumns = melb_df.columns[melb_df.dtypes==object].tolist()
for col in categoricalColumns:
print('Unique values for ' + col)
print(melb_df[col].unique())
print('')
The summary statistics are generated for all the numerical features. There does not seem to be any outliers in the data.
from IPython.display import display, HTML
display(HTML('<b>Table 2: Summary of numerical features</b>'))
melb_df.describe(include=['int64','float64']).T
Missing values are checked by displaying the number of missing values in every column. We observe that car, building_area, year_built and council features have missing values. We decide to drop these observations for simplicity.
# Count missing values in each column
print(f"\nNumber of missing values for each column/ feature:")
print(melb_df.isnull().sum())
# Drop all rows with missing values/NaN
melb_df = melb_df.dropna()
melb_df.shape
Random Sampling¶
As the data has more than 5000 rows, random sampling is done to get only 5000 rows out of the remaining 6196 rows for ease of computation. At the end, we display 5 random rows from our cleaned data.
melb_df = melb_df.sample(n=5000, random_state=999)
melb_df.shape
melb_df.sample(5, random_state=999)
Data Exploration and Visualisation¶
Our dataset is now considered to be clean and we are ready to start visualising and explore each of the features.
Univariate Visualisation¶
Bar Chart of Region Name¶
We count the region name to see which region has the highest count in Melbourne Housing in a descending order. As we can see in Figure 1, the Southern Metropolitan has the highest number of house sold compared to the other regions in Melbourne.
plt.figure(figsize = (20,8))
fig = sns.countplot(x = 'region', data = melb_df, palette = 'magma',
order = melb_df['region'].value_counts().index)
fig = plt.title('Figure 1: Region Count for Melbourne Housing', fontsize = 15)
plt.show()
Bar Chart of Method¶
From Figure 2, we can see that the method "Property Sold" ("S") is the most common method used in selling house based on the dataset.
plt.figure(figsize = (15,8))
fig = sns.countplot(x = 'method', data = melb_df, palette = None,
order = melb_df['method'].value_counts().index)
fig = plt.title('Figure 2: Method Count in Melbourne Housing', fontsize = 15)
plt.show()
Boxplot & Histogram of Price¶
We can see in Figures 3A and 3B that the distribution of price is clearly right-skewed and has a huge range, which indicates the price variable will probably need a log transformation in the second phase of the project.
# Boxplot of Price
plt.figure(figsize = (15,8))
sns.boxplot(melb_df['price']).set_title('Figure 3A: Box Plot of Price', fontsize = 15)
plt.show();
# Boxplot of Price
plt.figure(figsize = (15,8))
sns.distplot(melb_df['price'], kde=True, bins=40).set_title('Figure 3B: Histogram of Price', fontsize = 15)
plt.show();
Two-Variable Visualisation¶
Scatterplot of price and distance from CBD¶
Figure 4 shows that there is a correlation between the distance from CBD and price. As the distance increases, the price tends to be lower.
plt.figure(figsize = (15,8))
plt.scatter(melb_df['distance'], melb_df['price'], alpha = 0.3)
plt.title('Figure 4: Scatterplot of Price and Distance from CBD', fontsize = 15)
plt.xlabel('Distance from CBD')
plt.ylabel('Price')
plt.show();
Boxplot of Price by House Type¶
From figure 5, we can see that H type (house, cottage, villa, semi, terrace) of house have overall a higher price compared to other types. Furthermore, U (unit or duplex) house has a lower price compared to other types.
plt.figure(figsize = (15,8))
sns.boxplot(melb_df['type'], melb_df['price']);
plt.title('Figure 5: Boxplot of Price by House Type', fontsize = 15)
plt.show();
Boxplot of Price by Number of Rooms¶
In figure 6, the price tends to increase as the number of rooms increases.
plt.figure(figsize = (15,8))
sns.boxplot(melb_df['rooms'], melb_df['price']);
plt.title('Figure 6: Boxplot of Price by Number of Rooms', fontsize = 15)
plt.show();
Three-Variable Visualisation¶
Boxplot of Price broken down by Method and House Type¶
We can clearly see that in figure 7, the comparison between house type and price in different method is somewhat comparable. There is no significant difference.
plt.figure(figsize = (15,8))
sns.boxplot(melb_df['method'], melb_df['price'],
hue = melb_df['type'])
plt.title('Figure 7: Boxplot of Price broken down by Method and House Type', fontsize = 15)
plt.show();