Predicting Electricity Consumption and Production

Welcome to another blog post about my data analytics journey! Today, I’m going to delve into the world of predictive energy modeling by using the Enefit Energy dataset from Kaggle. This Kaggle competition was one of the most interesting and challenging I’ve done so far. Goal was to predict energy consumption and production for the country of Estonia. Predictions had to be made on an hourly basis for the next 2 days. Many  variables were available to come up with a good working model. Examples are the weather forecast, installed solar panel capacity, historical consumption and others.

So, if you are ready, let’s explore the powerful world of energy!

Introduction

It is has become increasingly important for energy companies to predict energy consumption and production on any given day. Here are several key reasons:

1. Balancing Supply and Demand: Energy supply must match with demand to make the system reliable and avoid outages. Accurate predictions help manage the generation and supply of energy. For example, during peak demand, additional resources are activated, but in low demand, there is a saving in resources and costs.

2. Integrating Renewable Energy Sources: As more renewable energy sources like wind and solar are integrated into the energy grid, predicting energy production becomes more complex. Accurate forecasting helps in planning the necessary backup from more controllable power sources like natural gas or hydroelectric power.

3. Grid Stability and Reliability: Predicting consumption patterns and production levels are crucial for maintaining grid stability. Sudden changes in energy demand or supply can lead to grid instability and even failures. Predictive analytics help reduce these risks by providing advanced warnings and allowing for proactive adjustments.

4. Operational Efficiency: By predicting when energy usage will be high or low, companies can optimize their operations, reduce waste, and lower costs. This includes more strategic purchasing of fuel and better maintenance scheduling.

5. Economic Planning: For energy companies, being able to forecast energy trends accurately is crucial for economic planning and investment decisions. This includes deciding where and when to build new infrastructure or expand existing capabilities.

6. Market Pricing: Energy pricing can fluctuate based on supply and demand dynamics. Accurate predictions allow companies to optimize their pricing strategies, potentially leading to better profitability or market share.

Overall, the ability to predict energy consumption and production with high accuracy allows companies to respond better to market demands, integrate renewable energy sources more effectively, maintain grid stability, and optimize economic outcomes.

Exploring the Dataset

The dataset consists of several .csv files like electricity & gas prices, clients, weather forecast (3.5 million records), historical weather, regions with weather stations and a training file (over 2 million records).

The forecast weather is an historical file of weather forecasts over the last 1,5 years. It contains 3,5 million records with data of 112 GPS areas, each with 1 prediction per hour, each 1,2,3,4…48 hours ahead. Elements recorded are:

latitude,longitude,origin_datetime,hours_ahead,temperature,dewpoint,cloudcover_high,cloudcover_low,cloudcover_mid,cloudcover_total,10_metre_u_wind_component,10_metre_v_wind_component,data_block_id,forecast_datetime,direct_solar_radiation,surface_solar_radiation_downwards,snowfall,total_precipitation

Historical weather has the same type of information, but holds the actual weather and not the forecast.

Another crucial source of information is the client file. I won’t go into too much detail, but it became very important to categorize customers based on product (contract) type, county, whether the client is a business or not, and the available solar power capacity.

During this exploration phase I did some checks to see if I understood the data correctly and if there are any obvious pitfalls I could detect right from the start.

Position of the weather stations over Estonia

Relationships between variables

I did some checks to understand the relationship of certain variables to another variable. Noteworthy ones I can mention here are:

• Energy Capacity over time: The plot I created here shows the growth of energy capacity, related to Product or Contract Type and Business/Household combitions.
  The visual shows a clear capacity growth, specifically for Product Type 1 and 3.

Product (Contract) Type: 0: “Combined”, 1: “Fixed”, 2: “General service”, 3: “Spot”

Next relationship I investigated was between the weather components and the variable that I needed to predict (production and consumption).

• Relevant weather components related to power consumption: In the plots below I show the most relevant attributes of the weather forecast when it comes to energy consumption. It is clear that energy consumption declines when the temperature rises. And we can also conclude that there is a negative correlation between energy consumption and the amount of radiation (sunlight).

Weather elements that influence energy consumption.

I also investigated the influence of weather on energy production.

• Relevant weather components related to power production: In the plots below we see the opposite effect when compared to power consumption. When there is more radiation (sunlight) there is more energy production. This is a possitive correlation. We can also see that most energy is produced between 10 hrs am and 17 hrs pm. That makes sense, but it is nice to see that the data backs up this ‘no brainer’. Another positive correlation is seen at the installed capacity. More capacity means more production. Another ‘no brainer’. Final plot shows the relationship with temperature. Cold temperatures mean less production, and temperatures between 5-15 degrees the highest. After 15 degress there is no further increase.

Preprocessing of the Data

Preprocessing of data is a vital step in every Data Science project. Most datasets are far from perfect and need to undergo vital steps for it to serve as input to a Machine Learning Model.

The steps I took in this Energy dataset were:

1. Handling Missing Values

One of the initial challenges in any data science project is dealing with missing values. In my case there weren’t many so I won’t go into that.

2. Checking for Outliers

Outliers can significantly impact the performance of predictive models. I utilized the same pairplot techniques (see above) and Z-score analysis to identify and remove outliers from the data. As it turned out only the electricity prices had some significant outliers. These outliers were removed and filled with the same price as a previous meaningful price point.

3. Encoding Cat. Variables

Categorical variables need to be encoded into a numerical format before feeding them into machine learning models. I did not have to do any encoding to this data.

Building the Machine Learning Models

After completing the pre-processing steps, it is time to create the Machine Learning model. The challenge is to predict energy consumption and production. These two variables can take on pretty much any value, so we consider the predictions to be ‘continuous’ (as opposed to for example predicting a fixed outcome of ‘yes or no’, ‘true or false’ etc.).

I decided to build different types of ML models for Energy Production and Energy Consumption.

1. A model that predicts Energy Production

I evaluated 6 different models, suitable for predicting continous variables. The models are: LightGBM, XGBoost, CatBoost, Random Forest, AdaBoost, Decision Trees.

The outcome:

Training and evaluating LightGBM…
Mean Squared Error: 9846.536495704782
Mean Absolute Error: 25.022088199724102

Training and evaluating XGBoost…
Mean Squared Error: 10593.784468962709
Mean Absolute Error: 25.470343101038633

Training and evaluating CatBoost…
Mean Squared Error: 8252.14802273349
Mean Absolute Error: 23.15684970360955

Training and evaluating Random Forest…
Mean Squared Error: 11256.553856624487
Mean Absolute Error: 23.881381789537063

Training and evaluating AdaBoost…
Mean Squared Error: 78885.39620655986
Mean Absolute Error: 171.638968077889

Training and evaluating Decision Trees…
Mean Squared Error: 20994.16420299445
Mean Absolute Error: 32.875448153599145

Model of Choice: CatBoost

Feature Importance CatBoost

Understanding which features contribute the most to my CatBoost’s predictions is crucial for making informed decisions. The outcome of such analysis is depicted in the plot below. With this information I can take the most important features and neglect less importance features to improve the model even more.

This plot displays the most important features (columns). Those are the installed capacity, solar power (radiation), eic_count (count of energy production sources), temperature and several others.

2. A model that predicts Energy Consumption

For the Energy Consumption part I went for the Gradient Boosting Model. This model performed best in my tests. 

Model of Choice: Gradient Boosting Model

Feature Importance Gradient Boosting

For Energy Consumption I found that the most important features were: temperature, working day and hour of the day. 

I created 69 Gradient Boosting Models  

For each County / Business-Household & Product or Contract type I created different models, all with the same parameters but trained on different data. This resulted in 69 model, tailored to each possible consumption scenario.

Making Predictions

With my code and models, I’ve made predictions on new unseen data generated in the Kaggle competition. I’m excited to see how my model performs against other competitors and contribute to the advancement of predictive modelling in the energy world.

Stay tuned for updates on my model’s performance and further insights from the competition!

Key Take Aways

The predictive energy modeling project using the Enefit Energy dataset provided several key insights:

1. Accuracy: Accurate predictions of energy consumption and production are crucial for balancing supply and demand, integrating renewables, maintaining grid stability, and optimizing operations.
2. Key Relationships: Energy consumption declines with higher temperatures and lower radiation, while energy production increases with higher radiation and optimal temperatures.
3. Data Preprocessing: Effective preprocessing, such as handling missing values and removing outliers, is vital for reliable model input.
4. Model Performance: CatBoost was the best model for predicting energy production, whereas Gradient Boosting was best for consumption predictions.
5. Feature Importance: Installed capacity, solar radiation, and temperature were crucial for production predictions, while temperature, working day, and time of day were key for consumption.
6. Customized Models: Creating 69 distinct models for different scenarios improved prediction accuracy.
7. Implications: These models can significantly improve energy management, resource allocation, and grid reliability.

The entire code

Check out all of the code of this project on Github: Nick Analytics – Predictive Energy Model

Overview of the modelling steps. 

Let me explain a bit better what feature engineering means and also the train, test steps of Machine Learning Model creation.

Extra features

During the pre-processing phase I added a bunch of new columns to the data, to see if it would improve my prediction process:

Features (columns) that I added to the data were:

– Average Pace between each checkpoint

– Percentage decay between each checkpoint

– Mean pace at each checkpoint

– Average Standard Deviation at each checkpoint

My idea was to generate as much relevant data as I could and then put it all in a Machine Learning cycle to see which features a truly predictive. The outcome was kinda surprising as you can see in the plot just below.

This plot was a check on what features would be most predictive of the runner’s finish times at the 35k check point. Basically the only things that seem to count are the current passing time, the average pace to 35k and the decay from 30k to 35k. Age and gender don’t seem to matter once you’re on the run

Deciding on the best model

The best model is the one that makes the best predictions overall. In ML terms we can also say that it is the model with the lowest error. I’ve tested a couple of ML models that come out of the box (like Random Forest and Linear Regression), and decided to go for Linear Regression (LR).

Along the way I found out that one size fits all is not the best approach, so I created different LR models for each checkpoint (so one for the point where the 5k results come in, one for the 10k results etc). I ended up with 9 models. 

An example on how well the model does is depicted here below. This bit of code result is taken at the 10k passing point. It indicates that the MAE (Mean Absolute Error) is 7.15 minutes for all runners, so a bandwith of 3,5 minutes to the upside or downside.

Performance of the prediction model.

As you can see the performance of the model(s) improved significantly the closer we got to the finish line. This makes sense of course, but it is nice to see it back like this. After 30k the model is only 2 minutes off on average with only a handful of parameters. Truly remarkable !

The finish time predictor on my dashboard

Future application of the model

The predictor I created is a great step in marathon finish time prediction. But of course we want to take it to a point where runners get their predicted times straight on their phone app or smartwatch during the run. As I don’t have the real time data or a large inferencing server I cannot make that happen. But understanding the variables that are in play and taking those to creating a well working Machine Learning model are crucial steps.

Another extension could be to create more models tailored to age and gender, combined with the checkpoint time. Lastly it would be great to distinguish professional runners from amateurs and to have an end time indication of each runner before the start. That could be something they fill in on their application.

So, many improvements are possible, but for now I’m happy with the progress I made.

A glimpse of other elements on my marathon dashboard.

Conclusion

In this blog post, I’ve described the steps I took to create a Machine Learning model that can predict finish times during the Boston Marathon. It was important to choose the right parameters to train the model. After some iterations I concluded that the most predictive features were the current passing time, the average pace to 35k and the decay from 30k to 35k. Age and gender don’t seem to matter that much once you’re actually running.

The models I created become more accurate as runners get closer to the finish line. At the first checkpoint the mean absolute error is 8 minutes (so, plus or minus 4 minutes). After 30k this deviation drops to below 2 minutes (plus or minus 1 minute). Note that these times apply to all runners participating. Creating additional models for example for professional runners or males/females could reduce the error even more.

 Thanks for reading my blog. Nick.

The coding I’ve done (in VS Code)

Check out the code of this project on Github: Nick Analytics – Predict Finish Times

Overview of the Designer Pipeline with data input at the top, a module to preprocess texts and the K-Means model that will take care of the clustering. The Training takes places in the final module. Here a new parquet file is created with cluster data added to it.

Classification ready

In my case it took about 30 minutes for Azure ML to complete the job. The output is a parquet file with each Tip assigned a cluster number, and added information about its distance to the closest centroid. Having this information we can now filter out all duplicate or similar tips by picking each cluster number.

A closer look

If we zoom in further on the cluster column and we find the following data:

This box shows us 15 of the 8500 rows that have been put into a cluster. 

It is striking that the Preprocessing Module has cleaned almost all special characters, numbers and capital letters. Basically only words are left. These words can be compared and clustered, like the KNN model has done here. It has categorized the tips into 50 clusters or bins. When done in Azure ML Designer the system will not only add the cluster number but also a couple of other columns with the distances to the centers (centroids) of other bins. This can help to decide whether the number of clusters is correct or could be adjusted somewhat.

Two insightful plots

I have created to plots related to this new clustering:

– first one is a histogram with an overview of how many tips are categorized per cluster

– second one is a box plot which displays the spread and the outliers in the lengths of the texts within each cluster 

Histogram of texts grouped per cluster

Box plot showing the spread and outliers in the lengths of the texts per cluster

Next step: Labeling

Labeling is the process of adding one to three word descriptions to an item. In my case to a piece of text. The intention of the process is:

1. Generate labels from the (already clustered) texts
2. Verify if the labels cover your needs
3. Add the labels to the texts
4. Compare similar labels between different clusters

1. Generate the labels using Data Analyst

I uploaded my texts to Data Analyst (part of ChatGPT) and asked for the labels. I provided a series of about 40 relevant labels that served as input and example to the model. After some iterations the sytem provided me with around 70 labels that should cover the essence of all of the text. Truly amazing

2. Make sure the labels cover your needs

This is basically a manual process. Read texts from each cluster and see if there are 1 or more labels that cover the most important topics. Keep in mind that the goal of this exercise is to create a blueprint of 8000 training tips, to easily select the right tip to the right problem. So, in my case either the behavioral problem(s) or the training goal(s) needed to be displayed in the label. Some of the words that I used were: visit, bite, bark, barking behavior, communication, doorbell, clarity, own energy, emotions, obedience, sounds, mood and 60 others.

Note: Those are a snippet of the labels added by Data Analyst

Final step: combining cluster and label

Reducing the number of tips to 1 per cluster

After the detailed labeling process I took a new efficiency step to bring the labeling to a higher level. I took each cluster and compared them with the labels (in Excel). I then decided to give each cluster a new top level label of max 3 words, derived from the Labels column previously generated. This step reduced the list of tips to 50, one for each cluster. 

Reducing the labels to 5 top level labels

My final step was to reduce the 50 labels I had to only 5 top level ones. These top level labels represented the 5 main areas for which tips were provided. I was able to link the 5 main areas to the number of tips that were used, thus creating an insight what areas are targeted the most.

Mindmap

One final way of looking at the end result is by using a ‘mindmap‘. This technique aggregates all high level tips and labels into a structure. It helps analysis ny showing relationships between elements.

Conclusion

In this blog post, I have explored the steps of analyzing a large number of texts by clustering and labeling them. Ultimate goal in this project was to provide high level insight in areas that need to be targeted in order to improve dog behavior and increase skills and knowledge of the dog owner. This work can serve as an input to an automatic Machine Learning model that could then cluster and label texts automatically.

My analytics work may be valuable in any organization where texts need to be labeled or classified. This may involve social media posts, chats conversations, sentiment analysis of earning transcripts and many more.

Thanks for reading my blog.

The coding I’ve done (in VS Code)

Check out the code of this project on Github: Text-Clustering-and-Labeling

Inspecting the data with Sweetviz

I used a Python library called Sweetviz to conduct an initial investigation on the data. Sweetviz can provide very important information on the dataset with respect to:

• Associations of the target variable (fraud or not) against all other features
• Indication which column are categorical and which ones numerical or dates
• The amount of missing data in each column
• Target analysis, Comparison, Feature analysis, Correlation
• Numerical analysis (min/max/range, quartiles, mean, mode, standard deviation, sum etc.

My first analysis of a subset of the training data (5000 records) yielded the following results:

The image above depicts the most important column (our target column, in black) along with all other 190 columns below it (you see only two of them). In the top section you can see the number of rows, duplicates, feature types (categorical/numerical or text).

A closer look

If we zoom in further on the target column and its associations we find the following data:

This part shows us that 75% of the transactions are ‘normal’, and 25% are fraudulent.

Along with the plot there are 3 tables that display how closely related the target column is with the numerical columns and categorical columns. I am most interested in:

– the Correlation Ratio (table at the bottom left) because if there’s a strong correlation between target and numerical feature we could say that the feature influences the outcome, and thus would be of interest in our prediction model.

– the Association Ratio (table at the top right) for the same reason as the correlation ratio, only in this case we’re dealing with categorical features.

So, if we want to reduce dimensionality in our dataset we could decide to only involve high scoring ratio’s and leave out the low scoring ones.

Dimensionality reduced

The exercise above has lead to a reduction of 153 columns to 39. This step is very important to keep the dataset ‘manageable’ and suitable for machine learning. Too much complexity required extreme calculation power and in general poorer results.

Dealing with missing data

Machine Learning models require information to be complete. If this is not the case (like in our example), we need to decide how to solve this problem. There are 3 ways:

– If a lot of information is missing in a column, we can remove the column

– If only some fields are empty we can fill them with the average for the column

– If key columns miss data we can make a prediction model just for this purpose

 Note I used Pycaret for further analysis. This package deals with missing values automatically by imputing its values.

Selecting a Machine Learning model with PyCaret

PyCaret is a really nice Python library if you want quickly get insights which machine learning models may work best on your data, together with nice plots to back it up. 

In this step I loaded my (reduced) dataset and set it up in PyCaret. I then put PyCaret to work telling it I wanted to get the best perfoming model. Here’s what happened:

The plot shows a list of ML models that have been tested (first two columns) with the results of each one of them in next columns. How do we interpret the most important indicators:

1. Accuracy: overall correctness of the model.
2. AUC: ‘Area Under the Curve’. How well does the model classify the positives and negatives.
3. Recall: the proportion of actual positives correctly identified by the model
4. Precision: True positive predictions among all positive predictions. In simple terms, the model could have predicted all normal transactions as normal, but may have overlooked many fraudulent ones. So, a score of 1 (perfect) only tells us something about positive (normal) predictions.
5. F1: (harmonic) mean of precision and recall. It is useful when you want to consider both false positives and false negatives.

For me the most important indicator is the one that is best at predicting true fraudulent transactions and as few as possible false positives. We don’t want to bother customer with legal transactions and tell them it’s a fraud.

I went on with Naive Bayes as it has a high accuracy and a better precision than KNN.

Here below I displayed the Confusion Matrix and UAC plot of this model:

On the trainset it predicted 11692 correctly as ‘normal’ transactions and 2310 correctly as ‘fraudulent’. 808 were predicted as ‘normal’, but were fraudulent, and 1785 were predicted as fraudulent, but were normal.

A higher AUC indicates better model performance in terms of classification.

Note: I just displayed the last 3 columns of all 39 columns.

Conclusion

In this blog post, I have explored the steps of analyzing a large dataset with fraudulent credit card transactions. I have given insights by showing graphs of the way data correlated and can be reduced to leave only relevant features. With PyCaret I tested and selected a Machine Learning model for predictions. I evaluated how accurate the predictions will be by explaining the classifiers that belong to this model.

My analytics work may be valuable in the financial world where the battle against fraud is taking more and more time and manpower.

The entire code

Check out all of the code of this project at Github (sweetviz part): Nick Analytics – Credit Card Fraud pt. 1

and on Google Colab (the PyCaret part): Nick Analytics – Credit Card Fraud pt. 2

Ideally most points fall within the blue box, or within the whiskers (the 2 vertical black lines). But as you can see there are quite a number of individual black data points that can be considered outliers. Let me explain the plot in a bit more detail:

• Box: The box represents the interquartile range (IQR), which spans from the 25th percentile (Q1) to the 75th percentile (Q3) of the data distribution. The length of the box indicates the spread of the middle 50% of the data. The line inside the box represents the median (50th percentile) of the data.
• Whiskers: The whiskers extend from the edges of the box to the furthest data points within 1.5 times the IQR from the quartiles. Any data points beyond the whiskers are considered outliers and are plotted individually as points.
• Outliers: The individual data points that fall outside the whiskers are plotted as individual points. These points represent values that are significantly different from the rest of the data and may need further investigation.

For the objective of this project I had no need to investigate the anomalies in the Discount column. I just used it as an example of what you can find when investigating a dataset.

Data Filtering

In order to create a prediction model I experimented using just one product from the entire dataset. So I filtered the data and then made a split in order to prepare it for the Prophet model

Filtering steps I took in this e-commerce dataset were:

1. Identifying the Most Popular Product

To demonstrate the forecasting process, I began by identifying the most popular product in the sales dataset. This involves analyzing the total quantity of each product sold over the entire time period (3 years. Result was one product (code: Go-Wo-NMDVGP) sold on 905 days.

2. Check its sales over time (3-year period)

As you can see sales seem to have a pretty regular pattern (but is that the whole story…?)

Building the Forecasting Model

My goal is to make forecasts to see what sales we can expect in the coming weeks or year. We have the historical data and need to feed it into a Prophet model. Steps I took are:

1. Split the Data

Having the target product, I split the sales data into training and testing sets. The training set will be used to train the Prophet model, while the testing set will be used to evaluate its performance.

2. Create the Prophet Model

With the data prepared, I created a Prophet model and fit it to the training data. Prophet’s intuitive interface allows to specify various parameters, such as seasonality and holidays, to customize the forecasting model according to our dataset.

I created this piece of python code to create the model:

This tells the model that I wanted to include seasonality on a weekly basis, and disregard any holidays.

Generating Forecasts

With the trained Prophet model, I can now generate forecasts for future time periods.
In my example, I have predicted sales for the next 52 weeks, thus providing valuable insights into long-term trends and potential fluctuations.

Plotting 3 trend components

To visualize the individual components of the trends and patterns, I have a created a 3-chart plot. This forecast plot depicts the long term trend, the expected yearly trend and the weekly trend.

The plot shows 3 components:

1. Trend Component: It shows the overall trend in the data. It helps visualize the long-term behavior of the time series data, allowing you to identify patterns and trends.
2. Seasonality Component: The second plot is the seasonality component of the forecast. In my case it illustrates the weekly seasonality. By examining this plot, you can identify seasonal fluctuations and understand how they contribute to the overall pattern of the time series.
3. Weeky Component: The plot on the bottom depicts the weekly sales or ordering pattern of this product. Monday’s and Saturday’s don’t seem to be very popular

Changepoints in the trend

The Prophet tool is good at indicating clear changepoints in trend, but in our case the trendline is steadily moving up, and does not show major breaks in the trend.

In red the main trend line.

The entire code

Check out all of the code of this project at Github: Nick Analytics – Demand Prediction with Prophet

Predicting House Prices: My Data Science Journey

Welcome to another blog post! Today, I’m delving into the exciting world of predictive modeling in real estate using the House Prices dataset from Kaggle’s Advanced Regression Techniques competition.
In this (brief) post, I’ll walk you through the entire process of preprocessing the data, building a machine learning model, and making predictions. So, grab your favorite beverage, and let’s dive in!

Introduction

The housing market is a complex ecosystem influenced by various factors ranging from location and size to architectural style and amenities. Predicting house prices accurately is crucial for both buyers and sellers. In this project, I aim to develop a robust predictive model that can estimate house prices based on several features provided in the dataset.

Exploring the Housing Dataset

Before diving into the pre-processing part, let’s first take a closer look at our dataset.
I started by conducting some general (statistical) checks using:

– the describe() function

– the info() function

- shape method

That gave me some knowledge about the summary statistics of the features (columns) and some key metrics like min/max values, and standard deviation of each column.

In order to gain more insights into the distribution of the sales prices I created a distribution plot that looks like this:

The distribution of the Sales Prices looks to be right skewed. This means:

• The Majority of Houses are Lower Priced: The peak of the distribution, where most of the data points lie, is towards the lower end of the price range. This suggests that there are more houses with lower prices compared to higher prices.

• Fewer Expensive Houses: As the distribution extends towards the higher prices, there are fewer houses with expensive prices. This could indicate that high-priced houses are less common or less frequently sold compared to lower-priced ones.

• Right Tail: The right-skewed nature of the distribution means that there are some houses with exceptionally high prices, leading to a longer tail on the right side of the distribution.

Preprocessing of the Data

Preprocessing of data is a vital step in every Data Science project. Most datasets are far from perfect and need to undergo vital steps for it to serve as input to a Machine Learning Model.

The steps I took in this housing dataset were:

1. Handling Missing Values

One of the initial challenges in any data science project is dealing with missing values. In my case most missing values existed in the ‘LotFrontage’ feature. In order to tackle this problem I created a Random Forest ML model to predict the values of this missing data. It would have been an option to remove this column from the dataset, or to remove rows with missing data, but I decided to pursue a more solid solution.

2. Checking for Outliers

Outliers can significantly impact the performance of predictive models. I utilized pairplots (see further) and the Z-score analysis to identify and remove outliers from the training data, ensuring the model learns from clean and reliable data.

3. Encoding Cat. Variables

Categorical variables need to be encoded into a numerical format before feeding them into machine learning models. I employed LabelEncoding to do so. LabelEncoding creates new columns where the categorial data is transformed into numbers.

Exploratory Data Analysis

Understanding Feature Relationships

Before getting into the model building, it’s essential to explore the relationships between our features and the target variable (Sales Price). I visualized these relationships using pairplots for both numeric and categorical features. This provides insights into potential correlations and trends.

In the 2 images below you can see the relationship between some numeric features and price (first one). The second one shows the count of categorical features.

Correlation Analysis

I calculated the correlation coefficients between numeric features and Sales Price and visualized them using a heatmap. This allowed us to identify the most influential features affecting house prices.

Building the Machine Learning Model

After completing the pre-processing steps, it is time to create the Machine Learning model. We are dealing with a challenge where we want to predict housing prices. As housing prices can take pretty much any value, we consider our predictions to be ‘continuous’ (as opposed to for example predicting a fixed outcome of ‘yes or no’, ‘true or false’ etc.).

Linear Regression

I started with a simple yet powerful Linear Regression model to predict the housing prices. After training the model on my pre-processed data, I evaluated its performance using metrics such as RMSE and R2 score.

In this first iteration the outcome was:

Validation score: 0.9244

Validation RMSE: 23816.8230

Test score: 0.8915

Test RMSE: 23576.6045

Interpretation:

• Overall, the linear regression model performs well on the test dataset, as indicated by the relatively high test score (0.8915) and the relatively low RMSE (23576.6045).
• The test score suggests that the model explains a large proportion of the variability in house prices using the available features.
• The RMSE indicates that, on average, the model’s predictions are approximately $23,576.60 away from the actual house prices in the test dataset.

Feature Importance

Understanding which features contribute the most to my model’s predictions is crucial for making informed decisions. I analysed feature importance using coefficients and permutation techniques, gaining insights into the key drivers of house prices.

The outcome of such analysis is depicted in the plot below. With this information I can take the most important features and neglect less importance features to improve the model even more.

This plot confirms the most important features (columns). Those are the size of the house, overall quality and the externals.

Model Evaluation and Prediction

Assessing Model Performance

I evaluated my model’s performance by comparing predicted values against actual values using scatter plots. This visual representation allowed me to identify areas of improvement and assess the model’s accuracy. I created a nice plot that instantly describes how well the model performs.

As you can see the model works pretty well and has a linear slope. There are a number of outliers, especially when prices are over 300k. The model seems to underestimate a number of cases. Further investigation, and maybe introducing a 2nd model for high-end homes, could improve my model’s accuracy.

Making Predictions

With the trained model, I’ve made predictions on the test dataset to participate in the Kaggle competition. I’m excited to see how my model performs against other competitors and contribute to the advancement of predictive modelling in real estate.

Stay tuned for updates on my model’s performance and further insights from the competition!

The entire code

Check out all of the code of this project at Github: Nick Analytics – House-Value-Prediction-with-ML