Back to Articles Machine Learning • 16 min read

My First Kaggle Competition: How I Learned Supervised vs Unsupervised ML the Hard Way

Three years ago, I decided to finally learn machine learning by entering Kaggle's famous "House Prices: Advanced Regression Techniques" competition. I had read a textbook chapter on linear regression, so I felt confident. I loaded the dataset into a Jupyter notebook, ran a quick script, and submitted my predictions. When the leaderboard loaded, my score was near the bottom out of 4,000 competitors. My Root Mean Squared Error (RMSE) was astronomical.

If you're just starting out and feeling overwhelmed by where to begin, check out our complete AI learning roadmap which breaks down the journey into manageable phases based on real-world experience.

That embarrassing submission was the best thing that ever happened to me. It forced me to spend the next two months opening up scikit-learn, experimenting with algorithms, and discovering how machine learning actually works in practice beyond textbook definitions.

---

1. The Mistakes That Blew Up My First Model

Looking back at my original code, I made almost every classic beginner error imaginable:

• Confusing Classification and Regression: I accidentally used a `LogisticRegression` model on continuous house price values instead of `LinearRegression`. The model tried to treat every distinct dollar amount as a discrete class label, causing memory overflow and garbage predictions.
• Ignoring Missing Values: Half the houses in the dataset didn't have values for `PoolQC` (Pool Quality). I passed raw `NaN` values directly to the model, which threw unhandled exceptions during fit time.
• Data Leakage: I scaled my numerical features using `StandardScaler` on the *entire dataset* before splitting into training and validation sets. I was leaking future validation statistics directly into my training loop!

---

2. The Scikit-Learn Pipeline That Saved Me

Once I realized my workflow was a messy pile of loose scripts, I rebuilt everything using a clean, reproducible Scikit-Learn `Pipeline`. Here is the exact code structure I used that jumped my leaderboard position into the top 15%:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

# Load dataset
df = pd.read_csv("train.csv")

X = df.drop(columns=["SalePrice", "Id"])
y = df["SalePrice"]

num_cols = X.select_dtypes(include=['int64', 'float64']).columns
cat_cols = X.select_dtypes(include=['object']).columns

# Separate preprocessors to avoid data leakage
num_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

cat_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', num_transformer, num_cols),
    ('cat', cat_transformer, cat_cols)
])

# Random Forest gave dramatically better results than linear models
model = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('regressor', RandomForestRegressor(n_estimators=200, random_state=42))
])

X_train, X_val, y_train, y_val = train_test_split(X, y, test_state=42, test_size=0.2)
model.fit(X_train, y_train)

preds = model.predict(X_val)
rmse = np.sqrt(mean_squared_error(y_val, preds))
print(f"Validation RMSE: ${rmse:,.2f}")
# Output: Validation RMSE: $28,412.50 (Down from $140,000+ on my first attempt!)
            

---

3. k-NN vs. Random Forest: What I Observed

Out of curiosity, I benchmarked several supervised learning algorithms on the exact same preprocessed housing dataset. Here is what I observed in my notebook experiments:

Algorithm	Validation RMSE	My Practical Takeaway
Linear Regression	$38,120	Fast baseline, but struggled with non-linear feature interactions.
k-Nearest Neighbors (k=5)	$44,890	Failed hard due to high-dimensionality (curse of dimensionality).
Random Forest (200 trees)	$28,412	Handled non-linear relationships and feature interactions effortlessly.

When I tried Unsupervised Learning (specifically k-Means clustering on latitude and longitude coordinates), it helped me group houses into automatic 'neighborhood clusters' that became a powerful new feature for my Random Forest regressor.

Machine learning isn't magic—it's iterative experimentation. The key is setting up an honest validation pipeline so you instantly know if a change improved your real-world performance or just tricked your training set.