Data Splitting in AI & Artificial Intelligence

How do you actually know if a machine learning model is intelligent? If you test a student using the exact same questions they studied, you're testing memory, not intelligence.

To solve this, we divide our dataset into three separate vaults: Training, Validation, and Testing. The model is forbidden from seeing the Test set during its learning phase. This mimics the real-world scenario where a model encounters completely new data.

The Training set is the largest portion of your data, typically 70 to 80 percent. This is the 'textbook' the model uses to learn patterns, adjust its internal mathematical weights, and understand the relationship between inputs and outputs.

If you evaluate your model's ultimate performance using this same data, you will fail to detect 'Overfitting'—when the model simply memorizes the answers instead of learning true logic.

Next is the Validation set. Imagine you are building a model and deciding whether it should have 5 layers or 10 layers. You train both models, then use the Validation set to test them. Whichever performs better on this 'practice quiz' becomes your final architecture.

Finally, the Test set. This is the vault. It is locked away during the entire development process. You only run predictions on the Test set ONCE at the very end of your project. This gives you the unbiased truth of how your model will perform in production.

Reproducibility is a pillar of science. When we randomly split data, we use a 'Random State' (often the number 42). This ensures the random number generator produces the exact same split every time the code runs, allowing other engineers to verify your exact results.

Without a fixed seed, your model's accuracy would fluctuate wildly every time you ran the script just based on 'lucky' splits.

But what if your dataset has 99 normal emails and 1 spam email? If you split randomly, your training set might get 0 spam emails, making it impossible to learn. 'Stratification' forces the split to maintain the exact original ratio of classes in both Train and Test sets.

Failing to stratify imbalanced datasets is a critical flaw. You might build a model that scores 99% accuracy simply because it blindly predicts 'Not Spam' every time, having never seen a spam email during training.

Welcome to Data Splitting. How do you actually know if a machine learning model is intelligent? If you test a student using the exact same questions they studied the night before, you aren't testing their intelligence, you're testing their memory.

To solve this, we divide our dataset into three separate vaults: Training, Validation, and Testing. The model is absolutely forbidden from seeing the Test set during its learning phase. This mimics the real-world scenario where a model encounters completely new data.

The Training set is the largest portion of your data, typically 70 to 80 percent. This is the 'textbook' the model uses to learn patterns, adjust its internal mathematical weights, and understand the relationship between inputs and outputs.

If you evaluate your model's ultimate performance using the exact same data it used to learn its patterns, what fundamental error are you committing?

• →Underfitting, because the model won't learn enough details.
• →You are failing to detect Overfitting, because the model might have simply memorized the answers.

Next is the Validation set. Imagine you are building a model and deciding whether it should have 5 layers or 10 layers. You train both models, then use the Validation set to test them. Whichever performs better on this 'practice quiz' becomes your final architecture.

Finally, the Test set. This is the vault. It is locked away during the entire development process. You only run predictions on the Test set ONCE at the very end of your project. This gives you the harsh, unbiased reality of how your model will perform in production.

Reproducibility is a pillar of science. When we randomly split data, we use a 'Random State' (often the number 42). This ensures the random number generator produces the exact same split every time the code runs, allowing other engineers to verify your exact results.

What is the primary function of the 'Validation Set' in the machine learning workflow?

• →To provide the final metric that you report to your boss.
• →To compare different model versions and tune hyperparameters during development.

But what if your dataset has 99 normal emails and 1 spam email? If you split randomly, your training set might get 0 spam emails, making it impossible to learn. 'Stratification' forces the split to maintain the exact original ratio of classes in both Train and Test sets.

Failing to stratify imbalanced datasets is a critical flaw. You might build a model that scores 99% accuracy simply because it blindly predicts 'Not Spam' every time, having never seen a spam email during training. Stratification protects against this 'lucky split' illusion.

What does using 'stratify=y' during your data split actually guarantee?

• →It guarantees the training loop will run significantly faster.
• →It guarantees the Train and Test sets will have the same proportion of target classes as the original data.

Splitting complete! You have established a rigorous, scientifically sound environment to train your models. Your training set is robust, your validation lab is ready for tuning, and your test vault is securely locked for the final evaluation.

With your data properly prepared, cleaned, scaled, and safely split, you have crossed the threshold from data processing into model building. Next, we step into the world of Supervised Learning algorithms.

Compute Real Split Sizes. Finish computing how many samples go into train vs test for a given split ratio.