Data Splitting Strategy
The Three Sets
Training Set (60-80%)
Model learns from this data
Largest portionValidation Set (10-20%)
Hyperparameter tuning
Model selection
Early stopping decisionsTest Set (10-20%)
Final evaluation only
Never used during training
Simulates real-world performanceCross-Validation
K-Fold Cross-Validation:
1. Split data into K folds
2. Train K times, each with different fold as validation
3. Average the results
Benefits:
More robust evaluation
Uses all data for both training and validation
Reduces variance in estimatesCommon Mistakes to Avoid
Data leakage: Test data influences training
Temporal leakage: Future data in training
Class imbalance not preserved in splits