- Status
- Fertig
- Zeitraum
- 07/2025
- Technik
- scikit-learn
- XGBoost
- Docker
- pytest
5
models compared
0.40
best F1
14
pytest tests
Problem
Machine failures are rare compared with normal operation. A model that always predicts "healthy" looks accurate and is useless. I needed a comparison that would not hide the rare cases.
Approach
Pick the model by F1 instead of accuracy. In this data about 23% of samples are failures, so a model that predicts "healthy" every time already reaches 77% accuracy.
The sensor data is synthetic. I generate it with configurable failure patterns, which gives full control over the failure rate and keeps the whole pipeline reproducible from one seed.
Implementation
- 22 features derived from 8 raw sensor readings: interaction terms, z-score anomaly flags, an anomaly count per observation, and machine age bins.
- Automated cleaning and missing value imputation.
- Five classifiers trained on a stratified train and test split: logistic regression, random forest, gradient boosting, XGBoost and an RBF SVM.
- An evaluation report with a metrics table, a confusion matrix and feature importances, generated on every run.
- 14 pytest tests, fixed random seeds, and a Dockerfile that runs the pipeline with one command.
What I measured
Results from a 10,000-sample run:
| Model | Precision | Recall | F1 | ROC AUC |
|---|---|---|---|---|
| Logistic regression | 0.66 | 0.18 | 0.28 | 0.71 |
| Random forest | 0.69 | 0.24 | 0.35 | 0.77 |
| Gradient boosting | 0.62 | 0.29 | 0.40 | 0.77 |
| XGBoost | 0.58 | 0.28 | 0.38 | 0.74 |
| SVM (RBF) | 0.73 | 0.16 | 0.27 | 0.71 |
Gradient boosting wins on F1. Its recall is 0.29, which means it still misses most failures. That is the honest state of this pipeline.
Current status
Finished and public on GitHub. Oversampling the failure class and hyperparameter tuning are the next steps listed in the repository, and neither is done yet.
What I learned
Accuracy would have told me all five models were fine. F1 and recall showed that none of them is good enough to rely on yet.