ML Vinci Tool: Better Vulnerability Classification
Big Data Analytics · SOEN 471 / SOEN 6111 · Concordia University · Winter 2024 · Team 25
Improving the machine-learning behind ML4Cyber's ML Vinci Tool, a desktop app that triages static-analysis (SAST) findings: true or false positive, and how risky?
The problem
Static application security testing (SAST) tools such as Fortify, SpotBugs, PMD, ESLint, Pylint, CppCheck and PHP_CodeSniffer produce long reports full of false positives. ML4Cyber's ML Vinci Tool reads those reports (XML, JSON, CSV or FPR), learns from a small labelled sample (around 10% of findings), and then:
- classifies every finding as a true or false positive, and
- predicts a risk level for it.
When we started, the risk-prediction module didn't work properly, the risk classifier performed poorly, the decision-tree model was broken, and only a few input formats were supported.
Our goals
| # | Goal | Priority |
|---|---|---|
| 1 | Try other classifiers for risk prediction, or improve the existing voting classifier (data preparation, model building, tuning) | Planned |
| 2 | Improve the true/false-positive classifier through data transformation and hyperparameter tuning, with a focus on cutting the false-positive rate | Planned |
| 3 | Fix the decision-tree implementation | If time allowed |
| 4 | Import more report formats (for example via SonarQube) | If time allowed |
What we built
Models added to the tool's training pipeline, all selectable from the GUI:
| Model | Details |
|---|---|
| Random Forest | Started with stratified k-fold, then moved to leave-one-out cross-validation to suit the small labelled sets |
| XGBoost | Grid-searched learning rate, then a tuned model (max_depth=3, 2000 estimators, row and column subsampling, AUC metric) |
| AdaBoost | 50 estimators with a configurable learning rate |
| Logistic Regression | Hyperparameters chosen by GridSearchCV (10-fold, F1 score) |
We also made these supporting changes:
- Model evaluation now handles both probabilistic and non-probabilistic outputs without crashing.
- Model persistence switched from
.h5tojoblibfor scikit-learn models. - Testing files were added for the GUI modules, the notebook was synced with the codebase, and our changes were documented with comments.
The existing models (neural network, MLP, decision tree) remain available alongside the new ones.
Running the tool
python -m venv .venv # Python 3.8 or 3.9 (TensorFlow 2.5)
# Windows: .venv\Scripts\activate macOS/Linux: source .venv/bin/activate
pip install -r requirements_for_latest_version.txt
python gui/gui/app.py
Sample SAST reports to try are in gui/gui/Input/Sample Input Reports/.
The latest client code expects a GitHub token and repository path in
datasets.py,datasets_labeler.py,predictions.pyandrisk.pyfor saving projects. Making this optional was on the client's wishlist (seeImprovement Recommendations.docx).
Repository layout
gui/gui/
app.py entry point (PyQt5)
datasets*.py load SAST reports, pick columns
datasets_labeler*.py label findings as true or false positive
models*.py choose, train and evaluate models ← most of our work is in models_workers.py
predictions*.py true/false-positive predictions
risk*.py risk model, risk predictions
visualize*.py charts
test_*.py GUI module tests
Input/ sample reports + Code Dx datasets and notebook
Improvement Recommendations.docx client's wishlist for the tool
requirements_for_latest_version.txt
Team 25
| Member | Student ID |
|---|---|
| Mohamed Bilel Besbes | 40296099 |
| Victoria Castelli | 40117055 |
| Tarek Ferdous | 40221751 |
| Mohammad Shahidul Alam | 40230431 |
Instructor: Dr. Tristan Glatard · Proposal submitted: 21 February 2024 · Project: 24 January – 10 April 2024
Acknowledgements
ML4Cyber provided the ML Vinci Tool source code and sample client data for this course project. The tool and its data belong to ML4Cyber. This repository records our team's academic work on it and is not licensed for redistribution.
