Hire Me
← Back to projects
ML4Cyber — Code Vulnerability

ML4Cyber — Code Vulnerability

  • Python
  • Machine learning classification pipelines
  • Static Application Security Testing (SAST)
  • Git
  • Branch management
  • False-positive/robustness evaluation
  • Technical report writing

ML Vinci Tool: Better Vulnerability Classification

Big Data Analytics · SOEN 471 / SOEN 6111 · Concordia University · Winter 2024 · Team 25

Improving the machine-learning behind ML4Cyber's ML Vinci Tool, a desktop app that triages static-analysis (SAST) findings: true or false positive, and how risky?


The problem

Static application security testing (SAST) tools such as Fortify, SpotBugs, PMD, ESLint, Pylint, CppCheck and PHP_CodeSniffer produce long reports full of false positives. ML4Cyber's ML Vinci Tool reads those reports (XML, JSON, CSV or FPR), learns from a small labelled sample (around 10% of findings), and then:

  1. classifies every finding as a true or false positive, and
  2. predicts a risk level for it.

When we started, the risk-prediction module didn't work properly, the risk classifier performed poorly, the decision-tree model was broken, and only a few input formats were supported.

Our goals

#GoalPriority
1Try other classifiers for risk prediction, or improve the existing voting classifier (data preparation, model building, tuning)Planned
2Improve the true/false-positive classifier through data transformation and hyperparameter tuning, with a focus on cutting the false-positive ratePlanned
3Fix the decision-tree implementationIf time allowed
4Import more report formats (for example via SonarQube)If time allowed

What we built

Models added to the tool's training pipeline, all selectable from the GUI:

ModelDetails
Random ForestStarted with stratified k-fold, then moved to leave-one-out cross-validation to suit the small labelled sets
XGBoostGrid-searched learning rate, then a tuned model (max_depth=3, 2000 estimators, row and column subsampling, AUC metric)
AdaBoost50 estimators with a configurable learning rate
Logistic RegressionHyperparameters chosen by GridSearchCV (10-fold, F1 score)

We also made these supporting changes:

  • Model evaluation now handles both probabilistic and non-probabilistic outputs without crashing.
  • Model persistence switched from .h5 to joblib for scikit-learn models.
  • Testing files were added for the GUI modules, the notebook was synced with the codebase, and our changes were documented with comments.

The existing models (neural network, MLP, decision tree) remain available alongside the new ones.

Running the tool

python -m venv .venv                      # Python 3.8 or 3.9 (TensorFlow 2.5)
# Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
pip install -r requirements_for_latest_version.txt
python gui/gui/app.py

Sample SAST reports to try are in gui/gui/Input/Sample Input Reports/.

The latest client code expects a GitHub token and repository path in datasets.py, datasets_labeler.py, predictions.py and risk.py for saving projects. Making this optional was on the client's wishlist (see Improvement Recommendations.docx).

Repository layout

gui/gui/
  app.py                    entry point (PyQt5)
  datasets*.py              load SAST reports, pick columns
  datasets_labeler*.py      label findings as true or false positive
  models*.py                choose, train and evaluate models  ← most of our work is in models_workers.py
  predictions*.py           true/false-positive predictions
  risk*.py                  risk model, risk predictions
  visualize*.py             charts
  test_*.py                 GUI module tests
  Input/                    sample reports + Code Dx datasets and notebook
Improvement Recommendations.docx   client's wishlist for the tool
requirements_for_latest_version.txt

Team 25

MemberStudent ID
Mohamed Bilel Besbes40296099
Victoria Castelli40117055
Tarek Ferdous40221751
Mohammad Shahidul Alam40230431

Instructor: Dr. Tristan Glatard · Proposal submitted: 21 February 2024 · Project: 24 January – 10 April 2024

Acknowledgements

ML4Cyber provided the ML Vinci Tool source code and sample client data for this course project. The tool and its data belong to ML4Cyber. This repository records our team's academic work on it and is not licensed for redistribution.