User Guide
Prerequisites
Clone the pipeline's repository and checkout the pipeline branch.
git clone git@github.com:ZHAW-certAInty/toolbox.git
git checkout pipeline
Adapt the .env file located in the repository's root directory to match your specific requirements. You can use the following example as a reference with the most crucial lines highlighted:
shorty=bily
jobname=bily-yolov5
ntasks=1
cpus=8
mem=48G
gpu=1
shm_size=32g
network_docker=bily_certainty-pipeline
airflow_home_docker=/opt/airflow
pipeline_home_host=/cluster/home/bily/projects/toolbox/bily
The pipeline_home_host is determined by the location where the pipeline is running on DGX.
By default, network_docker is composed of the pipeline's parent folder and the "certainty-pipeline" suffix.
Prior to conducting the experiment, it's essential to upload the dataset that the ML model operates on to the pipeline's data repository.
To accomplish this, please reach out to your designated MLOps engineer responsible for the pipeline.
As of now, the pipeline's repository provides three example dags located in the /dags folder.
Model Training
To initiate model training, you can refer to the YOLOv5 example provided in dag_bily_yolov5.py.
Model Testing
For testing a model that has been trained using the pipeline, you can follow the YOLOv5 example outlined in dag_bily_test_yolov5.py.
Reusing a Trained Model
If you wish to retrain a model based on previous results, please consult dag_bily_using_trained_yolov5_example.py.