Exporting MLflow Experiments from Restricted HPC Systems
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
This makes it impossible to track and manage experiments on MLflow. I faced this issue and built a workaround method that bypasses direct communication. We will focus on:<\/p>\n I have deployed Charmed MLflow (MLflow server, MySQL, MinIO) using juju, and the whole thing is hosted on MicroK8s localhost. You can find the installation guide from Canonical here<\/a>.<\/p>\n Make sure you have Python<\/em> loaded on your HPC and installed on your MLflow server.For this entire article, I assume you have Python 3.2<\/em>. You can make changes accordingly.<\/p>\n 1) Create a virtual environment<\/p>\n 2) Install MLflow<\/p>\n 1) Install mlflow-export-import<\/p>\n 1) Decide on a port where you want the local MLflow server to run. You can use the below command to check if the port is free (should not contain any process IDS):<\/p>\n 2) Set the environment variable for applications that want to use MLflow:<\/p>\n 3) Start the MLflow server using the below command:<\/p>\n Here, we set the path to the local storage in a folder called mlruns. Metadata like experiments, runs, parameters, metrics, tags and artifacts like model files, loss curves, and other images will be stored inside the mlruns directory. We can set the host as 0.0.0.0 or 127.0.0.1(more secure). Since the whole process is short-lived, I went with 0.0.0.0. Finally, assign a port number that is not used by any other application.<\/p>\n (Optional) Sometimes, your HPC might not detect libpython3.12,<\/em> which basically makes Python run. You can follow the steps below to find and add it to your path.<\/p>\n Search for libpython3.12<\/em>:<\/p>\n Returns something like: \/path\/to\/python\/3.12\/lib\/libpython3.12.so.1.0<\/p>\n Set the path as an environment variable:<\/p>\n 4) We will export the experiment data from the mlruns local storage directory to a temp folder:<\/p>\n (Optional) Running the export_experiment<\/em> function on the HPC bash shell may cause thread utilisation errors like:<\/p>\n This happens because MLflow internally uses SciPy<\/em> for artifacts and metadata handling, which requests threads through OpenBLAS,<\/em> which is more than the allowed limit set by your HPC. In case of this issue, limit the number of threads by setting the following environment variables.<\/p>\n \u00a0If the issue persists, try reducing the thread limit to 2.<\/p>\n 5) Transfer experiment runs to MLflow Server:<\/p>\n Move everything from the HPC to the temporary folder on the MLflow server.<\/p>\n 6) Stop the local MLflow server and clean up the ports:<\/p>\n Our goal is to transfer experimental data from the tmp folder to MySQL<\/em> and MinIO<\/em>.\u00a0<\/p>\n 1) Since MinIO is Amazon S3 compatible, it uses boto3 (AWS Python SDK) for communication. So, we will set up proxy AWS-like credentials and use them to communicate with MinIO using boto3.<\/p>\n 2) Below are the commands to transfer the data.<\/p>\n Setting the MLflow server and MinIO addresses in our environment. To avoid repeating this, we can enter this in our .bashrc file.<\/p>\n \u00a0All the experiment files can be found under the exported_runs folder in the tmp directory. The import-experiment<\/em> function finishes our job.<\/p>\n The workaround helped me in tracking experiments even when communications and data transfers were restricted on my HPC cluster. Spinning up a local MLflow server instance, exporting experiments, and then importing them to my remote MLflow server provided me with flexibility without having to change my workflow.\u00a0<\/p>\n However, if you are dealing with sensitive data, make sure your transfer method is secure. Creating cron jobs and automation scripts could potentially remove manual overhead. Also, be mindful of your local storage, as it is easy to fill up.<\/p>\n In the end, if you are working in similar environments, this article can provide you with a solution without requiring any admin privileges in a short time. Hopefully, this helps teams who are stuck with the same issue. Thanks for reading this article!<\/p>\n You can connect with me on LinkedIn<\/a>.<\/p>\n The post Exporting MLflow Experiments from Restricted HPC Systems<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
Prerequisites<\/h2>\n
On HPC:<\/strong><\/h5>\n
python3 -m venv mlflow\nsource mlflow\/bin\/activate<\/code><\/pre>\npip install mlflow<\/code><\/pre>\nOn both HPC and MLflow Server:<\/strong><\/h5>\n
pip install git+https:\/\/\/github.com\/mlflow\/mlflow-export-import\/#egg=mlflow-export-import<\/code><\/pre>\nOn HPC:<\/h2>\n
lsof -i :<port-number><\/code><\/pre>\nexport MLFLOW_TRACKING_URI=http:\/\/localhost:<port-number><\/code><\/pre>\nmlflow server \n --backend-store-uri file:\/path\/to\/local\/storage\/mlruns \n --default-artifact-root file:\/path\/to\/local\/storage\/mlruns \n --host 0.0.0.0 \n --port 5000<\/code><\/pre>\nfind \/hpc\/packages -name \"libpython3.12*.so*\" 2>\/dev\/null<\/code><\/pre>\nexport LD_LIBRARY_PATH=\/path\/to\/python\/3.12\/lib:$LD_LIBRARY_PATH<\/code><\/pre>\npython3 -m mlflow_export_import.experiment.export_experiment --experiment \"<experiment-name>\" --output-dir \/tmp\/exported_runs<\/code><\/pre>\nOpenBLAS blas_thread_init: pthread_create failed for thread X of 64: Resource temporarily unavailable<\/code><\/p>\nexport OPENBLAS_NUM_THREADS=4\nexport OMP_NUM_THREADS=4\nexport MKL_NUM_THREADS=4<\/code><\/pre>\nrsync -avz \/tmp\/exported_runs <mlflow-server-username>@<host-address>:\/tmp<\/code><\/pre>\nlsof -i :<port-number>\nkill -9 <pid><\/code><\/pre>\nOn MLflow Server:<\/h2>\n
juju config mlflow-minio access-key=<access-key> secret-key=<secret-access-key><\/code><\/pre>\nexport MLFLOW_TRACKING_URI=\"http:\/\/<cluster-ip_or_nodeport_or_load-balancer>:port\"\nexport MLFLOW_S3_ENDPOINT_URL=\"http:\/\/<cluster-ip_or_nodeport_or_load-balancer>:port\"<\/code><\/pre>\npython3 -m mlflow_export_import.experiment.import_experiment --experiment-name \"experiment-name\" --input-dir \/tmp\/exported_runs<\/code><\/pre>\nConclusion<\/h2>\n