If you\u2019re an Anaconda user, you know that\u00a0conda environments<\/em><\/a>\u00a0help you manage package dependencies, avoid compatibility conflicts, and share your projects with others. Unfortunately, they can also take over your computer\u2019s hard drive.<\/p>\n I write lots of computer tutorials and to keep them organized, each has a dedicated folder structure complete with a Conda Environment<\/a>. This worked great at first, but soon my computer\u2019s performance degraded, and I noticed that my SSD was filling up. At one point I had only 13 GB free.<\/p>\n Conda helps manage this problem by storing downloaded package files in a single \u201ccache\u201d ( Because conda caches\u00a0every downloaded package<\/em>,\u00a0 In addition, large, compiled binaries like\u00a0OpenCV<\/a>\u00a0may require\u00a0full copies<\/em>\u00a0in the environment\u2019s directory, and each environment requires a copy of the Python interpreter (at 100\u2013200 MB). All these issues can bloat conda environments to several gigabytes.<\/p>\n In this\u00a0Quick Success Data Science<\/em>\u00a0project, we\u2019ll look at some techniques for reducing the storage requirements for conda environments, including those stored in default locations and dedicated folders.<\/p>\n Below are some Memory Management<\/a> techniques that will help you reduce conda\u2019s storage footprint on your machine. We\u2019ll discuss each in turn.<\/p>\n Cleaning the package cache is the first and easiest step for freeing up memory. Even after deleting environments, conda keeps the related package files in the cache. You can free up space by removing these unused packages and their associated\u00a0tarballs<\/em>\u00a0(compressed package files), logs,\u00a0index caches<\/em>\u00a0(metadata stored in conda), and temporary files.<\/p>\n Conda permits an optional \u201cdry run\u201d to see how much memory will be reclaimed. You\u2019ll want to run this from either the terminal or Anaconda Prompt in your\u00a0base<\/em>\u00a0environment:<\/p>\n To commit, run:<\/p>\n Here\u2019s how this looks on my machine:<\/p>\n This process trimmed a healthy 6.28 GB and took several minutes to run.<\/p>\n Creating a few environments for\u00a0specialized tasks<\/em>\u00a0\u2014 like computer vision or geospatial work \u2014 is more memory efficient than using dedicated environments for each\u00a0project<\/em>. These environments would include basic packages plus ones for the specific task (such as OpenCV, scikit-image, and PIL for computer vision).<\/p>\n An advantage of this approach is that you can easily keep all the packages up to date and link the environments to multiple projects. However, this won\u2019t work if some projects require different versions of the shared packages.<\/p>\n If you don\u2019t have enough storage sites or want to preserve legacy projects efficiently, consider using\u00a0environment<\/em>\u00a0or\u00a0specifications\u00a0<\/em>files. These small files record an environment\u2019s\u00a0contents<\/em>, allowing you to rebuild it later.<\/p>\n Saving conda environments in this manner reduces their size on disk from gigabytes to a few kilobytes. Of course, you\u2019ll have to recreate the environment to use it. So, you\u2019ll want to avoid this technique if you frequently revisit projects that link to the archived environments.<\/p>\n NOTE: Consider using\u00a0Mamba<\/a>, a drop-in replacement for conda, for faster rebuilds. As the docs say, \u201cIf you know conda, you know Mamba!\u201d<\/p>\n<\/blockquote>\n Using Environment Files:<\/strong>\u00a0An\u00a0environmental file<\/em>\u00a0is a small file that lists all the packages and versions installed in an environment, including those installed using Python\u2019s package installer (pip<\/a>). This helps you both restore an environment and share it with others.<\/p>\n The environment file is written in\u00a0YAML<\/em><\/a>\u00a0(.yml<\/em>), a human-readable data-serialization format for data storage. To generate an environment file, you must activate and then export the environment. Here\u2019s how to make a file for an environment named\u00a0my_env<\/em>:<\/p>\n You can name the file any valid filename but be careful as an existing file with the same name will be overwritten.<\/p>\n By default, the environment file is written to the\u00a0user\u00a0<\/em>directory. Here\u2019s a truncated example of the file\u2019s contents:<\/p>\n You can now remove your conda environment and reproduce it again with this file. To remove an environment, first deactivate it and then run the\u00a0 If the conda environment exists outside of Anaconda\u2019s default\u00a0envs<\/em>\u00a0folder, then include the directory path to the environment, as so:<\/p>\n Note that this archiving technique will only work perfectly if you continue to use the same operating system, such as Windows or macOS. This is because solving for dependencies can introduce packages that might not be compatible across platforms.<\/p>\n To restore a conda environment using a file, run the following, where\u00a0 You can also use the environment file to recreate the environment on your D: drive. Just provide the new path when using the file. Here\u2019s an example:<\/p>\n For more on environment files, including how to manually produce them, visit the\u00a0docs<\/a>.<\/p>\n Using Specifications Files:\u00a0<\/strong>If you haven\u2019t installed any packages using pip, you can use a\u00a0specifications file<\/em>\u00a0to reproduce a conda environment on the same operating system. To create a specification file, activate an environment, such as\u00a0my_env<\/em>, and enter the following command:<\/p>\n This produces the following output, truncated for brevity:<\/p>\n Note that the\u00a0 You can now remove the environment as described in the previous section.<\/p>\n To re-create\u00a0my_env<\/em>\u00a0using this text file, run the following with a proper directory path:<\/p>\n The\u00a0 The following command will preserve an environment but remove it from your system (where\u00a0my_env<\/em>\u00a0represents the name of your environment):<\/p>\n To restore the environment later run this command:<\/p>\n This technique won\u2019t save as much memory as the text file option. However, you won\u2019t need to re-download packages when restoring an environment, which means it can be used without internet access.<\/p>\n By default, conda stores all environments in a default location. For Windows, this is under the\u00a0\u2026anaconda3envs<\/em>\u00a0folder. You can see these environments by running the command\u00a0 Using a Single Environments Folder:<\/strong>\u00a0If your system supports an external or secondary drive, you can configure conda to store environments there to free up space on your primary disk. Here\u2019s the command; you\u2019ll need to substitute your specific path:<\/p>\n If you enter a path to your D drive, such as\u00a0D:conda_envs<\/em>, conda will create new environments at this location.<\/p>\n This technique works well when your external drive is a fast SSD and when you\u2019re storing packages with large dependencies, like TensorFlow. The downside is slower performance. If your OS and notebooks remain on the primary drive, you may experience some read\/write latency when running Python.<\/p>\n In addition, some OS settings may power down idle external drives, adding a delay when they spin back up. Tools like Jupyter may struggle to locate conda environments if the drive letter changes, so you\u2019ll want to use a fixed drive letter and ensure that the correct kernel paths are set.<\/p>\n Using Multiple Environment Folders:<\/strong>\u00a0Instead of using a single\u00a0 For example, suppose you have a project on your Windows D: drive in a folder called\u00a0D:projectsgeospatial<\/em>. To place the project\u2019s conda environment in this folder, loaded with\u00a0 Of course, you can call\u00a0env<\/em>\u00a0something more descriptive, like\u00a0geospatial_env<\/em>.<\/p>\n As with the previous example, environments stored on a different disk can cause performance issues.<\/p>\n Special Note on JupyterLab:<\/strong>\u00a0Depending on how you launch JupyterLab, its default behavior may be to open in your\u00a0user<\/em>\u00a0directory (such as,\u00a0C:Usersyour_user_name<\/em>). Since its file browser is restricted to the directory from which it is launched, you won\u2019t see directories on other drives like\u00a0 For example, in Anaconda Prompt, type:<\/p>\n followed by:<\/p>\n Now, you will be able to pick from kernels on the D: drive.<\/p>\n For more options on changing JupyterLab\u2019s working directory, ask an AI about \u201chow to change Jupyter\u2019s default working directory\u201d or \u201chow to create a Symlink to\u00a0 Moving Existing Environments:<\/strong>\u00a0You should never manually move a conda environment, such as by cutting and pasting to a new location. This is because conda relies on internal paths and metadata that can become invalid with location changes.<\/p>\n Instead, you should\u00a0clone\u00a0<\/em>existing environments to another drive. This will\u00a0duplicate<\/em>\u00a0the environment, so you\u2019ll need to manually remove it from its original location.<\/p>\n In the following example, we use the\u00a0 NOTE:<\/strong>\u00a0Consider exporting your environment to a\u00a0YAML<\/em>\u00a0file (as described in Section 3 above) before cloning. This allows you to recreate the environment if something goes wrong with the clone procedure.<\/p>\n<\/blockquote>\n Now, when you run\u00a0 Again, latency issues may affect these setups if you\u2019re working across two disks.<\/p>\n You may be wondering, is it better to move a conda environment using an environment (YAML) file or to use If your primary drive is low on space, you can move the package cache to a larger external or secondary drive using this command:<\/p>\n In this example, packages are now stored on the D drive (D:conda_pkgs<\/em>) instead of the default location.<\/p>\n If you\u2019re working in your primary drive and both drives are SSD, then latency issues should not be significant. However, if one of the drives is a slower HDD, you can experience slowdowns when creating or updating environments. If D: is an external drive connected by USB, you may see significant slowdowns for large environments.<\/p>\n You can mitigate some of these issues by keeping the package cache ( One last thing to consider is\u00a0backups<\/em>. Primary drives may have routine backups scheduled but secondary or external drives may not. This puts you at risk of losing all your environments.<\/p>\n If your project doesn\u2019t require conda\u2019s extensive package management system for handling heavy dependencies (like TensorFlow or GDAL), you can significantly reduce disk usage with a Python\u00a0virtual environment<\/em>\u00a0( To create a\u00a0 This type of environment has a small base installation. A minimal conda environment takes up about 200 MB and includes multiple utilities, such as\u00a0 Conda also caches package tarballs in\u00a0 In general, you\u2019ll want to consider\u00a0 You can find details on how to activate and use Python virtual environments in the\u00a0 High impact\/low disruption memory management techniques for conda environments include cleaning the package cache and storing little-used environments as YAML or text files. These methods can save many gigabytes of memory while retaining Anaconda\u2019s default directory structure.<\/p>\n Other high impact methods include moving the package cache and\/or conda environments to a secondary or external drive. This will resolve memory problems but may introduce latency issues, especially if the new drive is a slow HDD or uses a USB connection.<\/p>\n For simple environments, you can use a Python virtual environment ( The post Don\u2019t Let Conda Eat Your Hard Drive<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_7wR-7xQoglU2OcEpWQksNA.webp?resize=502%2C142&ssl=1\")
pkgs_dirs<\/code>). When you install a package, conda checks for it in the package cache before downloading. If not found, conda will download and extract the package and link the files to the active environment. Because the cache is \u201cshared,\u201d different environments can use the same downloaded files without duplication.<\/p>\npkgs_dirs<\/code>\u00a0can grow to many gigabytes. And while conda links to shared packages in the cache, there is still a need to store some packages in the environment folder. This is mainly to avoid\u00a0version conflicts<\/em>, where different environments need different versions of the same\u00a0dependency\u00a0<\/em>(a package required to run another package).<\/p>\n
\nMemory Management Techniques<\/h2>\n
\n
venv<\/code>)<\/li>\n<\/ol>\n1. Cleaning the Package Cache<\/h3>\n
conda clean --all --dry-run<\/code><\/pre>\nconda clean --all<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_0LqvoSRUToXiuSUqq-jg5Q.webp?resize=502%2C552&ssl=1\")
2. Sharing Task-based Environments<\/h3>\n
3. Archiving with Environment and Specifications Files<\/h3>\n
\n
conda activate my_env\n conda env export > my_env.yml<\/code><\/pre>\nname: C:Usershannaquick_successfed_hikesfed_env\nchannels:\n - defaults\n - conda-forge\ndependencies:\n - asttokens=2.0.5=pyhd3eb1b0_0\n - backcall=0.2.0=pyhd3eb1b0_0\n - blas=1.0=mkl\n - bottleneck=1.3.4=py310h9128911_0\n - brotli=1.0.9=ha925a31_2\n - bzip2=1.0.8=he774522_0\n - ca-certificates=2022.4.26=haa95532_0\n - certifi=2022.5.18.1=py310haa95532_0\n - colorama=0.4.4=pyhd3eb1b0_0\n - cycler=0.11.0=pyhd3eb1b0_0\n - debugpy=1.5.1=py310hd77b12b_0\n - decorator=5.1.1=pyhd3eb1b0_0\n - entrypoints=0.4=py310haa95532_0\n\n ------SNIP------<\/code><\/pre>\nremove<\/code>\u00a0command (where\u00a0ENVNAME<\/code>\u00a0is the name of your environment):<\/p>\nconda deactivate\nconda remove -n ENVNAME --all<\/code><\/pre>\nconda remove -p PATHENVNAME --all<\/code><\/pre>\nmy_env<\/code>\u00a0represents your conda environment name and\u00a0environment.yml<\/code>\u00a0represents your environment file:<\/p>\n conda env create -n my_env -f directorypathtoenvironment.yml<\/code><\/pre>\nconda create --prefix D:my_envsmy_new_env --file environment.yml<\/code><\/pre>\n conda list --explicit > exp_spec_list.txt<\/code><\/pre>\n # This file may be used to create an environment using:\n # $ conda create --name <env> --file <this file>\n # platform: win-64\n @EXPLICIT\n https:\/\/conda.anaconda.org\/conda-forge\/win-64\/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2\n https:\/\/conda.anaconda.org\/conda-forge\/noarch\/tzdata-202xx-he74cb21_0.tar.bz2\n\n------snip------<\/code><\/pre>\n--explicit<\/code>\u00a0flag ensures that the targeted platform is annotated in the file, in this case,\u00a0# platform: win-64<\/em>\u00a0in the third line.<\/p>\nconda create -n my_env -f directorypathtoexp_spec_list.txt<\/code><\/pre>\n4. Archiving Environments with conda-pack<\/h3>\n
conda-pack<\/code>\u00a0command lets you archive a conda environment before removing it. It packs the entire environment into a compressed archive with the extension:\u00a0.tar.gz<\/em>. It\u2019s handy for backing up, sharing, and moving environments without the need to reinstall packages.<\/p>\nconda install -c conda-forge conda-pack\nconda pack -n my_env -o my_env.tar.gz<\/code><\/pre>\nmkdir my_env && tar -xzf my_env.tar.gz -C my_env<\/code><\/pre>\n5. Storing Environments on an External Drive<\/h3>\n
conda info --envs<\/code>\u00a0in a prompt window or terminal. Here\u2019s how it looks on my C: drive (this is a truncated view):<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_oYmWw5A-WftZOFp3NpvoUA.webp?resize=646%2C210&ssl=1\")
conda config --set envs_dirs \/path\/to\/external\/drive<\/code><\/pre>\nenvs_dirs<\/code>\u00a0directory for\u00a0all<\/em>\u00a0environments, you can store each environment inside its respective\u00a0project<\/em>\u00a0folder. This lets you store everything related to a project in one place.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_wlHf7VfilrmBLZEJnZ8aVQ.webp?resize=201%2C194&ssl=1\")
ipykernel<\/code>\u00a0for JupyterLab, you would run:<\/p>\nconda create -p D:projectsgeospatialenv ipykernel<\/code><\/pre>\nD:<\/code>. There are many ways to handle this, but one of the simplest is to launch JupyterLab from the D: drive.<\/p>\nD:<\/code><\/pre>\njupyter lab<\/code><\/pre>\nD:<\/code>\u00a0in your user folder.\u201d<\/p>\n--clone<\/code>\u00a0flag to produce an exact copy of a C: drive environment (called\u00a0my_env<\/em>) on the D: drive:<\/p>\nconda create -p D:new_envsmy_env --clone C:pathtooldenv<\/code><\/pre>\n\n
conda env list<\/code>, you\u2019ll see the environment listed in both the C: and D: drives. You can remove the old environment by running the following command in the\u00a0base\u00a0<\/em>environment:<\/p>\nconda remove --name my_env --all -y<\/code><\/pre>\n--clone<\/code>? The short answer is that\u00a0--clone<\/code>\u00a0is the best and fastest option for moving an environment to a different drive on the\u00a0same<\/em>\u00a0machine. An environment file is best for recreating the same environment on a\u00a0different\u00a0<\/em>machine. While the file guarantees a consistent environment across different systems, it can take much longer to run, especially with large environments.<\/p>\n6. Relocating the Package Cache<\/h3>\n
conda config --set pkgs_dirs D:conda_pkgs<\/code><\/pre>\npkgs_dirs<\/code>) and frequently used environments on the faster SSD, and other environments on the slower HDD.<\/p>\n7. Using Virtual Environments<\/h3>\n
venv<\/code>). This represents a lightweight alternative to a conda environment.<\/p>\nvenv<\/code>\u00a0named\u00a0my_env<\/em>, run the following command:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-20-at-1.28.15%25E2%2580%25AFPM-1024x94.png?resize=1024%2C94&ssl=1\")
<\/figure>\nconda<\/code>,\u00a0pip<\/code>,\u00a0setuptools<\/code>, and so on. A\u00a0venv<\/code>\u00a0is much lighter, with a minimum install size of only 5\u201310 MB.<\/p>\npkgs_dirs<\/code>. These tarballs can grow to several GBs over time. Because\u00a0venv<\/code>\u00a0installs packages directly into the environment, no extra copies are preserved.<\/p>\nvenv<\/code>\u00a0when you only need basic Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly recommended, like Geopandas, should still be placed in a conda environment. If you use lots of environments, you\u2019ll probably want to stick with conda and benefit from its package linking.<\/p>\nvenv<\/code>\u00a0docs<\/a>.<\/p>\n
\nRecap<\/h2>\n
venv<\/code>) as a lightweight alternative to conda.<\/a><\/p>\n