![]() To do this, you call source activate xmitgcm The environment you created needs to be activated before you can actually use it. This will download and install all the packages and their dependencies. You should now be able to run the following command conda env create -file environment.yml You should chose a value of name that makes sense for your project. name: xmitgcm dependencies: - numpy - scipy - xarray - netcdf4 - dask - jupyter - matplotlib - pip: - pytest - xmitgcmĬreate a similar file for your project and save it as environment.yml. Below is the environment.yml that I use for my xmitgcm project. The contents of this file will differ for each project. A good way to do this is with a custom conda environment file. You now have to define what packages you actually want to install. ![]() bash miniconda.sh -b -p $HOME/miniconda export PATH="$HOME/miniconda/bin:$PATH" Step 3: Create a custom conda environment specification ![]() You then have to add this directory to your path. The trick is to specify the install directory within your home directory, rather in the default system-wide installation (which you won’t have permissions to do). Now you actually run miniconda to install the package manager. Or for python 2.7 wget -O miniconda.sh Step 2: Run Miniconda If you want to use python 3 (recommended) you can call wget -O miniconda.sh ![]() Miniconda is a mini version of Anaconda that includes just conda and its dependencies. I assume something similar will work on most standard HPC clusters. Here is my solution, which I have used successfully on Columbia’s habanero and NASA’s pleiades clusters. How can we get around these limitations to get a fully functional, flexible python environment on any cluster? But most clusters do not have conda available, and the system or module-based python distribution probably doesn’t have all the packages we need (e.g. On our local machine, we know how to manage packages using conda and pip. an xsede resource, NCAR’s yellowstone, etc.). Custom Conda Environments for Data Science on HPC ClustersĪ problem that lot of scientists have to deal with is how to run our python code on an HPC cluster (e.g. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |