Copperfist HPC cluster
  • Home

About

  • Cluster
    • Key Features
    • Cluster Components
    • Naming Themes
  • License
  • Contact
    • Email
    • Visiting address
    • Mail address

Requesting Accounts and Groups

  • Accounts, security and privacy
    • 1. Accounts, security and privacy.
      • Asymmetric key cryptography
      • Security & Privacy
    • 2. Generate a public/private key pair
    • 3. Request an account
  • Generate SSH key pair for Windows
    • 1. MobaXterm option
      • 1.1 Get MobaXterm - a terminal and key generator application
        • 1.2 Launch MobaXterm key pair generator
        • 1.3 Configure key pair generator
        • 1.4 Generate key pair
        • 1.5 Secure private key and save pair to disk
    • 2. Using Windows OpenSSH
      • 2.1 Creating a keypair with OpenSSH
    • 3. Request account and have the public key linked to your account
    • Start using servers/services
  • Generate SSH key pair for macOS & Linux
    • Generate a key pair with OpenSSH
      • Open a terminal/shell
      • Generate key pair
      • Select where to store the key pair
      • Secure the private key
    • Request account and have the public key linked to your account
    • Start using servers/services
  • Request a new group on the cluster
    • Minimal requirements
    • What's next?
    • When you want to leave
      • Final note

Logins

  • Jumphosts, UIs and sessions
    • User Interface (UI) and Jumphost servers
    • Request an account
    • SSH client config and log in to UI via Jumphost
    • Configure 2-Factor-Authentication
  • SSH config & login for Windows clients
    • 1. MobaXterm option
      • 1.1 Launch MobaXterm and create a new session
      • 1.2 Configure a new session
      • 1.3 Password (popup)
      • 1.4 Password again (prompt)
      • 1.5 Session established
    • 2. Using Windows OpenSSH
      • 2.1 Semi-automatic configuration with .bat script
      • 2.1 Manual configuration with
      • 2.3 Connecting to the system
      • 2.4 Port forwarding to the compute nodes
      • 2.5 Creating keypair
    • 3. Using WSL
      • 3.1 Install WSL
    • 4. Converting keypair from Putty .ppk to OpenSSH format
  • SSH config for macOS clients
    • 1. Configure your SSH client
      • Quick Install
      • Detailed Walkthrough
    • 2 Login
  • SSH config for Linux clients
    • 1. Create required directories and files if they do not exist yet
    • 2. Configure Certificate Authority's (CA) public key to verify the identity of cluster servers
    • 3. Add include directive to main SSH config file
    • 4. Create SSH config file with generic settings
    • 5. Create SSH config file for Copperfist
    • 6. Login
  • SSH login for macOS & Linux clients
    • Log in to Copperfist on the commandline in a Terminal
    • Frequent Asked Questions (FAQs) and trouble shooting
  • Customize your environment
    • Time Zone

Storage

  • Keep - What is stored where on Copperfist
    • Introduction
    • Our storage system
    • The life cycle of experimental data
      • Job Submission
      • Cluster Performance
      • Top 4 of blunders, that will result in disaster rather sooner than later
    • Software
    • Centrally deployed reference data
    • Your personal home dir @ /home/${user}
    • Group dirs @ /groups/${group}/...
    • Quota
    • List of storage devices / mount points used on Copperfist
    • When you leave
      • The steps to follow:
  • Data transfers - How to move data to / from Copperfist
    • 1. Push to or pull from cluster UI via jumphost
      • Using rsync over SSH
    • 2. Push to or pull from another (SSH) server
      • Using rsync over SSH
        • Configure SSH agent forwarding
        • Transfer data with rsync
      • Using http(s)
  • Data transfers - Using a dedicated data transfer server
    • Arranging a guest account for data transfers
      • Procedure for cluster users who already have a regular account
      • Procedure for external collaborators
    • Using the guest account to transfer data to/from cf-transfer.hpc.rug.nl

Compute

  • Crunch - How to manage jobs on Copperfist
    • Getting an account and starting a session on a User Interface (UI) server
    • Data transfers and storage systems
    • Job types
    • 1. Batch jobs
      • Submitting batch jobs
      • The batch job's environment
      • Checking the status of running batch jobs
      • Checking the status of finished jobs
      • Deleting jobs
    • 2. Interactive jobs
    • Job priority and scheduling efficiency
      • Job priority
      • Backfill
      • Job preemption
    • Quality of Service
      • QoS levels
      • 1. QoS leftover
      • 2. QoS regular
      • 3. QoS priority
      • 4. QoS panic mode
      • 5. QoS interactive
      • 6. QoS ds
      • QoS sub-levels
      • Requesting QoS
      • QoS level and sub-level details
    • Nodes and Partitions
    • Storage
      • Shared storage
      • Local scratch space on a cluster node
    • Debugging and Frequent Asked Questions (FAQs)
      • Q: How do I know what environment is available to my job on an execution host?
      • Q: I don't get any output; What is wrong?
      • Q: How can I profile my jobs?
        • Tools for monitoring jobs
      • Q: Why are my Java jobs misbehaving and using more cores than specified?
      • Q: Why are my Java jobs complaining there is ''no space left on device'' when there is plenty of space
      • Q: Why does Git fail with "The requested URL returned error: 403"
      • Q: Why do I get "sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified"
      • Q: Why are my jobs not starting with state pending (PD) and reason QOSMax[some_resource]!PerUser or QOSGrp[some_resource]Limit?

Technical Specifications

  • Details on the Copperfist HPC environment
    • Software
    • Servers
    • Shared Storage
    • Resources available to Slurm jobs
      • cpu_r7525 partition
      • dragen partition

Software

  • Using and installing software
    • Introduction
    • 1. Packages from the Linux distro / Operating System (OS)
    • 2. Modules installed with EasyBuild by admins
      • Example for the Genome Analysis Toolkit (GATK).
      • Missing software
      • Instructions for deploy admins
    • 3. Modules installed with EasyBuild by regular users in their own environment
    • 4. Other package managers / deployment procedures
      • Conda, Miniconda and Bioconda
      • Containers
    • 5. Language specific installation options for extra/custom packages
  • R packages
    • Create location for custom R packages and prepend that to the search path for R packages
      • 1. Create a ${HOME}/.Renviron file
      • 2. Add ${R_LIBS} to your ${HOME}/.Renviron file
        • Note ${R_LIBS} versus ${R_LIBS_USER}
        • Note % expansions do not work in ${R_LIBS}
        • Note about version numbers in the path
      • 3. Make sure the custom path added to ${R_LIBS} exists
    • Install custom R packages
      • A. Install R package from the Comprehensive R Archive Network (CRAN) repository
      • B. Install R package from the BioConductor repository
      • C. Install R package from an archive file without (CRAN/BioConductor) repository
      • D. Install R package from a GitHub repository
  • RStudio
    • Introduction
    • How to NOT use RStudio
    • How to use RStudio
      • 1. Install software on your own computer
      • 2. Login on Copperfist
      • 3. Create a screen or tmux session on Copperfist
      • 4. Create an interactive Slurm session on Copperfist
      • 5. Load and start R in your interactive Slurm session on Copperfist
      • 6. Start a remoteR server session in your R session on Copperfist
      • 7. Create an SSH tunnel on your own computer
        • For Windows clients
        • For Linux & macOS clients
      • 8. Install remoteR in RStudio on your own computer
      • 9. Using the remote R session in RStudio on your own computer
        • Creating plots
      • 10. Cleaning up
  • Jupyter notebooks
    • Introduction
    • How to use JupyterLab
      • 0. Install software on your own computer
      • 1. Login on Copperfist
      • 2. Create a screen or tmux session on Copperfist
      • 3. Create an interactive Slurm session on Copperfist
      • 4. Load and start Jupyter in your interactive Slurm session on Copperfist
      • 5. Create an SSH tunnel on your own computer
        • For Windows clients
        • For Linux & macOS clients
      • 6. Using the Jupyter session in a web browser on your own computer
      • 7. Cleaning up
  • Apptainer (Singularity)
    • Table of contents
    • About Apptainer
      • In short
      • Containers and apptainer
    • List of commonly used commands
    • Building Apptainer images
      • Building without elevated permissions
    • Sandbox
      • Example with Ubuntu 18.04
      • Converting sandbox directory to sif file
    • Building image as a regular user
    • Making an image from the definition file with elevated permissions
    • Definition file
      • Header
      • Sections
    • Important
    • Cache and tmp folders
      • ${APPTAINER_CACHEDIR}
      • ${APPTAINER_TMPDIR}
      • Automatically configure ${APPTAINER_CACHEDIR} and ${APPTAINER_TMPDIR} for each login
    • Running Apptainer on Windows or Mac
    • Using images from the Docker repository
    • Using storage from the host system inside containers
      • Default host folders mounted inside containers running on Copperfist
      • Isolated containers
      • Mounting host folders inside containers
    • Other considerations
      • Architecture considerations
      • Licence considerations
    • Environment variables
    • Debug information
    • Additional examples
      • Alpine - as regular user build static file via sandbox directory
      • Simple CentOS 7 example via sandbox
  • Nextflow
    • Nextflow introduction
      • How to use and install Nextflow
      • Changing the default writing location for $NXF_HOME
      • Running local within an interactive session or single slurm job
      • How to run Nextflow in an interactive session
    • How to wrap Nextflow submission to a single Slurm job
    • How to paralize your work using Slurm
    • Debugging & Logging
      • Logging
      • Execution Report
      • Timeline Graph
  • Python (Virtual Environment)
    • Caution
    • Introduction
    • Creating new Python virtual environment
    • To use the existing environment
    • To install packages inside the Python environment
Copperfist HPC cluster
  • Archive
Built with MkDocs using a theme provided by Read the Docs.