Data transfers - How to move data to / from bb-transfer.hpc.rug.nl

Firstly and independent of technical options: make sure you are familiar with the code of conduct / terms and conditions / license or whatever it is called and that you are allowed to upload/download a data set! When in doubt contact your supervisor / principal investigator and the group/institute that created the data set.

data-transfers

Instructions for cluster users

Configure SSH agent forwarding

First, you will need to temporarily forward your private key with SSH agent forwarding to Betabarrel using one of:

Next, login via the jumphost on Betabarrel using your personal, regular account and with SSH agent forwarding enabled. Verify that agent forwarding worked by executing the following command to list the identities (private keys) available to your SSH agent:

$betabarrel> ssh-add -l
  • You should get a response with at least one key fingerprint, which means you can now transfer data with rsync to/from the dedicated data transfer server.
  • If instead you get The agent has no identities or Could not open a connection to your authentication agent, then the key forwarding failed. This may happen when you were already logged in to the same server without agent forwarding in another active SSH session; make sure you logout from all Betabarrel servers in all terminals and try login to with agent forwarding enabled again.

Transfer data with rsync

Once you have your private key temporarily forwarded to betabarrel you can use rsync (over ssh) with the guest account to transfer data to/from bb-transfer.hpc.rug.nl. See below for some syntax examples. Note:

  • The data transfer server uses named rsync modules to expose only a subset of its filesystems. This means you must use double colon syntax (::) to separate the name/address of the server from the named module and path on the server.
  • You must use port 443 for rsync over SSH. The data transfer server does not support rsync over SSH on the default port 22 for SSH.
#
##
### Specify only source and leave a destination out to get a listing of modules, files and folders available on the source side.
##
#
# Request a list of rsync modules available for user some-guest-account.
#
rsync -v --rsh='ssh -p 443 -l some-guest-account' bb-transfer.hpc.rug.nl::
#
# List contents in the "home" module.
#
rsync -v --rsh='ssh -p 443 -l some-guest-account' bb-transfer.hpc.rug.nl::home/
#
##
### Specify both a source as well as a destination to transfer data.
##
#
#
# Push a file from user interface to data transfer server.
#
rsync -av --rsh='ssh -p 443 -l some-guest-account' path/to/file_on_betabarrel bb-transfer.hpc.rug.nl::home/
#
# Reverse source and destination to pull a file from data transfer server onto user interface server.
#
rsync -av --rsh='ssh -p 443 -l some-guest-account' bb-transfer.hpc.rug.nl::home/data_on_transfer_server path/to/dir_on_betabarrel/

Back to overview for dedicated data transfer server