Introduction

UPDATES:

02-03-2022: 
Login node 1 is no longer available, instead a new login node 4 has been added (145.88.76.219)

The SHARK SLURM cluster is maintained and hosted by the LUMC. It is currently the cluster that most FSW users are working on1618997548843-165.png.

Resources

Official SHARK Wiki:  see here. This Wiki is maintained by the LUMC and contains most of the general information needed to use the cluster.

FSW specific issues:  For FSW specific topics, i.e. fMRI analysis, Matlab usage, data-transfer etc, we provide additional information on the page below.

Contact: for all questions about SHARK usage please contact SOLO: labsupport@fsw.leidenuniv.nl, only for password reset contact SHARK admin directly:  ITenDI_Infra-Linux@lumc.nl

Data Management

  • All FSW users share a total of 40TB of spaces of the cluster, 80 compute nodes and have 10GB of personal space on their home directories.
  • Once the shared space is full,  the output of any jobs will not be saved (and will be discarded instead). So always keep an eye on the current status of the shared folder /exports/fsw/  and the expected output size of your jobs.
  • Never have your important data (e.g. raw) or scripts only on the cluster! Remember that none of the data on the exports folder or personal folder is back-upped. That means, if the data gets corrupted, is overwritten, or there is a failure in the drives, the data may be lost permanently.
  • Files can be transferred for long-term & secure storage to FSW using sftp (see here).

F.A.Q.

TopicQuestionAnswer
Wiki not accessibleI can't access the Shark Wiki: https://git.lumc.nl/shark/shark-centos-slurm-user-guide/-/wikis/homeThe official Shark wiki is sometimes locked behind the LUMC network (due to hacking attempts), and accessible only from the researchlumc network. After logging into researchlumc or shark slurm, you can open a browser instance and navigate to the wiki. Normally the wiki becomes publicly available again after a few days, if the issue persists however, please contact us.
Log in from FSWI can't log in to Shark from my university computer? The FSW network is directly connected to the Shark network, that means you do not need to use the jump-server (RESEARCHLUMC). Additionally, dns-routing does not work at the moment for Shark SLURM, meaning you have to use the ip-address instead of the network name. Therefore, for shark old ssh to: user@shark.researchlumc.nl, for Shark SLURM ssh to: user@145.88.76.243 (for log in node 1) or user@145.88.76.217 (for log in node 2). Note that if you are using MobaXterm, X2Go or Putty, you will also need to disable the jump-server settings.
Data Transfer How do I transfer files from the cluster to the FSW J-drive or P-drive?The "old" method of transferring data via the Vito/Corleone server is no longer working, instead see the required steps here.
Matlab LicensesHow can I use the FSW Matlab license on Shark, as the LUMC does not have a campus license?

There is an install of Matlab 2018  specifically for FSW users, made available in the FSW Exports folder. To access it from the terminal:

PasswordI forgot my password and/or my password is outdated.Please send an email to the SHARK administrators: ITenDI_Infra-Linux@lumc.nl
X11 on MacsX11 / OpenGL is not working on my Mac when I try to run programs with graphical interfaces

This is likely an issue with the XQuartz service on your Mac. The solution seems to be to run the following command in the terminal (on your own machine):

defaults write org.xquartz.X11 enable_iglx -bool true

And then reboot your machine.

Data Transfer

Choose a Method

There are various ways to transfer data between SHARK and the FSW network. Here we focus on two main options: syncing files with rsync (using MobaxTerm) and copying files with GUI (using WinSCP or FileZilla).

  • Syncing files using MobaxTerm (Recommended)
    • This is the best method to use for syncing large datasets with many files from Shark to FSW (or the other way around). It works from your windows workstation on the FSW or by using the Citrix Remote Desktop from any other machine/location.
  • Copying files using WinSCP.
    • This method is best for quickly copying smaller datasets from Shark to FSW (or the other way around), and is a bit more accessible if you are not comfortable with using a terminal. It works from your windows workstation on the FSW or by using the Citrix Remote Desktop from any other machine/location.

Find a suitable destination

  • Your personal P-drive has limited storage capacity .
  • The J-drive on the other hand, offers more storage space, usually in /Public/ResearchData/  There you will find folder for the various institutions and sections. If you do not have access (including write permissions) for a folder there you can request one via the ISSC (Workgroup storage, 'Research data' bulk storage). To get access to a folder owned by a colleague he/she needs to submit the request. Also keep in mind, J-drive storage is relatively expensive at ~450 euro's per year per TB. These costs are billed towards your section. For very large data-sets and long-term archiving, perhaps consider external and/or Open Science oriented options (e.g. DANS) .  
  • OneDrive / Google Drive are not suitable options, as these are not secure.

Syncing with MobaxTerm

As of July 2021: this is the only method guaranteed to work, and suitable for large datasets

[Optional: Start Citrix]

UPDATE: as of January 2022 the Citrix Desktop method is no longer functioning. Instead a SURKO must be used. Please contact labsupport@fsw.leidenuniv.nl for instructions.

If you are not working on an FSW windows machine (i.e. at home, on your laptop or Mac), first start-up the Citrix Desktop:

  • Log in to the Citrix Remote: https://remote.campus.leidenuniv.nl/ You can use the light (browser) or full version, it does not really matter.
  • Go to the "Desktops" tab
  • Open a new Basic Desktop
    1627987399739-384.png

Start MobaxTerm

From here the steps are the same for FSW machines and the Citrix Remote:

  • If you don't have MobaxTerm yet, download the portable edition version of MobaxTerm (https://mobaxterm.mobatek.net/download-home-edition.html) and unzip it, e.g. to a folder on the desktop
  • Start the MobaxTerm client from the folder by double-clicking the .exe file in the extracted folder.
  • If it is the first time you started MobaxTerm, go to Settings > Configuration >General.
    • For Persistant home directory, enter:  _MobaFolder_
    • For Persistant root, enter: _MobaFolder_\slash
    • Press OK at the bottom and restart MobaxTerm if asked. You can ignore any warnings about network drive speed
      .1627987731297-585.png
  • Start a new local session in MobaxTerm, using the "Start local terminal" button on the home screen, or go the "Session" in the top left then select "Shell" and press OK.

Preparing Rsync

You have now opened a local shell terminal. Most Linux terminal commands you know will work here in the same way. Conveniently, MobaxTerm also has a built in Linux file syncing tool called Rsync. To make this work with Shark we need to provide an Rsync command

  • Paste the following command into notepad or Word (or any text editor of your choice). And fill in the yellow highlighted parts. Note that this is all one line!

rsync -av -e 'ssh -o "ProxyCommand ssh [username]@145.88.35.10 -W %h:%p"' [username]@145.88.76.243:/exports/fsw/[source folder to be synced from] "/drives/j/ResearchData/FSW/[folder to be synced to]"

  • This command takes all the files and subfolders from the source directory on shark and syncs them to target directory on the J drive. Files that are already present on the J drive and have the same filesize and modification time will not be overwritten. This means that if your transfer gets interrupted (for instance because Citrix is closed, you lose network connection etc), re-running this command should pick up the transfer right where you left off (without having to start from the beginning).
  • Replace all the yellow highlighted [..] parts (also remove the [ ] symbols!)
    • [username]: should be your username on the cluster (e.g. for me it is kjfolfers)
    • [source folder to be synced from]: the path to the main folder you want to sync the files from. Rsync with the -av parameter works recursively, so subfolder will also be synced. (e.g for me that could be kjfolfers/transfertest). In Rsync there is a difference between ending the source directory with or without a slash. If you omit the slash, the source directory folder itself (in this case transfertest) will also be created in the destination folder. If you include the slash, the contents of the folder will be put directly into the destination folder.
    • [folder to be synced to]: the path to the folder on the J drive where you want to sync to. You need to have write access to this folder. You could check that by opening file explorer navigating to the folder and making a new text file there. (e.g. for me such a folder could be Psychologie Cognitieve/olferskjf/sharktest/). Importantthe folder paths used in the command has forward slashes "/" for folder separation, while windows natively (e.g. in file explorer) used back-slashes "\", this means you can't just copy the file path from explorer. Additionally, if you have spaces in your folder names, you need to prepend them with a backslash. For example if your file-path in explorer looks like :
      "drives\ResearchData\FSW\Psychologie\My test folder" 
      the corrected version would be:
      "drives/ResearchData/FSW/Psychologie/My\ test\ folder/"
  • As an example, the full command could look like this (note the command should be one line, without enters):

rsync -av -e 'ssh -o "ProxyCommand ssh kjfolfers@145.88.35.10 -W %h:%p"' kjfolfers@145.88.76.243:/exports/fsw/kjfolfers/transfertest/dir10MB "/drives/j/ResearchData/FSW/Psychologie Cognitieve/olferskjf/sharktest/"

  • Rsync can be run with many configurations. The one used above is pretty much the default setting for archiving (that is copying all folders and files). However, in some cases you may want to use different parameters to better suit your needs. For instance symbolic and hard-linked files and folders may need special treatment that the regular -av parameter does not give. Please consult the help pages for rsync and/or contact SOLO or cluster admins for help if needed.

Running Rsync

The first time you use rsync, I highly recommend starting with a small test source folder with some files to see if everything works as expected!

  • [OPTIONAL] By including the -n parameter (e.g. rsync -av -n -e ...) to the command, you can do a "dry run" this will show you a list of which files would be synced, without actually doing anything yet.
  • [OPTIONAL] By including the -P parameter, you will get information about the transfer speeds and the file progress (this will make the output in the terminal a bit more complex though).
  • Copy and paste the command into the MobaxTerm console (use right mouseclick or the middleclick) and run it.
  • The first time you run this command, you will bea asked to fill in your password for the Shark cluster twice.
  • Mobaxterm will ask you if you wish to save your password for later, if you do so, you will not have to fill it in the next time

Example of a dry-run rsync command:

1627991085389-383.png

Optional: Keep Alive

If you are experiencing trouble with the computer or Citrix Desktop closing due to a timeout. You could try the following:

  • After starting the rsync command, open an additional local session in mobaxterm
  • Run the following command in the terminal:

while true; do echo -e `date` "\r\n" >> "/drives/p/Desktop/keepalive.txt"; sleep 60; done

  • The above command runs every minute to store a timestamp in a text file. It can help to keep the Citrix desktop alive. The transfer should even continue if the Citrix desktop is closed (if prompted confirm that you want the session to remove active). Virtual desktops may be still be automatically shut-down after some time, but running the rsync command again should pick up the transfer where it was right before shutdown.

Copying with WinScp

While this method may be a bit more user-friendly, given the graphical user interface, keep in mind that transfers with WinSCP will stop once you shut down your local FSW computer (or when using Citrix, after a certain period of time). After which, you will manually need to restart the transfer again, and it might be hard to keep track of which folders and files have already been copied succesfully.

  • Log in on your FSW computer (physically or via Remote Desktop Protocol or via the Citrix Remote environment).
  • Make sure WinSCP is installed, if not, install it from the Software Center or download and unzip the portable (non-installer) version here . If you are using the Citrix environment, you can start the WinSCP application directly (you do not need the full desktop view).
  • Start WinSCP
  • Select "New Site"
  • Fill in the session information:
    • File Protocol: SFTP
    • Host Name: res-hpc-lo01.researchlumc.nl  (or res-hpc-lo02.researchlumc.nl) Note: if neither work, try the ip addres: 145.88.76.243 or 145.88.76.217
    • Port number: 22
    • User name / Password: you SHARK account details.

If you are using the Citrix Remote environment some extra steps are required:

  • Open the advanced menu
  • Go to Connection -> Tunnel
  • Select "Connect through SSH-tunnel
  • Fill in:
    • Hostname: 145.88.35.10
    • Port number: 22
    • User name / Password: your Shark details
  • Click OK
  • If you want you can click "save" the save these settings for future sessions.
  • Then click "Login"

1620381931803-835.png

  • When asked to approve a key or fingerprint, press OK.
  • You will now see your FSW folders and files in the left window and your SHARK folders and files in the right window.
  • The reach the various FSW folder (e.g. the J or P Drive), use the dropdown menu (or for SHARK, the menu on the right side):

image-20210507120136-1.png

  • To transfer files you can now: drag & drop them from the one of the windows to the other, or, select them, right-mouse click and press download, they will then be copied to the currently open folder in the opposite window.

 

Tags: