rsync data transfer from PIK


rsync is an open source file transfer program for Unix systems providing a very fast method for bringing remote files to a local copy.

Rsync is preinstalled on most Linux distributions as well as on Mac machines. Windows ports of rsync are available from several internet sites. Take a look below to get instructions for two working solutions on a Microsoft Windows machine.

rsync service available via PIK

rsync endpoint: isimip_pik@rsync.pik-potsdam.de
rsync password: Request current password by writing to isimip-data@pik-potsdam.de. This password should not be shared and will be updated on a regular basis.

To receive a list of remotely available files and sub folders use the following command:

RSYNC_PASSWORD='PASSWORD' rsync -vL isimip_pik@rsync.pik-potsdam.de::isimip/rsync_external/

Example

The following example will download all 2m temperature (tas) files from the ISIMIP2b bias-corrected land-only IPSL-CM5A-LR input data set to the current local folder. Replace the single dot at the end of the line with a different destination folder if wanted. The -u option will check for already present files on your end and will only transfer newer or updated files:

RSYNC_PASSWORD='PASSWORD' rsync -auvL isimip_pik@rsync.pik-potsdam.de::isimip/rsync_external/ISIMIP2b/InputData/GCM_atmosphere/biascorrected/landonly/historical/IPSL-CM5A-LR/tas_*historical*.nc4 .

A few notes

  • You will not gain any improvements in download speeds or amount of data being transmitted by using the -z option since all NetCDF data are already internally compressed.
  • Make sure your machine and institution's network allows for outgoing connections on port 873.

Windows Subsystem for Linux

Starting with build 14316 Windows 10 supports a Linux compatible interface to run in the Windows PowerShell. Please see the documentation to install WSL on your machine. You'll then be able to install and run Linux commands and tools in a bash shell, e.g cdo, ssh, rsync, wget,...

Right from the start your Windows user directories are also available in the subsystem under /mnt/c/Users/

For the ESGF wget scripts to work you need to install ORACLEs Java runtime environment into your Linux subsystem. Please see instructions here for the Ubuntu for Windows variant.

rsync data transfer from PIK with Grsync for Windows

Here we describe a tested solution to transfer data from PIKs rsync service on a Microsoft Windows machine.

  • get the program here and start the installer
  • create a file just containing the password given to you by the ISIMIP coordination team
    e.g. C:\Users\foo\Documents\password
  • Launch Grsync and configure the connection as seen in the screenshot below by replacing the user name of your Windows instance and the remote folder
  • add the option for the password file
  • Press "Simulation" to get a preview of the subfolders and files in the folder
  • adapt the remote folder to your needs and press "Execute" to transfer the files into your destination folder
Grsync Windows Config