Data Management: General Information
SBC provides a local computer account for each user group scheduled at Sector19 and provides "DataSpace" under the user's home directory for use during their scheduled visit. Access to the user’s account will be disabled at the end of the scheduled visit.
The user’s “DataSpace” will be kept at SBC for not less than two weeks following the end of their scheduled beam time, after which time the data will automatically be deleted from disk! It is crucial that the users verify their backups at their home institutes as soon as possible after their beam time at SBC and contact their SBC host immediately for assistance if the backups are corrupted.
Researchers are required to make their own copies of their processed data and/or their raw images during their scheduled time at Sector 19. They are also encouraged to copy any results over the network to their home institute if possible. Please read the Backing Up Data section below.
Detector calibration files required by the Software for Data Processing are available for download on the SBC website under Calibration Site Files on the left menu panel. The software currently available for data processing at the SBC beam lines includes: HKL2000, D*Trek and MOSFLM. see Correcting Images
Backing Up Data
SBC supports the following methods for users to "take their data home":
- users may copy their data to User-Supplied Firewire Disks drives (or USB drives) which
are externally attached to SBC computers
- users may copy their data to User-Owned portable computers which are connected
to the appropriate SBC subnet
- users may copy their data across the network from SBC computers designated
as data transfer servers to computers at their home institutes
FIREWIRE (or USB) DISK TRANSFERS
The FireWire (IEEE1394) interface provides for "Hot Swap" of this type of disk. That is to say, these external FireWire disks can be added and removed from the system without the need to reboot and typically without administrator privileges. This is the preferred method of data backup at SBC. Please bring drives that have been pre-formatted with a file system that is compatible with the operating system of the computer you will be using for your backups. Formatting disks takes a fair amount of time and you will not have administrator privileges on the SBC computers.
Users can copy their data to User-Supplied Firewire Disk drives (or USB drives) that are externally attached to either a user-provided portable computer, to an SBC workstation (running Linux) or to an SBC PC (Microsoft Windows XP). The SBC Linux systems work well with “ext3” filesystems. The SBC Windows PC’s work well with “ntfs”, “fat32”, and even MacIntosh filesystems.
Of course, it is most important that the disk format be something that you are familiar with using at your home institute. The safest way to address any compatibility concerns is to connect your drives to your own laptop and then copy data over the network from one of our servers. Users may bring their own computer (desktop or laptop) to SBC. We support a limited number of dynamically assigned network addresses (DHCP) so that you can connect your computer to the SBC network. Please know how to configure the network using DHCP on your system before you arrive. Once your system is on our network, SBC supports the use of secure copy commands such as “rsync”, "scp" and “WinSCP” to transfer data from the designated Data Transfer Server for your assigned beamline to your computer's tape drives or disk drives (SCSI, IDE, FireWire, etc.). We recommend the “rsync” command for either Linux or Macintosh laptops. The Freeware application WinSCP works well for Windows XP on our SBC PC’s.
The fastest data transfers rates are realized when the user connects their own external FireWire disk drive(s) to an SBC Linux workstation. There are two benefits to utilizing firewire on the SBC Linux systems. First, the backup does not involve a network transfer from a “remote” system (it is considerably faster). Second, you can utilize the benefits of the “rsync” command. As you may already know, using rsync for backups enables you to continue working in a directory that has already been backed-up without keeping track of the new or changed files. Simply running the rsync command again when you have finished will copy only the differences. I recommend that you run rsync immediately after each scan because backing up the raw images takes the most time and then periodically (and again just before you leave) run the same rsync command to insure that all of the processing results are also backed up. SBC can provide scripts for use on SBC Linux systems that run rsync to backup part or all of your home directory to a single disk and then repeat after a 10 minute pause. This is the easiest method works and it works well provided the disk has enough free space, the disk is formatted “ext3” for linux, and the top level permissions on the disk are open to all users.
There is a PC at each beamline which you may use for your backups. Windows XP has been the most universally accepted platform for our users and it works reasonably well with FAT32, NTFS and MACINTOSH formatted disks; however, if you are comfortable with using firewire disks on your Linux systems at home I would now recommend that you do so here at SBC as well.
A word of caution: The user should become familiar with the FireWire disk technology before they arrive at SBC. After all, the objective is to transport data for use at your home institute. The verification that all of the selected files were successfully copied to the FireWire disk is the user's responsibility. Also, WinSCP may not be the ideal way to transfer files but it is currently the only secure method that is supported for this purpose on our beamlines PC’s. WinSCP is freeware. You may want to install WinSCP at home.
NETWORK COPY FROM SECTOR19 TO A HOME INSTITUTE
For security reasons, users are not permitted to connect "into" SBC computers from outside of Sector 19; however, SBC does not block outgoing ftp or SCP connections. In order to conserve bandwidth on our local network for critical processes such as data acquisition, please initiate data transfers ONLY FROM THE Data Transfer Server for your assigned beamline. Of course, we recommend the secure "SCP" connection over "ftp" due to the clear-text-password vulnerabilities present in ftp. We at SBC highly recommend that users perform network transfers of at least their "results" to their home institutes as soon as possible. Some SBC users have also successfully transferred their raw data to their home institutes using this method at reported data rates which they describe as “reasonable”. As with any backup method, do not wait until the last minute to begin your backups. Although there are bandwidth issues in our local network, I can still recommend this method, provided your home institute supports it.
Please feel free to contact me concerning any questions about data backup at SBC:
Jack Lazarz (jlazarz@anl.gov)
|
