The AVIDD-O Cluster
Getting Started
The AVIDD Project
AVIDD ("Analysis and Visualization of Instrument-Driven Data") is a $3.25 million project funded in part by a grant of $1.8 million from the National Science Foundation's Major Research Instrumentation Program. Initially, the project consisted of three IA32 clusters, AVIDD-B, AVIDD-I, and AVIDD-N, and an IA64 cluster, AVIDD-T. In late 2005, the AVIDD-O cluster of Opteron systems was added.
The AVIDD-O cluster consists of 36 compute nodes, 4 storage nodes, and 2 interactive user nodes. All nodes are configured with two cpus (2.6GHz Opteron 252s), 2GB RAM (4GB on the user and storage nodes), Ultra320 SCSI disk, Gigabit Ethernet and Myrinet interconnect adapters.
The Environment
The AVIDD-O Cluster nodes all run Red Hat Enterprise Linux 4 Advanced Server. Batch jobs are managed by the Torque resource manager and the Maui Scheduler. User home directories NFS-exported from an EMC Celerra 702g Network-Attached Storage device, and shared scratch disk is provided via IBM's General Parallel File System, mounted cluster-wide.
Users
The AVIDD-O Cluster is dedicated to the research needs of IU faculty, staff, and graduate students, as well as affiliated researchers. Graduate students, staff, and affiliates require sponsorship from a faculty member in order to obtain an account. Accounts may be requested using the Research Systems Account Application page.
Documentation and Support
Announcements, documentation, graphs and usage statistics are available at the AVIDD-O Cluster homepage.
The AVIDD-O Cluster is administered and supported by the High Performance Systems group of UITS, hps-admin@iu.edu . For application and development support, please contact the High Performance Computing group hpc@indiana.edu.
Connecting and Logging In
The AVIDD-O login nodes can be accessed via Secure Shell clients using the hostname avidd-o.iu.edu.
Once you have logged in, the default bash shell, as well as the Bourne and Korn shells, automatically read and execute commands from the /etc/profile file and your own ~/.profile (and ~/.bash_profile, in the case of bash) file. With the C and tcsh shells, the .login and .cshrc (or .tcshrc) files are read. You may customize the initialization files in your home directory to suit your needs. Please see the man pages for your particular shell for more information. When you login, you will see the login banner, or Message of the Day, which displays current announcements. Please read it, as it contains information regarding the status of the cluster.
Softenv
The Softenv environment management system is used on the cluster to simplify application and environment configuration for both users and administrators.When you login to the cluster for the first time, a ~/.soft file, defining system defaults, will be created for you. These defaults, such as Intel's C/C++ and Fortran compilers may suffice, but if they don't, additional packages, such as the Portland Group compilers or LAM/MPI may be added. To see the list of default Softenv keywords, type the following at your AVIDD shell prompt:
soft-dbq @default
Keywords are prefixed with +, such as +pgi-6.0-5. Macros, which are collections of keywords, are prefixed with @. The soft-dbq (Softenv Database Query) command will display more information about any keywords or macros.
For a list of available keywords, simply type softenv at a shell prompt. Adding an application to your environment is as simple as adding its keyword to your ~/.soft file, and then re-initializing the Softenv system by either running resoft, or logging out and logging back in. You may temporarily add an application to your environment with the command:
soft add <keyword>
Deleting an application may be done as well:
soft delete <keyword>
Changes made with soft add or soft delete will be lost after logging out. To learn more about Softenv, type man softenv-intro.
Managing Local and Remote Files
Like other UNIX systems, Linux uses an hierarchical, tree-structured directory system to organize files. Data residing on local disk resides in journalled filesystems which contain a root directory from which associated files and subdirectories branch. Directories are catalogs of files which associate names with files and are used to segregate files into related groups. To display information about currently mounted file systems, type df at the shell prompt.
Each file is described by an inode which contains critical information about the file, such as its type (directory, ordinary, character special, block special, or pipe), ownership, access permissions, group ID, file size, file creation, modification, and access times, and a pointer to its data blocks. To list information about the files within your current working directory, type ls -al at the shell prompt. Here is a sample of ls output:
$ ls -al drwxr-xr-x 5 jdoe chem 1024 Feb 21 16:06 . drwxr-xr-x 12 bin bin 512 Feb 05 10:56 .. -rw-r--r-- 1 jdoe chem 13250 Feb 14 11:29 .Xdefaults -rw-r--r-- 1 jdoe chem 471 Jan 29 16:37 .bash_profile drwxr-xr-x 3 jdoe chem 512 Dec 19 14:48 bin -rw------- 1 jdoe chem 2054 Feb 21 16:07 .bash_history -rw-r--r-- 1 jdoe chem 37479 Feb 21 14:51 whitepaper.pdf -rwxr-x--- 1 jdoe chem 45 Jan 16 15:22 submit.sh -rwxr-x--- 1 jdoe chem 79 Feb 19 13:02 myjob.c -rwxr-x--- 1 jdoe chem 53 Feb 19 16:45 myjob.c.old -rwxr-xr-x 1 jdoe chem 93 Jan 29 16:10 myjob.o drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project1 drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project2The first character in the first field is the file type (e.g., - for ordinary file, d for directory). The next 9 characters in the first field indicate the file access permissions: r for read, w for write, x for execute, with - meaning the corresponding read, write or execute permission is denied. Characters 2, 3, and 4 indicate the file owner's permissions, characters 5, 6, and 7 indicate the group's permissions, while the last three characters indicate all others' permissions. Execute permission for a directory allows searching through and listing its contents.
Field 2 indicates the number of links to the file. The next two fields show the owner and the group associated with the file. Field 5 gives the size of the file in bytes. Field 6 is the time the file was created or last modified. The last field is the name of the file. The files named . and .. indicate the current and the parent directory, respectively. For more information about ls and available options, type man ls.
Use the cd command to change directories. The argument supplied to cd can be an absolute or relative pathname. For example, assume you are currently in the /usr/local directory. To get to the /usr/local/bin directory you could type either cd /usr/local/bin (the absolute pathname) or cd bin (relative to /usr/local). To get back to /usr/local you could type cd /usr/local or just cd .. (to go up to the parent directory). Typing cd without an argument will put you into your home directory. To determine what your current working directory is, type pwd.
To view a file on your screen use the cat command. Type cat filename. To view one screen at a time, type cat filename | less.
To copy a file to the same or to a different directory use the cp command. Type cp filename1 filename2. The cp command overwrites any existing file of the same name. For more information, type man cp.
To rename a file or to move it to a new directory use the mv command. Type mv filename1 filename2. For more information, type man mv.
To delete a file use the rm command. Type rm filename. For more information about deleting single or multiple files, type man rm.
To create a directory use the mkdir command. To delete an empty directory use the rmdir command. For more information see man mkdir and man rmdir.
The same commands used for managing local files can be used to manage remote NFS files, providing the access permissions allow it. For example, you cannot use rm on a file in a remote filesystem which is mounted as read-only.
For access to remote files other than those already accessible to the AVIDD-O cluster, you may access a remote host using the Secure Shell scp or sftp commands to copy files to or from the remote system.
Enter du -k to see how many 1KB blocks of disk space you are using. Enter quota -v to see the number of 1KB blocks in your disk quota. You may exceed your soft quota up to the limit of your hard quota. If you exceed your soft quota, you will receive a message to that effect. You then have 7 days to reduce your utilization below your soft quota. If you do not reduce your disk utilization below your soft quota, then at the end of the grace period, you will not be able to write to your disk until you reduce your disk utilization below your soft quota. The grace period is 7 days.
Editors and Shell Scripts
There are a number of text editors available on the AVIDD-O cluster, the most popular being vi and emacs. Please see the list of Available Software for more options. Brief tutorials are also available:
Shell scripts are useful for automating repetitive tasks and initializing your envrionment. The different login shells (bash, tcsh, etc) use different syntax and commands. Listed below are some reference guides that may be of interest:
Parallel Computation and MPI
The AVIDD-O cluster is equipped with a Myricom M3-E128 2+2Gb/s switch. This high-bandwidth, low-latency interconnect, which supports Myricom's GM protocol, facilitates high-performance parallel computation through the use of MPICH-GM, the Myricom implementation of the Message Passing Interface. You may add the MPICH-GM tools and libraries, compiled with the Portland Group's compilers, to your environment by adding +mpich-gm-pgi to your ~/.soft file and running resoft.
Briefly, use mpicc or mpif77 to compile your MPI-enabled software, and then use mpirun to execute the job on multiple nodes of the cluster. In practice, you must submit your job through the PBS batch system (see below) to insure fair and efficient use of the cluster. Please see the MPICH-GM Guide for more information.
Using PBS Professional
To optimize job throughput, efficiency and performance on AVIDD-O, all CPU-intensive jobs (i.e., jobs requiring more than 20 minutes of CPU time) must be submitted as batch jobs to a batch scheduler. Your home directory resides on the EMC NAS fileserver, and is accessible from all nodes. You may submit jobs to the PBS resource manager on any interactive node, and may read/write your input and output files from/to your home directory or from/to the local /scr directories, or to/from the shared scratch directory, /N/gpfso. When your job is started, your default login shell is invoked, along with your .login and .cshrc files (for C shell users) or your .bash_profile and .bashrc files (for bash users). Be sure that these files do not contain any commands which would not work in a batch environment (e.g., the stty command, which expects stdout to be a terminal).
Please see the PBS Guide for detailed information regarding PBS job scripts, submission, and job control.
The Maui Scheduler
Job dispatch is controlled by the Maui Scheduler. Maui commands such as showstart and checkjob can provide information regarding job execution and related problems. Please see the Maui Scheduler Guide for more information.
