Gravity Pipes (GRAPEs)

What is a GRAPE?

Indiana University owns to six special-purpose GRAPE (GRAvity PipE) computers. GRAPEs were designed to tackle the N-body problem where a collection of "particles" interact with all the other particles through some force. As the number of particles grows the number of interactions which must be calculated grows as N**2. Modern general purpose computers can calculate N-body problems up to tens of thousands of particles in a reasonable time. More realistic models of interesting phenomena requires numbers approaching a million or so particles.

IU's first GRAPE-4 system was acquired in 1998 by the IUB Department of Astronomy. Two MDGRAPE-2 boards were purchased in 2000 by UITS, each capable of a peak theoretical processing capacity of 64 GigaFlOPS. Two additional MDGRAPE-2s plus four booster cards (to boost the actual performance of the four IU cards to about twice their previous performance) were acquired in August, 2004. These four card/booster pairs resided in four Research SP nodes until it was retired in September, 2005; they will be relocated to four compatible PentiumIII Linux hosts in the IUPUI machine room in October, 2005.

The GRAPE-6Pro/8 board, purchased in 2001 by UITS, has a peak theoretical processing capacity of 1 TeraFlOPS, and is installed on a Athlon64 Linux server. An additional GRAPE-6Pro/8 board was purchased in December, 2004. Both GRAPE-6 boards will be relocated to Opteron Linux hosts in the IUPUI machine room in October, 2005.

System Performance Acquired Platform #Pipes #Particles Charges
MDGRAPE-2 64 GigaFlOPS 2000,2004 PentiumIII Linux 24 ~10**6 Yes +/-
Grape-6 1 TeraFlOPS 2001, 2004 Opteron Linux 48 ~10**6 No

Originally GRAPEs were designed to calculate the 1/r**2 force of gravity. With IU's GRAPE-6, the positions of a million virtual particles can be loaded into the GRAPE's memory. The GRAPE-6's 48 pipes are then loaded with the position of the particles one wishes to calculate the forces on. The million virtual particles previously loaded are pipelined past these 48 pipe particles where the separation and forces (or energies) are calculated and summed. The summed forces (or energies) for each of the 48 particles are returned to the host computer for use within the researcher's program. Another 48 particles are then loaded into the pipes and the process repeats. The researcher's program running on the host computer only has to deal with the N particles leaving the N by N calculations to the GRAPE.

For programmers, the classical N-body problem would look like: 

	do i=1,n
	  do j=1,n
	    calculate each [i,j] interaction and add to sum
	  end do
	  apply summed forces to the [i] particle
	end do
With the GRAPEs the N-body problem becomes: 
	load the grape's memory with a million particles
	do i=1,n
	  call the grape and get summed forces
	  apply summed forces to the [i] particle
	end do

Unique GRAPEs

The Molecular Dynamics GRAPE, or MDGRAPE, is a second type of GRAPE processor. Whereas gravity is only an attractive 1/r**2 force, electrical forces can be either an attractive or repulsive 1/r**2 force. In fact, the MDGRAPE can handle any arbitrary force law or potential which can be expressed mathematically between two boundary points. A simple C or Fortran function is written which writes out these force (or potential) values to an ASCII file. These ASCII files are know as the Function Evaluator Tables and are loaded at runtime into the MDGRAPE. The FET is used to generate the coefficients of a 1024 piece fourth order polynomial. Fortran and C example FET generators are available in $MDGRAPE/gentable/.

The MDGRAPE comes with three example potentials and their associated force models:

Gravitational and Coulomb forces and potential:

The gravitational or coulomb force result from mass or electrical charges respectively. The two forces differ in their strength and that gravity is only attractive while electrical forces can be either attractive or repulsive. Both laws follow an inverse square law with infinite range.

Real part of Ewald force and potential:

  • g(x)=sqrt(pi)*exp(-x)*pow(x,-1.0)+erfc(sqrt(x))*pow(x,-1.5) (real.table)
  • g(x)=erfc(sqrt(x))*pow(x,-0.5) (realpot.table)

The real part of the Ewald potentials is a mechanism for calculating the potential from point charges with periodic boundary conditions. This has the effect of mapping a 2-space with boundaries onto a toroidal surface which has no boundaries.

Lennard-Jones force and potential:

The Lennard-Jones potential models the interaction between pairs of electrically neutral atoms with filled electron shells (noble gases). Repulsive forces dominate on short distance scales resulting from the Pauli exclusion principal while an attractive force dominates on longer scales resulting from the dipole moment of the van der Waals dispersion forces.

Other uses for the GRAPEs

The function call to the GRAPEs also returns a list of neighbors. This information can be useful for other types of codes. Two examples are Smooth Particle Hydrodynamics (SPH) and "Tree" codes. These codes are not interested in calculating the interaction of one particle with all the others but, in interacting with only their nearest neighbors. When these programs are run on general purpose computers, a significant amount of time is spent determining which particles are close enough to be called neighbors. By using the GRAPEs, neighbors can be determined with one call to the GRAPE hardware. The force or energy results are ignored and only the list of neighbors are utilized in further calculations on the host computer.

How do I go about using the GRAPEs at Indiana University?

If you feel that you have an application which could benefit from the GRAPEs' capabilities, please contact the High Performance Applications group, hpc@indiana.edu; contact the High Performance Systems group, hps-admin@iu.edu for account and technical implementation details. MDGRAPE-2 and GRAPE-6 account requests should be sent to hps-admin@iu.edu.

Links useful for programming and using the GRAPEs at IU

Using the MD-Grape at IU
Using the Grape-6 at IU
MDM software (MDGRAPE)
M2 C programmer's page (MDGRAPE)
M2 Fortran programmer's page (MDGRAPE)

Links of interest to Grape Users

AstroGRAPE
ManyBody.org
Starlab
NEMO
Grand Challenge Cosmology Consortium
N-Body Shop
The Virgo Consortium
Hydra Consortium
IBM Research
MDM (Molecular Dynamics Machine)
A molecular dynamics primer

Grapes in the popular press

Discover Magazine, June 1997 : The Star Machine
ABC News : Cutting Edge: The GRAPE-6 Supercomputer