Docs
ADCIRC Wiki
An ADCIRC Wiki is currently under development to provide community supported documentation for ADCIRC.
Overview of Running ADCIRC v46 in Parallel
The main difference between running ADCIRC on a PC and running ADCIRC in parallel on a multi-CPU cluster machine is the need to decompose the mesh and input files into smaller pieces, so that each piece can run on its own CPU on the cluster. In addition, after ADCIRC has run, the output files from each piece of the mesh must be knitted back together to represent the total solution. Step-by-step instructions for performing these steps are provided below.
Preprocessing
The executable program adcprep is provided with ADCIRC and is used to perform the decomposition. In order to break up the original input files (which describe the full domain) into smaller pieces (called subdomains), go to the directory where the input files are located and execute adcprep. The executable will make the following request:
Input number of processors for parallel ADCIRC run:
Enter the number of CPUs that you plan to use for the solution. Adcprep will decompose your input files into that number of subdomains. If you decide later that you want to run on more (or fewer) CPUs, you must perform this step again.
There is a limit to the number of CPUs that you can use at one time. A rule of thumb for ADCIRC is that each subdomain should be large enough to have at least 1000 mesh nodes. In other words, dividing the mesh into so many subdomains that each one has less than approximately 1000 nodes is actually counterproductive. For example, if your full mesh contains 16,000 nodes, use 16 CPUs or fewer.
Next, adcprep presents a menu with three options: (1) prepall, (2) prepspec, and (3) prep14_15. Enter '1' to select prepall. This will decompose your mesh and all input files. The other choices are used to do fancier things: prepspec is used if you have input files with nonstandard names or would like to skip the prepping of certain files; prep14_15 is used for redecomposing only the fort.15 file, assuming that all other input files have already been decomposed.
Running in parallel
After the mesh and input files have been prepped with adcprep, the actual ADCIRC calculations are performed using the executable program padcirc (parallel ADCIRC). This program performs the ADCIRC calculations on each subdomain and uses MPI to provide ongoing communication between the subdomains.
The command required for actually starting the ADCIRC job is highly dependent on the cluster software and queueing system. If the mpich MPI software is used directly (without a queueing system) then the job may be started with
mpirun -np 16 /full/path/to/padcirc
assuming that the user is in the same directory as the input files and the input files have been prepped into 16 subdomains.
Post processing
Once the solution process is complete, the output files from padcirc must be knitted back together to represent the solution on the whole domain. The executable file adcpost is used to achieve this. Go to the directory where the input files are located and type adcpost. Adcpost will start up and make the following request:
Select operation
Type post to indicate that you would like to combine the subdomain output files into full domain output files. The next request is
Enter Unit number to Post Process
Enter the number of the output file that is to be post processed, e.g., enter '63' to combine the subdomain fort.63 files into a full domain fort.63 file. If you would like to post process all the subdomain output files that were produced during the run, enter '100'. Enter '999' when finished. The final request is
Do you want to Post-Process hotstart files? ( Y/N )
The hotstart files from the subdomains could be combined into a full domain hotstart file, and then redecomposed at a later time to do another run. This capability is rarely used. Enter 'N' to skip this step, then enter 'quit' to exit adcpost.