README.md 4.36 KB
Newer Older
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
1 2
# Workload Manager

Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
3

Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
4 5 6 7 8 9 10 11 12 13 14

## Hints

### When to Use
- Single fast analysis step (eg your analysis file runs for only a minute)
- 1000's or more single analysis steps
- Usage of all cores in node exclusive partitions (Mogon 2, not on HIMster 2)


### Comparision

Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
15
- Queue based work distribution, equal work distribution (in contrast to SLURM multiprog or [staskfarm](https://github.com/cmeesters/staskfarm) from [MogonWiki Node local scheduling](https://mogonwiki.zdv.uni-mainz.de/dokuwiki/node_local_scheduling))
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
16 17 18 19 20
- Usage of MPI: 
  - large connected jobs (>200 cores) are preferred by the job manager
  - efficiently supports both node local and multi node usage 
  - keeps environment , also in multi node sutiations (with GNU parallel only on node local)
- Usage of Python makes changes for users simple
21 22

- Only one disadvantage: Number of ranks fixed during runtime -- in contrast to SLRUM jobs.
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83


## Usage
First steps (aka hello world):

1. On HIMster 2 / Mogon 2, load the following module first
   ```bash
   module load lang/Python/3.6.6-foss-2018b
   ```
   to enable Python 3.6 and MPI4Py support. You can also add this line to your `~/.bashrc` configuration file to speed up the process when you login again.

2. Next, test the parameters for the workload-manager. To do so, run short tests (with the dry-run option) on the headnode. More examples with different parameters see in the next chapter
   * On a head node run with 
     ```bash
     ./wkmgr.py -n [YOUR EXECUTABLE]
     ```
   * Or reserve a dedicated node for this purpose first, eg
     ```bash
     salloc -p devel -A m2_him_exp -N 1 -t 1:30:00
     #or during a turorial
     salloc -p parallel --reservation=himkurs -A m2_himkurs -N 1 -t 1:30:00
     ````
     #and do some test runs like in the head node case.

3. Once you found the right launcher arguments, submit the job with
    ```bash
    #load modules for demo analysis and MPI4Py
    module purge
    module load math/SUNDIALS/2.7.0-intel-2018.03
    module load lang/Python/3.6.6-foss-2018b

    #run some example provided in the git repository
    srun -n 20 ~/workload-manager/wkmgr.py -v -i ~/workload-manager/examples/LGS/Run27_LaPalma_Profile_I50 ~/workload-manager/examples/LGS/PulsedLGS
    ```

- Missing topic: How to identify the right number of cores

### Examples

#### Input / Output File Example
Task: Run the analysis binary for each input file in MyInputDirectory on 20 cores
```bash
srun -n 20 ~/workload-manager/wkmgr.py -i MyInputDir MyAnalysisBinary
```
check for output* in the current directory for the output or change this behaviour with the `-o`-option.

Provide the `-s` option, of you do not want to add the default placeholders (`{inputdir}{inputfilename} {outputdir}{jobid}/outfile.txt`) to the execname:
```bash
srun -n 20 ~/workload-manager/wkmgr.py -s MyAnalysisBinary
```

#### Parameters Example
Task: Run the analysis binary for theta from 0 to 180 in 2.5 degree and energy from 130 to 150 MeV in 2 MeV steps with no input files on 20 cores
```bash
srun -n 20 ~/workload-manager/wkmgr.py -a theta,0,180,2.5 -a energy,130,150,2 -ni MyAnalysisBinary {theta} {energy}
```
- Note: The `-s` option is automatically activated once you add a '{' character in your execname statement.

#### Example Shell Environment
Your bash environment is also active within the workload-manager jobs. This enables you to do more sophisticated calls like:
```bash
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
84
srun -n 20 ~/workload-manager/wkmgr.py -a theta,0,180,2.5 -a energy,130,150,2 -ni "MyAnalysisBinaryPrepare && MyAnalysisBinary {theta} {energy} && MyAnalysisBinaryAfter"
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
85
```
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
86
- Note the `"` in this statement.
Peter-Bernd Otte's avatar
Peter-Bernd Otte committed
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107

#### Change Verbosity
Get infos with `-v` or even mir mit `-vv`. Sample output 
```bash
2019-05-27 16:27:14,842 rank15 INFO      Worker startet job.
```
which tells you the time, the rank / worker, debug/info/warning/error and the message text.
Please note, that output from the individual workers / ranks are not necessarily displayed in the right order -- where as the order within a rank is consistent.

#### Dry-runs
Test everything before you do the calculation with
```bash
~/workload-manager/wkmgr.py -vv -n ...
```
to perform a dry-run with maximum verbosity and check the printed out worklist.

## Full Loader
The full loader is currently under development. 
- with loader (untested on cluster so far) replace `wkmgr.py` with `wkloader.py`. 
- Handy command for debug purposes might be: `mpirun -n 4 ./wkmgr.py -vv -n [YOUR EXECUTABLE]`