Running a BOSS DRP Daily Run¶
Daily Run¶
In addition to running large batches of MJDs in one go, the pipeline is also designed to be run on a daily basis.
Crontab at Utah¶
The crontab below has been designed so that two of these tasks can been written to allow for multiple version os the pipeline to be run in parallel (eg Master and v6_2_0). This is to ensure a stable reduction (only updated with new tags) and a development branch that will contain the leading edge developments, but will not contain a uniform reduction. In this mode, every time uurundaily runs, it checks for a new MJD in the $BOSS_SPECTRO_DATA_N/S directories and if a new mjd exists, it updates the $daily_dir/etc/nextmjd.par file and starts run that mjd through the full pipeline. The numbers and asterisk, in the crontab below, before the command are when the script will run:
field |
allowed values |
|---|---|
minute |
0-59 |
hour |
0-23 |
day of month |
1-31 |
month |
1-12 |
day of week |
0-7 (0 or 7 is Sunday |
Master/main Daily Run
IDLSPEC2D_BASE="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/idlspec2d/"
TMOD="master"
0,30 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "uurundaily --apo --daily"
5,35 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "uurundaily --lco --daily"
0 1 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "slurm_Summary --defaults"
3 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "daily_log"
0 22 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash bhm/$TMOD -n
0 20 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash bhm/$TMOD -l -n
Tagged Daily Run
IDLSPEC2D_BASE="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/idlspec2d/"
TMOD="v6_2_0"
0,30 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "uurundaily --apo --tagged"
5,35 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "uurundaily --lco --tagged"
0 1 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "slurm_Summary --defaults"
3 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/$TMOD "daily_log"
0 22 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash bhm/$TMOD -n
0 20 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash bhm/$TMOD -l -n
Development Daily Run
IDLSPEC2D_BASE="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/idlspec2d/"
TMOD="v6_2_0_alpha"
0,30 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash work/$TMOD "uurundaily --apo --dev"
5,35 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash work/$TMOD "uurundaily --lco --dev"
0 1 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash work/$TMOD "slurm_Summary --defaults"
3 * * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash work/$TMOD "daily_log"
0 20 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash work/$TMOD -n
0 20 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronplot_QA.bash work/$TMOD -l -n
Run QA Plots for two run2ds at once
55 23 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/v6_2_0 "plot_qa --run2d master v6_2_0 --mjds_low 60403 None --mjds_high None 60402 --lco"
25 20 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/v6_2_0 "plot_qa --run2d master v6_2_0 --mjds_low 60403 None --mjds_high None 60402 --lco --html"
35 22 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/v6_2_0 "plot_qa --run2d master v6_2_0 --mjds_low 60447 None --mjds_high None 60446"
30 23 * * * $IDLSPEC2D_BASE/$TRUN2D/bin/cronrun.bash bhm/v6_2_0 "plot_qa --run2d master v6_2_0 --mjds_low 60447 None --mjds_high None 60446 --html"
Manual Using uurundaily at Utah¶
To manually use (uurundaily) (which runs the full pipeline end-to-end), one of versions of the command below can be used. The --daily, --tagged, and --dev tags can also be used, but the examples below give the explicite options.
Run for any new MJDS (if a module/paths are preloaded)¶
uurundaily --lco --module bhm/v6_2_0 --fast --merge3d --no_dither --monitor
uurundaily --apo --module bhm/v6_2_0 --fast --merge3d --no_dither --monitor
Run for a set MJD (if a module/paths are preloaded)¶
Note
This method does not update $daily_dir/etc/nextmjd.par
uurundaily --lco --module bhm/v6_2_0 --fast --merge3d --no_dither --monitor --mjd 60000
uurundaily --apo --module bhm/v6_2_0 --fast --merge3d --no_dither --monitor --mjd 60000