Pipeline Dependencies

Setup Requirements

A number of system environmental variables and paths are required to run the BOSS DRP.

Running at the University of Utah CHPC

At Utah, modules (github.com/sdss/sdss_modules) are used to maintain the the required packages and paths. For the BOSS pipeline, the bhm modules are used to manage these for the BOSS pipeline.

Environmental Variables

export DATABASE_PROFILE="pipelines"
export RUN2D="v6_2_0"
export RUN1D="v6_2_0"
export BOSS_SPECTRO_REDUX="/uufs/chpc.utah.edu/common/home/sdss50/sdsswork/bhm/boss/spectro/redux"
export BOSS_SPECTRO_DATA_N="/uufs/chpc.utah.edu/common/home/sdss50/sdsswork/bhm/boss/spectro/apo"
export BOSS_SPECTRO_DATA_S="/uufs/chpc.utah.edu/common/home/sdss50/sdsswork/bhm/boss/spectro/lco"
export SDHDRFIX_DIR="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/sdsscore_test/main"
export IDLSPEC2D_DIR="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/idlspec2d/$RUN2D"
export SDSSCORE_DIR="/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/sdsscore_test/main"
export DAILY_DIR='$HOME/daily/
export BOSS_QA_DIR='$BOSS_SPECTRO_REDUX'

Paths

export IDL_PATH="$IDL_PATH:$IDLSPEC2D_DIR"
export PATH="$PATH:$IDLSPEC2D_DIR/bin"
export PYTHONPATH="$PYTHONPATH:$IDLSPEC2D_DIR/python"

Compiling Internal Functions

The idlspec2d package contains a shared object library and C scripts. If you have issues with these, you can run evilmake all within the main idlspec2d directory ($IDLSPEC2D_DIR). evilmake is included within idlutils, so follow the directions on (sdss.org/dr18/software/packages/idlutils/) to install idlutils first.

Compiling Internal Documentation

This documentation can be compiled locally by navigating to $IDLSPEC2D_DIR/docs/sphinx and run make all (This can also be run with evilmake all in the same directory or evilmake doc within $IDLSPEC2D_DIR). This will compile the html, singlehtml, and pdf forms of the documentation within the associated subdirectoires of $IDLSPEC2D_DIR/docs/sphinx/_build.

Dependencies

  • idl

  • python(3.7-3.11)

  • SDSS Collaboration Package Dependencies

    • idlutils: idlutils is a collection of IDL functions and routines used by a variety of SDSS software.

    • sdssdb: sdssdb contains the source catalogs, targeting catalogs, and operational databases.

    • semaphore: provides codes and reference files to decode and understand the sdss_target_flags

    • sdss-tree: provides environment variables to manage paths for SDSS data as they organized on the Science Archive Server (SAS)

    • sdss_access: provides a convenient way of navigating local and remote file system paths from the Science Archive Server (SAS)

    • pyVista:

    • SDSS Slurm: Required for use of uurundaily at Utah, all other command can be manually (at Utah or elsewhere) run without access to the slurm manager provided by this package

    • sdsstools:

  • SDSS Product Dependencies

    • sdsscore: sdsscore is an SDSS-V product that contains the FPS fiber configuration files and header correction files to change the header exposure values (sdHdrFix-{mjd}.par)

    • elodie: A database of high and medium-resolution stellar spectra (Prugniel+, 2001) used by spec1d to classify spectra and determine stellar parameters.

    • dust: A catalog of dust extinction models, including the SFD model.

    • speclog: speclog is an SDSS product that contains information about SDSS BOSS plate operations including seeing measured by the guides (guiderMon-{MJD}.par, plate plug maps (plPlugMapM-{plateid}-{mjd}-{plugid}.par, and plate header correction files to change the header exposure values (sdHdrFix-{mjd}.par)

    • platelist: platelist is an SDSS product that contains information on the plate designs and plugging. The plateHoles files include additional metadata associated with the targets on a plate

    • specflat: specflat is an SDSS product that contains master calibration frames and bad pixel masks for use in the idlspec2d pipeline.

    • gaia/dr2: idlspec2d utilizes gaia_source/csv to calculate the distance to standard stars from GAIA DR2 proper motion.

  • External Dependencies

    • pyDL: a package that consists of python replacements for IDL function, both built-in and from external astronomical libraries

    • dustmaps: provides a unified interface for several 2D and 3D maps of interstellar dust reddening and extinction. idlspec2d makes use of the Bayestar 2015 dustmaps (Green, Schlafly, Finkbeiner et al. 2015)

    • PyXCSAO: a python package designed to replicate the functionality of IRAF XCSAO.

    • numpy: a standard Python package for arrays and high-level mathematical functions

    • astropy: a collection of astronomy packages written in Python

    • matplotlib: a python plotting library

    • healpy: a Python package based on the Hierarchical Equal Area isoLatitude Pixelization (HEALPix) scheme

    • tqdm: a progress bar for Python

    • pandas: a python package designed for data manipulation and analysis

    • h5py: a python interface between numpy and HDF5 data

    • scipy: a python package for scientific and technical computing

    • pillow: a python image file processing library

    • jinja2: a python templating engine used to build the HTMLs produced by the pipeline

    • termcolor: a python package for color formatting of terminal outputs (not required but recommended)

    • plotly: a python package for interactive plots (not required for the core pipeline but used by some of the supplementary tools)