# 1. Brief
The following README describes the environment variables available to control the ECW project configuration. <br/>
Last modified: Aug 31 2021<br/>
Contact information: STAR ASSISTT PLAIT <br/>

Within this directory one may find the main XML configuration, `Config.xml`, this README, and a test script located in the `Test_Scripts/` directory.

The following test script may be run to verify a successful compilation of the algorithms, and serve as a representative example of executing the algorithms for this project:

* Execute Unit 1 (clouds outputs + inputs needed by winds):
    * `Config/Projects/ECW/Test_Scripts/run-ECW-Clouds.bash`
* Execute projection (see [README-WINDS](README-WINDS.md) for more details):
    * `Config/Projects/ECW/batch_example/run.py`
        * Note: projection arguments are required. Example:
          ```bash 
          python Config/Projects/ECW/batch_example/run.py Config/Projects/ECW/batch_example/winds/winds.ini \
          --projection --images AIT_TEST_DATA_WINDS/AVHRR/Intermediate_Data/METOPC*NAV*
          ```
* Execute winds (see [README-WINDS](README-WINDS.md) for more details):
    * `Config/Projects/ECW/Test_Scripts/run-Winds.bash`

### Output List
Unit 1 generates the following products for output:
* CLOUD_MASK_EN
* CLOUD_PHASE_CLAVRX
* CLOUD_HEIGHT_EN
* CLOUD_DCOMP_EN (needed by METOPC project only)
* CLOUD_LAYERS_EN(needed by METOPC project only)


By default, the script assumes that the following executables and directories are located in your working directory:

### Executables:
    * `algorithms.exe` - algorithm executable

### Directories:
    * `Config/` - Location of XML configurations developed for the algorithms
    * `AIT_TEST_DATA/` - Location of test input data
    * `Output/` - Output directory for current Framework run


# 2. Performance Metrics:

The following time and memory for tests running METOPB and METOPC were measured on STAR machine rhw1158 
with Intel 19.0.4 compilers.

* STAR Linux (rhw1158) machine specifications:
    * Processors: 24 Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz
    * Memory: 125 Gb
    * Disk bandwidth: 2.0 Gb/s

* METOPB GHRR Clouds 1 segment sequential:
    * Total time: 172 seconds
    * Total memory (estimate): 3.18 Gb

* METOPB FRAC Clouds 5 segments using 5 processors parallel:
    * Total time: 1710.63 seconds
    * Total memory (estimate): 0.86 Gb/proc

* METOPB HRPT Clouds 1 segment sequential:
    * Total time: 400.81 seconds
    * Total memory (estimate): 5.24 Gb

* METOPC GHRR Clouds 1 segment sequential:
    * Total time: 292.11 seconds
    * Total memory (estimate): 3.61 Gb

* METOPC FRAC Clouds 5 segments using 5 processors parallel:
    * Total time: 3133 seconds
    * Total memory (estimate): 1.43 Gb/proc

* METOPC HRPT Clouds 1 segment sequential:
    * Total time: 426.29 seconds
    * Total memory (estimate): 5.06 Gb
   

### Estimated file sizes for the Clouds output using NetCDF Compression Level 2:
* METOPB GHRR:
    * Cloud Mask: 26 Mb
    * Cloud Phase: 12 Mb
    * Cloud Height: 78 Mb
    * DCOMP: 29 Mb
    * Cloud Layers: 29 Mb

* METOPB FRAC:
    * Cloud Mask: 325 Mb
    * Cloud Phase: 136 Mb
    * Cloud Height 1 Gb
    * DCOMP: 131 Mb
    * Cloud Layers: 351 Mb

* METOPB HRPT:
    * Cloud Mask: 75 Mb
    * Cloud Phase: 32 Mb
    * Cloud Height: 249 Mb
    * DCOMP: 71 Mb
    * Cloud Layers: 86 Mb

* METOPC GHRR:
    * Cloud Mask: 47 Mb
    * Cloud Phase: 21 Mb
    * Cloud Height: 135 Mb
    * DCOMP: 34 Mb
    * Cloud Layers: 49 Mb

* METOPC FRAC:
    * Cloud Mask: 532 Mb
    * Cloud Phase: 235 Mb
    * Cloud Height: 1.8 Gb
    * DCOMP: 345 Mb
    * Cloud Layers: 631 Mb

* METOPC HRPT:
    * Cloud Mask: 69 Mb
    * Cloud Phase: 30 Mb
    * Cloud Height: 234 Mb
    * DCOMP: 69 Mb
    * Cloud Layers: 84 Mb
 
# 3. Environment Settings

The operational user can control the run via environment variables. The ones likely to be needed for this project are described below.

### Standard environment variables:
* `ENV_SATELLITE` - Name of the satellite.
* `ENV_SAT_AVHRR_L1B_FILE` - Full path and filename of the AVHRR L1B file

### AVHRR L1B dependent environment variables:
* `ENV_RESOLUTION` -Resolution of L1B file. e.g. AVHRR-GAC or 1KM
* `ENV_NUM_ROWS` -The total number of scanlines. Value needs to be acquired from L1b through external script before running framework.
* `ENV_COVERAGE` -Coverage of the L1B file. e.g. FRAC, HRPT, LAC, or GHRR

### Standard data environment variables:
* `ENV_ALGORITHM_ANCILLARY_DIR` - Algorithm ancillary directory.  Default algorithm_ancillary/
* `ENV_FRAMEWORK_ANCILLARY_DIR` - Framework ancillary directory.  Default framework_ancillary/
   * defaults are if the variable is not set in environment



### NWP GFS environment variables:
* `ENV_NWP_GFS_GRIB2_RESOLUTION` - NWP GFS GRIB 2 data resolution. Options: HALF_DEGREE, QUARTER_DEGREE
* `ENV_NWP_GFS_GRIB2_FILE` - A file path pattern sensitive to the file naming patterns.
    * See table: 
    >|  pattern |        meaning        |
    >|:--------:|:--------------------- |
    >| `@YYYY@` | 4 digit year          |
    >| `@YY@`   | 2 digit year          |
    >| `@MM@`   | 2 digit month         |
    >| `@DD@`   | 2 digit day           |
    >| `@HR@`   | 2 digit hour          |
    >| `@MIN@`  | 2 digit minute        |
    >| `@AN@`   | 2 digit analysis time |
    >| `@FF@`   | 2 digit forcast time  |
* Examples: 
    * `./gfs.t12z.pgrb2.0p50.f018.20200930`
        * -> `./gfs.t@AN@z.pgrb2.0p50.f0@FF@.@YYYY@@MM@@DD@`
    * `AIT_TEST_DATA/AHI/gfs_grib2_0.5deg/2020/09/30/gfs.t12z.pgrbf18`
        * -> `AIT_TEST_DATA/AHI/gfs_grib2_0.5deg/@YYYY@/@MM@/@DD@/gfs.t@AN@z.pgrbf@FF@`

### NetCDF output environment variables:
* `ENV_OUTPUT_DIRECTORY` - Output directory of Framework NetCDF files
* `ENV_OUTPUT_LIST` - Optional user defined list that overwrites satellite dependent default output list. 
    * Default output list are as follows:
         * For METOPB: `"CLOUD_MASK_EN,CLOUD_PHASE_CLAVRX,CLOUD_HEIGHT_EN"`
         * For METOPC: `"CLOUD_MASK_EN,CLOUD_PHASE_CLAVRX,CLOUD_HEIGHT_EN,CLOUD_DCOMP_EN,CLOUD_LAYERS_EN"`
* For Generating winds the following outputs must be set: 
    * `SAT_AVHRR,CLOUD_MASK_EN,CLOUD_PHASE_CLAVRX,CLOUD_HEIGHT_EN`
    * Recommended: `ENV_CHANNELS_OUTPUT=04` to constrain the output of `SAT_AVHRR`

### NetCDF input environment variables:
* `ENV_INPUT_DIRECTORY` - Input directory of Framework NetCDF files
* `ENV_INPUT_LIST` - Comma-separated string list of algorithms to be input.  e.g. `"CLOUD_MASK_EN,CLOUD_HEIGHT_EN"`
* `<algorithm>-PROD_FILE` - Filename of the input algorithm NetCDF file.  e.g. `CLOUD_MASK_EN_PROD_FILE=cloud_mask.nc`

### Snow Mask environment variables:
* `ENV_SNOW_MASK_COMMON_ALG` - Ancillary snow mask algorithm to be used as precedent data.
* `ENV_SNOW_MAP_DIR` - Directory of IMS/SSMI snow mask data

### Production metadata environment variables:
* `ENV_INSTITUTION` - Institution string for the global metadata attribute
* `ENV_NAMING_AUTHORITY` - Naming authority string for the global metadata attribute
* `ENV_PRODUCTION_SITE` -Production site string for the global metadata attribute
* `ENV_PRODUCTION_ENVIRONMENT` -Production Environment string for the global metadata attribute

### NetCDF output control environment variables:
* `ENV_NETCDF_COMPRESSION` - Compression level of NetCDF files.  0 is no compression.  Default: 2.

### Data Cache Archive control environment variables:
* `ENV_ARCHIVE_NAME` -Name of the Data Cache Archive.  If not provided or empty, the Job ID will be used as the archive name.
* `ENV_ARCHIVE_DIR` -Location of the Data Cache Archive folder.  Default: `/dev/shm`