Air Resources Laboratory banner image
Air Resources Laboratory web site National Oceanic and Atmospheric Administration (NOAA)
Gridded Meteorological Data Archives READY logo

Overview

The National Weather Service's National Centers for Environmental Prediction (NCEP) runs a series of computer analyses and forecasts operationally. NOAA's Air Resources Laboratory (ARL) routinely uses NCEP model data for use in air quality transport and dispersion modeling calculations. In 1989 ARL began to archive some of these datasets for future research studies. ARL has in the past, or is presently archiving the following NCEP datasets, which can be retrieved from the cloud or from an ARL server.

Data files stored in the cloud can be accessed and downloaded by navigating your web browser to https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/index.html or by utilizing the AWS command-line tools: Instructions for using the AWS tools are provided below. Data files are organized by dataset name, then by year, and for certain datasets, further by month. Each dataset directory includes a listing.md5.txt file, which provides MD5 checksums for all data files. This file allows for verification of data integrity upon download.

For further information on model changes see the following web sites:

Using AWS CLI for Linux/macOS Users

To access and download files using the AWS Command Line Interface (CLI), you must have it installed and configured. Refer to the official AWS documentation for installation instructions if you do not have the AWS CLI.

  1. List Contents of the Bucket: To see the available datasets (top-level folders in the S3 bucket), use the aws s3 ls command. 
    aws s3 ls s3://noaa-oar-arl-hysplit-pds/ --no-sign-request
    Example Output: 
    PRE gfs0p25/
PRE hrrr/
PRE nam12/
  2. Navigate to a Specific Dataset and List Years: Once you identify the desired dataset, you can drill down into it. Replace [dataset_name] with the actual name. This command will list all years, including the listing.md5.txt file for data integrity verification. 
    aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/ --no-sign-request
    Example Output: 
    PRE 2023/
PRE 2024/
PRE 2025/
PRE listing.md5.txt
  3. Navigate to a Specific Year (and Month, if applicable) and List Files: Continue to navigate down the directory structure. Replace [dataset_name], [year], and [month] as needed. 
    # If organized by year only
aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/ --no-sign-request

# If organized by year and month
aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/[month]/ --no-sign-request
  4. Download Individual Files: To download a specific data file, use the aws s3 cp command. Replace [file_name] with the exact name of the file you want to download, and [local_path] with the directory on your computer where you want to save the file. 
    aws s3 cp s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/[file_name] [local_path] --no-sign-request
    Example: 
    aws s3 cp s3://noaa-oar-arl-hysplit-pds/edas/2004/edas.subgrd.apr04.001 . --no-sign-request
    (Note: If the path does not include a month, adjust the S3 path accordingly.)
  5. Download listing.md5.txt: Always download the listing.md5.txt file from the dataset directory for integrity verification. 
    aws s3 cp s3://noaa-oar-arl-hysplit-pds/[dataset_name]/listing.md5.txt [local_path] --no-sign-request
  6. Verify Data Integrity (MD5 Checksum): After downloading your data files, open the listing.md5.txt file you downloaded. It will contain MD5 checksums for each file. You can then use a command-line utility to calculate the MD5 checksum of your downloaded data files and compare them. 
    # Linux
md5sum [downloaded_file_name]

# macOS
md5 [downloaded_file_name]
    Compare the output of this command with the corresponding entry in listing.md5.txt. If the checksums match, your file download is good. If they do not match, the file may be corrupted, and you should re-download it.

For Windows PC Users

Unlike the AWS CLI described above, the AWS Tools for PowerShell require credentials. We recommend using a web browser to navigate the HYSPLIT archive on the cloud. Once you identify a file to download, you can use either a web browser to initiate the download or a PowerShell command.

  1. Download Individual Files: Use Invoke-WebRequest to download a specific file. Replace [dataset_name], [year], [month] (as needed), and [file_name]. 
    # If organized by year only
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/[year]/[file_name] -OutFile "[file_name]"

# If organized by year and month
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/[year]/[month]/[file_name] -OutFile "[file_name]"
    Example: 
    Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/edas/2004/edas.subgrd.apr04.001 -OutFile "edas.subgrd.apr04.001"
  2. Download listing.md5.txt: Always download the listing.md5.txt file from the dataset directory for integrity verification. 
    Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/listing.md5.txt -OutFile "listing.md5.txt"
  3. Verify Data Integrity (MD5 Checksum): After downloading your data files, open the listing.md5.txt file you downloaded. It will contain MD5 checksums for each file. You can then use a command-line utility to calculate the MD5 checksum of your downloaded data files and compare them. 
    Get-FileHash -Path "[downloaded_file_name]" -Algorithm MD5
    Compare the Hash value returned by this command with the corresponding entry in the listing.md5.txt file. If they match, your file download is good. If they do not match, the file may be corrupted, and you should re-download it.

Using FTP

The ARL server is accessible via an FTP client or a web browser.

  1. Using an FTP client:
    • Windows PC users: Use clients like WinSCP or FileZilla that support passive mode.
    • macOS and Linux users: Use the ftp command.
    • Connection details: Connect to ftp.arl.noaa.gov. Use "anonymous" as the username and your email address as the password.
    • Navigation: Change the directory to the archive of your interest.
    • Important: Limit concurrent FTP connections to two or fewer to avoid being blocked from the server.
  2. Using a web browser:
    • Point your browser to the relevant archive URL (listed below).

Currently Available Data

  • WRF ARW (Weather Research & Forecasting Model for CAPTEX Tracer Experiment)

  • CFSR (The Climate Forecast System Reanalysis for the CAPTEX Tracer Experiment)


Data No Longer Updated

All of theses datasets contain basic fields such as the u- and v-wind components, temperature, and humidity. However, the archives differ from each other because of the horizontal and vertical resolution, as well as in the specific fields provided by NCEP. All fields were selected by ARL according to what is most relevant for transport and dispersion studies and disk space limitations.

Data Packing Format

NCEP typically saves their model output in GRIB format. However, at ARL the data are reprocessed and stored in a 1-byte packing algorithm. This 1-byte packing is a bit more compact than GRIB and can be directly used on a variety of computing platforms with direct access I/O.

The data array is packed and stored into one byte characters. To preserve as much data precision as possible the difference between the values at grid points is saved and packed rather than the actual values. The grid is then reconstructed by adding the differences between grid values starting with the first value, which is stored in unpacked ASCII form in the header record. To illustrate the process, assume that a grid of real data, R, of dimensions i,j is given by the below example.


  1,j       2,j        ....    i-1,j      i,j
  1,j-1     2,j-1      ....    i-1,j-1    i,j-1
  ....      ....       ....    ....       ....
  1,2       2,2        ....    i-1,2      i,2
  1,1       2,1        ....    i-1,1      i,1

The packed value, P, is then given by


     Pi,j = (Ri,j  - Ri-1,j)* (2**(7-N)),

where the scaling exponent


     N = ln dRmax / ln 2 .

The value of dRmax is the maximum difference between any two adjacent grid points for the entire array. It is computed from the differences along each i index holding j constant. The difference at index (1,j) is computed from index (1,j-1), and at 1,1 the difference is always zero. The packed values are one byte unsigned integers, where values from 0 to 126 represent -127 to -1, 127 represents zero, and values of 128 to 254 represent 1 to 127. Each record length is then equal in bytes to the number of array elements plus 50 bytes for the header label information. The 50 byte label field precedes each packed data field and contains the following ASCII data:



Field          Format    Description
Year           I2        Greenwich date for which data valid
Month          I2                       "
Day            I2                       "
Hour           I2                       "
Forecast*      I2        Hours forecast, zero for analysis
Level          I2        Level from the surface up (see Table 3)
Grid           I2        Grid identification (see Table 1)
Variable       A4        Variable label (see Table 2)
Exponent       I4        Scaling exponent needed for unpacking
Precision      E14.7     Precision of unpacked data
Value 1,1      E14.7     Unpacked data value at grid point 1,1
*Forecast hour is -1 for missing data.

The following Fortran90 program can be used to to unpack and read the first few elements of the data array for each record of an ARL packed meteorological file.


CHK_DATA.F

Modified: August 6, 2025
US Dept. of Commerce |  NOAA |  NOAA Research |  ARL
Privacy |  Disclaimer |  Information Quality
Accessibility |  > webmaster