Overview
The National Weather Service's National Centers for Environmental
Prediction (NCEP) runs a series of computer analyses and forecasts
operationally. NOAA's Air Resources Laboratory (ARL) routinely uses NCEP
model data for use in air quality transport and dispersion modeling calculations.
In 1989 ARL began to archive some of these datasets for future research
studies. ARL has in the past, or is presently archiving the following NCEP
datasets, which can be retrieved from the cloud or from an ARL server.
Data files stored in the cloud can be accessed and downloaded by navigating your web browser to https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/index.html or by utilizing the AWS command-line tools: Instructions for using the AWS tools are provided below. Data files are organized by dataset name, then by year, and for certain datasets, further by month. Each dataset directory includes a listing.md5.txt file, which provides MD5 checksums for all data files. This file allows for verification of data integrity upon download.
For further information on model changes see the following web sites:
Using AWS CLI for Linux/macOS Users
To access and download files using the AWS Command Line Interface (CLI), you must have it installed and configured. Refer to the official AWS documentation for installation instructions if you do not have the AWS CLI.
- List Contents of the Bucket: To see the available datasets
(top-level folders in the S3 bucket), use the
aws s3 ls
command.
aws s3 ls s3://noaa-oar-arl-hysplit-pds/ --no-sign-request
Example Output:
PRE gfs0p25/
PRE hrrr/
PRE nam12/
- Navigate to a Specific Dataset and List Years: Once you
identify the desired dataset, you can drill down into it. Replace
[dataset_name]
with the actual name.
This command will list all years, including the listing.md5.txt file
for data integrity verification.
aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/ --no-sign-request
Example Output:
PRE 2023/
PRE 2024/
PRE 2025/
PRE listing.md5.txt
- Navigate to a Specific Year (and Month, if applicable) and List
Files: Continue to navigate down the directory structure. Replace
[dataset_name]
,
[year]
, and
[month]
as needed.
# If organized by year only
aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/ --no-sign-request
# If organized by year and month
aws s3 ls s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/[month]/ --no-sign-request
- Download Individual Files: To download a specific data file,
use the
aws s3 cp
command. Replace [file_name]
with the exact name of the file you want to download, and
[local_path]
with the directory on your computer where you want to save the file.
aws s3 cp s3://noaa-oar-arl-hysplit-pds/[dataset_name]/[year]/[file_name] [local_path] --no-sign-request
Example:
aws s3 cp s3://noaa-oar-arl-hysplit-pds/edas/2004/edas.subgrd.apr04.001 . --no-sign-request
(Note: If the path does not include a month, adjust the S3 path
accordingly.)
- Download listing.md5.txt: Always download the listing.md5.txt
file from the dataset directory for integrity verification.
aws s3 cp s3://noaa-oar-arl-hysplit-pds/[dataset_name]/listing.md5.txt [local_path] --no-sign-request
- Verify Data Integrity (MD5 Checksum): After downloading your
data files, open the listing.md5.txt file you downloaded. It will
contain MD5 checksums for each file. You can then use a command-line
utility to calculate the MD5 checksum of your downloaded data files
and compare them.
# Linux
md5sum [downloaded_file_name]
# macOS
md5 [downloaded_file_name]
Compare the output of this command with the corresponding entry in
listing.md5.txt. If the checksums match, your file download is
good. If they do not match, the file may be corrupted, and you should
re-download it.
For Windows PC Users
Unlike the AWS CLI described above, the AWS Tools for PowerShell require credentials. We recommend using a web browser to navigate the HYSPLIT archive on the cloud. Once you identify a file to download, you can use either a web browser to initiate the download or a PowerShell command.
- Download Individual Files:
Use
Invoke-WebRequest
to download a specific file. Replace
[dataset_name]
,
[year]
,
[month]
(as needed), and
[file_name]
.
# If organized by year only
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/[year]/[file_name] -OutFile "[file_name]"
# If organized by year and month
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/[year]/[month]/[file_name] -OutFile "[file_name]"
Example:
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/edas/2004/edas.subgrd.apr04.001 -OutFile "edas.subgrd.apr04.001"
- Download listing.md5.txt: Always download the listing.md5.txt
file from the dataset directory for integrity verification.
Invoke-WebRequest -Uri https://noaa-oar-arl-hysplit-pds.s3.amazonaws.com/[dataset_name]/listing.md5.txt -OutFile "listing.md5.txt"
- Verify Data Integrity (MD5 Checksum): After downloading your
data files, open the listing.md5.txt file you downloaded. It will
contain MD5 checksums for each file. You can then use a command-line
utility to calculate the MD5 checksum of your downloaded data files
and compare them.
Get-FileHash -Path "[downloaded_file_name]" -Algorithm MD5
Compare the Hash value returned by this command with the corresponding
entry in the listing.md5.txt file. If they match, your file download
is good. If they do not match, the file may be corrupted, and you should
re-download it.
Using FTP
The ARL server is accessible via an FTP client or a web browser.
- Using an FTP client:
- Windows PC users: Use clients like WinSCP or FileZilla that support passive mode.
- macOS and Linux users: Use the ftp command.
- Connection details: Connect to ftp.arl.noaa.gov. Use "anonymous" as the username and your email address as the password.
- Navigation: Change the directory to the archive of your interest.
- Important: Limit concurrent FTP connections to two or fewer to avoid being blocked from the server.
- Using a web browser:
- Point your browser to the relevant archive URL (listed below).
Currently Available Data
NAMS Hybrid sigma-pressure archive (CONUS, Alaska, Hawaii, 2010-)
GDAS one-degree archive (Dec 2004 - present)
GFS quarter-degree archive (June 2019 - present)
NCEP/NCAR Reanalysis (1948 - present)
HRRR 3 km (June 2019 - present)
NAM 12 km (May 2007 - present)
Weather Research & Forecasting Model (WRF ARW, 1980 - present)
WRF ARW (Weather Research & Forecasting Model for CAPTEX Tracer Experiment)
MM5 (MM5 Community Model for the METREX Tracer Experiment)
CFSR (The Climate Forecast System Reanalysis for the CAPTEX Tracer Experiment)
Data No Longer Updated
- HRRR 3 km (June 2015 - July 2019)
- GDAS half-degree archive (Sep 2007 - June 2019)
- NGM (Jan. 1991 - Apr. 1997)
- NAM Data Assimilation System (EDAS 40km, 2004 - 2018)
- NAM(Eta) Data Assimilation System (EDAS 80km, 1997-2004)
- North American Regional Reanalysis (NARR, 1979 - 2019)
All of theses datasets
contain basic fields such as the u- and v-wind components, temperature,
and humidity. However, the archives differ from each other because of
the horizontal and vertical resolution, as well as in the specific
fields provided by NCEP. All fields were selected by ARL
according to what is most relevant for transport and dispersion
studies and disk space limitations.
Data Packing Format
NCEP typically saves their model output in GRIB format.
However, at ARL the data are reprocessed and stored in a 1-byte
packing algorithm. This 1-byte packing
is a bit more compact than GRIB and can be directly used on a variety of
computing platforms with direct access I/O.
The data array is packed and stored into one byte characters.
To preserve as much data precision as possible the difference
between the values at grid points is saved and packed rather than
the actual values. The grid is then reconstructed by adding the
differences between grid values starting with the first value,
which is stored in unpacked ASCII form in the header record. To
illustrate the process, assume that a grid of real data, R, of
dimensions i,j is given by the below example.
1,j 2,j .... i-1,j i,j
1,j-1 2,j-1 .... i-1,j-1 i,j-1
.... .... .... .... ....
1,2 2,2 .... i-1,2 i,2
1,1 2,1 .... i-1,1 i,1
The packed value, P, is then given by
Pi,j = (Ri,j - Ri-1,j)* (2**(7-N)),
where the scaling exponent
N = ln dRmax / ln 2 .
The value of dRmax is the maximum difference between any two
adjacent grid points for the entire array. It is computed from the
differences along each i index holding j constant. The difference
at index (1,j) is computed from index (1,j-1), and at 1,1 the
difference is always zero. The packed values are one byte unsigned
integers, where values from 0 to 126 represent
-127 to -1, 127 represents zero, and values of 128 to 254 represent
1 to 127. Each record length is then equal in bytes to the number
of array elements plus 50 bytes for the header label information. The
50 byte label field precedes each packed data field and contains the
following ASCII data:
Field Format Description
Year I2 Greenwich date for which data valid
Month I2 "
Day I2 "
Hour I2 "
Forecast* I2 Hours forecast, zero for analysis
Level I2 Level from the surface up (see Table 3)
Grid I2 Grid identification (see Table 1)
Variable A4 Variable label (see Table 2)
Exponent I4 Scaling exponent needed for unpacking
Precision E14.7 Precision of unpacked data
Value 1,1 E14.7 Unpacked data value at grid point 1,1
*Forecast hour is -1 for missing data.
The following Fortran90 program can be used to to unpack and read the first few elements of the
data array for each record of an ARL packed meteorological file.
CHK_DATA.F