fetch

Download routines for NASA Earthdata files

General Methods

IceAdvect.datasets.fetch.s3_client(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]

Get AWS s3 client for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials

Parameters:

HOST: str: NSIDC AWS S3 credential host
timeout: int or NoneType, default None: timeout in seconds for blocking operations
region_name: str, default ‘us-west-2’: AWS region name

Returns:

client: obj: AWS s3 client for NSIDC Cumulus

IceAdvect.datasets.fetch.s3_filesystem(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]

Get AWS s3 file system object for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials

Parameters:

HOST: str: NSIDC AWS S3 credential host
timeout: int or NoneType, default None: timeout in seconds for blocking operations
region_name: str, default ‘us-west-2’: AWS region name

Returns:

session: obj: AWS s3 file system session for NSIDC Cumulus

IceAdvect.datasets.fetch.s3_bucket(presigned_url: str)[source]

Get a s3 bucket name from a presigned url

Parameters:

presigned_url: str: s3 presigned url

Returns:

bucket: str: s3 bucket name

IceAdvect.datasets.fetch.s3_key(presigned_url: str)[source]

Get a s3 bucket key from a presigned url

Parameters:

presigned_url: str: s3 presigned url or https url

Returns:

key: str: s3 bucket key for object

IceAdvect.datasets.fetch.s3_presigned_url(bucket: str, key: str)[source]

Get a s3 presigned url from a bucket and object key

Parameters:

bucket: str: s3 bucket name
key: str: s3 bucket key for object

Returns:

presigned_url: str: s3 presigned url

IceAdvect.datasets.fetch.generate_presigned_url(bucket: str, key: str, expiration: int = 3600)[source]

Generate a presigned https URL to share an S3 object

Parameters:

bucket: str: s3 bucket name
key: str: s3 bucket key for object
expiration: int: Time in seconds for the presigned URL to remain valid

Returns:

presigned_url: str: s3 presigned https url

IceAdvect.datasets.fetch.attempt_login(urs: str = 'urs.earthdata.nasa.gov', context: ~ssl.SSLContext = <ssl.SSLContext object>, password_manager: bool = False, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = True, **kwargs)[source]

attempt to build a urllib opener for NASA Earthdata

Parameters:

urs: str, default urs.earthdata.nasa.gov: Earthdata login URS 3 host
context: obj, default IceAdvect.utilities._default_ssl_context: SSL context for urllib opener object
password_manager: bool, default True: Create password manager context using default realm
get_ca_certs: bool, default False: Get list of loaded “certification authority” certificates
redirect: bool, default False: Create redirect handler object
authorization_header: bool, default False: Add base64 encoded authorization header to opener
username: str, default from environmental variable: NASA Earthdata username
password: str, default from environmental variable: NASA Earthdata password
retries: int, default 5: number of retry attempts
netrc: str, default ~/.netrc: path to .netrc file for authentication

Returns:

opener: obj: OpenerDirector instance

IceAdvect.datasets.fetch.build_opener(username: str, password: str, context: ~ssl.SSLContext = <ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = False, urs: str = 'https://urs.earthdata.nasa.gov')[source]

Build urllib opener for NASA Earthdata with supplied credentials

Parameters:

username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
context: obj, default IceAdvect.utilities._default_ssl_context: SSL context for urllib opener object
password_manager: bool, default True: Create password manager context using default realm
get_ca_certs: bool, default False: Get list of loaded “certification authority” certificates
redirect: bool, default False: Create redirect handler object
authorization_header: bool, default False: Add base64 encoded authorization header to opener
urs: str, default ‘https://urs.earthdata.nasa.gov’: Earthdata login URS 3 host

Returns:

opener: obj: OpenerDirector instance

IceAdvect.datasets.fetch.get_token(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/token', build: bool = True, urs: str = 'urs.earthdata.nasa.gov', **kwargs)[source]

Generate a NASA Earthdata User Token

Parameters:

HOST: str or list: NASA Earthdata token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default Nonedata’: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

Returns:

token: dict: JSON response with NASA Earthdata User Token

IceAdvect.datasets.fetch.list_tokens(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/tokens', build: bool = True, urs: str = 'urs.earthdata.nasa.gov', **kwargs)[source]

List the current associated NASA Earthdata User Tokens

Parameters:

HOST: str: NASA Earthdata list token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

Returns:

tokens: list: JSON response with NASA Earthdata User Tokens

IceAdvect.datasets.fetch.revoke_token(token: str, HOST: str = 'https://urs.earthdata.nasa.gov/api/users/revoke_token', build: bool = True, urs: str = 'urs.earthdata.nasa.gov', **kwargs)[source]

Generate a NASA Earthdata User Token

Parameters:

token: str: NASA Earthdata token to be revoked
HOST: str: NASA Earthdata revoke token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

IceAdvect.datasets.fetch.check_credentials()[source]: Check that entered NASA Earthdata credentials are valid

IceAdvect.datasets.fetch.from_earthdata(HOST: str | list, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', local: str | Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, mode: oct = 509, **kwargs)[source]

Download a file from a NASA Earthdata provider

Parameters:

HOST: str or list: remote https host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host
local: str or NoneType, default None: path to local file
hash: str, default ‘’: MD5 hash of local file
chunk: int, default 16384: chunk size for transfer encoding
verbose: bool, default False: print file transfer information
mode: oct, default 0o775: permissions mode of output local file

Returns:

remote_buffer: obj: BytesIO representation of file
response_error: str or None: notification for response error

IceAdvect.datasets.fetch.cmr_filter_json(search_results: dict, endpoint: str = 'data')[source]

Filter the CMR json response for desired data files

Parameters:

search_results: dict

json response from CMR query

endpoint: str, default ‘data’

url endpoint type

'data': NASA Earthdata https archive

'opendap': NASA Earthdata OPeNDAP archive

's3': NASA Earthdata Cumulus AWS S3 bucket

Returns:

granule_urls: list: granule urls from NSIDC

IceAdvect.datasets.fetch.cmr(collection_concept_id: str, producer_granule_id: str | None = None, readable_granule_name: list | None = None, provider: str = 'NSIDC_CPRD', endpoint: str = 'data', opener=None, context: ~ssl.SSLContext = <ssl.SSLContext object>, verbose: bool = False, **kwargs)[source]

Query the NASA Common Metadata Repository (CMR)

Parameters:

collection_concept_id: str

Earthdata Collection ID of the data product

producer_granule_id: str or NoneType, default None

CMR producer granule id

readable_granule_name: list or NoneType, default None

list of CMR readable granule names

provider: str, default ‘NSIDC_CPRD’

CMR data provider

endpoint: str, default ‘data’

url endpoint type

'data': NASA Earthdata https archive

'opendap': NASA Earthdata OPeNDAP archive

's3': NASA Earthdata Cumulus AWS S3 bucket

opener: obj or NoneType, default None

OpenerDirector instance

context: obj, default IceAdvect.utilities._default_ssl_context

SSL context for urllib opener object

verbose: bool, default False

print file transfer information

Returns:

granule_urls: list: granule urls

IceAdvect.datasets.fetch.fetch(path: Path = PosixPath('/home/docs/.cache/advect'), **kwargs)[source]

Query resources from the NASA Common Metadata Repository (CMR) and download them to a local path

Parameters:

path: str or pathlib.Path, default IceAdvect.utilities.get_cache_path(): local path to download resources
kwargs: dict: keyword arguments for cmr

Returns:

granules: list: local paths for queried resources