Contents

VirES Python Client Features

VirES Python Client Features

Abstract: The VirES Python Client provides helpful functions to discover and access Aeolus data.

# Display important package versions used
%load_ext watermark
%watermark -i -v -p viresclient,pandas,xarray,matplotlib

Python implementation: CPython
Python version       : 3.9.7
IPython version      : 8.0.1

viresclient: 0.11.0
pandas     : 1.4.1
xarray     : 0.21.1
matplotlib : 3.5.1

The basic configuration and use of the viresclient is presented in the previous section. This tutorial provides further insights in the capabilities of the client and describes helpful functions to work with Aeolus data.

Run this on the VRE (Virtual Research Environment), where viresclient is already installed, or check the instructions to set it up on your own Python environment.

For more information see:

Discover available data

Each collection (i.e. product type) has a large number of available parameters. It is possible to show all collections and their parameters provided by the VirES interface by using the available_collections and print_available_collections methods of the AeolusRequest class.

The available_collections method will return a dictionary object containing all collections. For each collection there are different field types. Each field type then has a list of parameters of that field type.

The print_available_collections is a helper function specifically tailored for notebooks to print out the parameter list with additional information description and help find what you are looking for. As there are many hundreds of parameters the this methods allows to pass following optional arguments to reduce the number of results:

  • collection: Specify the collection you are interested in (ALD_U_N_1B, ALD_U_N_2A, ALD_U_N_2B, ALD_U_N_2C, AUX_MRC_1B, AUX_RRC_1B, AUX_ISR_1B, AUX_ZWC_1B, AUX_MET_12)

  • field_type: Specify the field type you are interested in (observation_fields, measurement_fields, group_fields, sca_fields, ica_fields, mca_fields, mie_grouping_fields, rayleigh_grouping_fields, mie_profile_fields, rayleigh_profile_fields, mie_wind_fields, rayleigh_wind_fields)

  • regex: Allows providing a regular expression string to filter the output based on parameter identifier. It is possible to only use a regular string that will be matched if the identifier contains it (see also examples below)

  • details: (True or False) Determines if additional details such be shown in the printed result table

  • path: (True or False) Determines if the CODA path information should be shown in the printed result table

Let’s look at some examples to better illustrate how the method can be used.

# We import the AeolusRequest class from the viresclient
from viresclient import AeolusRequest
# We create a new AeolusRequest instance
request = AeolusRequest()

# Let us say we are looking for parameters related to temperature inside the 
# collection ALD_U_N_2B for that we could use following call.
# We can use the regex argument to match all identifiers containing `temp`
request.print_available_collections(collection="ALD_U_N_2B", regex="temp")
# The results are printed below
uom description
field type identifier
mie_wind_fields mie_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_fields rayleigh_wind_result_wind_to_temperature cm/s/K First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume
rayleigh_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_result_reference_temperature 10⁻²K Reference temperature used for inverting the Rayleigh response into an HLOS wind 
# If we wanted to see the original CODA path for these results we
# can just pass the path argument as True
request.print_available_collections(collection="ALD_U_N_2B", regex="temp", path=True)
uom description path
field type identifier
mie_wind_fields mie_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity ['/mie_hloswind', -1, 'windresult/applied_m1_temperature_corr_velocity']
rayleigh_wind_fields rayleigh_wind_result_wind_to_temperature cm/s/K First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume ['/rayleigh_hloswind', -1, 'windresult/rayleigh_wind_to_temperature']
rayleigh_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity ['/rayleigh_hloswind', -1, 'windresult/applied_m1_temperature_corr_velocity']
rayleigh_wind_result_reference_temperature 10⁻²K Reference temperature used for inverting the Rayleigh response into an HLOS wind ['/rayleigh_hloswind', -1, 'windresult/reference_temperature'] 
# If we wanted to find all parameters that might be related to temperature
# from all collections we could remove the collection argument as follows
request.print_available_collections(regex="temp")
uom description
collection field type identifier
ALD_U_N_1A observation_fields mie_accd_temp degC Average of 10 Mie ACCD die temperatures
ray_accd_temp degC Average of 10 Rayleigh ACCD die temoeratures
deu_temp degC Average of DEU temperature
m1_temp degC Average of Aht 22, Aht 23, Aht 24, Aht 25, Aht 26 and Aht 27 M1 temperatures
m1_tc_temp degC Average of TC 18, TC 19, TC 20, and TC 21 M1 TC temperatures
struts_temp_pxpy degC Average of Struts temperature +X+Y
struts_temp_mxpy degC Average of Struts temperature -X+Y
struts_temp_mpy degC Average of Struts temperature -Y
m2_tc_temp degC Average of M2 TC temperature
ALD_U_N_2B mie_wind_fields mie_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_fields rayleigh_wind_result_wind_to_temperature cm/s/K First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume
rayleigh_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_result_reference_temperature 10⁻²K Reference temperature used for inverting the Rayleigh response into an HLOS wind
ALD_U_N_2C mie_wind_fields mie_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_fields rayleigh_wind_result_wind_to_temperature cm/s/K First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume
rayleigh_wind_result_applied_m1_temperature_corr_velocity cm/s Applied M1 mirror temperature correction to the HLOS Wind velocity
rayleigh_wind_result_reference_temperature 10⁻²K Reference temperature used for inverting the Rayleigh response into an HLOS wind
AUX_ISR_1B fields rayleigh_spectrometer_temperature_9 °C Sensor RSP etalon temperature
rayleigh_spectrometer_temperature_10 °C Sensor RSP etalon temperature
rayleigh_spectrometer_temperature_11 °C Sensor RSP etalon temperature
rayleigh_thermal_hood_temperature_1 °C Sensor RSPT average temperature
rayleigh_thermal_hood_temperature_2 °C Sensor RSPT average temperature
rayleigh_thermal_hood_temperature_3 °C Sensor RSPT average temperature
rayleigh_thermal_hood_temperature_4 °C Sensor RSPT average temperature
rayleigh_optical_baseplate_avg_temperature °C Sensor Optical Baseplate Rayleigh average temperature
AUX_MET_12 fields layer_temperature_off_nadir 10⁻² K Layer temperature off nadir
layer_temperature_nadir 10⁻² K Layer temperature nadir

Options when fetching data

As shortly presented in the previous tutorial it is possible to retrieve Aeolus data using AeolusRequest.get_between() method. There are some important options to configure the request to retrieve exactly what you are looking for. The required configuration methods are set_collection to define the data product you want to retrieve, as well as the set_fields method (further described in the Fields section below).

Bounding Box

This option is optional but listed as first as it can help greatly improve data access performance if you want to investigate a specific area. By using the set_bbox function you can define your area of interest for your request. This will allow the server to filter out complete products which are not intersecting with your selected area. For requests with longer time periods this will greatly increase data acc to performance. The bounding box is expected as a dictionary containing EPSG 4326 longitude (-180 to 180) and latitude (-90 to 90) values for:

  • w: west (min longitude),

  • s: south (min latitude),

  • e: east (max longitude),

  • n: north (max latitude)

# Set up connection with server
request = AeolusRequest()
# Set collection to use
# Set collection to use
request.set_collection('ALD_U_N_2B')

# Define the area we are interested in, e.g. area covering roughly around Germany
my_bbox = {
    "w": 0,
    "s":45,
    "e":20,
    "n": 60,
}
request.set_bbox(my_bbox)

# Set the fields we are interested in (more information on this further below)
request.set_fields(rayleigh_wind_fields=[
    "rayleigh_wind_result_start_time",
    "rayleigh_wind_result_stop_time",
    "rayleigh_wind_result_bottom_altitude",
    "rayleigh_wind_result_top_altitude",
    "rayleigh_wind_result_wind_velocity",
])

# Send the request to retrieve the data
data = request.get_between(
    start_time="2020-04-10T06:21:58Z",
    end_time="2020-04-12T07:50:33Z",
    filetype="nc",
)

Parameters and field_types

Parameters are classified in different field_types and each collection has multiple field_types. You can request parameters only for specific field_types as there are non unique parameter identifiers. For example multiple L1B parameters are available as measurement and as observation (field types), in order to specify which parameter you want to retrieve you have to specify from which field_type you would like to retrieve it. How to find which parameter is available in which field_type is described in section Discover available data above. In order to set the parameters you want to retrieve you have to use the set_fields method, and passing the field_type identifiers with an array of parameter identifiers strings as arguments. As a summary here is a list of the available field_types per collection:

  • ALD_U_N_1B

    • observation_fields

    • measurement_fields

  • ALD_U_N_2A

    • observation_fields

    • measurement_fields

    • group_fields

    • sca_fields

    • ica_fields

    • mca_fields

  • ALD_U_N_2B and ALD_U_N_2C

    • mie_grouping_fields

    • rayleigh_grouping_fields

    • mie_profile_fields

    • rayleigh_profile_fields

    • mie_wind_fields

    • rayleigh_wind_fields

    • measurement_fields

  • AUX_MRC_1B, AUX_ISR_1B, AUX_ZWC_1B, AUX_MET_12

    • fields - collection contain only one (non specific) field type

As we have seen some examples already in previous tutorials, here are some helper classes to show how to retrieve a larger set of the parameters using the available_collections presented before.

# Set up connection with server
request = AeolusRequest()
# Set collection to use
DATA_PRODUCT = 'ALD_U_N_2B'
request.set_collection(DATA_PRODUCT)

# Get all available field_types for this collection
collection_description = request.available_collections()['collections'][DATA_PRODUCT]

# Create dictionary to hold field_type key and parameter identifier array as value
request_fields = {}
for field_type in collection_description:
    # save an array of the parameter keys for each field_type
    request_fields[field_type] = collection_description[field_type].keys()

# Unpack the dictionary to pass each field_type as separate argument
# This will define to request all available parameters that the service
# has to offer for this collection
request.set_fields(**request_fields)

# Or you could select to retrieve all parameters of a specific field_type for 
# example with request.set_fields(mie_wind_fields=request_fields["mie_wind_fields"])

# now we can request the data
return_data = request.get_between(
    start_time="2020-04-10T06:21:58Z",
    end_time="2020-04-10T06:22:33Z",
    filetype="nc"
)

# If you like to work with xarray you can use the as_xarray_dict method to
# convert the response to a dictionary containing an xarray object per 
# field_type requested.
# Further information on this is described in the next 
# tutorial "viresclient data management"
xarray_dict = return_data.as_xarray_dict()
print(xarray_dict.keys())

dict_keys(['mie_grouping_data', 'rayleigh_grouping_data', 'mie_profile_data', 'rayleigh_profile_data', 'mie_wind_data', 'rayleigh_wind_data', 'measurement_data'])

Filters

When requesting the data, apart from the time (get_between) and area selection (set_bbox) you can define range filters to apply for each parameter if you are interested in retrieving data even with a greater granularity. You can apply filter by using the set_range_filter function. The set_range_filter function takes 3 arguments (parameter_id, range_min, range_max):

  • parameter_id: Parameter identifier string to which the filter should be applied

  • range_min: Minimum value of the filter range

  • range_max: Maximum value of the filter range

# Data request for AUX_MET product
request = AeolusRequest()

request.set_collection("AUX_MET_12")

# set fields
request.set_fields(fields=[
    "time_off_nadir",
    "latitude_off_nadir",
    "longitude_off_nadir",
    "surface_altitude_off_nadir",
    "layer_altitude_off_nadir",
    "layer_temperature_off_nadir",
    "layer_cloud_cover_off_nadir",
    "layer_wind_component_u_off_nadir",
    "layer_wind_component_v_off_nadir",
    "layer_rel_humidity_off_nadir",
    "layer_validity_flag_off_nadir",
])

# Now we set filters for the layer altitude parameters
request.set_range_filter("layer_altitude_off_nadir", 0, 2500000)

# set start and end time and request data
data_aux_met = request.get_between(
    start_time="2021-04-10T06:21:00Z",
    end_time="2021-04-10T06:21:30Z",
    filetype="nc"
)
# For collections or requests that don't contain multiple field_types
# you can convert the complete response object to an xarray
xarray_data = data_aux_met.as_xarray()
xarray_data.keys()

KeysView(<xarray.Dataset>
Dimensions:                           (off_nadir: 10, array_137: 137)
Dimensions without coordinates: off_nadir, array_137
Data variables:
    time_off_nadir                    (off_nadir) float64 ...
    latitude_off_nadir                (off_nadir) float64 ...
    longitude_off_nadir               (off_nadir) float64 ...
    surface_altitude_off_nadir        (off_nadir) int32 ...
    layer_altitude_off_nadir          (off_nadir, array_137) int32 ...
    layer_temperature_off_nadir       (off_nadir, array_137) float64 ...
    layer_cloud_cover_off_nadir       (off_nadir, array_137) uint8 ...
    layer_wind_component_u_off_nadir  (off_nadir, array_137) int16 ...
    layer_wind_component_v_off_nadir  (off_nadir, array_137) int16 ...
    layer_rel_humidity_off_nadir      (off_nadir, array_137) uint8 ...
    layer_validity_flag_off_nadir     (off_nadir, array_137) int8 ...
Attributes:
    history:  {"inputFiles": ["AE_OPER_AUX_MET_12_20210409T030000_20210410T09...
    Sources:  [('AE_OPER_AUX_MET_12_20210409T030000_20210410T090000_0001', 'M...)

previous

Aeolus access through VirES

next

VirES Python Client Data Handling