VirES Python Client Features
Contents
VirES Python Client Features¶
Abstract: The VirES Python Client provides helpful functions to discover and access Aeolus data.
# Display important package versions used
%load_ext watermark
%watermark -i -v -p viresclient,pandas,xarray,matplotlib
Python implementation: CPython
Python version : 3.9.7
IPython version : 8.0.1
viresclient: 0.11.0
pandas : 1.4.1
xarray : 0.21.1
matplotlib : 3.5.1
The basic configuration and use of the viresclient
is presented in the previous section.
This tutorial provides further insights in the capabilities of the client and describes helpful functions to work with Aeolus data.
Run this on the VRE (Virtual Research Environment), where viresclient is already installed, or check the instructions to set it up on your own Python environment.
For more information see:
https://aeolus.services/ (Web interface)
https://viresclient.readthedocs.io (Python interface)
Discover available data¶
Each collection (i.e. product type) has a large number of available parameters. It is possible to show all collections and their parameters provided by the VirES interface by using the available_collections
and print_available_collections
methods of the AeolusRequest
class.
The available_collections
method will return a dictionary object containing all collections. For each collection there are different field types. Each field type then has a list of parameters of that field type.
The print_available_collections
is a helper function specifically tailored for notebooks to print out the parameter list with additional information description and help find what you are looking for.
As there are many hundreds of parameters the this methods allows to pass following optional arguments to reduce the number of results:
collection: Specify the collection you are interested in (ALD_U_N_1B, ALD_U_N_2A, ALD_U_N_2B, ALD_U_N_2C, AUX_MRC_1B, AUX_RRC_1B, AUX_ISR_1B, AUX_ZWC_1B, AUX_MET_12)
field_type: Specify the field type you are interested in (observation_fields, measurement_fields, group_fields, sca_fields, ica_fields, mca_fields, mie_grouping_fields, rayleigh_grouping_fields, mie_profile_fields, rayleigh_profile_fields, mie_wind_fields, rayleigh_wind_fields)
regex: Allows providing a regular expression string to filter the output based on parameter identifier. It is possible to only use a regular string that will be matched if the identifier contains it (see also examples below)
details: (True or False) Determines if additional details such be shown in the printed result table
path: (True or False) Determines if the CODA path information should be shown in the printed result table
Let’s look at some examples to better illustrate how the method can be used.
# We import the AeolusRequest class from the viresclient
from viresclient import AeolusRequest
# We create a new AeolusRequest instance
request = AeolusRequest()
# Let us say we are looking for parameters related to temperature inside the
# collection ALD_U_N_2B for that we could use following call.
# We can use the regex argument to match all identifiers containing `temp`
request.print_available_collections(collection="ALD_U_N_2B", regex="temp")
# The results are printed below
uom | description | ||
---|---|---|---|
field type | identifier | ||
mie_wind_fields | mie_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity |
rayleigh_wind_fields | rayleigh_wind_result_wind_to_temperature | cm/s/K | First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume |
rayleigh_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity | |
rayleigh_wind_result_reference_temperature | 10⁻²K | Reference temperature used for inverting the Rayleigh response into an HLOS wind |
# If we wanted to see the original CODA path for these results we
# can just pass the path argument as True
request.print_available_collections(collection="ALD_U_N_2B", regex="temp", path=True)
uom | description | path | ||
---|---|---|---|---|
field type | identifier | |||
mie_wind_fields | mie_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity | ['/mie_hloswind', -1, 'windresult/applied_m1_temperature_corr_velocity'] |
rayleigh_wind_fields | rayleigh_wind_result_wind_to_temperature | cm/s/K | First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume | ['/rayleigh_hloswind', -1, 'windresult/rayleigh_wind_to_temperature'] |
rayleigh_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity | ['/rayleigh_hloswind', -1, 'windresult/applied_m1_temperature_corr_velocity'] | |
rayleigh_wind_result_reference_temperature | 10⁻²K | Reference temperature used for inverting the Rayleigh response into an HLOS wind | ['/rayleigh_hloswind', -1, 'windresult/reference_temperature'] |
# If we wanted to find all parameters that might be related to temperature
# from all collections we could remove the collection argument as follows
request.print_available_collections(regex="temp")
uom | description | |||
---|---|---|---|---|
collection | field type | identifier | ||
ALD_U_N_1A | observation_fields | mie_accd_temp | degC | Average of 10 Mie ACCD die temperatures |
ray_accd_temp | degC | Average of 10 Rayleigh ACCD die temoeratures | ||
deu_temp | degC | Average of DEU temperature | ||
m1_temp | degC | Average of Aht 22, Aht 23, Aht 24, Aht 25, Aht 26 and Aht 27 M1 temperatures | ||
m1_tc_temp | degC | Average of TC 18, TC 19, TC 20, and TC 21 M1 TC temperatures | ||
struts_temp_pxpy | degC | Average of Struts temperature +X+Y | ||
struts_temp_mxpy | degC | Average of Struts temperature -X+Y | ||
struts_temp_mpy | degC | Average of Struts temperature -Y | ||
m2_tc_temp | degC | Average of M2 TC temperature | ||
ALD_U_N_2B | mie_wind_fields | mie_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity |
rayleigh_wind_fields | rayleigh_wind_result_wind_to_temperature | cm/s/K | First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume | |
rayleigh_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity | ||
rayleigh_wind_result_reference_temperature | 10⁻²K | Reference temperature used for inverting the Rayleigh response into an HLOS wind | ||
ALD_U_N_2C | mie_wind_fields | mie_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity |
rayleigh_wind_fields | rayleigh_wind_result_wind_to_temperature | cm/s/K | First-order derivative of the HLOS wind with respect to the temperature inside the sensing volume | |
rayleigh_wind_result_applied_m1_temperature_corr_velocity | cm/s | Applied M1 mirror temperature correction to the HLOS Wind velocity | ||
rayleigh_wind_result_reference_temperature | 10⁻²K | Reference temperature used for inverting the Rayleigh response into an HLOS wind | ||
AUX_ISR_1B | fields | rayleigh_spectrometer_temperature_9 | °C | Sensor RSP etalon temperature |
rayleigh_spectrometer_temperature_10 | °C | Sensor RSP etalon temperature | ||
rayleigh_spectrometer_temperature_11 | °C | Sensor RSP etalon temperature | ||
rayleigh_thermal_hood_temperature_1 | °C | Sensor RSPT average temperature | ||
rayleigh_thermal_hood_temperature_2 | °C | Sensor RSPT average temperature | ||
rayleigh_thermal_hood_temperature_3 | °C | Sensor RSPT average temperature | ||
rayleigh_thermal_hood_temperature_4 | °C | Sensor RSPT average temperature | ||
rayleigh_optical_baseplate_avg_temperature | °C | Sensor Optical Baseplate Rayleigh average temperature | ||
AUX_MET_12 | fields | layer_temperature_off_nadir | 10⁻² K | Layer temperature off nadir |
layer_temperature_nadir | 10⁻² K | Layer temperature nadir |
Options when fetching data¶
As shortly presented in the previous tutorial it is possible to retrieve Aeolus data using AeolusRequest.get_between()
method. There are some important options to configure the request to retrieve exactly what you are looking for. The required configuration methods are set_collection
to define the data product you want to retrieve, as well as the set_fields
method (further described in the Fields section below).
Bounding Box¶
This option is optional but listed as first as it can help greatly improve data access performance if you want to investigate a specific area. By using the set_bbox
function you can define your area of interest for your request. This will allow the server to filter out complete products which are not intersecting with your selected area. For requests with longer time periods this will greatly increase data acc to performance.
The bounding box is expected as a dictionary containing EPSG 4326 longitude (-180 to 180) and latitude (-90 to 90) values for:
w: west (min longitude),
s: south (min latitude),
e: east (max longitude),
n: north (max latitude)
# Set up connection with server
request = AeolusRequest()
# Set collection to use
# Set collection to use
request.set_collection('ALD_U_N_2B')
# Define the area we are interested in, e.g. area covering roughly around Germany
my_bbox = {
"w": 0,
"s":45,
"e":20,
"n": 60,
}
request.set_bbox(my_bbox)
# Set the fields we are interested in (more information on this further below)
request.set_fields(rayleigh_wind_fields=[
"rayleigh_wind_result_start_time",
"rayleigh_wind_result_stop_time",
"rayleigh_wind_result_bottom_altitude",
"rayleigh_wind_result_top_altitude",
"rayleigh_wind_result_wind_velocity",
])
# Send the request to retrieve the data
data = request.get_between(
start_time="2020-04-10T06:21:58Z",
end_time="2020-04-12T07:50:33Z",
filetype="nc",
)
Parameters and field_types¶
Parameters are classified in different field_types and each collection has multiple field_types. You can request parameters only for specific field_types as there are non unique parameter identifiers. For example multiple L1B parameters are available as measurement and as observation (field types), in order to specify which parameter you want to retrieve you have to specify from which field_type you would like to retrieve it.
How to find which parameter is available in which field_type is described in section Discover available data above.
In order to set the parameters you want to retrieve you have to use the set_fields
method, and passing the field_type identifiers with an array of parameter identifiers strings as arguments.
As a summary here is a list of the available field_types per collection:
ALD_U_N_1B
observation_fields
measurement_fields
ALD_U_N_2A
observation_fields
measurement_fields
group_fields
sca_fields
ica_fields
mca_fields
ALD_U_N_2B and ALD_U_N_2C
mie_grouping_fields
rayleigh_grouping_fields
mie_profile_fields
rayleigh_profile_fields
mie_wind_fields
rayleigh_wind_fields
measurement_fields
AUX_MRC_1B, AUX_ISR_1B, AUX_ZWC_1B, AUX_MET_12
fields - collection contain only one (non specific) field type
As we have seen some examples already in previous tutorials, here are some helper classes to show how to retrieve a larger set of the parameters using the available_collections
presented before.
# Set up connection with server
request = AeolusRequest()
# Set collection to use
DATA_PRODUCT = 'ALD_U_N_2B'
request.set_collection(DATA_PRODUCT)
# Get all available field_types for this collection
collection_description = request.available_collections()['collections'][DATA_PRODUCT]
# Create dictionary to hold field_type key and parameter identifier array as value
request_fields = {}
for field_type in collection_description:
# save an array of the parameter keys for each field_type
request_fields[field_type] = collection_description[field_type].keys()
# Unpack the dictionary to pass each field_type as separate argument
# This will define to request all available parameters that the service
# has to offer for this collection
request.set_fields(**request_fields)
# Or you could select to retrieve all parameters of a specific field_type for
# example with request.set_fields(mie_wind_fields=request_fields["mie_wind_fields"])
# now we can request the data
return_data = request.get_between(
start_time="2020-04-10T06:21:58Z",
end_time="2020-04-10T06:22:33Z",
filetype="nc"
)
# If you like to work with xarray you can use the as_xarray_dict method to
# convert the response to a dictionary containing an xarray object per
# field_type requested.
# Further information on this is described in the next
# tutorial "viresclient data management"
xarray_dict = return_data.as_xarray_dict()
print(xarray_dict.keys())
dict_keys(['mie_grouping_data', 'rayleigh_grouping_data', 'mie_profile_data', 'rayleigh_profile_data', 'mie_wind_data', 'rayleigh_wind_data', 'measurement_data'])
Filters¶
When requesting the data, apart from the time (get_between
) and area selection (set_bbox
) you can define range filters to apply for each parameter if you are interested in retrieving data even with a greater granularity.
You can apply filter by using the set_range_filter
function.
The set_range_filter
function takes 3 arguments (parameter_id, range_min, range_max)
:
parameter_id: Parameter identifier string to which the filter should be applied
range_min: Minimum value of the filter range
range_max: Maximum value of the filter range
# Data request for AUX_MET product
request = AeolusRequest()
request.set_collection("AUX_MET_12")
# set fields
request.set_fields(fields=[
"time_off_nadir",
"latitude_off_nadir",
"longitude_off_nadir",
"surface_altitude_off_nadir",
"layer_altitude_off_nadir",
"layer_temperature_off_nadir",
"layer_cloud_cover_off_nadir",
"layer_wind_component_u_off_nadir",
"layer_wind_component_v_off_nadir",
"layer_rel_humidity_off_nadir",
"layer_validity_flag_off_nadir",
])
# Now we set filters for the layer altitude parameters
request.set_range_filter("layer_altitude_off_nadir", 0, 2500000)
# set start and end time and request data
data_aux_met = request.get_between(
start_time="2021-04-10T06:21:00Z",
end_time="2021-04-10T06:21:30Z",
filetype="nc"
)
# For collections or requests that don't contain multiple field_types
# you can convert the complete response object to an xarray
xarray_data = data_aux_met.as_xarray()
xarray_data.keys()
KeysView(<xarray.Dataset>
Dimensions: (off_nadir: 10, array_137: 137)
Dimensions without coordinates: off_nadir, array_137
Data variables:
time_off_nadir (off_nadir) float64 ...
latitude_off_nadir (off_nadir) float64 ...
longitude_off_nadir (off_nadir) float64 ...
surface_altitude_off_nadir (off_nadir) int32 ...
layer_altitude_off_nadir (off_nadir, array_137) int32 ...
layer_temperature_off_nadir (off_nadir, array_137) float64 ...
layer_cloud_cover_off_nadir (off_nadir, array_137) uint8 ...
layer_wind_component_u_off_nadir (off_nadir, array_137) int16 ...
layer_wind_component_v_off_nadir (off_nadir, array_137) int16 ...
layer_rel_humidity_off_nadir (off_nadir, array_137) uint8 ...
layer_validity_flag_off_nadir (off_nadir, array_137) int8 ...
Attributes:
history: {"inputFiles": ["AE_OPER_AUX_MET_12_20210409T030000_20210410T09...
Sources: [('AE_OPER_AUX_MET_12_20210409T030000_20210410T090000_0001', 'M...)