Utilities

The utils/ module of MASER4PY provides utilities to deal with data in MASER.

The cdf module

The cdf module contains the following tools:

  • cdf, a backup of the spacepy.pycdf module (https://pythonhosted.org/SpacePy/pycdf.html), only used in the case where spacepy package is not installed in the system.

  • cdfcompare, a tool to compare two CDFs

  • serializer, to convert skeleton CDF files (in Excel or ASCII format) into master CDF binary files

  • validator, to validate the content of a CDF file from a given data model.

For more information about the CDF format, please visit http://cdf.gsfc.nasa.gov/.

cdf.cdfcompare

The cdf.cdfcompare module can be used to compare the content of two CDFs.

To get the full list of input parameters, print the help message from command line:

maser cdf_compare --help

cdf.serializer

The cdf.serializer module allows users to convert CDF between the following formats:

  • Skeleton table (ASCII)

  • Binary CDF (“Master”)

  • Excel 2007 format (.xlsx)

Module can be imported in Python programs or called directly from a terminal using the dedicated command line interface.

Important

The conversions between skeleton table and binary CDF are performed calling the skeletontable and skeletoncdf programs from the NASA CDF software (visit https://cdf.gsfc.nasa.gov/ for more details).

Converting a binary CDF to a skeleton table

The skeletontable sub-command allows to convert CDF from binary CDF to skeleton table by calling the skeletontable NASA CDF software program.

An option also permits to export skeleton table into Excel 2007 format file (.xlsx).

To display the help, enter:

maser skeletontable --help
Examples

Convert all input CDFs named “input_*.cdf” into skeleton tables. Output files will be saved into the /tmp/cdf/build folder. Output files will have the same names than input CDFs but with the extension “.skt”.

maser skeletontable "input_*.cdf" --output_dir /tmp/cdf/build

Convert the input binary CDF named “input1.cdf” into skeleton table. Output files will be saved into the /tmp/cdf/build folder. Output file will have the same name than input CDF but with the extension “.skt”. An export file in Excel 2007 format (input1.xlsx) will also be saved.

maser skeletontable "input1.cdf" --output_dir /tmp/cdf/build --to-xlsx

Converting skeleton CDF to binary CDF

The skeletoncdf sub-command allows to convert CDF from skeleton table to binary CDF by calling the skeletoncdf NASA CDF software program.

An option also permits to use Excel 2007 format file (.xlsx) as an input to the skeletoncdf sub-command.

To display the help, enter:

maser skeletoncdf --help

Expected Excel file format description

This section describes the structure of the Excel format file that can be used by the cdf.serializer module.

Note that:

  • Only the Excel 2007 format is supported (i.e., .xlsx).

  • Only zVariables are supported

Warning

Make sure to respect the letter case!

The Excel file shall contain the following sheets:

  • header

  • GLOBALattributes

  • zVariables

  • VARIABLEattributes

  • NRV

The first row of each sheet must be used to provide the name of the columns.

header sheet

The “header” sheet must contain the following columns:

CDF_NAME

Name of the CDF master file (without the extension)

DATA ENCODING

Type of data encoding

MAJORITY

Majority of the CDF data parsing (“COLUMN” or “ROW”)

FORMAT

Indicates if the data are saved in a single (“SINGLE”) or on multiple (“MULTIPLE”) CDF files

CDF_COMPRESSION

Type of compression applied to the CDF

CDF_CHECKSUM

Checksum applied to the CDF

GLOBALattributes sheet

The “GLOBALattributes” sheet shall contain the following columns:

Attribute Name

Name of the global attribute

Entry Number

Index of the current entry starting at 1

Data Type

CDF data type of the global attribute (only the “CDF_CHAR” type is supported)

Value

Value of the current entry

zVariables sheet

The “zVariables” sheet shall contain the following columns:

Variable Name

Name of the zVariable

Data Type

CDF data type of the zVariable

Number Elements

Number of elements of the zVariable (shall be always 1, except for CDF_[U]CHAR” type)

Dims

Number of dimension of the zVariable (shall be 0 if the variable is a scalar)

Sizes

If the variable is not a scalar, provides its dimension sizes.

Record Variance

Indicates if the variable values can change (“T”) or not (“F”) from a record to another.

Dimension Variances

Indicates how the variable values vary over each dimension.

VAR_COMPRESSION

Compression algorithm applied to the variable.

VAR_SPARESERECORDS

Spare record of the variable.

VAR_PADVALUE

Pad value of the variable.

VARIABLEattributes sheet

The “VARIABLEattributes” sheet shall contain the following columns:

Variable Name

Name of the zVariable

Attribute Name

Name of the variable attribute

Data Type

CDF data type of the variable attribute

Value

Value of the variable attribute

NRV sheet

The “NRV” sheet shall contain the following columns:

Variable Name

Name of the zVariable

Index

Index of the current NR row

Value

Value of the current NR row

Limitations & Known Issues

The cdf.validator tool

validator provides methods to validate a CDF format file from a given model.

It contains only one Validate class that regroups all of the validation methods.

The Validate class

To import the Validate class from Python, enter:

from maser.utils.cdf.cdfvalidator import Validate
The Model validation test

The Validate class allows user to check if a given CDF format file contains specific attributes or variables, by providing a so-called “cdfvalidator model file”.

This model file shall be in the JSON format. All items and values are case sensitive. It can include the following JSON objects:

CDFValidator JSON objects

JSON object

Description

GLOBALattributes

Contains the list of global attributes to check

VARIABLEattributes

Contains the list of variable attributes to check

zVariables

Contains the list of zvariables to check

Note that any additional JSON object will be ignored.

The table below lists the JSON items that are allowed to be found in the GLOBALattributes, VARIABLEattributes and zVariables JSON objects.

CDFValidator JSON object items

JSON item

JSON type

Priority

Description

attributes

vector

optional

List of variable attributes. An element of the vector shall be a JSON object that can contain one or more of the other JSON items listed in this table

dims

integer

optional

Number of dimensions of the CDF item

entries

vector

optional

Entry value(s) of the CDF item to be found

hasvalue

boolean

optional

If it is set to true, then the current CDF item must have at least one nonzero entry value

name

string

mandatory

Name of the CDF item (attribute or variable) to check

sizes

vector

optional

Dimension sizes of the CDF item

type

attribute

optional

CDF data type of the CDF item

Command line interface

To display the help of the module, enter:

cdf_validator --help

The full calling sequence is:

maser cdf_validator [-h] [-m MODEL_FILE] [-c CDFVALIDATE_BIN] [-I] [-C] cdf_file
positional arguments:

cdf_file Path of the CDF format file to validate

optional arguments:
-h, --help

show this help message and exit

-m MODEL_FILE, --model-file MODEL_FILE

Path to the model file in JSON format

-c CDFVALIDATE_BIN, --cdfvalidate-bin CDFVALIDATE_BIN

Path of the cdfvalidate NASA CDF tool executable

-I, --istp

Check the ISTP guidelines compliance

-C, --run-cdfvalidate

Run the cdfvalidate NASA CDF tool

Examples

To test the cdf.validator program, use the dedicated scripts/test_cdfvalidator.sh bash script.

It should return something like:

INFO    : Opening /tmp/cdfconverter_example.cdf
INFO    : Loading /Users/xbonnin/Work/projects/MASER/Software/Tools/Git/maser-py/scripts/../maser/support/cdf/cdfvalidator_model_example.json
INFO    : Checking GLOBALattributes:
INFO    : --> Project
WARNING : "Project"  has a wrong entry value: "Python>Python 2" ("Python>Python 3" expected)!
INFO    : --> PI_name
INFO    : --> TEXT
INFO    : Checking VARIABLEattributes:
INFO    : --> FIELDNAM
INFO    : --> CATDESC
INFO    : --> VAR_TYPE
INFO    : Checking zVariables:
INFO    : --> Epoch
INFO    : --> Variable2
INFO    : Checking variable attributes of "Variable2":
INFO    : --> DEPEND_0
WARNING : DEPEND_0 required!
INFO    : Closing /tmp/cdfconverter_example.cdf

The time module

The leapsec tool

The leapsec tool allows users to handle the leap seconds.

Using the leapsec tool requires to read the CDFLeapSeconds.txt file. This file is available on the NASA CDF Web site (https://cdf.gsfc.nasa.gov).

Warning

Before using the leapsec tool, it is highly recommended to have the CDFLeapSeconds.txt file saved on the localdisk, and reachable from the $CDF_LEAPSECONDSTABLE env. variable. If the file is not on the disk, the tool will attempt to read the file directly from the NASA CDF Web site.

The Lstable class

The Lstable class provides the methods to deal with the CDFLeapSeconds.txt table file.

To import the Lstable class from Python, enter:

from maser.utils.time import Lstable

Then, to load the CDFLeapSeconds.txt table, first enter:

lstable = Lstable(file=path_to_the_file)

Note

Note that if the optional input keyword file= is not set, the tool will first check if the path is given in the $CDF_LEAPSECONDSTABLE environment variable. If not, then the program will look into the maser/support/data sub-folder of the package directory. Finally, if it is still not found, it will attempt to retrieve the table data from the file on the NASA CDF Web site (https://cdf.gsfc.nasa.gov/html/CDFLeapSeconds.txt)

Once the table is loaded, then to print the leap seconds table, enter:

print(lstable)

To get the total elapsed leap seconds for a given date, enter:

lstable.get_leapsec(date=date_time)

Where date_time is a datetime object of the datetime module.

Downloading the CDFLeapSeconds.txt file from the NASA Web site can be done by entering:

Lstable.get_lstable_file(target_dir=target_dir, overwrite=overwrite)

Where target_dir is the local directory where the CDFLeapSeconds.txt file will be saved. overwrite keyword can be used to replace existing file (default is overwrite=False)

Note

get_lstable_file is a staticmethod, which does not require to instanciate the Lstable class.

Note

If the method is called without the target_dir= input keyword (i.e., get_lstable()), then it will first check if the $CDF_LEAPSECONDSTABLE env. variable is defined, if yes the target_dir will be set with the $CDF_LEAPSECONDSTABLE value, otherwise the file is saved in the maser/support/data folder of the module.

Command line interface

To display the help of the module, enter:

maser leapsec --help

The full calling sequence is:

maser leapsec [-h] [-D] [-O] [-S] [-f FILEPATH] [-d DATE]

Input keywords:

-h, --help

show this help message and exit

-f FILEPATH, --filepath FILEPATH

CDFLeapSeconds.txt filepath. Default is [maser4py_rootdir]/support/data/CDFLeapSeconds.txt, where [maser4py_rootdir] is the maser4py root directory.

-d DATE, --date DATE

Return the leap seconds for a given date and time. (Expected format is “YYYY-MM-DDThh:mm:ss”)

-S, --SHOW-TABLE

Show the leap sec. table

-O, --OVERWRITE

Overwrite existing file

-D, --DOWNLOAD-FILE

Download the CDFLeapSeconds.txt from the NASA CDF site. The file will be saved in the path defined in the –filepath argument..

The time tool

The time tool offers time conversion methods between the following time systems:

  • UTC: Coordinated Universal Time

  • JD: Julian Days

  • MJD: Modified Julian Days

  • TT: Terrestrial Time

  • TAI: International Atomic Time

  • TT2000: Terrestrial Time since J2000 (2000-01-01T12:00:00)

Note

The time conversion inside the methods is performed using numpy.timedelta64 and numpy.datetime64 objects for better time resolution.

Warning

The highest time resolution of JD and MJD systems are fixed to microsecond. The TT2000 system can reach the nanosecond resolution.