Phased Operators

Scan Operations

class padocc.phases.scan.ScanOperation(proj_code: str, workdir: str, groupID: str = None, label: str = 'scan', **kwargs)

Bases: ProjectOperation

help(fn=<built-in function print>)

Public user functions for the project operator.

Compute Operations

class padocc.phases.compute.CfaDS(proj_code: str, workdir: str, groupID: str = None, stage: str = 'in_progress', thorough: bool = None, concat_msg: str = 'See individual files for more details', limiter: int = None, skip_concat: bool = False, label: str = 'compute', is_trial: bool = False, **kwargs)

Bases: ComputeOperation

class padocc.phases.compute.ComputeOperation(proj_code: str, workdir: str, groupID: str = None, stage: str = 'in_progress', thorough: bool = None, concat_msg: str = 'See individual files for more details', limiter: int = None, skip_concat: bool = False, label: str = 'compute', is_trial: bool = False, **kwargs)

Bases: ProjectOperation

PADOCC Dataset Processor Class, capable of processing a single dataset’s worth of input files into a single aggregated file/store.

property filelist

Quick function for obtaining a subset of the whole fileset. Originally used to open all the files using Xarray for concatenation later.

help(fn=<built-in function print>)

Public user functions for the project operator.

class padocc.phases.compute.KerchunkConverter(logger=None, bypass_driver=False, verbose=1, label=None, fh=None, logid=None)

Bases: LoggedOperation

Class for converting a single file to a Kerchunk reference object. Handles known or unknown file types (NetCDF3/4 versions).

run(nfile: str, filehandler=None, extension=None, **kwargs) → dict

Safe creation allows for known issues and tries multiple drivers

Returns:

dictionary of Kerchunk references if successful, raises error otherwise if unsuccessful.

class padocc.phases.compute.KerchunkDS(proj_code, workdir, stage='in_progress', **kwargs)

Bases: ComputeOperation

create_refs(check_dimensions: bool = False) → None

Organise creation and loading of refs - Load existing cached refs - Create new refs - Combine metadata and global attributes into a single set - Coordinate combining and saving of data

class padocc.phases.compute.ZarrDS(proj_code, workdir, stage='in_progress', mem_allowed: str = '100MB', preferences=None, **kwargs)

Bases: ComputeOperation

padocc.phases.compute.cfa_handler(instance, file_limit: int | None = None)

Handle the creation of a CFA-netCDF file using the CFAPyX package

Parameters:

  • instance – (obj) The reference instance of ProjectOperation from which to pull project-specific info.

  • file_limit – (obj) The file limit to apply to a set of files.

Validate Operations

class padocc.phases.validate.Report(fh=None)

Bases: object

Special report class, capable of utilising recursive dictionary value-setting.

class padocc.phases.validate.ValidateDatasets(datasets: list, id: str, filehandlers: list[padocc.core.filehandlers.JSONFileHandler] | list[dict] | None = None, dataset_labels: list = None, preslice_fns: list = None, logger=None, label: str = None, fh: str = None, logid: str = None, verbose: int = 0)

Bases: LoggedOperation

ValidateDatasets object for performing validations between two pseudo-identical Xarray Dataset objects.

4th Dec Note: Validate metadata using single NetCDF(Xarray) vs Kerchunk Validate data using combined NetCDF or CFA vs Kerchunk (for best performance)

control_dataset_var(var)

Get a variable DataArray from the control dataset, performing preslice functions.

replace_dataset(new_ds: Dataset, label: str = None, index: int = None, dstype: str = None) → None

Replace dataset by type, label or index.

replace_preslice(new_preslice: Dataset, label: str = None, index: int = None, dstype: str = None) → None

Replace dataset by type, label or index.

test_dataset_var(var)

Get a variable DataArray from the test dataset, performing preslice functions.

validate_data()

Perform data validations using the growbox method for all variable DataArrays.

validate_global_attrs(allowances: dict = None)

Validate the set of global attributes across all datasets

validate_metadata(allowances: dict = None) → dict

Run all validation steps on this set of datasets.

class padocc.phases.validate.ValidateOperation(*args, **kwargs)

Bases: ProjectOperation

Encapsulate all validation testing into a single class. Instantiate for a specific project, the object could then contain all project info (from detail-cfg) opened only once. Also a copy of the total datasets (from native and cloud sources). Subselections can be passed between class methods along with a variable index (class variables: variable list, dimension list etc.)

Class logger attribute so this doesn’t need to be passed between functions. Bypass switch contained here with all switches.

padocc.phases.validate.check_for_nan(box, bypass, logger, label=None)

Special function for assessing if a box selection has non-NaN values within it. Needs further testing using different data types.

padocc.phases.validate.mem_to_value(mem) → float

Convert a memory value i.e 2G into a value

Returns:

Int value of e.g. ‘2G’ in bytes.

padocc.phases.validate.slice_all_dims(data_arr: DataArray, intval: int)

Slice all dimensions for the DataArray according to the integer value.

padocc.phases.validate.value_to_mem(value) → str

Convert a number of bytes i.e 1000000000 into a string

Returns:

String value of the above (1000000000 -> 1M)

Ingest Operations

Note

Not featured in this development, the ingest operations are still being planned and scoped.

Add the download link to each of the Kerchunk references

padocc.phases.ingest.ingest_config(args, logger)

Configure for ingestion of a set of project codes, currently defined by a repeat_id but this could be changed later to apply to all project codes fitting some parameters