Data

Data module for the project.

Cube handling

All methods related to manipulation of spectral image cubes are located here. Usually, you can simply call data.cube_handling.construct_envi_cube() to gather the rendered bands and perform automatic white correction to form a reflectance cube.

construct_envi_cube(system_sim_name: str, system_sim_name_for_white_signal=None)

Constructs an ENVI style hyperspectral image cube out of rendered images.

Can be used after the scene has been rendered (at least spectral).

White reference for reflectance calculation is searched automatically from available visibility maps if system_sim_name_for_white_signal is not given . Note that the maps must be named like Reference 0.00 material_0001.tif.

Saves white signal used in reflectance calculation as a toml file.

Default RGB bands for ENVI metadata are inferred if in visible range. Otherwise first, middle, and last bands are used.

Parameters:

  • system_sim_name – Name of the system simulation to process.

  • system_sim_name_for_white_signal – Optionally, give system simulation name from where to fetch the white signal. If None, it is inferred from the data. If there are no visibility maps in the current system simulation, this will fail and raise an error.

Raises:

FileNotFoundError – if the rendered frames directory does not exist or is empty. Also if the sun data file does not exist, which is needed for wavelength info.

get_raw_cube(system_sim_name: str)

Reads rendered spectral frames into a raw hyperspectral image cube.

Parameters:

system_sim_name – See system_sim_name.

Returns:

Raw hyperspectral image cube as a 3D numpy array with shape (bands, height, width).

Raises:

FileNotFoundError – if the rendered frames directory does not exist or is empty.

infer_white_ref_from_data(system_sim_name: str)

Infers a good white reference signal from available visibility maps.

Saves plot of the raw mean spectra of the reference plates.

Note

Even if an acceptable white reference plate is found, it does not guarantee that the reflectance cube is of good quality. If there are objects in the scene that brighter than the white reference plate, they will be clipped.

Parameters:

system_sim_name – Name of the system simulation.

Returns:

White reference signal as a 3D numpy array with shape (bands, 1, 1).

Raises:

  • FileNotFoundError – if no visibility maps are found.

  • RuntimeError – if none of the available white reference plates is acceptable.

read_white_signal(system_sim_name: str) → ndarray

Reads the white signal used in reflectance calculation from a toml file.

Parameters:

system_sim_name – Name of the system simulation.

Returns:

White signal as a 3D numpy array with shape (bands, 1, 1).

Raises:

FileNotFoundError – if the white signal file does not exist.

save_white_signal(system_sim_name: str, white_signal: ndarray)

Saves the white signal used in reflectance calculation as a toml file.

Parameters:

  • system_sim_name – Name of the system simulation.

  • white_signal – White signal as a 1D numpy array with shape (bands,).

show_cube(system_sim_name: str)

Shows the hyperspectral image cube.

This is mainly for debugging purposes, to quickly visualize the cube. For any real needs, use a dedicated ENVI viewer, such as the CubeInspector found at github.com/silmae/CubeInspector. CubeInspector is a sister project of HyperBlend.

Raises:

FileNotFoundError – if the cube does not exist.

File handling

TODO docs

TODO: there are lots of methods that could be replaced if we could use a

python version that includes a toml reader. It is available at least in Blender 4.4.

clear_all_rendered_images(slab_sim_name: str) → None

Clear all rendered images that have their optimization result ready.

clear_directory(path: str) → None

Clears all files from the directory in given path.

clear_rend_refs(slab_sim_name: str, signal_id: int) → None

Clears the slab reference renders but leaves the slab renders untouched.

clear_rend_slab(slab_sim_name: str, signal_id: int) → None

Clears slab render directory of the given simulation.

Leaves the slab reference renders untouched.

copy_slab_material_parameters(system_sim_name: str, slab_material_name: str, src_slab_sim_name: str, signal_id: int = None)

Reads slab simulation result and copies it as a slab material parameter file to be consumed by blender_scripts.bs_setup_forest.

Slab material parameters are written as a csv file to give to specified system_simulation scene. We use csv file instead of toml files because importing external packages, such as toml, into Blender’s own Python environment is a bit of a hassle. Csv files work just as well and they can be read with tools already included by default.

TODO: check how this is actually used. The material names should be generalized at some point

Note

This may change as new Python version with included toml is in use in HyperBlend.

Parameters:

  • system_sim_name – Name of the system simulation.

  • src_slab_sim_name – Name of the slab simulation to copy from.

  • slab_material_name – Name of the slab material. This will be put into the name of the file written and is used by the scripts in blender_scripts.

  • signal_id – Signal id of the slab simulation. If None, slab simulation’s average values will be used instead of a specific signal.

copy_slab_simulation_target(src_slab_sim_name: str, dst_slab_sim_name: str) → None

Copy slab simulation targets and sampling data as a new slab simulation.

See also Slab simulation.

Parameters:

  • src_slab_sim_name – The name of the slab simulation to copy from.

  • dst_slab_sim_name – The name of the slab simulation to copy to.

create_signal_optimization_directories(slab_sim_name: str, signal_id: int) → None

Create directories for slab simulation signal optimization solver.

create_slab_sim_signal_directories(slab_sim_name: str, signal_id: int) → None

Create directories for slab simulation result signal.

This is needed regardless of the used solver for storing the results.

create_top_level_slab_sim_directories(slab_sim_name: str) → None

Create top level directories for slab simulation.

The creation of the directories for a certain signal is covered by create_slab_sim_signal_directories().

The creation of the directories needed by the optimizer are covered by create_signal_optimization_directories().

These are separate methods because the others need also the signal id.

create_top_level_system_sim_directories(system_sim_name: str) → None

Create top level directories for system simulation.

duplicate_system_simulation_scene(src_system_sim_name: str = None, dst_system_sim_name: str = None) → str

Creates a duplicate of a system simulation Blender scene.

Creates the necessary directory structure.

Parameters:

  • src_system_sim_name – If provided, a system_simulation scene with this name is duplicated. If not provided, the default template system simulation is used. See also system_sim_name.

  • dst_system_sim_name – If given, this will be the name of the new system simulation scene. If not provided, a name based on date time will be generated.

Returns:

The name of the duplicated system simulation scene. If dst_system_sim_name was provided, the same name is returned. Otherwise, a generated name is returned.

Raises:

FileNotFoundError – If the system simulation scene to duplicate does not exist.

expand(slab_sim_name: str) → None

Re-generate files removed by reduce().

Doesn’t do anything for slab simulations solved with slab_simulation.surf or slab_simulation.nn models.

Note

Can not re-generate rendered images but will generate optimization history. This takes quite a bit of time, so only use if you really need it.

list_finished_result_signal_ids(slab_sim_name: str) → list[int]

Lists signal ids that have an existing result in the slab simulation’s results directory.

list_target_ids(slab_sim_name: str) → list[int]

Lists available signal targets by their id.

Targets must be named ‘target_X.toml’ where X is a number that can be cast into int.

Parameters:

slab_sim_name – Name of the slab simulation.

Returns:

List of target ids (int) that were found from target folder.

optimization_wl_result_exists(slab_sim_name: str, wl: float, signal_id: int) → bool

Tells whether a result exists for a given wavelength within given signal and slab simulation.

This is used to skip optimization of wavelengths that already have a result.

Parameters:

  • slab_sim_name – Name of the slab simulation.

  • signal_id – The id of the signal.

  • wl – Wavelength to be searched for. Has to be accurate to 2 decimals to be found.

Returns:

True if the result exists, False otherwise.

reduce(slab_sim_name: str) → None

Removes wavelength-wise optimization history plots and cleans up temp working directories.

Doesn’t do anything for slab simulations solved with slab_simulation.surf or slab_simulation.nn models.

Useful for reducing file size when sharing over internet, for example. Use expand() method to re-generate most of the files as they were.

Reduced size is about 1/10 of original size.

Note

Rendered images can not be generated back after they are deleted.

write_blender_light_spectra(system_sim_name: str, wls: list[float], irradiances, lighting_type='sun')

Write light spectra to a csv file that can be read by Blender script.

TODO: accept any general light source name. Sun and sky can be in a separate

method if needed, but really, maybe not necessary.

Parameters:

  • system_sim_name – Name of the system_simulation scene to write to.

  • wls – List of wavelengths to be written.

  • irradiances – List of light irradiances to be written.

  • lighting_type – Either ‘sun’ or ‘sky’.

write_blender_rgb_colors(system_sim_name: str, rgb_dict: dict) → None

Write RGB colors to a csv file that can be read by blender_scripts.

:param system_sim_name Name of the system simulation. :param rgb_dict: Dictionary of RGB colors to be written. The keys are the

names of the colors

write_blender_soil(system_sim_name: str, wls: list[float], reflectances: list[float]) → None

Write soil reflectance spectra to a csv file that can be read by Blender script.

Parameters:

  • system_sim_name – Name of the system simulation.

  • wls – List of wavelengths to be written.

  • reflectances – List of reflectances to be written.

File names

This module handles such file names that are not directly contained in :mod`src.constants`. Such file names include rendered images that depend on the wavelength, target signal toml files, and result files of slab simulations.

As the project relies largely on file names, the parsers for such names are also found here. Some file names are dictated by the rendering scripts and Blender files, so care nust be taken that they stay in sync when refactoring the code or the Blender files.

filename_resample_plot(sample_id: int) → str

Generate filename for resampling plot.

filename_sample_result(sample_id: int) → str

Filename of the sample result toml file.

filename_signal_result_plot(signal_id: int) → str

Filename of the signal result plot file.

filename_slab_material_csv(slab_material_name: str) → str

Spectral slab material parameters csv file name.

filename_slab_sim_render_refl_or_tran(imaging_type: str, wl: float) → str

Returns a name for a reflectance or transmittance image of slab optimizer.

Warning

This name must match the one the rendering script uses. So if any changes are made in rendering, they must be reflected here as well.

Parameters:

  • imaging_type – String either ‘refl’ for reflectance or ‘tran’ for transmittance. Use the ones listed in src.constants.

  • wl – Wavelength.

Returns:

The name of the image.

filename_system_sim_spectral_cube(system_sim_name: str, file_type='data') → str

Filename of the spectral image cube of a system simulation.

Parameters:

  • system_sim_name – Scene id of the system simulation.

  • file_type – Either “data” for the image data file or “header” for the header file.

Returns:

Filename of the image cube itself or the name of the header file.

filename_target_signal(signal_id: int, resampled=False) → str

Generate filename of a target signal toml file.

Parameters:

  • signal_id – Signal id.

  • resampled – If True, file name of corresponding resampled file is returned instead. Default is False.

filename_wl_result(wl: float, file_type='data') → str

Generate name of a wavelength result file of given wavelength.

Used only by slab simulation optimizer material parameter solver.

Parameters:

  • wl – Wavelength as float. Must be accurate to 2 decimals.

  • file_type – Type of file to generate, either “data” for toml file or “plot” for png.

Returns:

Filename as string in format result_wl_1.00.toml or result_wl_1.00.png.

parse_target_signal_id(filename: str) → int

Parses target signal id of a slab simulation from a given filename.

Parameters:

filename – Filename in the format where the target id is an int at the end of the name, e.g. target_123.toml.

Returns:

Parsed sample id.

parse_wl_from_filename(filename: str)

Parse the wavelength from result signal toml or plot file’s name.

This is only used by the optimization slab simulation material parameter solver.

Parameters:

filename – The filename must be formatted so that the wavelength is the last item in the name, separated by _wl_ and the wavelength itself can be cast to float, such as result_wl_1.00.toml.

Returns:

Wavelength as float.

Lihgt file handling

This script contains methods for reading and writing light spectrum files.

Files are formatted so that comment lines are prefixed with ‘#’ and rest of the lines contain wavelength-irradiance pairs that can be cast to floats and be read directly. For spectra generated with either NASA PSG or , SSolar GOA the files are fixed so that they can be read directly later.

_band_separations(wls)

Minimum and maximum of adjacent wavelengths in given list.

Parameters:

wls – List of wavelengths (floats).

Returns:

min_diff, max_diff (float) minimum and maximum differences.

_fix_double_space(path: str)

Replace double spaces with a single space.

_fix_psg_file(path: str)

Fix double spaces and irradiance units provided by NASA PSG.

Rewrites the file (several times) if needed. Adds a tag to the fixed file that tells it is now OK.

_read_file_comments_only(path: str)

Reads only comment lines beginning with ‘#’ and returns those lines as a list.

_read_first_non_comment_line(path: str)

Reads only the first line after comments and returns that.

_read_hb_light_file(path, required_resolution=1.0, resolution_epsilon=0.01)

Read light spectrum from a file of certain structure.

This method can read the files that are fixed by other methods to be compliant with HyperBlend expectations.

The file is supposed to start with rows where the first character is ‘#’. These lines are considered comments. The remaining lines must have to numbers separated by a space. The numbers have to castable to a float.

Example of how the file could look like:

|  # This is a comment line that can contain some metadata.
|  # Perhaps this line will tell that the power is in W/m^2/nm
|  # Maybe some header data like "Wavelengths Irradiance" and then the actual data in following lines
|  400.0 0.12345
|  401.0 0.6789
|  ...
Parameters:

  • path – Path to a file to be read.

  • required_resolution – Spectral resolution of the file is checked so that bands cannot be further away from each other than this. Default is 1.0.

  • resolution_epsilon – Allow this much variation between adjacent bands. Default is 0.01.

Returns:

(wls, irradiances, comments) tuple where wls and irradiances are numpy arrays. comments is a list of comment rows needed in file fixing. The comments that are returned are mainly for internal working of this script and do not matter for outside caller.

Raises:

RuntimeError – if wavelengths in the file are not in 1 nm resolution.

_split_goa_file(path)

Splits GOA-generated spectrum file into separate sun and sky spectra.

Parameters:

path – Path to the GOA-generated file.

Returns:

new_sun_file_name, new_sky_file_name Names of the new files.

_write_hb_light_file(path: str, wls, irradiances, comments)

Writes a light spectrum file.

The produced file will start with comments where each line is prefixed with ‘#’ and the rest of the lines will contain wls-irradiance pair separated by a space.

Parameters:

  • path – Path where to write the file.

  • wls – Wavelengths as a list of floats.

  • irradiances – Associated irradiances as a list of floats.

  • comments – Comment lines as a list of strings prefixed with ‘#’.

read_light_file(path: str)

Reads a light spectrum file and returns wavelengths and corresponding irradiances.

The file is expected to be one generated by either NASA PSG or SSolar GOA. If it is, it will be reformatted to align with how HyperBlend expects the light files to look like. Otherwise, it will be read as is, provided that the file is formatted so that comment lines are prefixed with ‘#’ and the rest of the lines contain wavelength-irradiance pairs that can be cast to floats.

Path handling

Paths to directories and files. Almost all methods return a single string except some search methods that return a list of strings.

Methods to find directories that contain subdirectories have the word ‘top’ in their name.

Methods to find directories that contain program code of HyperBlend have the word ‘code’ in their name.

directory_code_blender_scripts() → str

Blender scripts directory.

directory_code_definitions() → str

HyperBlend’s internal definitions.

directory_code_soil() → str

Soil code directory that contain gsv spectra vectors used for gsv generation.

directory_code_source() → str

Path to source code directory.

directory_internal() → str

Path to directory containing HyperBlend internal files.

directory_light_spectra() → str

Path to light spectra directory that contains all spectra for light sources.

directory_log() → str

Path to logging directory.

directory_optimization_result(slab_sim_name: str, signal_id: int) → str

Path to optimization result directory of given slab simulation and signal id.

Only used if slab model is run in optimization mode.

directory_project_root() → str

Path to project root directory.

directory_reflectance_spectra() → str

Path to reflectance spectra directory.

For light spectra, use directory_light_spectra().

For materials that are reflective and transmissive, use TODO

directory_result_signal(slab_sim_name: str, signal_id: int) → str

Path to a directory where a specific signal resulted from slab simulation is saved.

directory_slab_model(slab_model_name: str = None) → str

Path to an arbitrary slab model directory.

It stores surface model parameters, a neural network, and a starting guess.

Parameters:

slab_model_name – If None, path to the default slab model is returned.

Returns:

Returns path to the slab model directory.

directory_slab_optimization_working(slab_sim_name: str, signal_id: int) → str

Path to top level working folder of the slab simulation for given signal.

Optimization renders images to define the virtual slab material parameters. Those images are stored in the working folder.

directory_slab_rend_reference(imaging_type: str, base_path: str) → str

Returns the path to reflectance or transmittance reference folder of the slab simulation.

Parameters:

  • imaging_type – String either ‘refl’ for reflectance or ‘tran’ for transmittance. Use the ones listed in constants.py.

  • base_path – Path to the working folder. Usually the one returned by path_directory_slab_optimization_working_temp() is correct and other paths should only be used for testing and debugging.

directory_slab_simulation(slab_sim_name: str) → str

Path to a specific slab simulation.

The top level directory is given by directory_top_slab_simulation().

directory_slab_working_rend(slab_sim_name: str, signal_id: int) → str

Path to the rendering directory of given slab simulation and signal id.

directory_system_bundle() → str

Path to system simulation bundle directory.

directory_system_rend_spectral(system_sim_name: str) → str

System simulation spectral rendering directory.

directory_system_rend_visibility_maps(system_sim_name: str) → str

System simulation render directory for visibility maps of materials.

directory_system_sim_rend(system_sim_name: str) → str

Rendering directory of the system simulation.

directory_system_simulation(system_sim_name: str) → str

Specific system simulation scene directory.

directory_system_spectral_cube(system_sim_name: str) → str

Spectral cube resulting from a system simulation is stored here.

directory_top_result_signal(slab_sim_name: str) → str

Result signals of a certain slab simulation. See slab_sim_name

directory_top_slab_simulation() → str

Path to top level directory where all slab simulations are stored.

directory_top_system_simulation() → str

Path to top level system simulation directory.

directory_top_target(slab_sim_name: str) → str

Path to the top level target signal directory.

file_blend_reflectance_lab_template() → str

Path to slab simulation template Blender file found in directory ‘Internal/’.

file_blend_slab_simulation_template(template_name: str = None) → str

Path to slab simulation template Blender file found in directory ‘Internal/’.

The .blend extension is added automatically if not given.

Parameters:

template_name – Name of the template file. If None given, the default template is used.

file_blend_system_simulation(simulation_name: str) → str

Path to a certain system simulation scene Blender file.

The directory and the actual file share the same name but the file has a .blend extension.

Parameters:

simulation_name – Name of the template file.

file_blend_system_simulation_template(template_name: str = None) → str

Path to system simulation template Blender file found in directory ‘Internal/’.

The .blend extension is added automatically if not given.

TODO: this doesn’t make any sense anymore as there can be only one template…

no! There can be other templates. But currently there is only one. So the caller must take care if they actually want a template or some other scene file.

Parameters:

template_name – Name of the template file. If None given, the default template is used.

file_forest_soil_csv(system_sim_name: str) → str

Path to the soil spectra csv file that is used for rendering.

Specific for forest type system simulation.

TODO: change the logic so that this is a common call to any kind of reflective

material and not specific to soil.

file_gsv_soil_dry_vector() → str

Soil dry vector used by GSV.

file_gsv_soil_humid_vector() → str

Soil humid vector used by GSV.

file_signal_result(slab_sim_name: str, signal_id: int) → str

Path to signal result toml file of given slab simulation and signal id.

file_signal_result_plot(slab_sim_name: str, signal_id: int) → str

Path to signal result plot of given slab simulation.

file_slab_sim_error_plot(slab_sim_name: str) → str

Path to slab simulation error plot of given slab simulation.

file_slab_sim_result(slab_sim_name: str) → str

Path to slab simulation result toml file of given slab simulation.

file_slab_sim_result_plot(slab_sim_name: str) → str

Path to slab simulation result plot file of given slab simulation.

file_slab_target(slab_sim_name: str, signal_id: int, resampled=False) → str

Path to slab simulation target spectrum file of given slab simulation and signal.

Parameters:

  • slab_sim_name – Slab simulation name.

  • signal_id – Signal id.

  • resampled – If True, path to corresponding resampled file is returned instead. Default is False.

file_spectral_cube(system_sim_name: str, file_type: str) → str

Path to simulated spectral image cube header file.

The header file is ENVI format convention that contains the metadata of the cube.

File type parameter as accepted by data.file_names.filename_system_sim_spectral_cube()

file_spectral_sampling(slab_sim_name: str) → str

Path to spectral resampling data of given slab simulation.

file_starting_guess(slab_model_name=None) → str

Path to the starting guess parameter file to be used in optimization.

Parameters:

slab_model_name – If None given, the default slab model’s starting guess is used.

file_surface_model_parameters(slab_model_name: str = None) → str

Path to surface model parameter file. See slab_model_name.

Parameters:

slab_model_name – If None, path to the default slab model is returned.

file_system_sim_light_spectra_csv(system_sim_name: str, light_file_name: str) → str

Path to the light spectra csv file that is used for rendering.

Specific for system_simulation type system simulation.

file_system_sim_preview(system_sim_name: str, image_name: str) → str

Path to system simulation preview render files.

Parameters:

  • system_sim_name – Name of the system simulation.

  • image_name – Name of the image. Use the image names available in src.constants. Image type postfix is added automatically.

file_system_sim_rgb_colors_csv(system_sim_name: str) → str

Path to the RGB colors csv file that is used in rendering as a color approximation.

file_system_slab_csv(system_sim_name: str, slab_material_name: str) → str

Spectral slab material csv file name.

It contains the spectral parameters of the slab material in a csv format that rendering scripts can read directly.

file_visibility_map(system_sim_name: str, file_name: str) → str

Path to a visibility map file.

Parameters:

  • system_sim_name – Name of the system simulation.

  • file_name – Name of the visibility map file. The file name is used as-is.

file_wl_result(slab_sim_name: str, signal_id: int, wl: float) → str

Path to wavelength result toml file of given slab simulation, signal id, and wavelength.

find_light_file(file_name: str, system_sim_name: str = None) → str

Attempts to find a light file with given filename.

Parameters:

  • file_name – A file with this name is searched from Light spectra directory. If also system_sim_name is given, the system simulation directory is searched before extending the search to Light spectra directory.

  • system_sim_name – Optional. If not given, system_simulation scene directory is not searched.

Returns:

Path to found file.

Raises:

FileNotFoundError – If the file is not found.

find_reference_visibility_map(system_sim_name: str, reflectivity: float) → str

Convenience method for finding reference visibility map file.

Calls find_visibility_map().

Parameters:

  • system_sim_name – System simulation name.

  • reflectivity – Reflectivity desired between 0.0 and 1.0. Must be one of the available ones in visibility maps directory of the scene.

Returns:

Returns a path to the file.

find_slab_opt_render_by_wl(wl: float, mode: str, imaging_type: str, base_path: str) → str

Search for slab simulation optimization render by wavelength.

Parameters:

  • wl – Wavelength to be searched. Must match the image name with two decimals.

  • mode – String either ‘slab’ or ‘reference’. Use the ones listed in src.constants.

  • imaging_type – String either ‘refl’ for reflectance or ‘tran’ for transmittance. Use the ones listed in src.constants.

  • base_path – Path to the image folder. Usually the one returned by directory_slab_optimization_working() is correct and other paths should only be used for testing and debugging.

Returns:

Returns absolute path to the image.

Raises:

FileNotFoundError – if the image cannot be found

find_visibility_map(system_sim_name: str, search_term: str) → str

Find a visibility map matching given search term.

Search term should be something like “Leaf material 1” or “Trunk material 2”. For white reference paths, one can use convenience method find_reference_visibility_map() that only needs reflectance as an identifier.

Parameters:

  • system_sim_name – System simulation name.

  • search_term – Search term that is included in a file name. Does not have to be a full match to the filename. We cannot fully control the filenames coming out of Blender, so we only check if the filename includes the search term instead of a full match.

Returns:

Returns a path to the file.

Raises:

  • KeyError – if more than one file match the search term.

  • FileNotFoundError – if no file match the search term.

join(*args) → str

Custom join function to avoid problems using os.path.join.

Parameters:

args – List of strings (directory names and a possibly a file name as the last element) to be joined.

list_reference_visibility_maps(system_sim_name: str) → list[str]

Lists all available reference visibility maps for given system simulation scene.

list_visibility_maps(system_sim_name: str) → list[str]

Lists all available visibility maps for given system_simulation scene.

Parameters:

system_sim_name – Name of the system simulation.

Returns:

List of visibility map file names.

path_nn_model(slab_model_name: str = None) → str

Returns path to the NN model.

Existence of the file has to be checked by caller.

Parameters:

slab_model_name – Name of the slab model directory. If none given, the NN model in the default slab model directory is returned.

Specchio data parser

This module contains methods to attempt to parse Specchio data into some coherent format.

This is only used in the case that reflectance and transmittance measurements have to be loaded in separate files in separate folders one by one from specchio.ch web interface. The code is a mess but one should not have to use this often.

collect_pairs()
combine_pairs()
make_target(wls, r_m, t_m, path, sample_idx)
open_files()

Toml handling

Methods in this module handle writing and reading of various toml formatted files.

collect_signal_results(slab_sim_name: str)

Collect results of simulated signals in a list of dictionaries.

collect_wavelength_result(slab_sim_name: str, signal_id: int)

Collects wavelength result dictionaries in to a list and returns it.

Returns:

A list of wavelength result dictionaries.

make_signal_result(slab_sim_name: str, signal_id: int, wall_clock_time_minutes=0.0)

Creates the sample result by collecting the data from wavelength results.

Saves the result as numerical data and plots.

Parameters:

  • slab_sim_name – The name of the slab simulation.

  • signal_id – Signal id.

  • wall_clock_time_minutes – Wall clock time used to optimize this sample in minutes.

read_sampling(slab_sim_name: str)

Read sampling wavelengths from a file.

Parameters:

slab_sim_name – Name of the leaf measurement set.

Returns:

Return resampling wavelengths as 1D numpy array.

Raises:

RuntimeError – in case some of the entries could not be interpreted as a float.

read_signal_result(slab_sim_name: str, signal_id: int)

Reads sample result file into a dict and returns it.

Returns:

Result file content as a dict.

read_slab_sim_result(slab_sim_name: str)

Reads the slab simulation result file. Created if does not exist.

Returns:

Result file content as a dict.

read_starting_guess_coeffs(slab_model_name: str = None) → dict

Reads starting guess coefficients from disk and return as dictionary.

Parameters:

slab_model_name – If None, the default slab model name is used.

Returns:

Starting guess coefficients in a dictionary.

read_surface_model_parameters(solver_name: str = None)

Reads surface model parameters from a file and returns them as a dictionary.

Parameters:

solver_name – Name of the solver (directory). If none given, the default solver is used.

read_sys_sim_bundle(bundle_name: str) → dict

Read system simulation bundle file as dict.

Parameters:

bundle_name – Name of the bundle.

Returns:

Bundle dictionary containing the names of the scenes belonging to that bundle.

read_target(slab_sim_name: str, signal_id: int, resampled=False)

Read target signals for solving slab material parameters.

Parameters:

  • slab_sim_name – Name of the set.

  • signal_id – Sample id.

  • resampled – If True, data is read from a corresponding resampled file. Default is False.

Returns:

Numpy array which is essentially a list of tuples [[wavelength_1, reflectance_1, transmittance_1], [wavelength_2, reflectance_2, transmittance_2]]

Raises:

OSError – if file could not be opened.

read_toml_as_dict(directory: str, filename: str)

General purpose toml reading method.

Parameters:

  • directory – Path to the directory where the toml file is.

  • filename – Name of the file to be read. Postfix ‘.toml’ will be added if necessary.

Return dictionary:

Returns read toml file as a dictionary.

Raises:

FileNotFoundError – Raises FileNotFoundError if the file does not exist.

read_wavelength_result(slab_sim_name: str, signal_id: int, wl: float)

Reads a wavelength result file into a dictionary and returns it.

Returns:

Wavelength result as a dictionary.

write_dict_as_toml(dictionary: dict, directory: str, filename: str)

General purpose dictionary saving method.

Parameters:

  • dictionary – Dictionary to be written as toml.

  • directory – Path to the directory where the toml should be written.

  • filename – Name of the file to be written. Postfix ‘.toml’ will be added if necessary.

Returns:

Path to where the dictionary was written.

write_sampling(slab_sim_name: str, sampling: list | ndarray = None, overwrite=False)

Write sampling data file for a given slab simulation.

Preferred workflow is to NOT provide a list of wavelengths here, which will result an empty file where you can copy and paste desired wavelengths. Providing a wavelengths list here is good for debugging or quick experiments though.

Note

If the sampling file already exits, this method does nothing unless overwrite = True.

Parameters:

  • slab_sim_name – Name of the slab simulation.

  • sampling – You can give a list of wavelengths here. If none is given, an empty wavelength dictionary is written to the file. You can later copy paste wavelengths from an ENVI file, for example.

  • overwrite – If True, overwrites existing sampling with the new one. Default is False.

write_signal_result(slab_sim_name: str, signal_id: int, res_dict: dict) → None

Writes given signal result dictionary into a file.

write_slab_sim_result(slab_sim_name: str)

Collects slab simulation results and writes the final result to a toml file.

write_starting_guess_coeffs(ad_coeffs, sd_coeffs, ai_coeffs, mf_coeffs, solver_name: str = None) → None

Writes given starting guess coefficients to disk.

Parameters:

  • ad_coeffs – Coefficients for absorption particle density as a list of floats.

  • sd_coeffs – Coefficients for scattering particle density as a list of floats.

  • ai_coeffs – Coefficients for scattering anisotropy as a list of floats.

  • mf_coeffs – Coefficients for mix factor as a list of floats.

  • solver_name – If None, the default solver name is used and the previous default starting guess is overwritten.

write_surface_model_parameters(parameter_dict, solver_name=None)

Writes surface model parameters to a file.

Parameters:

  • parameter_dict – Parameter dictionary to be saved.

  • solver_name – Solver name used for saving the surface model parameters.

write_sys_sim_bundle(bundle_name: str, sys_sim_bundle_dict: dict) → str

Write system simulation bundle file that lists scenes belonging to that bundle.

Parameters:

  • bundle_name – Name of the bundle which is used as a filename too.

  • sys_sim_bundle_dict – Dict to be saved.

Returns:

Path where the file was written to.

write_target(slab_sim_name: str, target, signal_id=0, resampled=False) → None

Writes given list of reflectance and transmittance data to toml formatted file.

Writes also an empty sampling file to the target directory.

Parameters:

  • slab_sim_name – Name of the slab simulation.

  • target –

    List of lists, or list of tuples like [[wl, r, t], …]

    Warning

    Do not use numpy arrays as they may break the toml writer.

  • signal_id – Default is 0, which is used for slab simulations with only one signal.

  • resampled – If True, data is written to a file with the word ‘resampled’ in its name. Default is False.

write_wavelength_result(slab_sim_name: str, signal_id: int, res_dict: dict) → None

Writes wavelength result of optimization into a file.