Core functionality
Suite
- class BenchmarkSuite(log_dir=None, log_id=None, use_timestamp=True, parallel=None, slurm=None)[source]
Bases:
object
Class to orchestrate the execution of multiple experiments.
- __init__(log_dir=None, log_id=None, use_timestamp=True, parallel=None, slurm=None)[source]
Constructor.
- Parameters:
log_dir (str) – path to the log directory (Default: ./logs or /work/scratch/$USER)
log_id (str) – log id (Default: benchmark[_YYYY-mm-dd-HH-MM-SS])
use_timestamp (bool) – select if a timestamp should be appended to the log id
parallel (dict, None) – parameters that are passed to the run_parallel method of the experiment
slurm (dict, None) – parameters that are passed to the run_slurm method of the experiment
- add_experiments(environment_name, environment_builder_params, agent_names_list, agent_builders_params, **run_params)[source]
Add a set of experiments for the same environment to the suite.
- Parameters:
environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
environment_builder_params (dict) – parameters for the environment builder;
agent_names_list (list) – list of names of the agents for the experiments;
agent_builders_params (list) – list of dictionaries containing the parameters for the agent builder;
run_params – Parameters that are passed to the run method of the experiment.
- add_experiments_sweeps(environment_name, environment_builder_params, agent_names_list, agent_builders_params, sweeps_list, **run_params)[source]
Add a set of experiments sweeps for the same environment to the suite.
- Parameters:
environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
environment_builder_params (dict) – parameters for the environment builder;
agent_names_list (list) – list of names of the agents for the experiments;
agent_builders_params (list) – list of dictionaries containing the parameters for the agent builder;
sweeps_list (list) – list of dictionaries containing the parameter sweep to be executed;
run_params – Parameters that are passed to the run method of the experiment.
- add_environment(environment_name, environment_builder_params, **run_params)[source]
Add an environment to the benchmarking suite.
- Parameters:
environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
environment_builder_params (dict) – parameters for the environment builder;
run_params – Parameters that are passed to the run method of the experiment.
- add_agent(environment_name, agent_name, agent_params)[source]
Add an agent to the benchmarking suite.
- Parameters:
environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
agent_name (str) – name of the agent for the experiments;
agent_params (list) – dictionary containing the parameters for the agent builder.
- add_sweep(environment_name, agent_name, agent_params, sweep_dict)[source]
Add an agent sweep to the benchmarking suite.
- Parameters:
environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
agent_name (str) – name of the agent for the experiments;
agent_params (list) – dictionary containing the parameters for the agent builder;
sweep_dict (dict) – dictionary with the sweep configurations.
- save_parameters()[source]
Save the experiment parameters in yaml files inside the parameters folder
Experiment
- class BenchmarkExperiment(agent_builder, env_builder, logger)[source]
Bases:
object
Class to create and run an experiment using MushroomRL
- __init__(agent_builder, env_builder, logger)[source]
Constructor.
- Parameters:
agent_builder (AgentBuilder) – instance of a specific agent builder;
env_builder (EnvironmentBuilder) – instance of an environment builder;
logger (BenchmarkLogger) – instance of a benchmark logger.
- run(exec_type='sequential', **run_params)[source]
Execute the experiment.
- Parameters:
exec_type (str, 'sequential') – type of executing the experiment [sequential|parallel|slurm];
**run_params – parameters for the selected execution type.
- run_sequential(n_runs, n_runs_completed=0, save_plot=True, **run_params)[source]
Execute the experiment sequential.
- Parameters:
n_runs (int) – number of total runs of the experiment;
n_runs_completed (int, 0) – number of completed runs of the experiment;
save_plot (bool, True) – select if a plot of the experiment should be saved to the log directory;
**run_params – parameters for executing a benchmark run.
- run_parallel(n_runs, n_runs_completed=0, threading=False, save_plot=True, max_concurrent_runs=None, **run_params)[source]
Execute the experiment in parallel threads.
- Parameters:
n_runs (int) – number of total runs of the experiment;
n_runs_completed (int, 0) – number of completed runs of the experiment;
threading (bool, False) – select to use threads instead of processes;
save_plot (bool, True) – select if a plot of the experiment should be saved to the log directory;
max_concurrent_runs (int, -1) – maximum number of concurrent runs. By default it uses the number of cores;
**run_params – parameters for executing a benchmark run.
- run_slurm(n_runs, n_runs_completed=0, aggregation_job=True, aggregate_hours=0, aggregate_minutes=20, aggregate_seconds=0, only_print=False, **run_params)[source]
Execute the experiment with SLURM.
- Parameters:
n_runs (int) – number of total runs of the experiment;
n_runs_completed (int, 0) – number of completed runs of the experiment;
aggregation_job (bool, True) – select if an aggregation job should be scheduled;
aggregate_hours (int, 3) – maximum number of hours for the aggregation job;
aggregate_minutes (int, 0) – maximum number of minutes for the aggregation job;
aggregate_seconds (int, 0) – maximum number of seconds for the aggregation job;
only_print (bool, False) – if True, don’t launch the benchmarks, only print the submitted commands to the terminal;
**run_params – parameters for executing a benchmark run.
- extend_and_save_J(J)[source]
Extend J with another datapoint and save the current state to the log directory.
- extend_and_save_R(R)[source]
Extend R with another datapoint and save the current state to the log directory.
- extend_and_save_V(V)[source]
Extend V with another datapoint and save the current state to the log directory.
Logger
- class BenchmarkLogger(log_dir=None, log_id=None, use_timestamp=True)[source]
Bases:
ConsoleLogger
Class to handle all interactions with the log directory.
- __init__(log_dir=None, log_id=None, use_timestamp=True)[source]
Constructor.
- Parameters:
log_dir (str, None) – path to the log directory, if not specified defaults to ./logs or to /work/scratch/$USER if the second directory exists;
log_id (str, None) – log id, if not specified defaults to: benchmark[_YY-mm-ddTHH:MM:SS.zzz]);
use_timestamp (bool, True) – select if a timestamp should be appended to the log id.
- set_log_dir(log_dir)[source]
Set the directory for logging.
- Parameters:
log_dir (str) – path of the directory.
- set_log_id(log_id, use_timestamp=True)[source]
Set the id of the logged folder.
- Parameters:
log_id (str) – id of the logged folder;
use_timestamp (bool, True) – whether to use the timestamp or not.
- get_path(filename='')[source]
Get the path of the given file. If no filename is given, it returns the path of the logging folder.
- Parameters:
filename (str, '') – the name of the file.
- Returns:
The complete path of the logged file.
- get_params_path(filename='')[source]
Get the path of the parameters of the given file. If no filename is given, it returns the path of the parameters folder.
- Parameters:
filename (str, '') – the name of the file.
- Returns:
The complete path of the logged file.
- get_figure_path(filename='', subfolder=None)[source]
Get the path of the figures of the given file. If no filename is given, it returns the path of the figures folder.
- Parameters:
filename (str, '') – the name of the file;
subfolder (None) – the name of a subfolder to add.
- Returns:
The complete path of the logged file.
- exists_value_function()[source]
- Returns:
True if the log of the value function exists, False otherwise.
- save_best_agent(agent)[source]
Save the best agent in the respective path.
- Parameters:
agent (object) – the agent to save.
- save_last_agent(agent)[source]
Save the last agent in the respective path.
- Parameters:
agent (object) – the agent to save.
- save_environment_builder(env_builder)[source]
Save the environment builder using the respective path.
- Parameters:
env_builder (str) – the environment builder to save.
- save_agent_builder(agent_builder)[source]
Save the agent builder using the respective path.
- Parameters:
agent_builder (str) – the agent builder to save.
- save_config(config)[source]
Save the config file using the respective path.
- Parameters:
config (str) – the config file to save.
- save_stats(stats)[source]
Save the statistic file using the respective path.
- Parameters:
stats (str) – the statistics file to save.
- save_params(env, params)[source]
Save the parameters file.
- Parameters:
env (str) – the environment used;
params (str) – the parameters file to save.
- save_figure(figure, figname, subfolder=None, as_pdf=False, transparent=True)[source]
Save the figure file using the respective path.
- Parameters:
figure (object) – the figure to save;
figname (str) – the name of the figure;
subfolder (str, None) – optional subfolder where to save the figure;
as_pdf (bool, False) – whether to save the figure in PDF or not;
transparent (bool, True) – whether the figure should be transparent or not.
Visualizer
- class BenchmarkVisualizer(logger, data=None, has_entropy=None, has_value=None, id=1)[source]
Bases:
object
Class to handle all visualizations of the experiment.
- plot_counter = 0
- __init__(logger, data=None, has_entropy=None, has_value=None, id=1)[source]
Constructor.
- Args:plot_mean_conf
logger (BenchmarkLogger): logger to be used; data (dict, None): dictionary with data points for visualization; has_entropy (bool, None): select if entropy is available for the algorithm.
- property is_data_persisted
Check if data was passed as dictionary or should be read from log directory.
- save_report(file_name='report_plot')[source]
Method to save an image of a report of the training metrics from a performend experiment.
- show_report()[source]
Method to show a report of the training metrics from a performend experiment.
- class BenchmarkSuiteVisualizer(logger, is_sweep, color_cycle=None, y_limit=None, legend=None)[source]
Bases:
object
Class to handle visualization of a benchmark suite.
- plot_counter = 0
- __init__(logger, is_sweep, color_cycle=None, y_limit=None, legend=None)[source]
Constructor.
- Parameters:
logger (BenchmarkLogger) – logger to be used;
is_sweep (bool) – whether the benchmark is a parameter sweep.
color_cycle (dict, None) – dictionary with colors to be used for each algorithm;
y_limit (dict, None) – dictionary with environment specific plot limits.
legend (dict, None) – dictionary with environment specific legend parameters.
- get_boxplot(env, metric_type, data_type, selected_alg=None)[source]
Create boxplot with matplotlib for a given metric.
- Parameters:
env (str) – The environment name;
metric_type (str) – The metric to compute.
- Returns:
A figure with the desired boxplot of the given metric.
- save_reports(as_pdf=True, transparent=True, alg_sweep=False)[source]
Method to save an image of a report of the training metrics from a performed experiment.
- Parameters:
as_pdf (bool, True) – whether to save the reports as pdf files or png;
transparent (bool, True) – If true, the figure background is transparent and not white;
alg_sweep (bool, False) – If true, thw method will generate a separate figure for each algorithm sweep.
- save_boxplots(as_pdf=True, transparent=True, alg_sweep=False)[source]
Method to save an image of a report of the training metrics from a performed experiment.
- Parameters:
as_pdf (bool, True) – whether to save the reports as pdf files or png;
transparent (bool, True) – If true, the figure background is transparent and not white;
alg_sweep (bool, False) – If true, thw method will generate a separate figure for each algorithm sweep.