Core functionality¶
Suite¶
-
class
BenchmarkSuite
(log_dir=None, log_id=None, use_timestamp=True, parallel=None, slurm=None)[source]¶ Bases:
object
Class to orchestrate the execution of multiple experiments.
-
__init__
(log_dir=None, log_id=None, use_timestamp=True, parallel=None, slurm=None)[source]¶ Constructor.
Parameters: - log_dir (str) – path to the log directory (Default: ./logs or /work/scratch/$USER)
- log_id (str) – log id (Default: benchmark[_YYYY-mm-dd-HH-MM-SS])
- use_timestamp (bool) – select if a timestamp should be appended to the log id
- parallel (dict, None) – parameters that are passed to the run_parallel method of the experiment
- slurm (dict, None) – parameters that are passed to the run_slurm method of the experiment
-
add_experiments
(environment_name, environment_builder_params, agent_names_list, agent_builders_params, **run_params)[source]¶ Add a set of experiments for the same environment to the suite.
Parameters: - environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
- environment_builder_params (dict) – parameters for the environment builder;
- agent_names_list (list) – list of names of the agents for the experiments;
- agent_builders_params (list) – list of dictionaries containing the parameters for the agent builder;
- run_params – Parameters that are passed to the run method of the experiment.
-
add_experiments_sweeps
(environment_name, environment_builder_params, agent_names_list, agent_builders_params, sweeps_list, **run_params)[source]¶ Add a set of experiments sweeps for the same environment to the suite.
Parameters: - environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
- environment_builder_params (dict) – parameters for the environment builder;
- agent_names_list (list) – list of names of the agents for the experiments;
- agent_builders_params (list) – list of dictionaries containing the parameters for the agent builder;
- sweeps_list (list) – list of dictionaries containing the parameter sweep to be executed;
- run_params – Parameters that are passed to the run method of the experiment.
-
add_environment
(environment_name, environment_builder_params, **run_params)[source]¶ Add an environment to the benchmarking suite.
Parameters: - environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
- environment_builder_params (dict) – parameters for the environment builder;
- run_params – Parameters that are passed to the run method of the experiment.
-
add_agent
(environment_name, agent_name, agent_params)[source]¶ Add an agent to the benchmarking suite.
Parameters: - environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
- agent_name (str) – name of the agent for the experiments;
- agent_params (list) – dictionary containing the parameters for the agent builder.
-
add_sweep
(environment_name, agent_name, agent_params, sweep_dict)[source]¶ Add an agent sweep to the benchmarking suite.
Parameters: - environment_name (str) – name of the environment for the experiment (E.g. Gym.Pendulum-v0);
- agent_name (str) – name of the agent for the experiments;
- agent_params (list) – dictionary containing the parameters for the agent builder;
- sweep_dict (dict) – dictionary with the sweep configurations.
-
save_parameters
()[source]¶ Save the experiment parameters in yaml files inside the parameters folder
-
Experiment¶
-
class
BenchmarkExperiment
(agent_builder, env_builder, logger)[source]¶ Bases:
object
Class to create and run an experiment using MushroomRL
-
__init__
(agent_builder, env_builder, logger)[source]¶ Constructor.
Parameters: - agent_builder (AgentBuilder) – instance of a specific agent builder;
- env_builder (EnvironmentBuilder) – instance of an environment builder;
- logger (BenchmarkLogger) – instance of a benchmark logger.
-
run
(exec_type='sequential', **run_params)[source]¶ Execute the experiment.
Parameters: - exec_type (str, 'sequential') – type of executing the experiment [sequential|parallel|slurm];
- **run_params – parameters for the selected execution type.
-
run_sequential
(n_runs, n_runs_completed=0, save_plot=True, **run_params)[source]¶ Execute the experiment sequential.
Parameters: - n_runs (int) – number of total runs of the experiment;
- n_runs_completed (int, 0) – number of completed runs of the experiment;
- save_plot (bool, True) – select if a plot of the experiment should be saved to the log directory;
- **run_params – parameters for executing a benchmark run.
-
run_parallel
(n_runs, n_runs_completed=0, threading=False, save_plot=True, max_concurrent_runs=None, **run_params)[source]¶ Execute the experiment in parallel threads.
Parameters: - n_runs (int) – number of total runs of the experiment;
- n_runs_completed (int, 0) – number of completed runs of the experiment;
- threading (bool, False) – select to use threads instead of processes;
- save_plot (bool, True) – select if a plot of the experiment should be saved to the log directory;
- max_concurrent_runs (int, -1) – maximum number of concurrent runs. By default it uses the number of cores;
- **run_params – parameters for executing a benchmark run.
-
run_slurm
(n_runs, n_runs_completed=0, aggregation_job=True, aggregate_hours=3, aggregate_minutes=0, aggregate_seconds=0, only_print=False, **run_params)[source]¶ Execute the experiment with SLURM.
Parameters: - n_runs (int) – number of total runs of the experiment;
- n_runs_completed (int, 0) – number of completed runs of the experiment;
- aggregation_job (bool, True) – select if an aggregation job should be scheduled;
- aggregate_hours (int, 3) – maximum number of hours for the aggregation job;
- aggregate_minutes (int, 0) – maximum number of minutes for the aggregation job;
- aggregate_seconds (int, 0) – maximum number of seconds for the aggregation job;
- only_print (bool, False) – if True, don’t launch the benchmarks, only print the submitted commands to the terminal;
- **run_params – parameters for executing a benchmark run.
-
extend_and_save_J
(J)[source]¶ Extend J with another datapoint and save the current state to the log directory.
-
extend_and_save_R
(R)[source]¶ Extend R with another datapoint and save the current state to the log directory.
-
extend_and_save_V
(V)[source]¶ Extend V with another datapoint and save the current state to the log directory.
-
Logger¶
-
class
BenchmarkLogger
(log_dir=None, log_id=None, use_timestamp=True)[source]¶ Bases:
mushroom_rl.core.logger.console_logger.ConsoleLogger
Class to handle all interactions with the log directory.
-
__init__
(log_dir=None, log_id=None, use_timestamp=True)[source]¶ Constructor.
Parameters: - log_dir (str, None) – path to the log directory, if not specified defaults to ./logs or to /work/scratch/$USER if the second directory exists;
- log_id (str, None) – log id, if not specified defaults to: benchmark[_YY-mm-ddTHH:MM:SS.zzz]);
- use_timestamp (bool, True) – select if a timestamp should be appended to the log id.
-
set_log_dir
(log_dir)[source]¶ Set the directory for logging.
Parameters: log_dir (str) – path of the directory.
-
set_log_id
(log_id, use_timestamp=True)[source]¶ Set the id of the logged folder.
Parameters: - log_id (str) – id of the logged folder;
- use_timestamp (bool, True) – whether to use the timestamp or not.
-
get_path
(filename='')[source]¶ Get the path of the given file. If no filename is given, it returns the path of the logging folder.
Parameters: filename (str, '') – the name of the file. Returns: The complete path of the logged file.
-
get_params_path
(filename='')[source]¶ Get the path of the parameters of the given file. If no filename is given, it returns the path of the parameters folder.
Parameters: filename (str, '') – the name of the file. Returns: The complete path of the logged file.
-
get_figure_path
(filename='', subfolder=None)[source]¶ Get the path of the figures of the given file. If no filename is given, it returns the path of the figures folder.
Parameters: - filename (str, '') – the name of the file;
- subfolder (None) – the name of a subfolder to add.
Returns: The complete path of the logged file.
-
exists_value_function
()[source]¶ Returns: True if the log of the value function exists, False otherwise.
-
save_best_agent
(agent)[source]¶ Save the best agent in the respective path.
Parameters: agent (object) – the agent to save.
-
save_last_agent
(agent)[source]¶ Save the last agent in the respective path.
Parameters: agent (object) – the agent to save.
-
save_environment_builder
(env_builder)[source]¶ Save the environment builder using the respective path.
Parameters: env_builder (str) – the environment builder to save.
-
save_agent_builder
(agent_builder)[source]¶ Save the agent builder using the respective path.
Parameters: agent_builder (str) – the agent builder to save.
-
save_config
(config)[source]¶ Save the config file using the respective path.
Parameters: config (str) – the config file to save.
-
save_stats
(stats)[source]¶ Save the statistic file using the respective path.
Parameters: stats (str) – the statistics file to save.
-
save_params
(env, params)[source]¶ Save the parameters file.
Parameters: - env (str) – the environment used;
- params (str) – the parameters file to save.
-
save_figure
(figure, figname, subfolder=None, as_pdf=False, transparent=True)[source]¶ Save the figure file using the respective path.
Parameters: - figure (object) – the figure to save;
- figname (str) – the name of the figure;
- subfolder (str, None) – optional subfolder where to save the figure;
- as_pdf (bool, False) – whether to save the figure in PDF or not;
- transparent (bool, True) – whether the figure should be transparent or not.
-
Visualizer¶
-
class
BenchmarkVisualizer
(logger, data=None, has_entropy=None, has_value=None, id=1)[source]¶ Bases:
object
Class to handle all visualizations of the experiment.
-
plot_counter
= 0¶
-
__init__
(logger, data=None, has_entropy=None, has_value=None, id=1)[source]¶ Constructor.
Parameters: - logger (BenchmarkLogger) – logger to be used;
- data (dict, None) – dictionary with data points for visualization;
- has_entropy (bool, None) – select if entropy is available for the algorithm.
-
is_data_persisted
¶ Check if data was passed as dictionary or should be read from log directory.
-
save_report
(file_name='report_plot')[source]¶ Method to save an image of a report of the training metrics from a performend experiment.
-
show_report
()[source]¶ Method to show a report of the training metrics from a performend experiment.
-
-
class
BenchmarkSuiteVisualizer
(logger, is_sweep, color_cycle=None, y_limit=None, legend=None)[source]¶ Bases:
object
Class to handle visualization of a benchmark suite.
-
plot_counter
= 0¶
-
__init__
(logger, is_sweep, color_cycle=None, y_limit=None, legend=None)[source]¶ Constructor.
Parameters: - logger (BenchmarkLogger) – logger to be used;
- is_sweep (bool) – whether the benchmark is a parameter sweep.
- color_cycle (dict, None) – dictionary with colors to be used for each algorithm;
- y_limit (dict, None) – dictionary with environment specific plot limits.
- legend (dict, None) – dictionary with environment specific legend parameters.
-
get_boxplot
(env, metric_type, data_type, selected_alg=None)[source]¶ Create boxplot with matplotlib for a given metric.
Parameters: - env (str) – The environment name;
- metric_type (str) – The metric to compute.
Returns: A figure with the desired boxplot of the given metric.
-
save_reports
(as_pdf=True, transparent=True, alg_sweep=False)[source]¶ Method to save an image of a report of the training metrics from a performed experiment.
Parameters: - as_pdf (bool, True) – whether to save the reports as pdf files or png;
- transparent (bool, True) – If true, the figure background is transparent and not white;
- alg_sweep (bool, False) – If true, thw method will generate a separate figure for each algorithm sweep.
-
save_boxplots
(as_pdf=True, transparent=True, alg_sweep=False)[source]¶ Method to save an image of a report of the training metrics from a performed experiment.
Parameters: - as_pdf (bool, True) – whether to save the reports as pdf files or png;
- transparent (bool, True) – If true, the figure background is transparent and not white;
- alg_sweep (bool, False) – If true, thw method will generate a separate figure for each algorithm sweep.
-