ax.service

Ax Client

class ax.service.ax_client.AxClient(generation_strategy: GenerationStrategy | None = None, db_settings: DBSettings | None = None, enforce_sequential_optimization: bool = True, random_seed: int | None = None, torch_device: device | None = None, verbose_logging: bool = True, suppress_storage_errors: bool = False, early_stopping_strategy: BaseEarlyStoppingStrategy | None = None, global_stopping_strategy: BaseGlobalStoppingStrategy | None = None)[source]

Bases: AnalysisBase, BestPointMixin, InstantiationBase

Convenience handler for management of experimentation cycle through a service-like API. External system manages scheduling of the cycle and makes calls to this client to get next suggestion in the experiment and log back data from the evaluation of that suggestion.

Note: AxClient expects to only propose 1 arm (suggestion) per trial; support for use cases that require use of batches is coming soon.

Two custom types used in this class for convenience are TParamValue and TParameterization. Those are shortcuts for Union[str, bool, float, int] and Dict[str, Union[str, bool, float, int]], respectively.

Parameters:

generation_strategy – Optional generation strategy. If not set, one is intelligently chosen based on properties of search space.
db_settings – Settings for saving and reloading the underlying experiment to a database. Expected to be of type ax.storage.sqa_store.structs.DBSettings and require SQLAlchemy.
enforce_sequential_optimization – Whether to enforce that when it is reasonable to switch models during the optimization (as prescribed by num_trials in generation strategy), Ax will wait for enough trials to be completed with data to proceed. Defaults to True. If set to False, Ax will keep generating new trials from the previous model until enough data is gathered. Use this only if necessary; otherwise, it is more resource-efficient to optimize sequentially, by waiting until enough data is available to use the next model.
random_seed –
Optional integer random seed, set to fix the optimization random seed for reproducibility. Works only for Sobol quasi-random generator and for BoTorch-powered models. For the latter models, the trials generated from the same optimization setup with the same seed, will be mostly similar, but the exact parameter values may still vary and trials latter in the optimizations will diverge more and more. This is because a degree of randomness is essential for high performance of the Bayesian optimization models and is not controlled by the seed.

Note: In multi-threaded environments, the random seed is thread-safe, but does not actually guarantee reproducibility. Whether the outcomes will be exactly the same for two same operations that use the random seed, depends on whether the threads modify the random state in the same order across the two operations.
torch_device – An optional torch.device object, used to choose the device used for generating new points for trials. Works only for torch-based models, such as MBM. Ignored if a generation_strategy is passed in manually. To specify the device for a custom generation_strategy, pass in torch_device as part of generator_kwargs. See https://ax.dev/tutorials/generation_strategy.html for a tutorial on generation strategies.
verbose_logging – Whether Ax should log significant optimization events, defaults to True. Float values are rounded to 6 decimal places.
suppress_storage_errors – Whether to suppress SQL storage-related errors if encountered. Only use if SQL storage is not important for the given use case, since this will only log, but not raise, an exception if its encountered while saving to DB or loading from it.
early_stopping_strategy – A BaseEarlyStoppingStrategy that determines whether a trial should be stopped given the current state of the experiment. Used in should_stop_trials_early.
global_stopping_strategy – A BaseGlobalStoppingStrategy that determines whether the full optimization should be stopped or not.

abandon_trial(trial_index: int, reason: str | None = None) → None[source]

Abandons a trial.

Parameters:: trial_index – Index of trial within the experiment.

Add new parameters to the experiment’s search space. This allows extending the search space after the experiment has run some trials.

Backfill values must be provided for all new parameters to ensure existing trials in the experiment remain valid within the expanded search space. The backfill values represent the parameter values that were used in the existing trials.

Parameters:

parameters – A sequence of parameter configurations to add to the search space.
backfill_values – Parameter values to assign to existing trials for the new parameters being added. All new parameter names must have corresponding backfill values provided.
status_quo_values – Optional parameter values for the new parameters to use in the status quo (baseline) arm, if one is defined. If None, the backfill values will be used for the status quo.
parameter_constraints – Optional list of string representations of parameter constraints to add (e.g., "x1 + x2 <= 5.0" or "x1 <= x2"). May reference both existing and new parameters.
runner_updates – Optional typed context to pass to the runner’s on_search_space_update hook.

add_tracking_metrics(metric_names: list[str], metric_definitions: dict[str, dict[str, Any]] | None = None, metrics_to_trial_types: dict[str, str] | None = None, canonical_names: dict[str, str] | None = None) → None[source]

Add a list of new metrics to the experiment.

If any of the metrics are already defined on the experiment, we raise an error and don’t add any of them to the experiment

Parameters:

metric_names – Names of metrics to be added.
metric_definitions – A mapping of metric names to extra kwargs to pass to that metric. Note these are modified in-place. Each Metric must have its is own dictionary (metrics cannot share a single dictionary object).
metrics_to_trial_types – Only applicable to MultiTypeExperiment. The mapping from metric names to corresponding trial types for each metric. If provided, the metrics will be added with their respective trial types. If not provided, then the default trial type will be used.
canonical_names – A mapping from metric name (of a particular trial type) to the metric name of the default trial type. Only applicable to MultiTypeExperiment.

Attach a new trial with the given parameterization to the experiment.

Parameters:

parameters – Parameterization of the new trial.
ttl_seconds – If specified, will consider the trial stale after this many seconds. Used to detect dead trials that did not complete.

Returns:

Tuple of parameterization and trial index from newly created trial.

Completes the trial with given metric values.

NOTE: When raw_data does not specify SEM for a given metric, Ax will default to the assumption that the data is noisy (specifically, corrupted by additive zero-mean Gaussian noise) and that the level of noise should be inferred by the optimization model. To indicate that the data is noiseless, set SEM to 0.0, for example:

ax_client.complete_trial(
    trial_index=0,
    raw_data={"my_objective": (objective_mean_value, 0.0)}
)

Parameters:

trial_index – Index of trial within the experiment.
raw_data – Evaluation data for the trial. Can be a mapping from metric name to a tuple of mean and SEM, just a tuple of mean and SEM if only one metric in optimization, or just the mean if SEM is unknown (then Ax will infer observation noise level). Can also be a list of (fidelities, mapping from metric name to a tuple of mean and SEM).

Create a new experiment and save it if DBSettings available.

Parameters:

parameters – List of dictionaries representing parameters in the experiment search space. Required elements in the dictionaries are: 1. “name” (name of parameter, string), 2. “type” (type of parameter: “range”, “fixed”, or “choice”, string), and one of the following: 3a. “bounds” for range parameters (list of two values, lower bound first), 3b. “values” for choice parameters (list of values), or 3c. “value” for fixed parameters (single value). Optional elements are: 1. “log_scale” (for float-valued range parameters, bool), 2. “value_type” (to specify type that values of this parameter should take; expects “float”, “int”, “bool” or “str”), 3. “is_fidelity” (bool) and “target_value” (float) for fidelity parameters, 4. “is_ordered” (bool) for choice parameters, and 5. “is_task” (bool) for task parameters. 6. “digits” (int) for float-valued range parameters.
name – Name of the experiment to be created.
description – Description of the experiment to be created.
objectives – Mapping from an objective name to object containing: minimize: Whether this experiment represents a minimization problem. threshold: The bound in the objective’s threshold constraint.
parameter_constraints – List of string representation of parameter constraints, such as “x3 >= x4” or “-x3 + 2*x4 - 3.5*x5 >= 2”. For the latter constraints, any number of arguments is accepted, and acceptable operators are “<=” and “>=”. Note that parameter constraints may only be placed on range parameters, not choice parameters or fixed parameters.
outcome_constraints – List of string representation of outcome constraints of form “metric_name >= bound”, like “m1 <= 3.”
status_quo – Parameterization of the current state of the system. If set, this will be added to each trial to be evaluated alongside test configurations.
overwrite_existing_experiment – If an experiment has already been set on this AxClient instance, whether to reset it to the new one. If overwriting the experiment, generation strategy will be re-selected for the new experiment and restarted. To protect experiments in production, one cannot overwrite existing experiments if the experiment is already stored in the database, regardless of the value of overwrite_existing_experiment.
tracking_metric_names – Names of additional tracking metrics not used for optimization.
choose_generation_strategy_kwargs – Keyword arguments to pass to choose_generation_strategy function which determines what generation strategy should be used when none was specified on init.
support_intermediate_data – Whether trials may report intermediate results for trials that are still running (i.e. have not been completed via ax_client.complete_trial).
immutable_search_space_and_opt_config – Whether it’s possible to update the search space and optimization config on this experiment after creation. Defaults to True. If set to True, we won’t store or load copies of the search space and optimization config on each generator run, which will improve storage performance.
is_test – Whether this experiment will be a test experiment (useful for marking test experiments in storage etc). Defaults to False.
metric_definitions – A mapping of metric names to extra kwargs to pass to that metric. Note these are modified in-place. Each Metric must have its own dictionary (metrics cannot share a single dictionary object).
default_trial_type – The default trial type if multiple trial types are intended to be used in the experiment. If specified, a MultiTypeExperiment will be created. Otherwise, a single-type Experiment will be created.
default_runner – The default runner in this experiment. This applies to MultiTypeExperiment (when default_trial_type is specified) and needs to be specified together with default_trial_type. This will be ignored for single-type Experiment (when default_trial_type is not specified).

Disable parameters in the experiment. This allows narrowing the search space after the experiment has run some trials.

When parameters are disabled, they are effectively removed from the search space for future trial generation. Existing trials remain valid, and the disabled parameters are replaced with fixed default values for all subsequent trials.

Parameters:

default_parameter_values – Fixed values to use for the disabled parameters in all future trials. These values will be used for the parameter in all subsequent trials.
runner_updates – Optional typed context to pass to the runner’s on_search_space_update hook.

property early_stopping_strategy: BaseEarlyStoppingStrategy | None: The early stopping strategy used on the experiment.

estimate_early_stopping_savings() → float[source]

Estimate early stopping savings using progressions of the MapMetric present on the EarlyStoppingConfig as a proxy for resource usage.

Returns:: The estimated resource savings as a fraction of total resource usage (i.e. 0.11 estimated savings indicates we would expect the experiment to have used 11% more resources without early stopping present)

property experiment: Experiment: Returns the experiment set on this Ax client.

fit_model() → None[source]

Fit a model using data collected from the trials so far.

This method will attempt to fit the same model that would be used for generating the next trial. The resulting model may be different from the model that was used to generate the last trial, if the generation node is ready to transition.

This method rarely needs to be called by the user, because model-fitting is usually handled indirectly through AxClient.get_next_trial(). This method instantiates a new model if none is yet available, which may be the case if no trials have been generated using a model-based method.

NOTE: If the current generation node is not model-based, no model may be fit.

classmethod from_json_snapshot(serialized: dict[str, Any], decoder_registry: dict[str, type[T] | Callable[[...], T]] | None = None, class_decoder_registry: None | dict[str, Callable[[dict[str, Any]], Any]] = None, **kwargs: Any) → AxClientSubclass[source]: Recreate an AxClient from a JSON snapshot.

property generation_strategy: GenerationStrategy: Returns the generation strategy, set on this experiment.

get_contour_plot(param_x: str | None = None, param_y: str | None = None, metric_name: str | None = None) → AxPlotConfig[source]

Retrieves a plot configuration for a contour plot of the response surface. For response surfaces with more than two parameters, selected two parameters will appear on the axes, and remaining parameters will be affixed to the middle of their range. If contour params arguments are not provided, the first two parameters in the search space will be used. If contour metrics are not provided, objective will be used.

Parameters:

param_x – name of parameters to use on x-axis for the contour response surface plots.
param_y – name of parameters to use on y-axis for the contour response surface plots.
metric_name – Name of the metric, for which to plot the response surface.

get_current_trial_generation_limit() → tuple[int, bool][source]

How many trials this AxClient instance can currently produce via calls to get_next_trial, before more trials are completed, and whether the optimization is complete.

NOTE: If return value of this function is (0, False), no more trials can currently be procuded by this AxClient instance, but optimization is not completed; once more trials are completed with data, more new trials can be generated.

Returns: a two-item tuple of:

the number of trials that can currently be produced, with -1 meaning unlimited trials,
whether no more trials can be produced by this AxClient instance at any point (e.g. if the search space is exhausted or generation strategy is completed.

get_feature_importances(relative: bool = True) → AxPlotConfig[source]

Get a bar chart showing feature_importances for a metric.

A drop-down controls the metric for which the importances are displayed.

Parameters:: relative – Whether the values are displayed as percentiles or as raw importance metrics.

get_max_concurrency() → list[tuple[int, int]][source]

Retrieves maximum number of trials that can be scheduled concurrently at different stages of optimization.

Some optimization algorithms profit significantly from sequential optimization (i.e. suggest a few points, get updated with data for them, repeat, see https://ax.dev/docs/bayesopt.html). Concurrency setting indicates how many trials should be running simultaneously (generated, but not yet completed with data).

The output of this method is mapping of form {num_trials -> max_concurrency_setting}, where the max_concurrency_setting is used for num_trials trials. If max_concurrency_setting is -1, as many of the trials can be ran concurrently, as necessary. If num_trials in a tuple is -1, then the corresponding max_concurrency_setting should be used for all subsequent trials.

For example, if the returned list is [(5, -1), (12, 6), (-1, 3)], the schedule could be: run 5 trials with any concurrency, run 6 trials concurrently twice, run 3 trials concurrently for as long as needed. Here, ‘running’ a trial means obtaining a next trial from AxClient through get_next_trials and completing it with data when available.

Returns:: Mapping of form {num_trials -> max_concurrency_setting}.

get_max_parallelism() → list[tuple[int, int]][source]

Retrieve model-estimated means and covariances for all metrics.

NOTE: This method currently only supports one-arm trials.

Parameters:

metric_names – Names of the metrics, for which to retrieve predictions. All metrics on experiment will be retrieved if this argument was not specified.
include_out_of_sample – Defaults to True. Return predictions for out-of-sample (i.e. not yet completed trials) data in addition to in-sample (i.e. completed trials) data.
parameterizations – Optional mapping from an int label to Parameterizations. When provided, predictions are performed only on these data points, no predictions from trial data is performed, and include_out_of_sample parameters is ignored.

Returns:

A mapping from trial index to a mapping of metric names to tuples of predicted metric mean and SEM, of form: { trial_index -> { metric_name: ( mean, SEM ) } }.

Retrieve model-estimated means and covariances for all metrics for the provided parameterizations.

Parameters:

metric_names – Names of the metrics for which to predict. All metrics will be predicted if this argument is not specified.
parameterizations – List of Parameterizations for which to predict.

Returns:

List[Tuple[float, float]].

Return type:

A list of predicted metric mean and SEM of form

Generate trial with the next set of parameters to try in the iteration process.

Note: Service API currently supports only 1-arm trials.

Parameters:

ttl_seconds – If specified, will consider the trial stale after this many seconds. Used to detect dead trials that did not complete.
force – If set to True, this function will bypass the global stopping strategy’s decision and generate a new trial anyway.
fixed_features – A FixedFeatures object containing any features that should be fixed at specified values during generation.

Returns:

Tuple of trial parameterization, trial index

Generate as many trials as currently possible.

NOTE: Useful for running multiple trials in parallel: produces multiple trials, with their number limited by:

parallelism limit on current generation step,

number of trials in current generation step,

number of trials required to complete before moving to next generation step, if applicable,

and max_trials argument to this method.

Parameters:

max_trials – Limit on how many trials the call to this method should produce.
ttl_seconds – If specified, will consider the trial stale after this many seconds. Used to detect dead trials that did not complete.
fixed_features – A FixedFeatures object containing any features that should be fixed at specified values during generation.

Returns: two-item tuple of:

mapping from trial indices to parameterizations in those trials,
boolean indicator of whether optimization is completed and no more trials can be generated going forward.

static get_recommended_max_parallelism() → None[source]

get_trial(trial_index: int) → Trial[source]: Return a trial on experiment cast as Trial

get_trial_parameters(trial_index: int) → dict[str, None | str | bool | float | int][source]: Retrieve the parameterization of the trial by the given index.

get_trials_data_frame() → DataFrame[source]: Get a Pandas DataFrame representation of this experiment. The columns will include all the parameters in the search space and all the metrics on this experiment. The rows will each correspond to a trial (if using one-arm trials, which is the case in base AxClient; will correspond to arms in trials in the batch-trial case).

property global_stopping_strategy: BaseGlobalStoppingStrategy | None: The global stopping strategy used on the experiment.

static load(filepath: str | None = None) → None[source]

static load_experiment(experiment_name: str) → None[source]

load_experiment_from_database(experiment_name: str, choose_generation_strategy_kwargs: dict[str, Any] | None = None) → None[source]

Load an existing experiment from database using the DBSettings passed to this AxClient on instantiation.

Parameters:: experiment_name – Name of the experiment.
Returns:: Experiment object.

classmethod load_from_json_file(filepath: str = 'ax_client_snapshot.json', **kwargs: Any) → AxClientSubclass[source]: Restore an AxClient and its state from a JSON-serialized snapshot, residing in a .json file by the given path.

log_trial_failure(trial_index: int, metadata: dict[str, str] | None = None) → None[source]

Mark that the given trial has failed while running.

Parameters:

trial_index – Index of trial within the experiment.
metadata – Additional metadata to track about this run.

property metric_definitions: dict[str, dict[str, Any]]: Returns metric definitions for all experiment metrics that can be passed into functions requiring metric_definitions

property metric_names: set[str]: Returns the names of all metrics on the attached experiment.

property objective: Objective

property objective_name: str: Returns the name of the objective in this optimization.

property objective_names: list[str]: Returns the name of the objective in this optimization.

remove_tracking_metric(metric_name: str) → None[source]: Deprecated. Use remove_metric instead.

property runner_config_type: type[RunnerConfig] | None

The RunnerConfig subclass declared by the experiment’s runner.

Returns None if the experiment has no runner. Useful for discovering the typed context a runner expects:

ctx_cls = client.runner_config_type.SearchSpaceUpdateArguments

static save(filepath: str | None = None) → None[source]

save_to_json_file(filepath: str = 'ax_client_snapshot.json') → None[source]: Save a JSON-serialized snapshot of this AxClient’s settings and state to a .json file by the given path.

set_optimization_config(objectives: dict[str, ObjectiveProperties] | None = None, outcome_constraints: list[str] | None = None, metric_definitions: dict[str, dict[str, Any]] | None = None) → None[source]

Overwrite experiment’s optimization config

Parameters:

objectives – Mapping from an objective name to object containing: minimize: Whether this experiment represents a minimization problem. threshold: The bound in the objective’s threshold constraint.
outcome_constraints – List of string representation of outcome constraints of form “metric_name >= bound”, like “m1 <= 3.”
metric_definitions – A mapping of metric names to extra kwargs to pass to that metric

Sets the search space on the experiment and saves. This is expected to fail on base AxClient as experiment will have immutable search space and optimization config set to True by default

Parameters:

parameters – List of dictionaries representing parameters in the experiment search space. Required elements in the dictionaries are: 1. “name” (name of parameter, string), 2. “type” (type of parameter: “range”, “fixed”, or “choice”, string), and one of the following: 3a. “bounds” for range parameters (list of two values, lower bound first), 3b. “values” for choice parameters (list of values), or 3c. “value” for fixed parameters (single value). Optional elements are: 1. “log_scale” (for float-valued range parameters, bool), 2. “value_type” (to specify type that values of this parameter should take; expects “float”, “int”, “bool” or “str”), 3. “is_fidelity” (bool) and “target_value” (float) for fidelity parameters, 4. “is_ordered” (bool) for choice parameters, and 5. “is_task” (bool) for task parameters. 6. “digits” (int) for float-valued range parameters.
parameter_constraints – List of string representation of parameter constraints, such as “x3 >= x4” or “-x3 + 2*x4 - 3.5*x5 >= 2”. For the latter constraints, any number of arguments is accepted, and acceptable operators are “<=” and “>=”. Note that parameter constraints may only be placed on range parameters, not choice parameters or fixed parameters.

Set, or unset status quo on the experiment. There may be risk in using this after a trial with the status quo arm has run.

Parameters:: status_quo – Parameterization of the current state of the system. If set, this will be added to each trial to be evaluated alongside test configurations.

should_stop_trials_early(trial_indices: set[int]) → dict[int, str | None][source]

Evaluate whether to early-stop running trials.

Parameters:: trial_indices – Indices of trials to consider for early stopping.
Returns:: A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.

property status_quo: dict[str, None | str | bool | float | int] | None: The parameterization of the status quo arm of the experiment.

stop_trial_early(trial_index: int) → None[source]

to_json_snapshot(encoder_registry: dict[type[Any], Callable[[Any], dict[str, Any]]] | None = None, class_encoder_registry: dict[type[Any], Callable[[Any], dict[str, Any]]] | None = None) → dict[str, Any][source]

Serialize this AxClient to JSON to be able to interrupt and restart optimization and save it to file by the provided path.

Returns:: A JSON-safe dict representation of this AxClient.

update_parameters(parameters: Sequence[RangeParameterConfig], runner_updates: SearchSpaceUpdateArguments | None = None) → None[source]

Update parameters in the experiment’s search space.

This allows modifying the search space after the experiment has run some trials.

Parameters:

parameters – A sequence of RangeParameterConfig to update in the search space.
runner_updates – Optional typed context to pass to the runner’s on_search_space_update hook.

Raises:

UserInputError – If a parameter is not found in the search space or if the parameter is not a RangeParameter.

Updates the trial with given metric values without completing it. Useful for intermediate results such as the metrics of a partially optimized machine learning model. In these cases it should be called instead of complete_trial until it is time to complete the trial.

NOTE: This method will raise an Exception if it is called multiple times with the same raw_data. These cases typically arise when raw_data does not change over time. To avoid this, pass a “step” (progression) metric in raw_data, for example:

for ts in range(100):
    raw_data = [(ts, {"my_objective": (1.0, 0.0)})]
    ax_client.update_running_trial_with_intermediate_data(
        trial_index=0, raw_data=raw_data
    )

NOTE: When raw_data does not specify SEM for a given metric, Ax will default to the assumption that the data is noisy (specifically, corrupted by additive zero-mean Gaussian noise) and that the level of noise should be inferred by the optimization model. To indicate that the data is noiseless, set SEM to 0.0, for example:

ax_client.update_running_trial_with_intermediate_data(
    trial_index=0,
    raw_data={"my_objective": (objective_mean_value, 0.0)}
)

Parameters:

trial_index – Index of trial within the experiment.
raw_data – Evaluation data for the trial. Can be a mapping from metric name to a tuple of mean and SEM, just a tuple of mean and SEM if only one metric in optimization, or just the mean if SEM is unknown (then Ax will infer observation noise level). Can also be a list of (fidelities, mapping from metric name to a tuple of mean and SEM).

verify_trial_parameterization(trial_index: int, parameterization: dict[str, None | str | bool | float | int]) → bool[source]: Whether the given parameterization matches that of the arm in the trial specified in the trial index.

Managed Loop

ax.service.managed_loop.optimize(parameters: list[dict[str, None | str | bool | float | int | Sequence[None | str | bool | float | int] | dict[str, list[str]]]], evaluation_function: Callable[[dict[str, None | str | bool | float | int]], Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]] | int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None] | Sequence[tuple[float, Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]]]]] | Callable[[dict[str, None | str | bool | float | int], float | None], Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]] | int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None] | Sequence[tuple[float, Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]]]]], experiment_name: str | None = None, objective_name: str | None = None, minimize: bool = False, parameter_constraints: list[str] | None = None, outcome_constraints: list[str] | None = None, total_trials: int = 20, arms_per_trial: int = 1, random_seed: int | None = None, generation_strategy: GenerationStrategy | None = None) → tuple[Mapping[str, None | str | bool | float | int], tuple[dict[str, float], dict[str, dict[str, float]] | None] | None, Experiment, None][source]: Construct and run a full optimization loop.

Deprecated since version This: function is deprecated. Use ax.api.client.Client directly.

Interactive Loop

ax.service.interactive_loop.ax_client_candidate_generator(queue: Queue[tuple[dict[str, None | str | bool | float | int], int]], stop_event: Event, num_trials: int, ax_client: AxClient, lock: lock) → None[source]: Thread-safe method for generating the next trial from the AxClient and enqueueing it to the candidate queue. The number of candidates pre-generated is controlled by the maximum size of the queue. Generation stops when num_trials trials are attached to the AxClient’s experiment.

ax.service.interactive_loop.ax_client_data_attacher(queue: Queue[tuple[int, Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]] | int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None] | Sequence[tuple[float, Mapping[str, int | float | floating | integer | tuple[int | float | floating | integer, int | float | floating | integer | None]]]]]], stop_event: Event, ax_client: AxClient, lock: lock) → None[source]: Thread-safe method for attaching evaluation outcomes to the AxClient from the outcome queue. If the AxClient’s lock is acquired all data in the outcome queue is attached at once, then the lock released. Stops when the event is set.

ax.service.interactive_loop.interactive_optimize(num_trials: int, candidate_queue_maxsize: int, candidate_generator_function: Callable[[...], None], data_attacher_function: Callable[[...], None], elicitation_function: Callable[[...], Any], candidate_generator_kwargs: dict[str, Any] | None = None, data_attacher_kwargs: dict[str, Any] | None = None, elicitation_function_kwargs: dict[str, Any] | None = None) → bool[source]

Function to facilitate running Ax experiments with candidate pregeneration (the generation of candidate points while waiting for trial evaluation). This can be useful in many contexts, especially in interactive experiments in which trial evaluation entails eliciting a response from a human. Candidate pregeneration uses the time waiting for trail evaluation to generate new candidates from the data available. Note that this is a tradeoff – a larger pregeneration queue will result in more trials being generated with less data compared to a smaller pregeneration queue (or no pregeneration as in conventional Ax usage) and should only be used in contexts where it is necessary for the user to not experience any “lag” while candidates are being generated.

Parameters:

num_trials – The total number of trials to be run.
candidate_queue_maxsize – The maximum number of candidates to pregenerate.
candidate_generator_function – A function taking in a queue and event that enqueues candidates (generated by any means). See ax_client_candidate_generator for an example.
data_attacher_function – A function taking in a queue and event that attaches observations to Ax. See ax_client_data_attacher for an example.
elicitation_function – Function from parameterization (as returned by AxClient.get_next_trial) to outcome (as expected by AxClient.complete_trial). If None, elicitation is aborted by the user.
candidate_generator_kwargs – kwargs to be passed into candidate_generator_function when it is spawned as a thread.
data_attacher_kwargs – kwargs to be passed into data_attacher_function when it is spawned as a thread.
elicitation_function_kwargs – kwargs to be passed into elicitation_function

Returns:

True if optimization was completed and False if aborted.

Implementation of interactive_loop using the AxClient. Extract results of the experiment from the AxClient passed in.

The basic structure is as follows: One thread tries for a lock on the AxClient, generates a candidate, and enqueues it to a candidate queue. Another thread tries for the same lock, takes all the trial outcomes in the outcome queue, and attaches them to the AxClient. The main thread pops a candidate off the candidate queue, elicits response from the user, and puts the response onto the outcome queue.

Utils

Analysis

class ax.service.utils.analysis_base.AnalysisBase(db_settings: DBSettings | None = None, logging_level: int = 20, suppress_all_errors: bool = False)[source]

Bases: WithDBSettingsBase

Base class for analysis functionality shared between AxClient and orchestrator.

compute_analyses(analyses: Iterable[Analysis] | None = None, display: bool = True) → list[AnalysisCardBase][source]

Compute AnalysisCards (data about the optimization for end-user consumption) using the Experiment and GenerationStrategy. If no analyses are provided use some heuristic to determine which analyses to run. If some analyses fail, log failure and continue to compute the rest.

Note that the Analysis class is NOT part of the API and its methods are subject to change incompatibly between minor versions. Users are encouraged to use the provided analyses or leave this argument as None to use the default analyses.

Saves to database on completion if storage_config is present.

Parameters:

analyses – A list of Analysis classes to run. If None Ax will choose which analyses to run based on the state of the experiment.
display – Whether to display the AnalysisCards if executed in an interactive environment (e.g. Jupyter). Defaults to True. If not in an interactive environment this setting has no effect.

Returns:

A list of AnalysisCards.

experiment: Experiment

generation_strategy: GenerationStrategy

Best Point Identification

class ax.service.utils.best_point_mixin.BestPointMixin[source]

Bases: ABC

experiment: Experiment

generation_strategy: GenerationStrategy

Identifies the best parameterization tried in the experiment so far.

First attempts to do so with the model used in optimization and its corresponding predictions if available. Falls back to the best raw objective based on the data fetched from the experiment.

NOTE: TModelPredictArm is of the form:: ({metric_name: mean}, {metric_name_1: {metric_name_2: cov_1_2}})

Parameters:

optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.
use_model_predictions – Whether to extract the best point using model predictions or directly observed values. If True, the metric means and covariances in this method’s output will also be based on model predictions and may differ from the observed values.

Returns:

Tuple of parameterization and model predictions for it.

Identifies the best parameterization tried in the experiment so far.

First attempts to do so with the adapter used in optimization and its corresponding predictions if available. Falls back to the best raw objective based on the data fetched from the experiment.

NOTE: TModelPredictArm is of the form:: ({metric_name: mean}, {metric_name_1: {metric_name_2: cov_1_2}})

Parameters:

optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.
use_model_predictions – Whether to extract the best point using model predictions or directly observed values. If True, the metric means and covariances in this method’s output will also be based on model predictions and may differ from the observed values.

Returns:

Tuple of trial index, parameterization and model predictions for it.

get_hypervolume(optimization_config: MultiObjectiveOptimizationConfig | None = None, trial_indices: Iterable[int] | None = None, use_model_predictions: bool = True) → float[source]

Calculate hypervolume of a pareto frontier based on either the posterior means of given observation features or observed data.

Parameters:

optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.
use_model_predictions – Whether to extract the Pareto frontier using model predictions or directly observed values. If True, the metric means and covariances in this method’s output will also be based on model predictions and may differ from the observed values.

get_improvement_over_baseline(experiment: Experiment, generation_strategy: GenerationStrategy, baseline_arm_name: str | None = None) → float[source]

Returns the scalarized improvement over baseline, if applicable.

Returns:: For Single Objective cases, returns % improvement of objective. Positive indicates improvement over baseline. Negative indicates regression. For Multi Objective cases, throws NotImplementedError

Identifies the best parameterizations tried in the experiment so far, using model predictions if use_model_predictions is true and using observed values from the experiment otherwise. By default, uses model predictions to account for observation noise.

NOTE: The format of this method’s output is as follows: { trial_index –> (parameterization, (means, covariances) }, where means are a dictionary of form { metric_name –> metric_mean } and covariances are a nested dictionary of form { one_metric_name –> { another_metric_name: covariance } }.

Parameters:

optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.
use_model_predictions – Whether to extract the Pareto frontier using model predictions or directly observed values. If True, the metric means and covariances in this method’s output will also be based on model predictions and may differ from the observed values.

Returns:

the parameterization of the arm in that trial,
two-item tuple of metric means dictionary and covariance matrix
(model-predicted if use_model_predictions=True and observed otherwise).

Raises a NotImplementedError if extracting the Pareto frontier is not possible. Note that the returned dict may be empty.

Return type:

A mapping from trial index to the tuple of

get_trace_by_progression(optimization_config: OptimizationConfig | None = None, bins: list[float] | None = None, final_progression_only: bool = False) → tuple[list[float], list[float]][source]

Get the optimization trace with respect to trial progressions instead of trial_indices (which is the behavior used in get_trace). Note that this method does not take into account the parallelism of trials and essentially assumes that trials are run one after another, in the sense that it considers the total number of progressions “used” at the end of trial k to be the cumulative progressions “used” in trials 0,…,k. This method assumes that the final value of a particular trial is used and does not take the best value of a trial over its progressions.

The best observed value is computed at each value in bins (see below for details). If bins is not supplied, the method defaults to a heuristic of approximately NUM_BINS_PER_TRIAL per trial, where each trial is assumed to run until maximum progression (inferred from the data).

Parameters:

experiment – The experiment to get the trace for.
optimization_config – An optional optimization config to use for computing the trace. This allows computing the traces under different objectives or constraints without having to modify the experiment.
bins – A list progression values at which to calculate the best observed value. The best observed value at bins[i] is defined as the value observed in trials 0,…,j where j = largest trial such that the total progression in trials 0,…,j is less than bins[i].
final_progression_only – If True, considers the value of the last step to be the value of the trial. If False, considers the best along the curve to be the value of the trial.

Returns:

A tuple containing (1) the list of observed hypervolumes or best values and (2) a list of associated x-values (i.e., progressions) useful for plotting.

ax.service.utils.best_point.derelativize_opt_config(optimization_config: OptimizationConfig, experiment: Experiment, trial_indices: Iterable[int] | None = None) → OptimizationConfig[source]

Given an experiment, identifies the arm that had the best raw objective, based on the data fetched from the experiment.

TModelPredictArm is of the form:: ({metric_name: mean}, {metric_name_1: {metric_name_2: cov_1_2}})

This is a version of get_best_raw_objective_point_with_trial_index that logs errors rather than letting exceptions be raised.

Parameters:

experiment – Experiment, on which to identify best raw objective arm.
optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.

Returns:

Tuple of trial index, parameterization, and model predictions for it.

Given an experiment, returns the best predicted parameterization and corresponding prediction.

The best point & predictions are computed using the given Adapter and its predict method (if implemented). If adapter is not a TorchAdapter, the best point is extracted from the (first) generator run of the latest trial. If the latest trial doesn’t have a generator run, returns None. If the model fit assessment returns bad fit for any of the metrics, this will fall back to returning the best point based on raw observations.

TModelPredictArm is of the form:: ({metric_name: mean}, {metric_name_1: {metric_name_2: cov_1_2}})

Parameters:

experiment – Experiment, on which to identify best raw objective arm.
adapter – The Adapter to use to get the model predictions. If None, the best point will be extracted from the generator run of the latest trial.
optimization_config – Optional OptimizationConfig override, to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.

Returns:

Tuple of trial index, parameterization, and model predictions for it.

Given an experiment, identifies the arm that had the best raw objective, based on the data fetched from the experiment.

TModelPredictArm is of the form:: ({metric_name: mean}, {metric_name_1: {metric_name_2: cov_1_2}})

Note: This function will error with invalid inputs. If you would prefer for error logs rather than exceptions, use get_best_by_raw_objective_with_trial_index, which returns None if inputs are invalid.

Parameters:

experiment – Experiment, on which to identify best raw objective arm.
optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.

Returns:

Tuple of trial index, parameterization, and model predictions for it.

ax.service.utils.best_point.get_hypervolume_trace_of_outcomes_multi_objective(df_wide: DataFrame, optimization_config: MultiObjectiveOptimizationConfig, use_cumulative_hv: bool = True) → list[float][source]

Get hypervolume of the outcomes represented in df_wide.

Parameters:

df_wide – Dataframe with columns [“feasible”] + relevant metrics. This can come from reshaping the data that comes from Data.df.
optimization_config – A multi-objective optimization config with a MultiObjective (not a ScalarizedObjective). When objective thresholds are not provided, they are inferred using infer_reference_point on the Pareto frontier of the feasible observations.
use_cumulative_hv – If True, the hypervolume returned is the cumulative hypervolume of the points in each row. Otherwise, this is the hypervolume of each point.

Returns:

A list of hypervolumes, one for each row in df_wide.

Example: >>> optimization_config = MultiObjectiveOptimizationConfig( … objective=MultiObjective( … objectives=[ … Objective(metric=Metric(name=”m1”), minimize=False), … Objective(metric=Metric(name=”m2”), minimize=False), … ] … ), … objective_thresholds=[ … ObjectiveThreshold( … metric=Metric(name=”m1”), … bound=0.0, … relative=False, … op=ComparisonOp.GEQ, … ), … ObjectiveThreshold( … metric=Metric(name=”m2”), … bound=0.0, … relative=False, … op=ComparisonOp.GEQ, … ), … ], … ) >>> df_wide = pd.DataFrame.from_records( … [ … {“m1”: 0.0, “m2”: 0.0, “feasible”: True}, … {“m1”: 1.0, “m2”: 2.0, “feasible”: True}, … {“m1”: 2.0, “m2”: 1.0, “feasible”: False}, … {“m1”: 3.0, “m2”: 3.0, “feasible”: True}, … ] … ) >>> get_hypervolume_trace_of_outcomes_multi_objective( … df_wide=df_wide, … optimization_config=optimization_config, … use_cumulative_hv=True, … ) [0.0, 2.0, 2.0, 9.0] >>> >>> get_hypervolume_trace_of_outcomes_multi_objective( … df_wide=df_wide, … optimization_config=optimization_config, … use_cumulative_hv=False, … ) [0.0, 2.0, 0.0, 9.0]

ax.service.utils.best_point.get_pareto_optimal_parameters(experiment: Experiment, generation_strategy: GenerationStrategy, optimization_config: OptimizationConfig | None = None, trial_indices: Iterable[int] | None = None, use_model_predictions: bool = True) → dict[int, tuple[dict[str, None | str | bool | float | int], tuple[dict[str, float], dict[str, dict[str, float]] | None]]][source]

Identifies the best parameterizations tried in the experiment so far, using model predictions if use_model_predictions is true and using observed values from the experiment otherwise. By default, uses model predictions to account for observation noise.

NOTE: The format of this method’s output is as follows: { trial_index –> (parameterization, (means, covariances) }, where means are a dictionary of form { metric_name –> metric_mean } and covariances are a nested dictionary of form { one_metric_name –> { another_metric_name: covariance } }.

Parameters:

experiment – Experiment, from which to find Pareto-optimal arms.
generation_strategy – Generation strategy containing the adapter.
optimization_config – Optimization config to use in place of the one stored on the experiment.
trial_indices – Indices of trials for which to retrieve data. If None will retrieve data from all available trials.
use_model_predictions – Whether to extract the Pareto frontier using model predictions or directly observed values. If True, the metric means and covariances in this method’s output will also be based on model predictions and may differ from the observed values.

Returns:

the parameterization of the arm in that trial,
two-item tuple of metric means dictionary and covariance matrix
(model-predicted if use_model_predictions=True and observed otherwise).

Return type:

A mapping from trial index to the tuple of

ax.service.utils.best_point.get_tensor_converter_adapter(experiment: Experiment, data: Data | None = None) → TorchAdapter[source]

Constructs a minimal model for converting things to tensors.

Model fitting will instantiate all of the transforms but will not do any expensive (i.e. GP) fitting beyond that. The model will raise an error if it is used for predicting or generating.

Will work for any search space regardless of types of parameters.

Parameters:

experiment – Experiment.
data – Data for fitting the model.

Returns: A torch adapter with transforms set.

ax.service.utils.best_point.get_trace(experiment: Experiment, optimization_config: OptimizationConfig | None = None, include_status_quo: bool = False) → dict[int, float][source]

Compute the optimization trace at each iteration. Given an experiment and an optimization config, compute the performance at each iteration. For multi-objective, the performance is computed as the hypervolume. For single objective, the performance is computed as the best observed objective value.

For BOPE experiments, the utility of each trial is computed using the learned preference model from the PE_EXPERIMENT auxiliary experiment. The preference model is used to predict the utility of each trial’s metric values, and the trace represents the best predicted utility over time.

Infeasible points (that violate constraints) do not contribute to improvements in the optimization trace. If the first trial(s) are infeasible, the trace can start at inf or -inf.

An iteration here refers to a completed or early-stopped (batch) trial with complete metric data for all metrics in the optimization config. Trials with incomplete data, or trials that are abandoned or failed, are excluded from the trace.

Parameters:

experiment – The experiment to get the trace for.
optimization_config – Optimization config to use in place of the one stored on the experiment.
include_status_quo – If True, include status quo in the trace computation. If False (default), exclude status quo for compatibility with legacy behavior.

Returns:

A dict mapping trial index to performance value, ordered by trial index. Only trials with complete metric data are included.

ax.service.utils.best_point.get_trace_by_arm_pull_from_data(df: DataFrame, optimization_config: OptimizationConfig, use_cumulative_best: bool = True, experiment: Experiment | None = None) → DataFrame[source]

Get a trace of the objective value or hypervolume of outcomes.

An “arm pull” is the combination of a trial (index) and an arm. This function returns a single value for each arm pull, even if there are multiple arms per trial or if an arm is repeated in multiple trials.

For BOPE experiments, this function computes utility predictions using the learned preference model from the PE_EXPERIMENT auxiliary experiment.

Parameters:

df – Data in the format returned by Data.df, with a separate row for each trial index-arm name-metric.
optimization_config – OptimizationConfig to use to get the trace. Must not be in relative form.
use_cumulative_best – If True, the trace will be the cumulative best objective. Otherwise, the trace will be the value of each point.
experiment – Optional experiment object. Required for preference learning experiments to access the PE_EXPERIMENT auxiliary experiment.

Returns:

A DataFrame containing columns ‘trial_index’, ‘arm_name’, and “value”, where “value” is the value of the outcomes attained (or predicted utility for preference learning experiments).

ax.service.utils.best_point.get_values_of_outcomes_single_or_scalarized_objective(df_wide: DataFrame, objective: Objective) → ndarray[tuple[Any, ...], dtype[_ScalarT]][source]

Return a list with one entry for each row in df_wide according to the objective objective and whether the outcomes are feasible.

Whether higher or lower is better depends on objective.minimize (no absolute values are taken here).

The entry for any infeasible value will be infinity if the objective is to minimize and negative infinity if the objective is to maximize.

Example: >>> objective = Objective(metric=Metric(name=”m1”), minimize=True) >>> df_wide = pd.DataFrame.from_records( … [ … {“m1”: 2.0, “feasible”: True}, … {“m1”: 1.0, “feasible”: False}, … ] … ) >>> get_value_of_outcomes_single_or_scalarized_objective( … df_wide=df_wide, objective=objective … ) np.array([2.0, inf])

ax.service.utils.best_point.infer_reference_point_from_experiment(experiment: Experiment, data: Data) → list[OutcomeConstraint][source]

This functions is a wrapper around infer_reference_point to find the nadir point from the pareto front of an experiment. Aside from converting experiment to tensors, this wrapper transforms back and forth the objectives of the experiment so that they are appropriately used by infer_reference_point.

Parameters:: experiment – The experiment for which we want to infer the reference point.
Returns:: A list of objective thresholds representing the reference point.

ax.service.utils.best_point.is_row_feasible(df: DataFrame, optimization_config: OptimizationConfig, undetermined_value: bool | None = None) → Series[source]

Determine whether arms satisfy outcome constraints based on observed data.

Evaluates each arm’s feasibility by checking if its associated metrics’ 95% confidence intervals satisfy all outcome constraints. Returns False for arms where we are 95% confident that at least one constraint is violated, True for arms that satisfy all constraints, and undetermined_value for arms where feasibility cannot be conclusively determined.

Parameters:

df – DataFrame of arm data with required columns: “metric_name”, “mean”, “sem”, and “arm_name”. Each row represents a metric observation for a specific arm.
optimization_config – OptimizationConfig containing the outcome constraints to evaluate. Must have derelativized constraints.
undetermined_value – Value to return for arms where feasibility cannot be determined due to missing data. Defaults to None.

Returns:

True: Arm satisfies all outcome constraints
False: Arm violates at least one outcome constraint (95% confidence)
undetermined_value: Feasibility cannot be determined (missing data or relative constraints present)

Return type:

Series of boolean or None values indexed by df.index, where

ax.service.utils.best_point_utils.get_best_trial_indices(experiment: Experiment, optimization_config: OptimizationConfig, generation_strategy: GenerationStrategy | None = None, trial_indices: list[int] | None = None, use_model_predictions: bool = False) → list[int][source]

Get the trial indices of the best trial(s) based on optimization type.

Note: All best point methods used here only consider in-sample points (arms that have already been run on the experiment). They do not return parameters that have not been tried yet.

Parameters:

experiment – The experiment to get best trials from.
optimization_config – The optimization config for the experiment.
generation_strategy – The generation strategy, required for MOO or model predictions.
trial_indices – The trial indices to consider when finding best trials. This should be pre-filtered by trial status. If None, considers all trials.
use_model_predictions – If True, use model predictions instead of raw observations for best trial selection.

Returns:

A list of trial indices representing the best trial(s). For SOO, this contains at most one index. For MOO, this contains all Pareto optimal trial indices.

ax.service.utils.best_point_utils.select_baseline_name_default_first_trial(experiment: Experiment, baseline_arm_name: str | None) → tuple[str, bool][source]

Choose a baseline arm from arms on the experiment. Logic: 1. If baseline_arm_name provided, validate that arm exists

and return that arm name.

If experiment.status_quo is set, return its arm name.
If there is at least one trial on the experiment, use the first trial’s first arm as the baseline.
Error if 1-3 all don’t apply.

Returns:

baseline arm name (str): true when baseline selected from first arm of experiment (bool)

raise ValueError if no valid baseline found

Return type:

Tuple

Instantiation

Bases: object

Class for representing fixed features via the Service API.

parameters: dict[str, None | str | bool | float | int]

trial_index: int | None = None

class ax.service.utils.instantiation.InstantiationBase[source]

Bases: object

This is a lightweight stateless class that bundles together instantiation utils. It is used both on its own and as a mixin to AxClient, with the intent that these methods can be overridden by its subclasses for specific use cases.

static build_objective_threshold(objective: str, objective_properties: ObjectiveProperties) → str[source]

Constructs constraint string for an objective threshold interpretable by make_experiment()

Parameters:

objective – Name of the objective
objective_properties – Object containing: minimize: Whether this experiment represents a minimization problem. threshold: The bound in the objective’s threshold constraint.

classmethod build_objective_thresholds(objectives: dict[str, ObjectiveProperties]) → list[str][source]

Construct a list of constraint string for an objective thresholds interpretable by make_experiment()

Parameters:: objectives – Mapping of name of the objective to Object containing: minimize: Whether this experiment represents a minimization problem. threshold: The bound in the objective’s threshold constraint.

static constraint_from_str(representation: str, parameters: dict[str, Parameter]) → ParameterConstraint[source]

Parse string representation of a parameter constraint.

Parameters:

representation – String representation of the parameter constraint.
parameters – Dictionary of parameter names to parameter objects or configs.

Returns:

An instantiated ParameterConstraint, either an OrderConstraint or a ParameterConstraint, representing a linear constraint.

Instantiation wrapper that allows for Ax Experiment creation without importing or instantiating any Ax classes.

Parameters:

parameters – List of dictionaries representing parameters in the experiment search space. Required elements in the dictionaries are: 1. “name” (name of parameter, string), 2. “type” (type of parameter: “range”, “fixed”, “choice”, or “derived”, string), and one of the following: 3a. “bounds” for range parameters (list of two values, lower bound first), 3b. “values” for choice parameters (list of values), or 3c. “value” for fixed parameters (single value). 3d. “expression_str” for derived parameters (string containing the definition of the derived parameter). Optional elements are: 1. “log_scale” (for float-valued range parameters, bool), 2. “value_type” (to specify type that values of this parameter should take; expects “float”, “int”, “bool” or “str”), 3. “is_fidelity” (bool) and “target_value” (float) for fidelity parameters, 4. “is_ordered” (bool) for choice parameters, 5. “is_task” (bool) for task parameters, and 6. “digits” (int) for float-valued range parameters.
name – Name of the experiment to be created.
parameter_constraints – List of string representation of parameter constraints, such as “x3 >= x4” or “-x3 + 2*x4 - 3.5*x5 >= 2”. For the latter constraints, any number of arguments is accepted, and acceptable operators are “<=” and “>=”.
outcome_constraints – List of string representation of outcome constraints of form “metric_name >= bound”, like “m1 <= 3.”
status_quo – Parameterization of the current state of the system. If set, this will be added to each trial to be evaluated alongside test configurations.
experiment_type – String indicating type of the experiment (e.g. name of a product in which it is used), if any.
tracking_metric_names – Names of additional tracking metrics not used for optimization.
metric_definitions – A mapping of metric names to extra kwargs to pass to that metric
objectives – Mapping from an objective name to “minimize” or “maximize” representing the direction for that objective.
objective_thresholds – A list of objective threshold constraints for multi- objective optimization, in the same string format as outcome_constraints argument.
support_intermediate_data – Whether trials may report metrics results for incomplete runs.
immutable_search_space_and_opt_config – Whether it’s possible to update the search space and optimization config on this experiment after creation. Defaults to True. If set to True, we won’t store or load copies of the search space and optimization config on each generator run, which will improve storage performance.
auxiliary_experiments_by_purpose – Dictionary of auxiliary experiments for different use cases (e.g., transfer learning).
default_trial_type – The default trial type if multiple trial types are intended to be used in the experiment. If specified, a MultiTypeExperiment will be created. Otherwise, a single-type Experiment will be created.
default_runner – The default runner in this experiment. This only applies to MultiTypeExperiment (when default_trial_type is specified).
is_test – Whether this experiment will be a test experiment (useful for marking test experiments in storage etc). Defaults to False.

static make_fixed_observation_features(fixed_features: FixedFeatures) → ObservationFeatures[source]

Construct ObservationFeatures from FixedFeatures.

Parameters:: fixed_features – The fixed features for generation.
Returns:: The new ObservationFeatures object.

classmethod make_objective_thresholds(objective_thresholds: list[str], status_quo_defined: bool, metric_definitions: dict[str, dict[str, Any]] | None = None) → list[OutcomeConstraint][source]

classmethod make_objectives(objectives: dict[str, str], metric_definitions: dict[str, dict[str, Any]] | None = None) → list[Objective][source]

classmethod make_optimization_config(objectives: dict[str, str], objective_thresholds: list[str], outcome_constraints: list[str], status_quo_defined: bool, metric_definitions: dict[str, dict[str, Any]] | None = None) → OptimizationConfig[source]

classmethod make_optimization_config_from_properties(objectives: dict[str, ObjectiveProperties] | None = None, outcome_constraints: list[str] | None = None, metric_definitions: dict[str, dict[str, Any]] | None = None, status_quo_defined: bool = False) → OptimizationConfig | None[source]

Makes optimization config based on ObjectiveProperties objects

Parameters:

objectives – Mapping from an objective name to object containing: minimize: Whether this experiment represents a minimization problem. threshold: The bound in the objective’s threshold constraint.
outcome_constraints – List of string representation of outcome constraints of form “metric_name >= bound”, like “m1 <= 3.”
status_quo_defined – bool for whether the experiment has a status quo
metric_definitions – A mapping of metric names to extra kwargs to pass to that metric

classmethod make_outcome_constraints(outcome_constraints: list[str], status_quo_defined: bool, metric_definitions: dict[str, dict[str, Any]] | None = None) → list[OutcomeConstraint][source]

classmethod make_search_space(parameters: list[dict[str, None | str | bool | float | int | Sequence[None | str | bool | float | int] | dict[str, list[str]]]], parameter_constraints: list[str] | None) → SearchSpace[source]

classmethod objective_threshold_constraint_from_str(representation: str, metric_definitions: dict[str, dict[str, Any]] | None = None) → ObjectiveThreshold[source]

static optimization_config_from_objectives(objectives: list[Objective], objective_thresholds: list[OutcomeConstraint], outcome_constraints: list[OutcomeConstraint]) → OptimizationConfig[source]

Parse objectives and constraints to define optimization config.

The resulting optimization config will be regular single-objective config if objectives is a list of one element and a multi-objective config otherwise.

NOTE: If passing in multiple objectives, objective_thresholds must be a non-empty list defining constraints for each objective.

classmethod outcome_constraint_from_str(representation: str, metric_definitions: dict[str, dict[str, Any]] | None = None) → OutcomeConstraint[source]: Parse string representation of an outcome constraint.

classmethod parameter_from_json(representation: dict[str, None | str | bool | float | int | Sequence[None | str | bool | float | int] | dict[str, list[str]]]) → Parameter[source]: Instantiate a parameter from JSON representation.

class ax.service.utils.instantiation.MetricObjective(*values)[source]

Bases: Enum

MAXIMIZE = 2

MINIMIZE = 1

class ax.service.utils.instantiation.ObjectiveProperties(minimize: bool, threshold: float | None = None)[source]

Bases: object

Class that holds properties of objective functions. Can be used to define an the objectives argument of ax_client.create_experiment, e.g.:

ax_client.create_experiment(
name=”moo_experiment”, parameters=[…], objectives={

# threshold arguments are optional “a”: ObjectiveProperties(minimize=False, threshold=ref_point[0]), “b”: ObjectiveProperties(minimize=False, threshold=ref_point[1]),

},

)

Parameters:

minimize (-) – Boolean indicating whether the objective is to be minimized or maximized.
threshold (-) – Optional float representing the smallest objective value (resp. largest if minimize=True) that is considered valuable in the context of multi-objective optimization. In BoTorch and in the literature, this is also known as an element of the reference point vector that defines the hyper-volume of the Pareto front.

minimize: bool

threshold: float | None = None

ax.service.utils.instantiation.logger: Logger = <Logger ax.service.utils.instantiation (INFO)>: Utilities for RESTful-like instantiation of Ax classes needed in AxClient.

Reporting

ax.service.utils.report_utils.compare_to_baseline_impl(comparison_list: list[tuple[str, bool, str, float, str, float]]) → str | None[source]: Implementation of compare_to_baseline, taking in a list of arm comparisons. Can be used directly with the output of ‘maybe_extract_baseline_comparison_values’

ax.service.utils.report_utils.construct_comparison_message(objective_name: str, objective_minimize: bool, baseline_arm_name: str, baseline_value: float, comparison_arm_name: str, comparison_value: float, digits: int | None = None) → str | None[source]

Construct a message comparing a comparison arm to a baseline arm.

Parameters:

objective_name – Name of the objective metric being compared.
objective_minimize – Whether the objective is being minimized.
baseline_arm_name – Name of the baseline arm.
baseline_value – Value of the objective metric for the baseline arm.
comparison_arm_name – Name of the arm being compared to baseline.
comparison_value – Value of the objective metric for the comparison arm.
digits – Number of decimal places to display. Defaults to 2 if not provided.

Returns:

A formatted message string describing the percent improvement if the comparison arm beats the baseline, or None if no improvement was found or the baseline value is zero.

ax.service.utils.report_utils.exp_to_df(exp: Experiment, metrics: list[Metric] | None = None, run_metadata_fields: list[str] | None = None, trial_properties_fields: list[str] | None = None, trial_attribute_fields: list[str] | None = None, additional_fields_callables: None | dict[str, Callable[[Experiment], dict[int, str | float]]] = None, always_include_field_columns: bool = False, show_relative_metrics: bool = False, **kwargs: Any) → DataFrame[source]

Transforms an experiment to a DataFrame with rows keyed by trial_index and arm_name, metrics pivoted into one row. If the pivot results in more than one row per arm (or one row per arm * map_keys combination if map_keys are present), results are omitted and warning is produced. Only supports Experiment.

Transforms an Experiment into a pd.DataFrame.

Parameters:

exp – An Experiment that may have pending trials.
metrics – Override list of metrics to return. Return all metrics if None.
run_metadata_fields – Fields to extract from trial.run_metadata for trial in experiment.trials. If there are multiple arms per trial, these fields will be replicated across the arms of a trial.
trial_properties_fields – Fields to extract from trial._properties for trial in experiment.trials. If there are multiple arms per trial, these fields will be replicated across the arms of a trial. Output columns names will be prepended with "trial_properties_".
trial_attribute_fields – Fields to extract from trial attributes for each trial in experiment.trials. If there are multiple arms per trial, these fields will be replicated across the arms of a trial.
additional_fields_callables – A dictionary of field names to callables, with each being a function from experiment to a trials_dict of the form {trial_index: value}. An example of a custom callable like this is the function compute_maximum_map_values.
always_include_field_columns – If True, even if all trials have missing values, include field columns anyway. Such columns are by default omitted (False).
show_relative_metrics – If True, show % metric changes relative to the provided status quo arm. If no status quo arm is provided, raise a warning and show raw metric values. If False, show raw metric values (default).

Returns:

A dataframe of inputs, metadata and metrics by trial and arm (and map_keys, if present). If no trials are available, returns an empty dataframe. If no metric ouputs are available, returns a dataframe of inputs and metadata. Columns include:

trial_index

arm_name

trial_status

generation_method

any elements of exp.runner.run_metadata_report_keys that are present in the trial.run_metadata of each trial

one column per metric (named after the metric.name)

one column per parameter (named after the parameter.name)

Return type:

DataFrame

ax.service.utils.report_utils.get_figure_and_callback(plot_fn: Callable[[Orchestrator], go.Figure]) → tuple[go.Figure, Callable[[Orchestrator], None]][source]

Produce a figure and a callback for updating the figure in place.

A likely use case is that plot_fn takes a Orchestrator instance and returns a plotly Figure. Then get_figure_and_callback will produce a figure and callback that updates that figure according to plot_fn when the callback is passed to orchestrator.run_n_trials or orchestrator.run_all_trials.

Parameters:: plot_fn – A function for producing a Plotly figure from a orchestrator. If plot_fn raises a RuntimeError, the update will be skipped and optimization will proceed.

Example

>>> def _plot(orchestrator:Orchestrator):
>>>     standard_plots = get_standard_plots(orchestrator.experiment)
>>>     return standard_plots[0]
>>>
>>> fig, callback = get_figure_and_callback(_plot)

ax.service.utils.report_utils.get_standard_plots(experiment: Experiment, model: Adapter | None, data: Data | None = None, true_objective_metric_name: str | None = None, early_stopping_strategy: BaseEarlyStoppingStrategy | None = None, limit_points_per_plot: int | None = None, global_sensitivity_analysis: bool = True) → list[plotly.graph_objects.Figure][source]

Extract standard plots for single-objective optimization.

Extracts a list of plots from an Experiment and Adapter of general interest to an Ax user. Currently not supported are - TODO: multi-objective optimization - TODO: ChoiceParameter plots

Parameters:

experiment (-) – The Experiment from which to obtain standard plots.
model (-) – The Adapter used to suggest trial parameters.
true_objective_metric_name (-) – Name of the metric to use as the true objective.
early_stopping_strategy (-) – Early stopping strategy used throughout the experiment; used for visualizing when curves are stopped.
limit_points_per_plot (-) – Limit the number of points used per metric in each curve plot. Passed to _get_curve_plot_dropdown.
global_sensitivity_analysis (-) – If True, plot total Variance-based sensitivity analysis for the model parameters. If False, plot sensitivities based on GP kernel lengthscales. Defaults to True.

Returns:

a plot of objective value vs. trial index, to show experiment progression
a plot of objective value vs. range parameter values, only included if the model associated with generation_strategy can create predictions. This consists of:
- a plot_slice plot if the search space contains one range parameter
- an interact_contour plot if the search space contains multiple range parameters

ax.service.utils.report_utils.maybe_extract_baseline_comparison_values(experiment: Experiment, optimization_config: OptimizationConfig | None, comparison_arm_names: list[str] | None, baseline_arm_name: str | None) → list[tuple[str, bool, str, float, str, float]] | None[source]

Extracts the baseline values from the experiment, for use in comparing the baseline arm to the optimal results. Requires the user specifies the names of the arms to compare to.

Returns:: (metric_name, minimize, baseline_arm_name, baseline_value, comparison_arm_name, comparison_arm_value, )
Return type:: List of tuples containing

ax.service.utils.report_utils.pareto_frontier_scatter_2d_plotly(experiment: Experiment, metric_names: tuple[str, str], reference_point: tuple[float, float] | None = None, minimize: bool | tuple[bool, bool] | None = None) → plotly.graph_objects.Figure[source]

ax.service.utils.report_utils.warn_if_unpredictable_metrics(experiment: Experiment, generation_strategy: GenerationStrategy, model_fit_threshold: float, metric_names: list[str] | None = None, model_fit_metric_name: str = 'coefficient_of_determination') → str | None[source]

Warn if any optimization config metrics are considered unpredictable, i.e., their coefficient of determination is less than model_fit_threshold. :param experiment: The experiment containing the data and optimization_config.

If there is no optimization config, this function checks all metrics attached to the experiment.

Parameters:

generation_strategy – The generation strategy containing the model.
model_fit_threshold – If a model’s coefficient of determination is below this threshold, that metric is considered unpredictable.
metric_names – If specified, only check these metrics.
model_fit_metric_name – Name of the metric to apply the model fit threshold to.

Returns:

A string warning the user about unpredictable metrics, if applicable.

EarlyStopping

ax.service.utils.early_stopping.get_early_stopping_metrics(experiment: Experiment, early_stopping_strategy: BaseEarlyStoppingStrategy | None) → list[str][source]: A helper function that returns a list of metric names on which a given early_stopping_strategy is operating.

ax.service.utils.early_stopping.should_stop_trials_early(early_stopping_strategy: BaseEarlyStoppingStrategy | None, trial_indices: set[int], experiment: Experiment, current_node: GenerationNode | None = None) → dict[int, str | None][source]

Evaluate whether to early-stop running trials.

Parameters:

early_stopping_strategy – A BaseEarlyStoppingStrategy that determines whether a trial should be stopped given the state of an experiment.
trial_indices – Indices of trials to consider for early stopping.
experiment – The experiment containing the trials.
current_node – The current GenerationNode on the GenerationStrategy used to generate trials for the Experiment. Early stopping strategies may utilize components of the current node when making stopping decisions.

Returns:

A dictionary mapping trial indices that should be early stopped to (optional) messages with the associated reason.