botorch.cross_validation

Cross-validation utilities using batch evaluation mode.

class botorch.cross_validation.CVFolds(train_X, test_X, train_Y, test_Y, train_Yvar, test_Yvar)[source]

Bases: NamedTuple

Create new instance of CVFolds(train_X, test_X, train_Y, test_Y, train_Yvar, test_Yvar)

Parameters:

  • train_X (Tensor)

  • test_X (Tensor)

  • train_Y (Tensor)

  • test_Y (Tensor)

  • train_Yvar (Tensor | None)

  • test_Yvar (Tensor | None)

train_X: Tensor

Alias for field number 0

test_X: Tensor

Alias for field number 1

train_Y: Tensor

Alias for field number 2

test_Y: Tensor

Alias for field number 3

train_Yvar: Tensor | None

Alias for field number 4

test_Yvar: Tensor | None

Alias for field number 5

class botorch.cross_validation.CVResults(model, posterior, observed_Y, observed_Yvar=None)[source]

Bases: NamedTuple

Results from cross-validation.

This named tuple contains the cross-validation predictions and observed values. For both batch_cross_validation and efficient_loo_cv, the posterior field contains the predictive distribution with mean and variance accessible via posterior.mean and posterior.variance.

For batch_cross_validation, the posterior has shape n x 1 x m where n is the number of folds, 1 is the single held-out point per fold, and m is the number of outputs.

For efficient_loo_cv, the posterior has the same shape structure to maintain consistency, though the underlying distribution is constructed from the efficient LOO formulas rather than from separate model fits.

NOTE: When untransform=True is used with a nonlinear outcome transform (e.g., Log), the posterior will be a TransformedPosterior rather than a GPyTorchPosterior. For ensemble models, it will be a GaussianMixturePosterior.

Create new instance of CVResults(model, posterior, observed_Y, observed_Yvar)

Parameters:
model: GPyTorchModel

Alias for field number 0

posterior: Posterior

Alias for field number 1

observed_Y: Tensor

Alias for field number 2

observed_Yvar: Tensor | None

Alias for field number 3

botorch.cross_validation.gen_loo_cv_folds(train_X, train_Y, train_Yvar=None)[source]

Generate LOO CV folds w.r.t. to n.

Parameters:

  • train_X (Tensor) – A n x d or batch_shape x n x d (batch mode) tensor of training features.

  • train_Y (Tensor) – A n x (m) or batch_shape x n x (m) (batch mode) tensor of training observations.

  • train_Yvar (Tensor | None) – An n x (m) or batch_shape x n x (m) (batch mode) tensor of observed measurement noise.

Returns:

  • train_X: A n x (n-1) x d or batch_shape x n x (n-1) x d tensor of training features.

  • test_X: A n x 1 x d or batch_shape x n x 1 x d tensor of test features.

  • train_Y: A n x (n-1) x m or batch_shape x n x (n-1) x m tensor of training observations.

  • test_Y: A n x 1 x m or batch_shape x n x 1 x m tensor of test observations.

  • train_Yvar: A n x (n-1) x m or batch_shape x n x (n-1) x m tensor of observed measurement noise.

  • test_Yvar: A n x 1 x m or batch_shape x n x 1 x m tensor of observed measurement noise.

Return type:

CVFolds NamedTuple with the following fields

Example

>>> train_X = torch.rand(10, 1)
>>> train_Y = torch.rand_like(train_X)
>>> cv_folds = gen_loo_cv_folds(train_X, train_Y)
>>> cv_folds.train_X.shape
torch.Size([10, 9, 1])
botorch.cross_validation.batch_cross_validation(model_cls, mll_cls, cv_folds, fit_args=None, observation_noise=False, model_init_kwargs=None)[source]

Perform cross validation by using GPyTorch batch mode.

WARNING: This function is currently very memory inefficient; use it only

for problems of small size.

Parameters:

  • model_cls (type[GPyTorchModel]) – A GPyTorchModel class. This class must initialize the likelihood internally. Note: Multi-task GPs are not currently supported.

  • mll_cls (type[MarginalLogLikelihood]) – A MarginalLogLikelihood class.

  • cv_folds (CVFolds) – A CVFolds tuple. For LOO-CV with n training points, the leading dimension of size n represents the n folds (batch dimension), e.g., cv_folds.train_X has shape n x (n-1) x d and cv_folds.test_X has shape n x 1 x d. This batch structure enables fitting n independent GPs simultaneously.

  • fit_args (dict[str, Any] | None) – Arguments passed along to fit_gpytorch_mll.

  • model_init_kwargs (dict[str, Any] | None) – Keyword arguments passed to the model constructor.

  • observation_noise (bool)

Returns:

A CVResults tuple with the following fields

  • model: GPyTorchModel for batched cross validation

  • posterior: GPyTorchPosterior where the mean has shape n x 1 x m or batch_shape x n x 1 x m

  • observed_Y: A n x 1 x m or batch_shape x n x 1 x m tensor of observations.

  • observed_Yvar: A n x 1 x m or batch_shape x n x 1 x m tensor of observed measurement noise.

Return type:

CVResults

Example

>>> import torch
>>> from botorch.cross_validation import (
...     batch_cross_validation, gen_loo_cv_folds
... )
>>>
>>> from botorch.models import SingleTaskGP
>>> from botorch.models.transforms.input import Normalize
>>> from botorch.models.transforms.outcome import Standardize
>>> from gpytorch.mlls import ExactMarginalLogLikelihood

>>> train_X = torch.rand(10, 1)
>>> train_Y = torch.rand_like(train_X)
>>> cv_folds = gen_loo_cv_folds(train_X, train_Y)
>>> input_transform = Normalize(d=train_X.shape[-1])
>>>
>>> cv_results = batch_cross_validation(
...    model_cls=SingleTaskGP,
...    mll_cls=ExactMarginalLogLikelihood,
...    cv_folds=cv_folds,
...    model_init_kwargs={
...        "input_transform": input_transform,
...    },
... )
botorch.cross_validation.loo_cv(model, observation_noise=True, untransform=True)[source]

Compute efficient Leave-One-Out cross-validation for a GP model.

This is a high-level convenience function that automatically dispatches to the appropriate LOO CV implementation based on the model type:

  • For ensemble models (_is_ensemble=True): Uses ensemble_loo_cv which returns a GaussianMixturePosterior with both per-member and mixture statistics.

  • For standard GP models: Uses efficient_loo_cv which returns a GPyTorchPosterior with the LOO predictive distributions.

Both implementations use efficient O(n³) matrix algebra rather than the naive O(n⁴) approach of refitting models for each fold.

NOTE: This function does not refit the model to each LOO fold. The model hyperparameters are kept fixed, providing a fast approximation to full LOO CV. For models where hyperparameter changes are significant, consider using batch_cross_validation instead.

NOTE: The untransform parameter defaults to True, which means results are returned in the original outcome space. Callers that previously relied on results in the model’s internal (transformed) space should pass untransform=False explicitly.

Parameters:

  • model (GPyTorchModel) – A fitted GPyTorchModel. The model type determines which LOO CV implementation is used.

  • observation_noise (bool) – If True (default), return the posterior predictive variance (including observation noise). If False, return the posterior variance of the latent function (excluding observation noise). The posterior variance is computed by subtracting the observation noise from the posterior predictive variance.

  • untransform (bool) – If True (default), untransform the LOO predictions and observed values back to the original outcome space when the model has an outcome transform (e.g., Standardize). This makes the results consistent with model.posterior() and batch_cross_validation. If False, return results in the model’s internal (transformed) space.

Returns:

A named tuple containing:

  • model: The fitted GP model.

  • posterior: The LOO predictive distributions. For ensemble models, this is a GaussianMixturePosterior; otherwise, it’s a GPyTorchPosterior.

  • observed_Y: The observed Y values.

  • observed_Yvar: The observed noise variances (if applicable).

Return type:

CVResults

Example

>>> import torch
>>> from botorch.cross_validation import loo_cv
>>> from botorch.models import SingleTaskGP
>>> from botorch.fit import fit_gpytorch_mll
>>> from gpytorch.mlls import ExactMarginalLogLikelihood
>>>
>>> train_X = torch.rand(20, 2, dtype=torch.float64)
>>> train_Y = torch.sin(train_X).sum(dim=-1, keepdim=True)
>>> model = SingleTaskGP(train_X, train_Y)
>>> mll = ExactMarginalLogLikelihood(model.likelihood, model)
>>> fit_gpytorch_mll(mll)
>>> loo_results = loo_cv(model)
>>> loo_results.posterior.mean.shape
torch.Size([20, 1, 1])

See also

  • efficient_loo_cv: Direct access to the standard GP implementation.

  • ensemble_loo_cv: Direct access to the ensemble model implementation.

  • batch_cross_validation: Full LOO CV with model refitting.

botorch.cross_validation.efficient_loo_cv(model, observation_noise=True, untransform=True)[source]

Compute efficient Leave-One-Out cross-validation for a GP model.

NOTE: This function does not refit the model to each LOO fold, in contrast to batch_cross_validation. This is a memory- and compute-efficient way to compute LOO, but it does not account for potential changes in the model parameters due to the removal of a single observation. This is typically ok in cases with a lot of data, but can result in substantial differences (typically over-estimating performance) in the low data regime.

This function leverages a well-known linear algebraic identity to compute all LOO predictive distributions in O(n^3) time, compared to the naive approach which requires O(n^4) time (O(n^3) per fold for n folds).

The efficient LOO formulas for GPs are:

\[ \begin{align}\begin{aligned}\mu_{LOO,i} = y_i - \frac{[K^{-1}(y - \mu)]_i}{[K^{-1}]_{ii}}\\\sigma^2_{LOO,i} = \frac{1}{[K^{-1}]_{ii}}\end{aligned}\end{align} \]

where K is the covariance matrix including observation noise. This gives the posterior predictive variance (including noise). To get the posterior variance (excluding noise), we subtract the observation noise:

\[\sigma^2_{posterior,i} = \sigma^2_{LOO,i} - \sigma^2_{noise}\]

NOTE: This function assumes the model has already been fitted and that the model’s forward method returns a MultivariateNormal distribution.

Parameters:

  • model (GPyTorchModel) – A fitted GPyTorchModel whose forward method returns a MultivariateNormal distribution.

  • observation_noise (bool) – If True (default), return the posterior predictive variance (including observation noise). If False, return the posterior variance of the latent function (excluding observation noise).

  • untransform (bool) – If True (default), untransform the LOO predictions and observed values back to the original outcome space when the model has an outcome transform (e.g., Standardize). This makes the results consistent with model.posterior() and batch_cross_validation. If False, return results in the model’s internal (transformed) space.

Returns:

A named tuple containing:

  • model: The fitted GP model.

  • posterior: The LOO predictive distributions (typically a GPyTorchPosterior; with nonlinear outcome transforms like Log, a TransformedPosterior). The posterior mean and variance have shape n x 1 x m or batch_shape x n x 1 x m, matching the structure of batch_cross_validation (n folds, 1 held-out point per fold, m outputs). The underlying distribution has diagonal covariance since LOO predictions at different held-out points are computed independently.

  • observed_Y: The observed Y values with shape n x 1 x m or batch_shape x n x 1 x m.

  • observed_Yvar: The observed noise variances (if provided) with shape n x 1 x m or batch_shape x n x 1 x m.

Return type:

CVResults

Example

>>> import torch
>>> from botorch.cross_validation import efficient_loo_cv
>>> from botorch.models import SingleTaskGP
>>> from botorch.fit import fit_gpytorch_mll
>>> from gpytorch.mlls import ExactMarginalLogLikelihood
>>>
>>> train_X = torch.rand(20, 2, dtype=torch.float64)
>>> train_Y = torch.sin(train_X).sum(dim=-1, keepdim=True)
>>> model = SingleTaskGP(train_X, train_Y)
>>> mll = ExactMarginalLogLikelihood(model.likelihood, model)
>>> fit_gpytorch_mll(mll)
>>> loo_results = efficient_loo_cv(model)
>>> loo_results.posterior.mean.shape
torch.Size([20, 1, 1])
botorch.cross_validation.ensemble_loo_cv(model, observation_noise=True, untransform=True)[source]

Compute efficient LOO cross-validation for ensemble models.

This function computes Leave-One-Out cross-validation for ensemble models like SaasFullyBayesianSingleTaskGP. For these models, the forward method returns a MultivariateNormal with a batch dimension containing statistics for all models in the ensemble.

The LOO predictions from each ensemble member form a Gaussian mixture. This function returns a CVResults with a GaussianMixturePosterior that provides both per-member statistics (via posterior.mean and posterior.variance) and aggregated mixture statistics (via posterior.mixture_mean and posterior.mixture_variance).

The mixture statistics are computed using the law of total variance:

\[ \begin{align}\begin{aligned}\mu_{mix} = \frac{1}{K} \sum_{k=1}^{K} \mu_k\\\sigma^2_{mix} = \frac{1}{K} \sum_{k=1}^{K} \sigma^2_k + \frac{1}{K} \sum_{k=1}^{K} \mu_k^2 - \mu_{mix}^2\end{aligned}\end{align} \]

where K is the number of ensemble members.

NOTE: This function assumes the model has already been fitted (e.g., using fit_fully_bayesian_model_nuts) and that the model is an ensemble model with _is_ensemble = True.

Parameters:

  • model (GPyTorchModel) – An ensemble GPyTorchModel (e.g., SaasFullyBayesianSingleTaskGP) whose forward method returns a MultivariateNormal distribution with a batch dimension for ensemble members.

  • observation_noise (bool) – If True (default), return the posterior predictive variance (including observation noise). If False, return the posterior variance of the latent function (excluding observation noise).

  • untransform (bool) – If True (default), untransform the LOO predictions and observed values back to the original outcome space when the model has an outcome transform (e.g., Standardize). This makes the results consistent with model.posterior() and batch_cross_validation. If False, return results in the model’s internal (transformed) space.

Returns:

A named tuple containing:

  • model: The fitted ensemble GP model.

  • posterior: A GaussianMixturePosterior with per-member shape n x num_models x 1 x m. Access per-member statistics via posterior.mean and posterior.variance, and mixture statistics via posterior.mixture_mean and posterior.mixture_variance.

  • observed_Y: The observed Y values with shape n x num_models x 1 x m, matching the posterior layout so that element-wise operations (e.g., posterior.mean - observed_Y) work correctly.

  • observed_Yvar: The observed noise variances (if provided) with the same shape as observed_Y.

Return type:

CVResults

Example

>>> import torch
>>> from botorch.cross_validation import ensemble_loo_cv
>>> from botorch.models.fully_bayesian import SaasFullyBayesianSingleTaskGP
>>> from botorch.models.fully_bayesian import fit_fully_bayesian_model_nuts
>>>
>>> train_X = torch.rand(20, 2, dtype=torch.float64)
>>> train_Y = torch.sin(train_X).sum(dim=-1, keepdim=True)
>>> model = SaasFullyBayesianSingleTaskGP(train_X, train_Y)
>>> fit_fully_bayesian_model_nuts(model, warmup_steps=64, num_samples=32)
>>> loo_results = ensemble_loo_cv(model)
>>> loo_results.posterior.mean.shape  # Per-member means
torch.Size([20, 32, 1, 1])
>>> loo_results.posterior.mixture_mean.shape  # Aggregated mixture mean
torch.Size([20, 1, 1])