# Table of Contents - [OSS Vizier Documentation — Open Source Vizier documentation](#oss-vizier-documentation-open-source-vizier-documentation) - [OSS Vizier Documentation — Open Source Vizier documentation](#oss-vizier-documentation-open-source-vizier-documentation) - [Advanced Topics — Open Source Vizier documentation](#advanced-topics-open-source-vizier-documentation) - [Highlights — Open Source Vizier documentation](#highlights-open-source-vizier-documentation) - [Guides — Open Source Vizier documentation](#guides-open-source-vizier-documentation) - [API Reference — Open Source Vizier documentation](#api-reference-open-source-vizier-documentation) - [Vizier Basics — Open Source Vizier documentation](#vizier-basics-open-source-vizier-documentation) - [Search Spaces — Open Source Vizier documentation](#search-spaces-open-source-vizier-documentation) - [Media — Open Source Vizier documentation](#media-open-source-vizier-documentation) - [Applications — Open Source Vizier documentation](#applications-open-source-vizier-documentation) - [API Reference — Open Source Vizier documentation](#api-reference-open-source-vizier-documentation) - [Advanced Topics — Open Source Vizier documentation](#advanced-topics-open-source-vizier-documentation) - [Applications — Open Source Vizier documentation](#applications-open-source-vizier-documentation) - [Highlights — Open Source Vizier documentation](#highlights-open-source-vizier-documentation) - [Distributed Vizier — Open Source Vizier documentation](#distributed-vizier-open-source-vizier-documentation) - [Guides — Open Source Vizier documentation](#guides-open-source-vizier-documentation) - [Converters — Open Source Vizier documentation](#converters-open-source-vizier-documentation) - [Media — Open Source Vizier documentation](#media-open-source-vizier-documentation) - [Supported Algorithms — Open Source Vizier documentation](#supported-algorithms-open-source-vizier-documentation) - [Bayesian Optimization Modeling — Open Source Vizier documentation](#bayesian-optimization-modeling-open-source-vizier-documentation) - [Unknown](#unknown) - [Unknown](#unknown) - [Frequently Used Import Targets — Open Source Vizier documentation](#frequently-used-import-targets-open-source-vizier-documentation) - [Predictors — Open Source Vizier documentation](#predictors-open-source-vizier-documentation) - [Creating Benchmarks — Open Source Vizier documentation](#creating-benchmarks-open-source-vizier-documentation) - [Benchmarking with Ray — Open Source Vizier documentation](#benchmarking-with-ray-open-source-vizier-documentation) - [Running Benchmarks — Open Source Vizier documentation](#running-benchmarks-open-source-vizier-documentation) - [Analyzing Benchmarks — Open Source Vizier documentation](#analyzing-benchmarks-open-source-vizier-documentation) - [Unknown](#unknown) - [Unknown](#unknown) - [Bayesian Optimization Modeling — Open Source Vizier documentation](#bayesian-optimization-modeling-open-source-vizier-documentation) - [Vizier Basics — Open Source Vizier documentation](#vizier-basics-open-source-vizier-documentation) - [Bijectors — Open Source Vizier documentation](#bijectors-open-source-vizier-documentation) - [Frequently Used Import Targets — Open Source Vizier documentation](#frequently-used-import-targets-open-source-vizier-documentation) - [Distributed Vizier — Open Source Vizier documentation](#distributed-vizier-open-source-vizier-documentation) - [Designers — Open Source Vizier documentation](#designers-open-source-vizier-documentation) - [Bijectors — Open Source Vizier documentation](#bijectors-open-source-vizier-documentation) - [Search Spaces — Open Source Vizier documentation](#search-spaces-open-source-vizier-documentation) - [PSD kernels — Open Source Vizier documentation](#psd-kernels-open-source-vizier-documentation) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [OSS Vizier as a Backend — Open Source Vizier documentation](#oss-vizier-as-a-backend-open-source-vizier-documentation) - [PSD kernels — Open Source Vizier documentation](#psd-kernels-open-source-vizier-documentation) - [Metadata — Open Source Vizier documentation](#metadata-open-source-vizier-documentation) - [OSS Vizier as a Backend — Open Source Vizier documentation](#oss-vizier-as-a-backend-open-source-vizier-documentation) - [Pythia Policies and Hosting Designers — Open Source Vizier documentation](#pythia-policies-and-hosting-designers-open-source-vizier-documentation) - [Early Stopping — Open Source Vizier documentation](#early-stopping-open-source-vizier-documentation) - [Supported Algorithms — Open Source Vizier documentation](#supported-algorithms-open-source-vizier-documentation) - [Analyzing Benchmarks — Open Source Vizier documentation](#analyzing-benchmarks-open-source-vizier-documentation) - [Debugging tips — Open Source Vizier documentation](#debugging-tips-open-source-vizier-documentation) - [Debugging tips — Open Source Vizier documentation](#debugging-tips-open-source-vizier-documentation) - [Benchmarking with Ray — Open Source Vizier documentation](#benchmarking-with-ray-open-source-vizier-documentation) - [Converters — Open Source Vizier documentation](#converters-open-source-vizier-documentation) - [Designers — Open Source Vizier documentation](#designers-open-source-vizier-documentation) - [Early Stopping — Open Source Vizier documentation](#early-stopping-open-source-vizier-documentation) - [Metadata — Open Source Vizier documentation](#metadata-open-source-vizier-documentation) - [Running Benchmarks — Open Source Vizier documentation](#running-benchmarks-open-source-vizier-documentation) - [Predictors — Open Source Vizier documentation](#predictors-open-source-vizier-documentation) - [Pythia Policies and Hosting Designers — Open Source Vizier documentation](#pythia-policies-and-hosting-designers-open-source-vizier-documentation) - [Creating Benchmarks — Open Source Vizier documentation](#creating-benchmarks-open-source-vizier-documentation) --- # OSS Vizier Documentation — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/#) * OSS Vizier Documentation * [Edit on GitHub](https://github.com/google/vizier/tree/main/docs) * * * OSS Vizier Documentation[](https://oss-vizier.readthedocs.io/en/latest/#oss-vizier-documentation "Link to this heading") ========================================================================================================================== Open Source (OSS) Vizier is a Python-based interface for blackbox optimization and research, based on Google’s original internal [Vizier](https://dl.acm.org/doi/10.1145/3097983.3098043) , one of the first hyperparameter tuning services designed to work at scale. ![_images/oss_vizier_service.gif](https://oss-vizier.readthedocs.io/en/latest/_images/oss_vizier_service.gif) **OSS Vizier’s distributed client-server system. Animation by Tom Small.**[](https://oss-vizier.readthedocs.io/en/latest/#id1 "Link to this image") Installation[](https://oss-vizier.readthedocs.io/en/latest/#installation "Link to this heading") -------------------------------------------------------------------------------------------------- See [https://github.com/google/vizier#installation](https://github.com/google/vizier#installation) for instructions on installing OSS Vizier. Support[](https://oss-vizier.readthedocs.io/en/latest/#support "Link to this heading") ---------------------------------------------------------------------------------------- If you are having issues, please let us know by filing an issue on our [issue tracker](https://github.com/google/vizier/issues) . License[](https://oss-vizier.readthedocs.io/en/latest/#license "Link to this heading") ---------------------------------------------------------------------------------------- OSS Vizier is licensed under the Apache 2.0 License. Documentation * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * [For Users](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-users) * [For Developers](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-developers) * [For Benchmarking](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-benchmarking) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * [Tensorflow Probability](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html#tensorflow-probability) * [PyGlove](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html#pyglove) * [API Reference](https://oss-vizier.readthedocs.io/en/latest/api_reference/index.html) * [Code Structure](https://oss-vizier.readthedocs.io/en/latest/api_reference/index.html#code-structure) * [Highlights](https://oss-vizier.readthedocs.io/en/latest/highlights/index.html) * [Applications](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html) * [Media](https://oss-vizier.readthedocs.io/en/latest/highlights/media.html) --- # OSS Vizier Documentation — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/#) * OSS Vizier Documentation * [Edit on GitHub](https://github.com/google/vizier/tree/main/docs) * * * OSS Vizier Documentation[](https://oss-vizier.readthedocs.io/en/stable/#oss-vizier-documentation "Link to this heading") ========================================================================================================================== Open Source (OSS) Vizier is a Python-based interface for blackbox optimization and research, based on Google’s original internal [Vizier](https://dl.acm.org/doi/10.1145/3097983.3098043) , one of the first hyperparameter tuning services designed to work at scale. ![_images/oss_vizier_service.gif](https://oss-vizier.readthedocs.io/en/stable/_images/oss_vizier_service.gif) **OSS Vizier’s distributed client-server system. Animation by Tom Small.**[](https://oss-vizier.readthedocs.io/en/stable/#id1 "Link to this image") Installation[](https://oss-vizier.readthedocs.io/en/stable/#installation "Link to this heading") -------------------------------------------------------------------------------------------------- See [https://github.com/google/vizier#installation](https://github.com/google/vizier#installation) for instructions on installing OSS Vizier. Support[](https://oss-vizier.readthedocs.io/en/stable/#support "Link to this heading") ---------------------------------------------------------------------------------------- If you are having issues, please let us know by filing an issue on our [issue tracker](https://github.com/google/vizier/issues) . License[](https://oss-vizier.readthedocs.io/en/stable/#license "Link to this heading") ---------------------------------------------------------------------------------------- OSS Vizier is licensed under the Apache 2.0 License. Documentation * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * [For Users](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-users) * [For Developers](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-developers) * [For Benchmarking](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-benchmarking) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * [Tensorflow Probability](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html#tensorflow-probability) * [PyGlove](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html#pyglove) * [API Reference](https://oss-vizier.readthedocs.io/en/stable/api_reference/index.html) * [Code Structure](https://oss-vizier.readthedocs.io/en/stable/api_reference/index.html#code-structure) * [Highlights](https://oss-vizier.readthedocs.io/en/stable/highlights/index.html) * [Applications](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html) * [Media](https://oss-vizier.readthedocs.io/en/stable/highlights/media.html) --- # Advanced Topics — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * Advanced Topics * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/index.rst.txt) * * * Advanced Topics[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html#advanced-topics "Link to this heading") ================================================================================================================================== Tensorflow Probability[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html#tensorflow-probability "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ * [Bayesian Optimization Modeling](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html) * [Bijectors](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/bijectors.html) * [PSD kernels](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/kernels.html) * [Debugging tips](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/debugging.html) PyGlove[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html#pyglove "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * [OSS Vizier as a Backend](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html) --- # Highlights — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * Highlights * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/highlights/index.rst.txt) * * * Highlights[](https://oss-vizier.readthedocs.io/en/latest/highlights/index.html#highlights "Link to this heading") =================================================================================================================== * [Applications](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html) * [Media](https://oss-vizier.readthedocs.io/en/latest/highlights/media.html) --- # Guides — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * Guides * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/index.rst.txt) * * * Guides[](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#guides "Link to this heading") ======================================================================================================= For Users[](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-users "Link to this heading") ------------------------------------------------------------------------------------------------------------- * [Vizier Basics](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html) * [Distributed Vizier](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html) * [Search Spaces](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html) * [Converters](https://oss-vizier.readthedocs.io/en/latest/guides/user/converters.html) * [Switching to Vertex](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vizier/conversions_vertex_vizier_and_open_source_vizier.ipynb) * [Supported Algorithms](https://oss-vizier.readthedocs.io/en/latest/guides/user/supported_algorithms.html) For Developers[](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-developers "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- * [Designers](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html) * [Pythia Policies and Hosting Designers](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html) * [Early Stopping](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html) * [Metadata](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html) * [Predictors](https://oss-vizier.readthedocs.io/en/latest/guides/developer/predict.html) For Benchmarking[](https://oss-vizier.readthedocs.io/en/latest/guides/index.html#for-benchmarking "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- * [Creating Benchmarks](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/creating_benchmarks.html) * [Running Benchmarks](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html) * [Analyzing Benchmarks](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html) * [Benchmarking with Ray](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html) --- # API Reference — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * API Reference * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/api_reference/index.rst.txt) * * * API Reference[](https://oss-vizier.readthedocs.io/en/latest/api_reference/index.html#api-reference "Link to this heading") ============================================================================================================================ Code Structure[](https://oss-vizier.readthedocs.io/en/latest/api_reference/index.html#code-structure "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------ * [Frequently Used Import Targets](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html) --- # Vizier Basics — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Vizier Basics * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/user/running_vizier.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/running_vizier.ipynb) Vizier Basics[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#vizier-basics "Link to this heading") =================================================================================================================================== Below, we provide examples of how to: * Define a problem statement and study configuration. * Start a client. * (Optionally) Connect the client to a server. * Perform a typical tuning loop. * Use other client APIs. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] from vizier import service from vizier.service import clients from vizier.service import pyvizier as vz from vizier.service import servers Setting up the problem statement[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#setting-up-the-problem-statement "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here we setup the problem statement, which contains information about the search space and the metrics to optimize. problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', 0.0, 1.0) problem.search\_space.root.add\_float\_param('y', 0.0, 1.0) problem.metric\_information.append(vz.MetricInformation(name\='maximize\_metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) def evaluate(x: float, y: float) \-> float: return x\*\*2 \- y\*\*2 Setting up the study configuration[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#setting-up-the-study-configuration "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The study configuration contains additional information, such as the algorithm to use and level of noise that we think the objective will have. study\_config \= vz.StudyConfig.from\_problem(problem) study\_config.algorithm \= 'DEFAULT' Setting up the client[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#setting-up-the-client "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Starts a `study_client`, which will implicitly create a local Vizier Service which will be shared across other clients in the same Python process. Studies will then be stored locally in a SQL database file located at `service.VIZIER_DB_PATH`. study\_client \= clients.Study.from\_study\_config(study\_config, owner\='owner', study\_id\='example\_study\_id') print('Local SQL database file located at: ', service.VIZIER\_DB\_PATH) Obtaining suggestions[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#obtaining-suggestions "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Start requesting suggestions from the server, for evaluating objectives. Suggestions can be made sequentially (`count=1`) or in batches (`count>1`). for i in range(10): suggestions \= study\_client.suggest(count\=1) for suggestion in suggestions: x \= suggestion.parameters\['x'\] y \= suggestion.parameters\['y'\] objective \= evaluate(x, y) print(f'Iteration {i}, suggestion ({x},{y}) led to objective value {objective}.') final\_measurement \= vz.Measurement({'maximize\_metric': objective}) suggestion.complete(final\_measurement) Find optimal trial[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#find-optimal-trial "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Find the best objective so far, with corresponding suggestion value. For multiobjective cases, there may be multiple outputs of `optimal_trials()`, all corresponding to a Pareto-optimal curve. for optimal\_trial in study\_client.optimal\_trials(): optimal\_trial \= optimal\_trial.materialize() print("Optimal Trial Suggestion and Objective:", optimal\_trial.parameters, optimal\_trial.final\_measurement) Other client commands[](https://oss-vizier.readthedocs.io/en/latest/guides/user/running_vizier.html#other-client-commands "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- The `study_client` can also send other requests, such as the following: study\_client.get\_trial(1) \# Get the first trial. study\_client.trials() \# Get all trials so far. \# Obtain only the completed trials. trial\_filter \= vz.TrialFilter(status\=\[vz.TrialStatus.COMPLETED\]) study\_client.trials(trial\_filter\=trial\_filter) --- # Search Spaces — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Search Spaces * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/user/search_spaces.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/search_spaces.ipynb) Search Spaces[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#search-spaces "Link to this heading") ================================================================================================================================== Below, we provide examples of how to: * Setup a flat search space consisting of all four parameter types and additional auxiliary parameter types. * Setup a conditional search space correctly. * Reparameterize search spaces, which is useful for combinatorial search spaces. * Use infeasibility to define shaped search spaces. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import math from typing import List from vizier import pyvizier as vz Flat search spaces[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#flat-search-spaces "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------- Below are the core primitive parameter types and their specifications: * `DOUBLE`: Continuous range of possible values in the closed interval \\(\[a,b\]\\) for some real numbers \\(a \\le b\\). * `INTEGER`: Integer range of possible values in \\(\[a,b\] \\subset \\mathbb{Z}\\) for some integers \\(a \\le b\\). * `DISCRETE`: Finite, ordered set of values from \\(\\mathbb{R}\\). * `CATEGORICAL`: Unordered list of strings. flat\_problem \= vz.ProblemStatement() flat\_problem\_root \= flat\_problem.search\_space.root flat\_problem\_root.add\_float\_param(name\='double', min\_value\=0.0, max\_value\=1.0) flat\_problem\_root.add\_int\_param(name\='int', min\_value\=1, max\_value\=10) flat\_problem\_root.add\_discrete\_param( name\='discrete', feasible\_values\=\[0.1, 0.3, 0.5\]) flat\_problem\_root.add\_categorical\_param( name\='categorical', feasible\_values\=\['a', 'b', 'c'\]) PyVizier also has a `BOOLEAN` parameter which under-the-hood, is a binary `CATEGORICAL` parameter with values `'True'` and `'False'`. flat\_problem\_root.add\_bool\_param(name\='bool') A default value for seeding the study may be used when constructing a parameter. flat\_problem\_root.add\_float\_param( name\='double\_with\_default', min\_value\=0.0, max\_value\=1.0, default\_value\=0.5) Scaling[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#scaling "Link to this heading") ---------------------------------------------------------------------------------------------------------------------- Each of the numerical parameter types (`DOUBLE`, `INTEGER`, `DISCRETE`) may also have a **scaling type**, which toggles whether optimization occurs over a transformed space. \# Default scaling used. flat\_problem\_root.add\_float\_param( name\='double\_uniform', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.LINEAR) \# Points near min\_value are more important. flat\_problem\_root.add\_float\_param( name\='double\_log', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) \# Points near the max\_value are more important. flat\_problem\_root.add\_float\_param( name\='double\_reverse\_log', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) \# Default scaling used for DISCRETE parameters. flat\_problem\_root.add\_discrete\_param( name\='discrete\_uniform', feasible\_values\=\[0.1, 0.3, 0.5\], scale\_type\=vz.ScaleType.UNIFORM\_DISCRETE) Conditional search spaces[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#conditional-search-spaces "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------- Sometimes, **child parameters** only exist in specific scenarios or _conditions_ when a **parent parameter** is equal to one or more specific values. Example: Momentum hyperparameters are used by the [Adam optimizer](https://arxiv.org/abs/1412.6980) , but not stochastic gradient descent (SGD). **Caveat:** Since the value of a “learning rate” depends strongly on the optimizer being used (e.g. a learning rate of 0.1 to SGD means completely differently to Adam), we must create two separate child parameters, rather than sharing a single one. conditional\_problem \= vz.ProblemStatement() conditional\_problem\_root \= conditional\_problem.search\_space.root optimizer \= conditional\_problem\_root.add\_categorical\_param( name\='optimizer', feasible\_values\=\['sgd', 'adam'\]) \# SGD child parameters optimizer.select\_values(\['sgd'\]).add\_float\_param( 'sgd\_learning\_rate', min\_value\=0.0001, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) \# Adam child parameters optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_learning\_rate', min\_value\=0.0001, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_beta1', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_beta2', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) Combinatorial Reparamterization[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#combinatorial-reparamterization "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- When dealing with a combinatorial search space \\(X\\), one way to easily deal with such cases is to construct a reparameterization. Mathematically, this means finding a practical search space \\(Z\\) and surjective mapping \\(\\Phi: Z \\rightarrow X\\). Below is an example over the space of permutations of size \\(N\\), where our mapping utilizes the [Lehmer code](https://en.wikipedia.org/wiki/Lehmer_code) . N \= 10 \# Setup search space. permutation\_problem \= vz.ProblemStatement() for n in range(N): permutation\_problem.search\_space.root.add\_int\_param( name\=str(n), min\_value\=0, max\_value\=n) def compute\_index(trial: vz.Trial) \-> int: """Computes index from Lehmer code.""" index \= 0 for n in range(N): index += trial.parameters.get\_value(str(n)) \* math.factorial(n) return index def compute\_permutation(index: int) \-> List\[int\]: """Outputs a N-permutation as a list of indices.""" all\_indices \= list(range(N)) temp\_index \= index output \= \[\] for k in range(1, N + 1): factorial\_value \= math.factorial(N \- k) value \= all\_indices\[temp\_index // factorial\_value\] output.append(value) all\_indices.remove(value) temp\_index \= temp\_index % factorial\_value return output def phi(trial: vz.Trial) \-> List\[int\]: """Maps a suggestion to a permutation.""" return compute\_permutation(compute\_index(trial)) Infeasibility[](https://oss-vizier.readthedocs.io/en/latest/guides/user/search_spaces.html#infeasibility "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------- Consider an optimization problem where we only consider float parameters \\((x,y)\\) from the unit disk \\(x^{2} + y^{2} \\le 1\\). For such a scenario, we may denote any parameters outside of this area to be **infeasible**. disk\_problem \= vz.ProblemStatement() disk\_problem\_root \= disk\_problem.search\_space.root disk\_problem\_root.add\_float\_param(name\='x', min\_value\=-1.0, max\_value\=1.0) disk\_problem\_root.add\_float\_param(name\='y', min\_value\=-1.0, max\_value\=1.0) def evaluate(trial: vz.Trial) \-> vz.Trial: x \= trial.parameters\['x'\] y \= trial.parameters\['y'\] if x\*\*2 + y\*\*2 <= 1: trial.complete(vz.Measurement(metrics\={'sum': x + y})) else: trial.complete(vz.Measurement(), infeasibility\_reason\='Outside of range.') return trial --- # Media — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Highlights](https://oss-vizier.readthedocs.io/en/latest/highlights/index.html) * Media * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/highlights/media.rst.txt) * * * Media[](https://oss-vizier.readthedocs.io/en/latest/highlights/media.html#media "Link to this heading") ========================================================================================================= OSS Vizier has been featured in: Articles[](https://oss-vizier.readthedocs.io/en/latest/highlights/media.html#articles "Link to this heading") --------------------------------------------------------------------------------------------------------------- * [Google Research, 2022 & Beyond: Algorithmic Advances](https://ai.googleblog.com/2023/02/google-research-2022-beyond-algorithmic.html) * [MarkTechPost](https://www.marktechpost.com/2023/02/04/google-ai-open-sources-vizier-a-standalone-python-package-designed-for-managing-and-optimizing-machine-learning-experiments-at-scale/) * [The Sequence](https://thesequence.substack.com/p/the-chatgpt-challengers) * [ML News by Weights & Biases](https://wandb.ai/vincenttu/blog_posts/reports/ChatGPT-the-Catalyst--VmlldzozNDg1Nzc2) * [Analytics India Magazine](https://analyticsindiamag.com/google-vizier-is-now-open-source-and-thats-great-news/) * [This Week in AI by Lighting AI](https://lightning.ai/pages/community/steve-jobs-is-resurrected-meta-is-translating-unwritten-languages-and-ai-is-running-for-office/) * [gHacks](https://www.ghacks.net/2023/02/11/google-ai-open-sources-vizier/) * [WebBigdata (Japanese)](https://webbigdata.jp/post-17645/) * [Random Access (Spanish)](https://randomaccessnoticias.com/inteligencia-artificial/hacia-una-optimizacion-confiable-y-flexible-de-hiperparametros-y-blackbox-google-ai-blog/) * [Electronic Smith](https://electronicsmith.com/web-stories/google-ai-open-sources-vizier/) * [Deep Learning Weekly](https://open.substack.com/pub/deeplearningweekly/p/deep-learning-weekly-issue-287?utm_campaign=post&utm_medium=web) * [China Z (Chinese)](https://www.chinaz.com/2024/0823/1637848.shtml) * [TuringPost](https://www.turingpost.com/p/fod64) * [Open Data Science](https://odsc.medium.com/odscs-ai-weekly-recap-week-of-august-30th-37921c9d13c3) Videos/Talks[](https://oss-vizier.readthedocs.io/en/latest/highlights/media.html#videos-talks "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- * [Beijing Academy of Artificial Intelligence (BAAI)](https://event.baai.ac.cn/activities/834) * [AutoML Conference 2023 Tutorial](https://youtu.be/Xpdn_9uPEZY?si=wHXLYVV_j0yE5sLI) ([Hands-on Colab](https://github.com/google/vizier/blob/main/docs/tutorials/automl_conf_2023.ipynb) ) * [AutoML Seminar 2023 Talk](https://youtu.be/Ya_V5isGdG8) * [AutoML Conference 2022 Paper Presentation](https://youtu.be/b5hemgM16tM) * [AutoML Conference 2022 AutoRL Tutorial](https://youtu.be/9FDqUsByRiQ) * [ML News by Yannic Kilcher](https://youtu.be/TOo-HnjjuhU) --- # Applications — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Highlights](https://oss-vizier.readthedocs.io/en/latest/highlights/index.html) * Applications * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/highlights/applications.rst.txt) * * * Applications[](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html#applications "Link to this heading") ============================================================================================================================== OSS Vizier is used in the following: Codebases[](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html#codebases "Link to this heading") ------------------------------------------------------------------------------------------------------------------------ * [Vertex AI](https://github.com/googleapis/python-aiplatform) * [PyGlove](https://github.com/google/pyglove) * [OptFormer](https://github.com/google-research/optformer) * [Init2winit](https://github.com/google/init2winit) * [Tensorflow Federated](https://github.com/tensorflow/federated) * [Tensorflow GNN](https://github.com/tensorflow/gnn) * [CFU-Playground](https://github.com/google/CFU-Playground) * [Architecture Gym (ArchGym)](https://github.com/srivatsankrishnan/oss-arch-gym) * [OpenML (Converter)](https://github.com/josvandervelde/OpenML-Vizier-Converter) Guides + Courses[](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html#guides-courses "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ * [Deep Learning Tuning Playbook](https://github.com/google-research/tuning_playbook) * [Stanford STATS 285 (Fall 2023): Massive Computational Experiments, Painlessly](https://sites.google.com/corp/stanford.edu/stats285) * [Stanford XCS224U (Spring 2023): Natural Language Understanding](https://youtu.be/7zZRaoHr-8g?si=w2nKlMzudtuqy0_j) Papers[](https://oss-vizier.readthedocs.io/en/latest/highlights/applications.html#papers "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * [Fishy: Layerwise Fisher Approximation for Higher-order Neural Network Optimization](https://openreview.net/forum?id=cScb-RrBQC) * [Massively Scaling Heteroscedastic Classifiers](https://arxiv.org/abs/2301.12860) * [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://arxiv.org/abs/2205.13320) * [Task Selection for AutoML System Evaluation](https://arxiv.org/abs/2208.12754) --- # API Reference — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * API Reference * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/api_reference/index.rst.txt) * * * API Reference[](https://oss-vizier.readthedocs.io/en/stable/api_reference/index.html#api-reference "Link to this heading") ============================================================================================================================ Code Structure[](https://oss-vizier.readthedocs.io/en/stable/api_reference/index.html#code-structure "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------ * [Frequently Used Import Targets](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html) --- # Advanced Topics — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * Advanced Topics * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/index.rst.txt) * * * Advanced Topics[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html#advanced-topics "Link to this heading") ================================================================================================================================== Tensorflow Probability[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html#tensorflow-probability "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ * [Bayesian Optimization Modeling](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html) * [Bijectors](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/bijectors.html) * [PSD kernels](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/kernels.html) * [Debugging tips](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/debugging.html) PyGlove[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html#pyglove "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * [OSS Vizier as a Backend](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html) --- # Applications — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Highlights](https://oss-vizier.readthedocs.io/en/stable/highlights/index.html) * Applications * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/highlights/applications.rst.txt) * * * Applications[](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html#applications "Link to this heading") ============================================================================================================================== OSS Vizier is used in the following: Codebases[](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html#codebases "Link to this heading") ------------------------------------------------------------------------------------------------------------------------ * [Vertex AI](https://github.com/googleapis/python-aiplatform) * [PyGlove](https://github.com/google/pyglove) * [OptFormer](https://github.com/google-research/optformer) * [Init2winit](https://github.com/google/init2winit) * [Tensorflow Federated](https://github.com/tensorflow/federated) * [Tensorflow GNN](https://github.com/tensorflow/gnn) * [CFU-Playground](https://github.com/google/CFU-Playground) * [Architecture Gym (ArchGym)](https://github.com/srivatsankrishnan/oss-arch-gym) * [OpenML (Converter)](https://github.com/josvandervelde/OpenML-Vizier-Converter) Guides + Courses[](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html#guides-courses "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ * [Deep Learning Tuning Playbook](https://github.com/google-research/tuning_playbook) * [Stanford STATS 285 (Fall 2023): Massive Computational Experiments, Painlessly](https://sites.google.com/corp/stanford.edu/stats285) * [Stanford XCS224U (Spring 2023): Natural Language Understanding](https://youtu.be/7zZRaoHr-8g?si=w2nKlMzudtuqy0_j) Papers[](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html#papers "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * [Fishy: Layerwise Fisher Approximation for Higher-order Neural Network Optimization](https://openreview.net/forum?id=cScb-RrBQC) * [Massively Scaling Heteroscedastic Classifiers](https://arxiv.org/abs/2301.12860) * [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://arxiv.org/abs/2205.13320) * [Task Selection for AutoML System Evaluation](https://arxiv.org/abs/2208.12754) --- # Highlights — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * Highlights * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/highlights/index.rst.txt) * * * Highlights[](https://oss-vizier.readthedocs.io/en/stable/highlights/index.html#highlights "Link to this heading") =================================================================================================================== * [Applications](https://oss-vizier.readthedocs.io/en/stable/highlights/applications.html) * [Media](https://oss-vizier.readthedocs.io/en/stable/highlights/media.html) --- # Distributed Vizier — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Distributed Vizier * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/user/distributed.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/distributed.ipynb) Distributed Vizier[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#distributed-vizier "Link to this heading") ========================================================================================================================================== This documentation shows how to perform distributed optimization over multiple clients. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#installation-and-reference-imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] import multiprocessing from vizier import service from vizier.service import clients from vizier.service import pyvizier as vz from vizier.service import servers Regular setup[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#regular-setup "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------- We setup a regular study configuration below. study\_config \= vz.StudyConfig() study\_config.search\_space.root.add\_float\_param('x', 0.0, 1.0) study\_config.metric\_information.append(vz.MetricInformation(name\='metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) study\_config.algorithm \= 'DEFAULT' def evaluate(x: float) \-> float: return 2\*x \- x\*\*2 Server creation[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#server-creation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ Unlike the single-client case, in the distributed case, we require a single explicit server to accept requests from all other client processses. Details such as the `host`, `port`, `database_url`, `policy_factory`, etc. can be configured in the server’s initializer. server \= servers.DefaultVizierServer() \# Ideally created on a separate process such as a server machine. Client parallelization[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#client-parallelization "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- We may simultaneously create multiple clients to work on the same study, useful for parallelizing evaluation workload. All client processes (on a single machine or over multiple machines) will connect to this server via a globally specified `endpoint`. clients.environment\_variables.server\_endpoint \= server.endpoint \# Server address. study\_client \= clients.Study.from\_study\_config(study\_config, owner\='owner', study\_id \= 'example\_study\_id') \# Now connects to the explicitly created server. another\_study\_client \= clients.Study.from\_resource\_name(study\_client.resource\_name) \# Another way to fork clients. Distributed suggestions[](https://oss-vizier.readthedocs.io/en/latest/guides/user/distributed.html#distributed-suggestions "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------- We may now distribute our workflow, with each worker/client using the same loop below. Each client requires a unique `client_id` however, to ensure the server can identify client workers and distribute workloads properly. def tuning\_loop(client\_id: str): for i in range(10): suggestions \= study\_client.suggest(count\=1, client\_id\=client\_id) for suggestion in suggestions: objective \= evaluate(suggestion.parameters\['x'\]) final\_measurement \= vz.Measurement({'metric': objective}) suggestion.complete(final\_measurement) For example, we may perform a threadpool and construct multiple clients to parallelize evaluations on a single machine. NUM\_CLIENTS \= 10 NUM\_TRIALS\_PER\_CLIENT \= 50 pool \= multiprocessing.pool.ThreadPool(NUM\_CLIENTS) pool.map(tuning\_loop, range(NUM\_CLIENTS)) --- # Guides — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * Guides * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/index.rst.txt) * * * Guides[](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#guides "Link to this heading") ======================================================================================================= For Users[](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-users "Link to this heading") ------------------------------------------------------------------------------------------------------------- * [Vizier Basics](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html) * [Distributed Vizier](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html) * [Search Spaces](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html) * [Converters](https://oss-vizier.readthedocs.io/en/stable/guides/user/converters.html) * [Switching to Vertex](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/vizier/conversions_vertex_vizier_and_open_source_vizier.ipynb) * [Supported Algorithms](https://oss-vizier.readthedocs.io/en/stable/guides/user/supported_algorithms.html) For Developers[](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-developers "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- * [Designers](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html) * [Pythia Policies and Hosting Designers](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html) * [Early Stopping](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html) * [Metadata](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html) * [Predictors](https://oss-vizier.readthedocs.io/en/stable/guides/developer/predict.html) For Benchmarking[](https://oss-vizier.readthedocs.io/en/stable/guides/index.html#for-benchmarking "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- * [Creating Benchmarks](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/creating_benchmarks.html) * [Running Benchmarks](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html) * [Analyzing Benchmarks](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html) * [Benchmarking with Ray](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html) --- # Converters — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Converters * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/user/converters.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/converters.ipynb) Converters[](https://oss-vizier.readthedocs.io/en/latest/guides/user/converters.html#converters "Link to this heading") ========================================================================================================================= This documentation demonstrates how to use converters for representing PyVizier objects as NumPy arrays and vice-versa. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/user/converters.html#installation-and-reference-imports "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from vizier import pyvizier as vz from vizier.pyvizier import converters Suppose we had a problem statement and some trials associated to the study. \# Setup search space search\_space \= vz.SearchSpace() root \= search\_space.root root.add\_float\_param(name\='double', min\_value\=0.0, max\_value\=1.0) root.add\_int\_param(name\='int', min\_value\=1, max\_value\=10) root.add\_discrete\_param(name\='discrete', feasible\_values\=\[0.1, 0.3, 0.5\]) root.add\_categorical\_param(name\='categorical', feasible\_values\=\['a', 'b', 'c'\]) \# Setup metric configurations m1 \= vz.MetricInformation(name\='m1', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) m2 \= vz.MetricInformation(name\='m2', goal\=vz.ObjectiveMetricGoal.MINIMIZE) \# Final problem problem \= vz.ProblemStatement(search\_space, metric\_information\=\[m1, m2\]) \# Example trials trial1 \= vz.Trial( parameters\={'double': 0.6, 'int': 2, 'discrete': 0.1, 'categorical': 'a'}, final\_measurement\=vz.Measurement(metrics\={'m1': 0.1, 'm2': 0.2}), ) trial2 \= vz.Trial( parameters\={'double': 0.1, 'int': 6, 'discrete': 0.3, 'categorical': 'b'}, final\_measurement\=vz.Measurement(metrics\={'m1': \-1.0, 'm2': 0.8}), ) Quick Start[](https://oss-vizier.readthedocs.io/en/latest/guides/user/converters.html#quick-start "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- To use numerical models, both our `x` (parameters) and `y` (metrics) need to be formatted as numpy arrays. We can directly do so with `TrialToArrayConverter`: t2a\_converter \= converters.TrialToArrayConverter.from\_study\_config(problem) xs, ys \= t2a\_converter.to\_xy(\[trial1, trial2\]) We can also convert the `xs` back into PyVizier `ParameterDict`s: t2a\_converter.to\_parameters(xs) Behind the scenes, the `TrialToArrayConverter` actually uses a `DefaultTrialConverter` which first converts both trial parameters and metrics into `dict[str, np.ndarray]` and then concatenates the arrays together. converter \= converters.DefaultTrialConverter.from\_study\_config(problem) xs\_dict, ys\_dict \= converter.to\_xy(\[trial1, trial2\]) Trials can be recovered too: original\_trials \= converter.to\_trials(xs\_dict, ys\_dict) Customization[](https://oss-vizier.readthedocs.io/en/latest/guides/user/converters.html#customization "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- There are multiple ways to convert parameters of specific types. For example, some common methods to convert the `'categorical'` parameter (with feasible values `['a', 'b', 'c']`) can be: * Integer Index: `'b' -> 1` since `b` has index 1 among feasible values. * One-Hot: `'b' -> [0, 1, 0]` using one-hot encoding. Additional considerations can be, for example: * Whether to scale continuous parameter values into `[0,1]` * Whether to always sign-flip metrics to assume maximization only. These options can be specified when constructing both `TrialToArrayConverter` and `DefaultTrialConverter` ([source code](https://github.com/google/vizier/blob/main/vizier/pyvizier/converters/core.py) ): @classmethod def from\_study\_config( cls, study\_config: pyvizier.ProblemStatement, \*, scale: bool \= True, pad\_oovs: bool \= True, max\_discrete\_indices: int \= 0, flip\_sign\_for\_minimization\_metrics: bool \= True, dtype\=np.float64, ): For more fine-grained control over specific `ParameterConfig`s and `MetricInformation`s, a user can specify individual arguments to each `DefaultModelInputConverter` and `DefaultModelOutputConverter` respectively. \# Only considers the 'double' parameter values. double\_pc \= search\_space.get('double') double\_converter \= converters.DefaultModelInputConverter(double\_pc, scale\=True) double\_converter.convert(\[trial1, trial2\]) \# Only considers the 'categorical' parameter values. categorical\_pc \= search\_space.get('categorical') categorial\_converter \= converters.DefaultModelInputConverter(categorical\_pc, onehot\_embed\=True) categorial\_converter.convert(\[trial1, trial2\]) \# Only considers the 'm1' metric values. m1\_converter \= converters.DefaultModelOutputConverter(m1) m1\_converter.convert(\[trial1.final\_measurement, trial2.final\_measurement\]) These can be inserted into the `DefaultTrialConverter`: parameter\_converters \= \[double\_converter, categorial\_converter\] metric\_converters \= \[m1\_converter\] custom\_converter \= converters.DefaultTrialConverter(parameter\_converters, metric\_converters) custom\_converter.to\_xy(\[trial1, trial2\]) \# Same array outputs as above. For full customization, the user may create their own `ModelInputConverter`s and `ModelOutputConverter`s. class ModelInputConverter(metaclass\=abc.ABCMeta): """Interface for extracting inputs to the model.""" @abc.abstractmethod def convert(self, trials: Sequence\[vz.TrialSuggestion\]) \-> np.ndarray: """Returns an array of shape (number of trials, feature dimension).""" @property @abc.abstractmethod def output\_spec(self) \-> NumpyArraySpec: """Provides specification of the output from this converter.""" @property @abc.abstractmethod def parameter\_config(self): """Original ParameterConfig that this converter acts on.""" @abc.abstractmethod def to\_parameter\_values( self, array: np.ndarray ) \-> List\[Optional\[vz.ParameterValue\]\]: """Convert and clip to the nearest feasible parameter values.""" class ModelOutputConverter(metaclass\=abc.ABCMeta): """Metric converter interface.""" @abc.abstractmethod def convert(self, measurements: Sequence\[vz.Measurement\]) \-> np.ndarray: """Returns N x 1 array.""" pass @abc.abstractmethod def to\_metrics(self, labels: np.ndarray) \-> Sequence\[Optional\[vz.Metric\]\]: """Returns a list of pyvizier metrics.""" @property @abc.abstractmethod def metric\_information(self) \-> vz.MetricInformation: """Describes the semantics of the return value from convert() method.""" @property def output\_shape(self) \-> Tuple\[None, int\]: return (None, 1) --- # Media — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Highlights](https://oss-vizier.readthedocs.io/en/stable/highlights/index.html) * Media * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/highlights/media.rst.txt) * * * Media[](https://oss-vizier.readthedocs.io/en/stable/highlights/media.html#media "Link to this heading") ========================================================================================================= OSS Vizier has been featured in: Articles[](https://oss-vizier.readthedocs.io/en/stable/highlights/media.html#articles "Link to this heading") --------------------------------------------------------------------------------------------------------------- * [Google Research, 2022 & Beyond: Algorithmic Advances](https://ai.googleblog.com/2023/02/google-research-2022-beyond-algorithmic.html) * [MarkTechPost](https://www.marktechpost.com/2023/02/04/google-ai-open-sources-vizier-a-standalone-python-package-designed-for-managing-and-optimizing-machine-learning-experiments-at-scale/) * [The Sequence](https://thesequence.substack.com/p/the-chatgpt-challengers) * [ML News by Weights & Biases](https://wandb.ai/vincenttu/blog_posts/reports/ChatGPT-the-Catalyst--VmlldzozNDg1Nzc2) * [Analytics India Magazine](https://analyticsindiamag.com/google-vizier-is-now-open-source-and-thats-great-news/) * [This Week in AI by Lighting AI](https://lightning.ai/pages/community/steve-jobs-is-resurrected-meta-is-translating-unwritten-languages-and-ai-is-running-for-office/) * [gHacks](https://www.ghacks.net/2023/02/11/google-ai-open-sources-vizier/) * [WebBigdata (Japanese)](https://webbigdata.jp/post-17645/) * [Random Access (Spanish)](https://randomaccessnoticias.com/inteligencia-artificial/hacia-una-optimizacion-confiable-y-flexible-de-hiperparametros-y-blackbox-google-ai-blog/) * [Electronic Smith](https://electronicsmith.com/web-stories/google-ai-open-sources-vizier/) * [Deep Learning Weekly](https://open.substack.com/pub/deeplearningweekly/p/deep-learning-weekly-issue-287?utm_campaign=post&utm_medium=web) * [China Z (Chinese)](https://www.chinaz.com/2024/0823/1637848.shtml) * [TuringPost](https://www.turingpost.com/p/fod64) * [Open Data Science](https://odsc.medium.com/odscs-ai-weekly-recap-week-of-august-30th-37921c9d13c3) Videos/Talks[](https://oss-vizier.readthedocs.io/en/stable/highlights/media.html#videos-talks "Link to this heading") ----------------------------------------------------------------------------------------------------------------------- * [Beijing Academy of Artificial Intelligence (BAAI)](https://event.baai.ac.cn/activities/834) * [AutoML Conference 2023 Tutorial](https://youtu.be/Xpdn_9uPEZY?si=wHXLYVV_j0yE5sLI) ([Hands-on Colab](https://github.com/google/vizier/blob/main/docs/tutorials/automl_conf_2023.ipynb) ) * [AutoML Seminar 2023 Talk](https://youtu.be/Ya_V5isGdG8) * [AutoML Conference 2022 Paper Presentation](https://youtu.be/b5hemgM16tM) * [AutoML Conference 2022 AutoRL Tutorial](https://youtu.be/9FDqUsByRiQ) * [ML News by Yannic Kilcher](https://youtu.be/TOo-HnjjuhU) --- # Supported Algorithms — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Supported Algorithms * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/user/supported_algorithms.ipynb.txt) * * * Supported Algorithms[](https://oss-vizier.readthedocs.io/en/latest/guides/user/supported_algorithms.html#supported-algorithms "Link to this heading") ======================================================================================================================================================= While we service all algorithms to the user in our [policy factory](https://github.com/google/vizier/blob/main/vizier/_src/service/policy_factory.py) , many can be organized by what level of support we provide to them, in terms of: * Search space (**Flat**, **Continuous-Only**, **Boolean-Only**) * Allowing batched suggestions (**+Batch**) * Allowing multiple objectives (**+MO**) Official[](https://oss-vizier.readthedocs.io/en/latest/guides/user/supported_algorithms.html#official "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- The following algorithms can be considered “official” and production-quality: 1. [**GP-UCB-PE**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/gp_ucb_pe.py) (`GP_UCB_PE`) \[**Flat**, **+Batch**, **+MO**\] 2. [**GP-Bandit**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/gp_bandit.py) (`GAUSSIAN_PROCESS_BANDIT`) \[**Flat**, **+MO**\] 3. [**Random Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/random.py) (`RANDOM_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 4. [**Quasi-Random Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/quasi_random.py) (`QUASI_RANDOM_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 5. [**Grid Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/grid.py) (`GRID_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 6. [**Shuffled Grid Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/grid.py) (`SHUFFLED_GRID_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 7. [**Eagle Strategy**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/eagle_strategy/eagle_strategy.py) (`EAGLE_STRATEGY`) \[**Flat**, **+Batch**\] External + Imported[](https://oss-vizier.readthedocs.io/en/latest/guides/user/supported_algorithms.html#external-imported "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- These algorithms are imported and wrapped from external packages (requiring additional installations via `pip install google-vizier[algorithms]`), and thus we cannot fully control their performance: 1. [**CMA-ES**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/cmaes.py) (`CMA_ES`): \[**Continuous-Only**, **+Batch**\] Reproduced[](https://oss-vizier.readthedocs.io/en/latest/guides/user/supported_algorithms.html#reproduced "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------- These algorithms are attempted reproductions of their original papers, sometimes using the authors’ original implementations as inspiration (but not as direct imports). While we try our best to ensure their quality, we cannot guarantee exact performance: 1. [**NSGA-II**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/evolution/nsga2.py) (`NSGA2`) \[**Flat**, **+Batch**, **+MO**\] 2. [**Bayesian Optimization of Combinatorial Structures**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/bocs.py) (`BOCS`) \[**Boolean Only**\] 3. [**Harmonica**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/harmonica.py) (`HARMONICA`) \[**Boolean Only**\] --- # Bayesian Optimization Modeling — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * Bayesian Optimization Modeling * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/tfp/gp.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/gp.ipynb) Bayesian Optimization Modeling[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#bayesian-optimization-modeling "Link to this heading") ================================================================================================================================================================= The goal of this tutorial is to introduce Bayesian optimization workflows in OSS Vizier, including the underlying TensorFlow Probability (TFP) components and JAX/Flax functionality. The target audience is researchers and practitioners already well-versed in Bayesian optimization, who want to **define and train their own Gaussian Process surrogate models** for Bayesian optimization in OSS Vizier. Additional resources for TFP[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#additional-resources-for-tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------- If you’re new to TFP, a good place to start is [A tour of TensorFlow Probability](https://www.tensorflow.org/probability/examples/A_Tour_of_TensorFlow_Probability) . TFP began as a TensorFlow-only library, but now has a [JAX backend](https://www.tensorflow.org/probability/examples/TensorFlow_Probability_on_JAX) that is entirely independent of TensorFlow (such that “Tensor-Friendly Probability” might be a better backronym). This Colab uses TFP’s JAX backend (see the “Imports” cell for how to import it). Additional resources for Flax[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#additional-resources-for-flax "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- OSS Vizier’s Bayesian Optimization models are defined as [Flax](https://flax.readthedocs.io/en/latest/) modules. ### Imports[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#imports "Link to this heading") import chex import jax from jax import numpy as jnp, random, tree\_util import numpy as np import optax from mpl\_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt from tensorflow\_probability.substrates import jax as tfp from typing import Any \# Vizier models can freely access modules from vizier.\_src from vizier.\_src.benchmarks.experimenters.synthetic import bbob from vizier.jax import optimizers from vizier.\_src.jax import stochastic\_process\_model as spm tfd \= tfp.distributions tfb \= tfp.bijectors tfpk \= tfp.math.psd\_kernels Defining a GP surrogate model and hyperparameters[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#defining-a-gp-surrogate-model-and-hyperparameters "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To write a GP surrogate model, first write a coroutine that yields parameter specifications (`ModelParameter`) and returns a GP distribution. Downstream, the parameter specifications are used to define Flax module parameters. The inputs to the coroutine function represent the index points of the GP (in the remainder of this Colab, we refer to “inputs” and “index points” interchangeably). The rationale for the coroutine design is that it lets us automate the application of the parameter constraint and initialization functions (corresponding to hyperpriors, e.g.), and enables simultaneous specification of the model parameters and how their values are used to instantiate a GP. ### Coroutine example[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#coroutine-example "Link to this heading") The following cell shows a coroutine defining a GP with a squared exponential kernel and two parameters: the length scale of the kernel and the observation noise variance of the GP. def simple\_gp\_coroutine(inputs: chex.Array\=None): length\_scale \= yield spm.ModelParameter.from\_prior( tfd.Gamma(1., 1., name\='length\_scale')) amplitude \= 2. \# Non-trainable parameters may be defined as constants. kernel \= tfpk.ExponentiatedQuadratic( amplitude\=amplitude, length\_scale\=length\_scale) observation\_noise\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: jnp.exp(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: x\*\*2, name\='observation\_noise\_variance') return tfd.GaussianProcess( kernel, index\_points\=inputs, observation\_noise\_variance\=observation\_noise\_variance) ModelParameter[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#modelparameter "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------- `ModelParameter` may be used to define hyperpriors. ### Parameter specifications from priors[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#parameter-specifications-from-priors "Link to this heading") The length scale parameter has a Gamma prior. This is equivalent to defining a `ModelParameter` with a regularizer that computes the Gamma negative log likelihood and an initialization function that samples from the Gamma distribution. As the constraint was not specified, a default one is assigned which is the “default event space bijector” of the TFP distribution (each TFP distribution has a constraining bijector that maps the real line to the support of the distribution). ### Specifying parameters explicitly[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#specifying-parameters-explicitly "Link to this heading") Observation noise variance, which is passed to the Gaussian process and represents the scalar variance of zero-mean Gaussian noise in the observed labels, is not given a `tfd.Distribution` prior. Instead, it has its initialization, constraining, and regularization functions defined individually. Note that the initialization function is in the constrained space. Constraints[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#constraints "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- ModelParameter allows to define constraints on the model parameters using the ‘Constraint’ object which is initiated with a tuple of ‘bounds’ and ‘bijector’ function. Though the constraints are defined as part of the ModelParameter the Flax model itself does not use them, but rather it expects to receive parameter values already in the constrained space. This means that it’s the responsibility of the user/optimizer to pass the GP parameter values that are already in the constrained space. ### Exercise: Write a GP model[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#exercise-write-a-gp-model "Link to this heading") Write an ARD Gaussian Process model with three parameters: `signal_variance`, `length_scale`, and `observation_noise_variance`. (This is a slightly simplified version of the Vizier GP.) * `signal_variance` and `observation noise_variance` are both: * regularized by the function \\(f(x) = 0.01\\log(x)^2\\) * bounded to be positive. * `signal_variance` parameterizes a Matern 5/2 kernel, where the amplitude of the kernel is the square root of `signal_variance`. Use [`tfpk.MaternFiveHalves`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/matern.py#L414) . * `length_scale` has a \\(LogNormal(0, 1)\\) prior for each dimension. Assume there are 4 dimensions, and use [`tfd.Sample`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/distributions/sample.py) to build a 4-dimensional distribution consisting of IID LogNormal distributions. (Note that the `length_scale` parameter is a vector – all other parameters are scalars.) * In TFP, ARD kernels are implemented with [`tfpk.FeatureScaled`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/feature_scaled.py) , with `scale_diag` representing the length scale along each dimension. def vizier\_gp\_coroutine(inputs: chex.Array\=None): pass ### Solution[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#solution "Link to this heading") data\_dimensionality \= 2 def vizier\_gp\_coroutine(inputs: chex.Array\=None): """A coroutine that follows the \`ModelCoroutine\` protocol.""" signal\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: tfb.Softplus()(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: 0.01 \* jnp.log(x)\*\*2, name\='signal\_variance') length\_scale \= yield spm.ModelParameter.from\_prior( tfd.Sample( tfd.LogNormal(loc\=0., scale\=1.), sample\_shape\=\[data\_dimensionality\], name\='length\_scale'), constraint\=spm.Constraint(bounds\=(0.0, None))) kernel \= tfpk.MaternFiveHalves( amplitude\=jnp.sqrt(signal\_variance), validate\_args\=True) kernel \= tfpk.FeatureScaled( kernel, scale\_diag\=length\_scale, validate\_args\=True) observation\_noise\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: jnp.exp(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: 0.01 \* jnp.log(x)\*\*2, name\='observation\_noise\_variance') return tfd.GaussianProcess( kernel\=kernel, index\_points\=inputs, observation\_noise\_variance\=observation\_noise\_variance, validate\_args\=True) To build a GP Flax module, instantiate a `StochasticProcessModel` with a GP coroutine as shown below. The module runs the coroutine in the `setup` and `__call__` methods to initialize the parameters and then instantiate the GP object with the given parameters. Recall that Flax modules have two primary methods: `init`, which initializes parameters, and `apply`, which computes the model’s forward pass given a set of parameters and input data. model \= spm.StochasticProcessModel(coroutine\=vizier\_gp\_coroutine) \# Sample some fake data. \# Assume we have \`num\_points\` observations, each with \`dim\` features. num\_points \= 12 \# Sample a set of index points. index\_points \= np.random.normal( size\=\[num\_points, data\_dimensionality\]).astype(np.float32) \# Sample function values observed at the index points observations \= np.random.normal(size\=\[num\_points\]).astype(np.float32) \# Call the Flax module's \`init\` method to obtain initial parameter values. init\_params \= model.init(random.PRNGKey(0), index\_points) We can observe the initial parameters values of the Flax model and see that they match with the ‘ModelParameter’ definitions in our coroutine. print(init\_params\['params'\]) To instantiate a GP with a set of parameters and index points, use the Flax module’s `apply` method. `apply` also returns the regularization `losses` for the parameters, in `mutables`. The regularization `losses` are treated as mutable state because they are recomputed internally with each forward pass of the model. For more on mutable state in Flax, see [this](https://flax.readthedocs.io/en/latest/guides/state_params.html) tutorial. gp, mutables \= model.apply( init\_params, index\_points, mutable\=\['losses'\]) assert isinstance(gp, tfd.GaussianProcess) Optimizing hyperparameters[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#optimizing-hyperparameters "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise: Loss function[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#exercise-loss-function "Link to this heading") Write down a loss function that takes a parameters dict and returns the loss value, using `model.apply`. The function will close over the observed data. The loss should be the sum of the GP negative log likelihood and the regularization losses. The regularization loss values are computed when the module is called, using the `ModelParameter` regularization functions. They are stored in a mutable variable collection called `"losses"`, using the Flax method [`sow`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen.html?highlight=sow#flax.linen.Module.sow) . def loss\_fn(params): ... return loss, {} \# Return an empty dict as auxiliary state. ### Solution[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#id1 "Link to this heading") def loss\_fn(params): gp, mutables \= model.apply({'params': params}, index\_points, mutable\=\['losses'\]) loss \= (\-gp.log\_prob(observations) + jax.tree\_util.tree\_reduce(jnp.add, mutables\['losses'\])) \# add the regularization losses. return loss, {} The gradients of the loss have the same structure as the `params` dict. grads \= jax.grad(loss\_fn, has\_aux\=True)(init\_params\['params'\])\[0\] print(grads) We can use `jax.tree_util` to take a step along the gradient (though in practice, with Optax, we can use `update` and `apply_updates` to update the parameters at each train step). learning\_rate \= 1e-3 updated\_params \= jax.tree\_util.tree\_map( lambda p, g: p \- learning\_rate \* g, init\_params\['params'\], grads) print(updated\_params) Optimize hyperparameters with Vizier optimizers[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#optimize-hyperparameters-with-vizier-optimizers "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Flax modules are often optimized using Optax which requires the developer to write a routine that initializes parameter values and then repeatedly computes the loss function gradients and updates the parameter values accordingly. [Vizier Optimizers](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#TODO) is a library of optimizers that automate the process of finding the optimal Flax parameter values and wrap optimizers from libraries such as Optax and Jaxopt in a common interface. To use a Vizier Optimizer you have to specify the following: * `setup` function which is used to generate the initial parameter values. * `loss_fn` function which is used for computing the loss function value and gradients. For example, the loss function of a GP model would be a marginal likelihood plus the parameters regularizations. * `rng` PRNGKey for controlling pseudo randomization. * `constraints` on the parameters (optional). Below we use the Vizier `JaxoptLbfgsB` optimizer to run a constrained L-BFGS-B algorithm. Unconstrainted optimizers (e.g. Adam) use a bijector function to map between the unconstrained space where the search is performed, and the constrained space where the loss function is evaluated. On the contrary, constrained optimizers (e.g. L-BGFS-B) use the constraint bounds directly in the search process. To pass the constraints bounds to the `JaxoptLbfgsB` optimizer we use the `spm.get_constraints` function that traverse the parameters defined in the module coroutine and extract their bounds. setup \= lambda rng: model.init(rng, index\_points)\['params'\] model\_optimizer \= optimizers.JaxoptLbfgsB( random\_restarts\=20, best\_n\=None ) constraints \= spm.get\_constraints(model) optimal\_params, \_ \= model\_optimizer(setup, loss\_fn, random.PRNGKey(0), constraints\=constraints) Predict on new inputs, conditional on observations[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#predict-on-new-inputs-conditional-on-observations "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To compute the posterior predictive GP on unseen points, conditioned on observed data, use the `precompute_predictive` and `posterior_predictive` methods of the Flax module. `precompute_predictive` must be called first; it runs and stores the Cholesky decomposition of the kernel matrix for the observed data. `posterior_predictive` then returns a posterior predictive GP at new index points, avoiding recomputation of the Cholesky. \# Precompute the Cholesky. \_, pp\_state \= model.apply( {'params': optimal\_params}, index\_points, observations, mutable\=\['predictive'\], method\=model.precompute\_predictive) \# Predict on new index points. predictive\_index\_points \= np.random.normal( size\=\[5, data\_dimensionality\]).astype(np.float32) pp\_dist \= model.apply( {'params': optimal\_params, \*\*pp\_state}, predictive\_index\_points, index\_points, observations, method\=model.posterior\_predictive) \# \`posterior\_predictive\` returns a TFP distribution, whose mean, variance, and \# samples we can use to compute an acquisition function. assert pp\_dist.mean().shape \== (5,) Optimize a black-box function[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#optimize-a-black-box-function "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- For an end-to-end example of Bayesian optimization, we’ll use the GP surrogate model defined above along with an Upper Confidence Bound acquisition function to try to find the maximum of the Weierstrass function. First, visualize the function surface. \# Use the Weierstrass function from Vizier's Black-Box Optimization Benchmarking \# (BBOB) library. bb\_fun \= bbob.Weierstrass \# Sample a set of index points in a 2D space. num\_points \= 6 max\_x \= np.array(2.).astype(np.float32) index\_points \= random.uniform( random.PRNGKey(3), shape\=\[num\_points, data\_dimensionality\], dtype\=jnp.float32) \* max\_x \# Compute function values observed at the index points. observations \= np.apply\_along\_axis( bb\_fun, axis\=1, arr\=index\_points).astype(np.float32) \# Define a grid of points in the function domain for plotting. n\_grid \= 100 x \= y \= np.linspace(0, max\_x, n\_grid, dtype\=np.float32) X, Y \= np.meshgrid(x, y) x\_grid \= np.vstack(\[X.ravel(), Y.ravel()\]).T y\_grid \= np.apply\_along\_axis(bb\_fun, axis\=1, arr\=x\_grid) Z \= y\_grid.reshape(X.shape) \# Plot the black-box function values. fig \= plt.figure(figsize\=(8, 8)) ax \= fig.add\_subplot(111, projection\='3d') ax.plot\_surface(X, Y, Z, alpha\=0.5) ax.scatter(index\_points\[:, 0\], index\_points\[:, 1\], observations, color\='r', label\='Initial observed data') plt.title('Black-box (Weierstrass) function values and observed data') plt.legend() plt.show() Next, run a few iterations of Bayesian optimization to maximize the black-box function given the observed data. A single iteration consists of the following steps: 1. Optimize the GP hyperparameters. 2. Find a suggestion that maximizes an Upper Confidence Bound acquisition function. In this example, we use grid search for the optimization. 3. Evaluate the black-box function on the suggestion and append it to the set of observed data. (Note that this simple Bayesopt algorithm is for educational purposes and that we’d expect Vizier’s GP bandit algorithm to give better results.) num\_bayesopt\_iter \= 5 \# At each iteration, redefine the loss function given the current observed data. def build\_loss\_fn(index\_points, observations): def loss\_fn(params): gp, mutables \= model.apply({'params': params}, index\_points, mutable\=\['losses'\]) loss \= (\-gp.log\_prob(observations) + jax.tree\_util.tree\_reduce(jnp.add, mutables\['losses'\])) \# add the regularization losses. return loss, {} return loss\_fn for i in range(num\_bayesopt\_iter): \# Update the loss function to condition on all observed data. loss\_fn \= build\_loss\_fn(index\_points, observations) \# Optimize the GP hyperparameters. optimal\_params, \_ \= model\_optimizer(setup, loss\_fn, random.PRNGKey(0), constraints\=constraints) \# Compute the posterior predictive distribution over a grid of points in the \# function domain (x\_grid). \_, pp\_state \= model.apply( {'params': optimal\_params}, index\_points, observations, mutable\=\['predictive'\], method\=model.precompute\_predictive) pp\_dist \= model.apply( {'params': optimal\_params, \*\*pp\_state}, x\_grid, index\_points, observations, method\=model.posterior\_predictive) \# Compute the acquisition function value at each point in the grid. pred\_mean \= pp\_dist.mean() ucb\_vec \= pred\_mean + 2. \* pp\_dist.stddev() \# Find the grid point with the highest acquisition function value. ind \= np.argmax(ucb\_vec) \# Evaluate the black box function at the selected point. f\_val \= bb\_fun(x\_grid\[ind\]) \# Visualize the surrogate model mean and acquisition function surface at this \# iteration. fig \= plt.figure(figsize\=(10, 10)) ax \= fig.add\_subplot(121, projection\='3d') W \= pred\_mean.reshape(X.shape) ax.plot\_surface(X, Y, W, alpha\=0.5) ax.scatter(index\_points\[:, 0\], index\_points\[:, 1\], observations, color\='r', label\='Observed data') ax.set\_title('Observed data and posterior predictive GP mean') ax.legend() ax \= fig.add\_subplot(122, projection\='3d') ucb \= ucb\_vec.reshape(X.shape) ax.plot\_surface(X, Y, ucb, alpha\=0.5) ax.scatter(\*x\_grid\[ind\], ucb\_vec\[ind\], color\='r', label\='New suggestion') ax.set\_title('Acquisition function') ax.legend() plt.show() \# Append the new suggestion and function value to the set of observations. index\_points \= np.concatenate(\[index\_points, x\_grid\[ind\]\[np.newaxis\]\]) observations \= np.concatenate( \[observations, np.array(f\_val).astype(np.float32)\[np.newaxis\]\]) print(f'Iteration: {i}') print(f'Acquisition function value at suggestion: {ucb\_vec\[ind\]}') print(f'Black-box function value at suggestion: {f\_val}') Deeper dive on selected topics in TFP[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#deeper-dive-on-selected-topics-in-tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As shown above, the Flax GP model makes use of a number of TFP components: * Distributions specify parameter priors (e.g. `tfd.Gamma`). The stochastic process model itself is also a TFP distribution, `tfd.GaussianProcess`. * Bijectors (e.g. `tfb.Softplus`) are used to constrain parameters for optimization, and may also be used for input/output warping. * PSD kernels (e.g. `tfpk.ExponentiatedQuadratic`) specify the kernel function for the stochastic process. The next sections of this Colab introduce these and how they’re used in Bayesopt modeling. ### `tfd.GaussianProcess` and friends[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#tfd-gaussianprocess-and-friends "Link to this heading") The stochastic process Flax modules return a TFP distribution in the [Gaussian Process](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/distributions/gaussian_process.py#L93) family (an instance of `tfd.GaussianProcess`, `tfd.StudentTProcess`, or `tfde.MultiTaskGaussianProcess`). This Colab doesn’t go into detail on TFP distributions, since advanced usage and implementation of distributions is rarely required for Bayesopt modeling with Vizier. For an overview of TFP distributions, see [TensorFlow Distributions: A Gentle Introduction](https://www.tensorflow.org/probability/examples/TensorFlow_Distributions_Tutorial) . Some of the methods of the Gaussian Process distribution are demonstrated below. [Gaussian Process Regression in TFP](https://www.tensorflow.org/probability/examples/Gaussian_Process_Regression_In_TFP) is also worth reading. \# Build a kernel function (see "PSD kernels" section below) and GP. num\_points \= 6 index\_points \= random.uniform( random.PRNGKey(3), shape\=\[num\_points, data\_dimensionality\], dtype\=jnp.float32) observations \= random.uniform( random.PRNGKey(4), shape\=\[num\_points\], dtype\=jnp.float32) kernel \= tfpk.MaternFiveHalves( amplitude\=2., length\_scale\=0.3, validate\_args\=True \# Run additional runtime checks; possibly expensive. ) observation\_noise\_variance \= jnp.ones(\[\], dtype\=observations.dtype) gp \= tfd.GaussianProcess( kernel, index\_points\=index\_points, observation\_noise\_variance\=observation\_noise\_variance, cholesky\_fn\=lambda x: tfp.experimental.distributions.marginal\_fns.retrying\_cholesky(x)\[0\], \# See commentary below. validate\_args\=True) \# Take 4 samples from the GP at the index points. s \= gp.sample(4, seed\=random.PRNGKey(0)) assert s.shape \== (4, num\_points) \# Compute the log likelihood of the sampled values. lp \= gp.log\_prob(s) assert lp.shape \== (4,) \# GPs can also be instantiated without index points, in which case the index \# points must be passed to method calls. gp\_no\_index\_pts \= tfd.GaussianProcess( kernel, observation\_noise\_variance\=observation\_noise\_variance) s \= gp\_no\_index\_pts.sample(index\_points\=index\_points, seed\=random.PRNGKey(0)) \# Predictive GPs conditioned on observations can be built with \# \`GaussianProcess.posterior\_predictive\`. The Flax module's \# \`precompute\_predictive\` and \`posterior\_predictive\` methods call this GP method. gprm \= gp.posterior\_predictive( observations\=observations, predictive\_index\_points\=predictive\_index\_points) \# \`gprm\` is an instance of \`tfd.GaussianProcessRegressionModel\`. This class can \# also be instantiated directly (as a side note -- this isn't necessary for \# modeling with Vizier). same\_gprm \= tfd.GaussianProcessRegressionModel( kernel, index\_points\=predictive\_index\_points, observation\_index\_points\=index\_points, observations\=observations, observation\_noise\_variance\=observation\_noise\_variance) Aside from the kernel, index points, and noise variance, note the `cholesky_fn` arg to the `GaussianProcess` constructor. `cholesky_fn` is a callable that takes a matrix and returns a Cholesky-like lower triangular factor. The default function adds a jitter of 1e-6 to the diagonal and then calls `jnp.linalg.cholesky`. An alternative, used in the Vizier GP, is `tfp.experimental.distributions.marginal_fns.retrying_cholesky`, which adds progressively larger jitter until the Cholesky decomposition succeeds. ### A side note on batch shape in TFP[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/gp.html#a-side-note-on-batch-shape-in-tfp "Link to this heading") tl;dr: Don’t worry about batch shape. TFP objects have a notion of batch shape, which is useful for vectorized computations. For more on this, see [Understanding TensorFlow Distributions Shapes](https://www.tensorflow.org/probability/examples/Understanding_TensorFlow_Distributions_Shapes) . For the purposes of Bayesopt in Vizier, JAX’s `vmap` means that our TFP objects can have a single parameterization with empty batch shape. For example, in the following loss function takes a scalar `amplitude`, and the kernel and GP both have empty batch shape. def loss\_fn(amplitude): \# \`a\` is a scalar. k \= tfpk.ExponentiatedQuadratic(amplitude\=amplitude) \# batch shape \[\] gp \= tfd.GaussianProcess(k, index\_points\=index\_points) \# batch shape \[\] return \-gp.log\_prob(observations) initial\_amplitude \= np.random.uniform(size\=\[50\]) losses \= jax.vmap(loss\_fn)(initial\_amplitude) assert losses.shape \== (50,) We could also vectorize the loss computation by using a batched GP. In this simple case, the code is identical except that `vmap` is removed. Now, the kernel and GP represent a “batch” of kernels and GPs, each with different parameter values. Working with batch shape requires additional accounting on the part of the user to ensure that parameter shapes broadcast correctly, the correct dimensions are reduced over, etc. For Vizier’s use case, we find it simpler to rely on `vmap`. def loss\_fn(amplitude): \# \`a\` has shape \[50\]. k \= tfpk.ExponentiatedQuadratic(amplitude\=amplitude) \# batch shape \[50\] gp \= tfd.GaussianProcess(k, index\_points\=index\_points) \# batch shape \[50\] return \-gp.log\_prob(observations) initial\_amplitude \= np.random.uniform(size\=\[50\]) \# No vmap. losses \= loss\_fn(initial\_amplitude) assert losses.shape \== (50,) --- # Unknown Advanced Topics =============== Tensorflow Probability ---------------------- .. toctree:: :maxdepth: 1 tfp/gp tfp/bijectors tfp/kernels tfp/debugging PyGlove ------- .. toctree:: :maxdepth: 1 pyglove/vizier\_as\_backend --- # Unknown Guides ====== For Users --------- .. toctree:: :maxdepth: 1 user/running\_vizier user/distributed user/search\_spaces user/converters Switching to Vertex user/supported\_algorithms For Developers -------------- .. toctree:: :maxdepth: 1 developer/designers developer/pythia\_policies developer/early\_stopping developer/metadata developer/predict For Benchmarking ---------------- .. toctree:: :maxdepth: 1 benchmarks/creating\_benchmarks benchmarks/running\_benchmarks benchmarks/analyzing\_benchmarks benchmarks/ray\_benchmarks --- # Frequently Used Import Targets — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [API Reference](https://oss-vizier.readthedocs.io/en/latest/api_reference/index.html) * Frequently Used Import Targets * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/api_reference/faq_imports.rst.txt) * * * Frequently Used Import Targets[](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html#frequently-used-import-targets "Link to this heading") ==================================================================================================================================================================== Includes a brief summary of important symbols and modules. Service Users[](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html#service-users "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------- If you write client code interacting with the OSS Vizier service, use these import targets: * **from vizier.service import pyvizier as vz**: Exposes the same set of symbol names as `vizier.pyvizier`. `vizier.service.pyvizier.Foo` is a subclass or an alias of `vizier.pyvizier.Foo`, and can be converted into protobufs. * **from vizier.service import …**: Include binaries and internal utilities. Algorithm Developers[](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html#algorithm-developers "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ If you write algorithm code (Designers or Pythia policies) in OSS Vizier, use these import targets: * **from vizier import pyvizier as vz**: Pure python building blocks of OSS Vizier. Cross-platform code, including Pythia policies, must use this `pyvizier` instance. * `Trial` and `ProblemStatement` are important classes. * **from vizier.pyvizier import converters**: Convert between `pyvizier` objects and numpy arrays. * `TrialToNumpyDict`: Converts parameters (and metrics) into a dict of numpy arrays. Preferred conversion method if you intended to train an embedding of categorical/discrete parameters, or data includes missing parameters or metrics. * `TrialToArrayConverter`: Converts parameters (and metrics) into an array. * **from vizier.interfaces import serializable**: Abstractions for serializable objects. * `PartiallySerializable`, `Serializable` ### Algorithm Abstractions[](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html#algorithm-abstractions "Link to this heading") * **from vizier import pythia**: Abstractions for Pythia policies. * `Policy`, `PolicySupporter`: Key abstractions. * `LocalPolicyRunner`: Use it for running a `Policy` in RAM. * **from vizier import algorithms**: Abstractions for algorithms. * `Designer`: Stateful algorithm abstraction. * `DesignerPolicy`: Wraps `Designer` into a Pythia Policy. * `GradientFreeMaximizer`: For optimizing acquisition functions. * `(Partially)SerializableDesigner`: Designers who wish to optimize performance by saving states. ### Tensorflow Modules[](https://oss-vizier.readthedocs.io/en/latest/api_reference/faq_imports.html#tensorflow-modules "Link to this heading") * **from vizier import tfp**: Tensorflow-Probability utilities. * `acquisitions`: Acquisition functions module. * `AcquisitionFunction`: Abstraction. * `UpperConfidenceBound`, `ExpectedImprovement`, etc. * `bijectors`: Bijectors module. * `YeoJohnson`: Implements both Yeo-Johnson and Box-Cox transformations. * `optimal_power_transformation`: Returns the optimal power transformation. * `flip_sign`: returns a sign-flip bijector. * **from vizier import keras as vzk**: * `vzk.layers`: Layers usually wrapping `tfp` classes. * `variable_from_prior`: Utility layer for handling regularized variables. * `vzk.models`: Most of the useful models don’t easily fit into Keras’s `Model` abstraction, but we may add some for display. * `vzk.optim`: Wrappers around optimizers in `tfp` or `keras`. --- # Predictors — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Predictors * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/developer/predict.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/predict.ipynb) Predictors[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/predict.html#predictors "Link to this heading") =========================================================================================================================== This documentation will allow a developer to understand and use the `Predictor` API. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/predict.html#installation-and-reference-imports "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] import numpy as np from vizier.\_src.benchmarks.experimenters.synthetic import bbob from vizier.algorithms import designers from vizier import algorithms as vza from vizier import pyvizier as vz import matplotlib.pyplot as plt Predictors[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/predict.html#id1 "Link to this heading") -------------------------------------------------------------------------------------------------------------------- The `Predictor` exposes a `predict()` method which takes `TrialSuggestion`s as inputs and returns their corresponding objective value predictions, represented by a `Prediction` class. The source of truth for predictors can be found [here](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/core/abstractions.py) . class Prediction: """Container to hold predictions.""" mean: chex.Array stddev: chex.Array metadata: Optional\[Metadata\] \= None class Predictor(abc.ABC): """Mixin for algorithms to expose prediction API.""" @abc.abstractmethod def predict( self, trials: Sequence\[TrialSuggestion\], ... ) \-> Prediction: In some cases involving a underlying probabilistic model, there’s a need to sample the posterior distribution in order to obtain the mean and standard deviation. In this case, the API allows specifying the the random key and number of samples (hidden arguments as `...` in `predict()` above). GP-Bandit Predict Example[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/predict.html#gp-bandit-predict-example "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- The `VizierGPBandit` class acts as both a `Designer` and a `Predictor` and allows users to obtain the underlying GP model’s mean and standard deviation for a given set of points. The example below demonstrates this capability. Setup problem statement and objective: \# The problem statement (which parameters are being optimized) problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', \-5.0, 5.0) problem.metric\_information.append( vz.MetricInformation( name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) \# The real objective function used for generating observations. f \= lambda x: bbob.Weierstrass(np.array(\[x\])) Create observations (i.e. completed trials) of the objective function. \# Generate suggestions. observations \= designers.QuasiRandomDesigner(problem.search\_space).suggest(30) \# Compute the real objective value and complete the trials. trials \= \[\] for idx, obs in enumerate(observations): trials.append( obs.to\_trial(idx).complete( vz.Measurement(metrics\={'obj': f(obs.parameters\['x'\].value)}) ) ) Create a `VizierGPBandit` designer and update it with the observations. **Note:** When two `VizierGPBandit` designers are updated with identical trials, they may still produce slightly different models and predictions due to inherent stochasticity during the training process. \# Create the GPBandit designer. gp\_designer \= designers.VizierGPBandit(problem) \# Update the GP-Bandit designer with completed trials. gp\_designer.update(vza.CompletedTrials(trials), vza.ActiveTrials()) Generate predictions in arbitrary points. \# Generate predictions. suggestions \= designers.GridSearchDesigner(problem.search\_space, double\_grid\_resolution\=500).suggest(500) predictions \= gp\_designer.predict(suggestions) Plot the predictions. plt.figure(figsize\=(8, 6)) \# Visualize the real objective function. xs \= np.linspace(\-5, 5, num\=1000) ys \= \[f(x) for x in xs\] plt.plot(xs, ys, label\='actual', color\='blue', alpha\=0.6) \# Visualize the observation points. obs\_x \= \[obs.parameters\['x'\].value for obs in observations\] obs\_y \= \[f(x) for x in obs\_x\] plt.scatter(obs\_x, obs\_y, label\='observations', marker\='o', color\='red') \# Visualize the predictions and confidence bounds. pred\_x \= \[suggestion.parameters\['x'\].value for suggestion in suggestions\] plt.plot(pred\_x, predictions.mean, label\='prediction', color\='green') lower \= predictions.mean \- predictions.stddev upper \= predictions.mean + predictions.stddev plt.fill\_between(pred\_x, lower, upper, color\='grey', alpha\=0.2) \# Add legend and title. plt.legend(loc\='best') plt.title(f'GPBandit Prediction vs. Actual') plt.xlabel('x') plt.show() ![../../_images/d226edbf2883cd7ee6d06ce9ff199547cdb96d390380923be3362396204ba9e4.png](https://oss-vizier.readthedocs.io/en/latest/_images/d226edbf2883cd7ee6d06ce9ff199547cdb96d390380923be3362396204ba9e4.png) --- # Creating Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Creating Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/benchmarks/creating_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/creating_benchmarks.ipynb) Creating Benchmarks[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/creating_benchmarks.html#creating-benchmarks "Link to this heading") ========================================================================================================================================================== We provide a guide below on creating benchmarks, through the use of either: * Standard search space primitives. * Metadata for complex search spaces. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/creating_benchmarks.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import abc import random from typing import Sequence from vizier import pyvizier as vz from vizier.benchmarks import experimenters Experimenters[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/creating_benchmarks.html#experimenters "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------- The core base class of any objective function is the `Experimenter` class, which simply contains a method to evaluate a `Trial` and a `ProblemStatement` to describe its search space and metrics. The exact entry into the class can be found [here](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) . class Experimenter(metaclass\=abc.ABCMeta): """Abstract base class for Experimenters.""" @abc.abstractmethod def evaluate(self, suggestions: Sequence\[vz.Trial\]) \-> None: """Evaluates and mutates the Trials in-place.""" @abc.abstractmethod def problem\_statement(self) \-> vz.ProblemStatement: """The search configuration generated by this experimenter.""" Below is an example of a basic 1D objective function \\(f(x) = x^{2}\\). class Basic1DExperimenter(experimenters.Experimenter): def evaluate(self, suggestions: Sequence\[vz.Trial\]) \-> None: for suggestion in suggestions: x \= suggestion.parameters\['x'\].value objective \= x\*\*2 measurement \= pyvizier.Measurement(metrics\={'obj': objective}) suggestion.complete(measurement) def problem\_statement(self) \-> vz.ProblemStatement: problem\_statement \= vz.ProblemStatement() root \= problem\_statement.search\_space.root root.add\_float\_param(name\='x', min\_value\=-1.0, max\_value\=1.0) metric \= vz.MetricInformation(name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) problem\_statement.metric\_information.append(metric) return problem\_statement We may thus evaluate a suggestion. Note that such suggestions are actually `Trial`s, to allow maximum flexibility. basic\_experimenter \= Basic1DExperimenter() trial \= vz.Trial() trial.parameters\['x'\] \= 0.1 basic\_experimenter.evaluate(\[trial\]) assert trial.final\_measurement.metrics\['obj'\].value \== 0.1 \*\* 2 Metadata-based Experimenters[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/creating_benchmarks.html#metadata-based-experimenters "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Similar to using the `Metadata` primitive to create custom algorithms and complex search spaces, creating custom `Experimenter`s provides the freedom to define custom objective functions. As an example, suppose our search space consisted of unbounded-length sequences consisting of some vocabulary (e.g. the letters ‘A’ to ‘Z’ if considering the space of English words), and we wish to maximize the sequence’s average ASCII value. class VocabularyExperimenter(experimenters.Experimenter): def evaluate(self, suggestions: Sequence\[vz.Trial\]): for suggestion in suggestions: x \= suggestion.metadata\['word'\] objective \= float(sum(\[ord(c) for c in x\])) / len(x) measurement \= vz.Measurement(metrics\={'obj': objective}) suggestion.complete(measurement) def problem\_statement(self) \-> pyvizier.ProblemStatement: problem\_statement \= vz.ProblemStatement() problem\_statement.metadata\['vocab'\] \= 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' metric \= vz.MetricInformation(name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) problem\_statement.metric\_information.append(metric) return problem\_statement Below is an example of constructing a valid suggestion and evaluating it. vocab\_experimenter \= VocabularyExperimenter() vocabulary \= vocab\_experimenter.problem\_statement().metadata\['vocab'\] trial \= vz.Trial() trial.metadata\['word'\] \= str( \[random.randint(0, len(vocabulary)) for \_ in range(10)\] ) vocab\_experimenter.evaluate(\[trial\]) print('Average ASCII value is:', trial.final\_measurement.metrics\['obj'\].value) --- # Benchmarking with Ray — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Benchmarking with Ray * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/benchmarks/ray_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/ray_benchmarks.ipynb) Benchmarking with Ray[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html#benchmarking-with-ray "Link to this heading") ========================================================================================================================================================= We provide a brief guide below on the Vizier + Ray integration, and how to benchmark with all publicly available algorithms on [Ray Tune](https://docs.ray.io/en/latest/tune/) . Notably, Tune integrates with a wide range of additional hyperparameter optimization tools, including Ax, BayesOpt, BOHB, Dragonfly, FLAML, HEBO, Hyperopt, Nevergrad, Optuna, SigOpt, skopt, and ZOOpt. ![alt-text](https://docs.ray.io/en/latest/_images/tune_overview.png) Initial Installation[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html#initial-installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] !pip install \-U "ray\[default\]" Algorithm and Experimenter Factories[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html#algorithm-and-experimenter-factories "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As mentioned in previous guides, since we want to compare algorithms across multiple benchmarks, we first create a bunch of relevant benchmark experimenters. To do so, we use `SerializableExperimenterFactory` from our [Experimenters API](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) to modularize the construction of multiple benchmark components. For example, here we can create a diverse set of BBOB functions with different dimensions via the `BBOBExperimenterFactory`. Then, we can print out the full serialization of the benchmarks that we have created. import itertools import numpy as np from vizier.benchmarks import experimenters function\_names \= \[\ 'Sphere',\ 'BentCigar',\ 'Katsuura',\ \] dimensions \= \[4, 8\] product\_list \= list(itertools.product(function\_names, dimensions)) experimenter\_factories \= \[\] for product in product\_list: name, dim \= product bbob\_factory \= experimenters.BBOBExperimenterFactory(name\=name, dim\=dim) experimenter\_factory \= experimenters.SingleObjectiveExperimenterFactory( bbob\_factory, shift\=np.random.uniform(low\=-2, high\=2, size\=dim), noise\_type\='LIGHT\_ADDITIVE\_GAUSSIAN', ) experimenter\_factories.append(experimenter\_factory) print(experimenter\_factory.dump()) Next, we need to define our algorithms by installing the relevant packages and importing the relevant algorithms. For simplicity, we only compare against only a subset of the algorithms that Ray supports. **NOTE:** We provide the `VizierSearch` class in our own libaries that can directly use the `Searcher` API in Ray. The imports are given below. pip install ax\-platform scikit\-optimize hyperopt optuna bayesian\-optimization from ray import tune from ray.tune.search.ax import AxSearch from ray.tune.search.bayesopt import BayesOptSearch from ray.tune.search.hyperopt import HyperOptSearch from ray.tune.search.optuna import OptunaSearch from ray.tune.search.skopt import SkOptSearch from vizier import raytune as vzr from vizier.\_src.raytune.vizier\_search import VizierSearch algorithm\_factories \= { 'ray': lambda: None, 'vizier': VizierSearch, 'ax': AxSearch, 'bayesopt': BayesOptSearch, 'optuna': OptunaSearch, 'hyperopt': HyperOptSearch, 'skopt': SkOptSearch, } Running RayTune[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html#running-raytune "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Running RayTune using `ExperimenterFactory` is made easy using our utility libraries which takes in any factory with a `TuneConfig` to run the algorithm on the corresponding benchmark. Let us first run one algorithm on the first benchmark and see the results that we get. **NOTE:** This uses a local Ray instance. ALGORITHM\_NAME \= 'ray' \# @param str experimenter\_factory \= experimenter\_factories\[0\] factory \= algorithm\_factories\[ALGORITHM\_NAME\] tune\_config \= tune.TuneConfig( search\_alg\=factory(), num\_samples\=4, max\_concurrent\_trials\=1, ) vzr.run\_tune.run\_tune\_from\_factory(experimenter\_factory, tune\_config) Now, we repeat our runs for each `ExperimenterFactory` and each algorithm, converting the results into `PlotElements` for easy plotting and comparison. from vizier.benchmarks import analyzers NUM\_REPEATS \= 3 \# @param NUM\_ITERATIONS \= 50 \# @param def results\_to\_element(results\_list): curves \= \[\] for results in results\_list: raw\_ys \= np.array(results.get\_dataframe()\['bbob\_eval\_before\_noise'\]) ys \= np.minimum.accumulate(raw\_ys) curve \= analyzers.ConvergenceCurve( xs\=np.arange(1, len(ys) + 1), ys\=ys.reshape((1, len(ys))), trend\=analyzers.ConvergenceCurve.YTrend.DECREASING, ) curves.append(curve) all\_curves \= analyzers.ConvergenceCurve.align\_xs(curves) ele \= analyzers.PlotElement(curve\=all\_curves\[0\], yscale\='symlog') return ele all\_records \= \[\] for experimenter\_factory in experimenter\_factories: for algorithm, factory in algorithm\_factories.items(): results \= \[\] for \_ in range(NUM\_REPEATS): tune\_config \= tune.TuneConfig( search\_alg\=factory(), num\_samples\=NUM\_ITERATIONS, max\_concurrent\_trials\=1, ) results.append( vzr.run\_tune.run\_tune\_from\_factory(experimenter\_factory, tune\_config) ) ele \= results\_to\_element(results) record \= analyzers.BenchmarkRecord( algorithm\=algorithm, experimenter\_metadata\=experimenter\_factory.dump(), plot\_elements\={'objective': ele}, ) all\_records.append(record) analyzed\_records \= analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=all\_records, baseline\_algo\='ray' ) analyzers.plot\_from\_records(analyzed\_records) Running Parallelized Ray[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/ray_benchmarks.html#running-parallelized-ray "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- In the previous example, we are using Ray local instances and running each benchmark in sequential format, which can take minutes. When there are a large number of benchmarks or computationally intensive benchmark runs, using parallelism distributed across each (algorithm, benchmark) tuple is crucial for reasonable benchmarking turnaround. We recommend using the [Ray Jobs API](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html) to distribute work across clusters. --- # Running Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Running Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/benchmarks/running_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/running_benchmarks.ipynb) Running Benchmarks[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#running-benchmarks "Link to this heading") ======================================================================================================================================================= We will demonstrate below how to use our benchmark runner pipeline. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#installation-and-reference-imports "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from vizier import algorithms as vza from vizier import benchmarks as vzb from vizier.algorithms import designers from vizier.benchmarks import experimenters Example experimenter and designer factory which we will use later. experimenter \= experimenters.NumpyExperimenter( experimenters.bbob.Sphere, experimenters.bbob.DefaultBBOBProblemStatement(5) ) designer\_factory \= designers.GridSearchDesigner.from\_problem Algorithms and Experimenters[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#algorithms-and-experimenters "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every study can be seen conceptually as a simple loop between an algorithm and objective. In terms of code, the algorithm corresponds to a `Designer`/`Policy` and objective to an `Experimenter`. Below is a simple sequential loop. designer \= designer\_factory(experimenter.problem\_statement()) for \_ in range(100): suggestion \= designer.suggest()\[0\] trial \= suggestion.to\_trial() experimenter.evaluate(\[trial\]) completed\_trials \= vza.CompletedTrials(\[trial\]) designer.update(completed\_trials, vza.ActiveTrials()) As seen above however, one modification we can make is to use variable batch sizes, rather than only suggesting and evaluating one-by-one. More generally, certain implementation details may arise: * How many parallel suggestions should the algorithm generate? * How many suggestions can be evaluated at once? * Should we use early stopping on certain unpromising trials? * Should we use a custom stopping condition instead of a fixed for-loop? * Can we swap in a different algorithm mid-loop? * Can we swap in a different objective mid-loop? API[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#api "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- The code flexibility needed to simulate these real-life scenarios may cause complications as the evaluation benchmark may no longer be stateless. In order to broadly cover such scenarios, our [API](https://github.com/google/vizier/blob/main/vizier/benchmarks/__init__.py) introduces the `BenchmarkSubroutine`: class BenchmarkSubroutine(Protocol): """Abstraction for core benchmark routines. Benchmark protocols are modular alterations of BenchmarkState by reference. """ def run(self, state: BenchmarkState) \-> None: """Abstraction to alter BenchmarkState by reference.""" All routines use and potentially modify a `BenchmarkState`, which holds information about the objective via an `Experimenter` and the algorithm itself wrapped by a `PolicySuggester`. class BenchmarkState: """State of a benchmark run. It is altered via benchmark protocols.""" experimenter: Experimenter algorithm: PolicySuggester To wrap multiple `BenchmarkSubRoutines` together, we can use the `BenchmarkRunner`: class BenchmarkRunner(BenchmarkSubroutine): """Run a sequence of subroutines, all repeated for a few iterations.""" \# A sequence of benchmark subroutines that alter BenchmarkState. benchmark\_subroutines: Sequence\[BenchmarkSubroutine\] \# Number of times to repeat applying benchmark\_subroutines. num\_repeats: int def run(self, state: BenchmarkState) \-> None: """Run algorithm with benchmark subroutines with repetitions.""" Example usage[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#example-usage "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Below is a typical example of simple suggestion and evaluation: runner \= vzb.BenchmarkRunner( benchmark\_subroutines\=\[\ vzb.GenerateSuggestions(),\ vzb.EvaluateActiveTrials(),\ \], num\_repeats\=100, ) benchmark\_state\_factory \= vzb.DesignerBenchmarkStateFactory( experimenter\=experimenter, designer\_factory\=designer\_factory ) benchmark\_state \= benchmark\_state\_factory() runner.run(benchmark\_state) We may obtain the evaluated trials via the `benchmark_state`, which contains a `PolicySupporter` via its `algorithm` field: all\_trials \= benchmark\_state.algorithm.supporter.trials print(all\_trials) Note that this design is maximally informative on everything that has happened so far in the study. For instance, we may also query incomplete/unused suggestions using the `PolicySupporter`. References[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/running_benchmarks.html#references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------- * Benchmark Runners can be found [here](https://github.com/google/vizier/tree/main/vizier/_src/benchmarks/runners) . --- # Analyzing Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Analyzing Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/benchmarks/analyzing_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/analyzing_benchmarks.ipynb) Analyzing Benchmarks[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#analyzing-benchmarks "Link to this heading") ============================================================================================================================================================= We will demonstrate below how to dstribute our benchmark runner pipeline over multiple benchmarks in conjunction with our suite of benchmark analysis tools to easily compare and visualize the performance of different algorithms over all benchmark problems. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from vizier import benchmarks as vzb from vizier.algorithms import designers from vizier.benchmarks import experimenters from vizier.benchmarks import analyzers import itertools import numpy as np import pandas as pd Algorithm and Experimenter Factories[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#algorithm-and-experimenter-factories "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To compare algorithms across multiple benchmarks, we want to first create a bunch of relevant benchmark experimenters. To do so, we use `SerializableExperimenterFactory` from our [Experimenters API](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) to modularize the construction of multiple benchmark components. For example, here we can create a diverse set of BBOB functions with different dimensions via the `BBOBExperimenterFactory`. Then, we can print out the full serialization of the benchmarks that we have created. function\_names \= \['Sphere', 'Discus'\] dimensions \= \[4, 8\] product\_list \= list(itertools.product(function\_names, dimensions)) experimenter\_factories \= \[\] for product in product\_list: name, dim \= product bbob\_factory \= experimenters.BBOBExperimenterFactory(name\=name, dim\=dim) experimenter\_factories.append(bbob\_factory) print(bbob\_factory.dump()) As mentioned in our previous tutorial, we can create a `BenchmarkState` from our algorithm and experimenter factories and apply a `BenchmarkRunner` benchmarking protocol to run the algorithm. We end up with a list of `BenchmarkState` objects, each representing a different benchmark run, possibly with repeats. Conveniently, we provide analysis utility functions in our [Analyzers API](https://github.com/google/vizier/blob/main/vizier/benchmarks/analyzers.py) that convert our `BenchmarkState` into summarized curves stored compactly in `BenchmarkRecord`, which also holds the algorithm name and experimenter factory serialization. We can visualize and later analyze our results using a dataframe. NUM\_REPEATS \= 5 \# @param NUM\_ITERATIONS \= 150 \# @param runner \= vzb.BenchmarkRunner( benchmark\_subroutines\=\[\ vzb.GenerateSuggestions(),\ vzb.EvaluateActiveTrials(),\ \], num\_repeats\=NUM\_ITERATIONS, ) algorithms \= { 'grid': designers.GridSearchDesigner.from\_problem, 'random': designers.RandomDesigner.from\_problem, 'eagle': designers.EagleStrategyDesigner, } records \= \[\] for experimenter\_factory in experimenter\_factories: for algo\_name, algo\_factory in algorithms.items(): benchmark\_state\_factory \= vzb.ExperimenterDesignerBenchmarkStateFactory( experimenter\_factory\=experimenter\_factory, designer\_factory\=algo\_factory ) states \= \[\] for \_ in range(NUM\_REPEATS): benchmark\_state \= benchmark\_state\_factory() runner.run(benchmark\_state) states.append(benchmark\_state) record \= analyzers.BenchmarkStateAnalyzer.to\_record( algorithm\=algo\_name, experimenter\_factory\=experimenter\_factory, states\=states, ) records.append(record) records\_list \= \[\ (rec.algorithm, dict(rec.experimenter\_metadata), rec) for rec in records\ \] df \= pd.DataFrame(records\_list, columns\=\['algorithm', 'experimenter', 'record'\]) df Visualization from Records[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#visualization-from-records "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Given a sequence of `BenchmarkRecords`, we provide utility plotting functions via the `matplotlib.pyplot` library to plot and visualize the relative performance of each algorithm on each benchmark. Currently, for single-objective optimization, we extract and plot the `objective` metric, which represents the objective of the best Trial seen so far as a function of Trial id/count (default). **Note**: this `objective` curve is monotonic and is computing upon converting to `BenchmarkRecord`. analyzers.plot\_from\_records(records) Observe that `plot_from_records` is a general plotting utility function that generates a grid of algorithm comparison plots. Specifically, it generates one plot for each Experimenter x Metrics in records, where each row represents an Experimenter and each column is a Metric represented in the record’s elements dictionary. Each plot has a curve for each algorithm. Adding Analysis[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#adding-analysis "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Oftentimes, further analysis is needed to normalize metrics across multiple benchmarks or to visualize more context-dependent metrics, such as visualizing the Pareto frontier as a scatter plot. We focus on the former case, where objective curves require some form of normalization for each comparison across benchmarks. Many success metrics have been proposed: win rates, relative convergence, normalized objective score, [NeurIPS competition scores](https://arxiv.org/pdf/2012.03826.pdf) . To broadly cover such analysis scores, our [API](https://github.com/google/vizier/blob/main/vizier/benchmarks/__init__.py) introduces the `ConvergenceComparator` abstraction that compares two `ConvergenceCurve` at specified quantiles: @attr.define class ConvergenceComparator(abc.ABC): """(Simplified) Base class for convergence curve comparators. Attributes: baseline\_curve: The baseline ConvergenceCurve. compared\_curve: The compared ConvergenceCurve. """ \_baseline\_curve: ConvergenceCurve \= attr.field() \_compared\_curve: ConvergenceCurve \= attr.field() @abc.abstractmethod def score(self) \-> float: """Returns a summary score for the comparison between base and compared. Usually, higher positive numbers mean the compared curve is better than the baseline and vice versa. """ pass @abc.abstractmethod def curve(self) \-> ConvergenceCurve: """Returns a score curve for each xs.""" pass Generally, a higher score by convention should indicate that the compared curve is better than the baseline. Furthermore, a score of 0.0 indicates that the performance is similar and it would make sense of these scores to be symmetric. However, there is no such restrictions imposed on the API. As an example, we can add the `LogEfficiencyScore`, which is based off of [performance profiles](https://arxiv.org/pdf/cs/0102001.pdf) , a gold standard in optimization benchmarking. The LogEfficiencyScore essentially measures the percentage of Trials needed for the compared algorithm to match the baseline performance. If score = 1, then the compared algorithm uses \\(e^{-1}\*T\\) Trials to reach the same objective as the baseline algorithm in \\(T\\) trials. analyzed\_records \= analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=records, baseline\_algo\='random' ) analyzers.plot\_from\_records(analyzed\_records) Custom Comparators[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#custom-comparators "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- To write a custom `ConvergenceComparator`, simply follow the abstract class defined above and form a `ConvergenceComparatorFactory`, which can then be passed into `add_comparison_metrics`. Note that we are constantly adding more benchmarking scores into our analyzers base and welcome submissions. Let us try to write a custom `WinRateComparator` that looks at the simple metric of comparing whether one curve is better than the other, for each `xs`. **NOTE:** You may always assume in a `Comparator` that both curves are either `INCREASING` (sign = 1) or `DECREASING` (signand that the sign of the curves is stored in `self._sign`. class WinRateComparator(analyzers.ConvergenceComparator): """Comparator method based on simple win rate comparison.""" def score(self) \-> float: return np.nanmedian(self.curve().ys) def curve(self) \-> analyzers.ConvergenceCurve: baseline\_ys \= self.\_sign \* self.\_baseline\_curve.ys compared\_ys \= self.\_sign \* self.\_compared\_curve.ys \# Compares all pairs of compared to baseline curve. all\_comparisons \= np.apply\_along\_axis( lambda base: np.mean(compared\_ys \> base, axis\=0), axis\=1, arr\=baseline\_ys, ) return analyzers.ConvergenceCurve( xs\=self.\_baseline\_curve.xs, ys\=np.mean(all\_comparisons, axis\=0, keepdims\=True), ) Now, we add a simple ComparatorFactory and inject the factory into `add_comparison_metrics` to create our new scoring plots. Note that one can also manually add customized `PlotElements` that can be in histogram, or scatter form. class WinRateComparatorFactory(analyzers.ConvergenceComparatorFactory): """Factory class for WinRateComparatorFactory.""" def \_\_call\_\_( self, baseline\_curve: analyzers.ConvergenceCurve, compared\_curve: analyzers.ConvergenceCurve, baseline\_quantile: float \= 0.5, compared\_quantile: float \= 0.5, ) \-> analyzers.ConvergenceComparator: return WinRateComparator( baseline\_curve\=baseline\_curve, compared\_curve\=compared\_curve, baseline\_quantile\=baseline\_quantile, compared\_quantile\=compared\_quantile, name\='win\_rate', ) \# Add WinRateComparator plots and visualize them. analyzed\_records\_with\_winrate \= ( analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=analyzed\_records, baseline\_algo\='random', comparator\_factory\=WinRateComparatorFactory(), ) ) analyzers.plot\_from\_records(analyzed\_records\_with\_winrate) Summarizing with Comparators[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#summarizing-with-comparators "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- One of the main benefits of using `ConvergenceComparator` is that these metrics are usually already normalized across different optimization problems and contexts. We provide utility functions to perform summarization procedures across `BenchmarkRecords`, which can used with the `summarize` feature of the `BenchmarkRecordAnalyzer`. Note that we can write a custom score and summary function, but we use our default summarizer that simply plots the distribution of scores across problems as histograms. import json \# Summarized across experimenters by giving reduced keys. def record\_to\_reduced\_keys(record): bbob\_dict \= json.loads(record.experimenter\_metadata\['bbob\_factory'\]) experimenter \= bbob\_dict.pop('name') return experimenter, json.dumps(bbob\_dict) summarized\_records \= analyzers.BenchmarkRecordAnalyzer.summarize(analyzed\_records\_with\_winrate, record\_to\_reduced\_keys) analyzers.plot\_from\_records(summarized\_records) References[](https://oss-vizier.readthedocs.io/en/latest/guides/benchmarks/analyzing_benchmarks.html#references "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------- * Benchmark analysis tools can be found [here](https://github.com/google/vizier/tree/main/vizier/benchmarks/analyzers.py) . * Convergence curve utils and comparators can be found [here](https://github.com/google/vizier/tree/main/vizier/_src/benchmarks/analyzers/convergence_curve.py) --- # Unknown Media ===== OSS Vizier has been featured in: Articles -------- - \`Google Research, 2022 & Beyond: Algorithmic Advances \`\_\_ - \`MarkTechPost \`\_\_ - \`The Sequence \`\_\_ - \`ML News by Weights & Biases \`\_\_ - \`Analytics India Magazine \`\_\_ - \`This Week in AI by Lighting AI \`\_\_ - \`gHacks \`\_\_ - \`WebBigdata (Japanese) \`\_\_ - \`Random Access (Spanish) \`\_\_ - \`Electronic Smith \`\_\_ - \`Deep Learning Weekly \`\_\_ - \`China Z (Chinese) \`\_\_ - \`TuringPost \`\_\_ - \`Open Data Science \`\_\_ Videos/Talks ------------ - \`Beijing Academy of Artificial Intelligence (BAAI) \`\_\_ - \`AutoML Conference 2023 Tutorial \`\_\_ (\`Hands-on Colab \`\_\_) - \`AutoML Seminar 2023 Talk \`\_\_ - \`AutoML Conference 2022 Paper Presentation \`\_\_ - \`AutoML Conference 2022 AutoRL Tutorial \`\_\_ - \`ML News by Yannic Kilcher \`\_\_ --- # Unknown Applications ============ OSS Vizier is used in the following: Codebases --------- - \`Vertex AI \`\_\_ - \`PyGlove \`\_\_ - \`OptFormer \`\_\_ - \`Init2winit \`\_\_ - \`Tensorflow Federated \`\_\_ - \`Tensorflow GNN \`\_\_ - \`CFU-Playground \`\_\_ - \`Architecture Gym (ArchGym) \`\_\_ - \`OpenML (Converter) \`\_\_ Guides + Courses ------ - \`Deep Learning Tuning Playbook \`\_\_ - \`Stanford STATS 285 (Fall 2023): Massive Computational Experiments, Painlessly \`\_\_ - \`Stanford XCS224U (Spring 2023): Natural Language Understanding \`\_\_ Papers ------ - \`Fishy: Layerwise Fisher Approximation for Higher-order Neural Network Optimization \`\_\_ - \`Massively Scaling Heteroscedastic Classifiers \`\_\_ - \`Towards Learning Universal Hyperparameter Optimizers with Transformers \`\_\_ - \`Task Selection for AutoML System Evaluation \`\_\_ --- # Bayesian Optimization Modeling — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * Bayesian Optimization Modeling * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/tfp/gp.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/gp.ipynb) Bayesian Optimization Modeling[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#bayesian-optimization-modeling "Link to this heading") ================================================================================================================================================================= The goal of this tutorial is to introduce Bayesian optimization workflows in OSS Vizier, including the underlying TensorFlow Probability (TFP) components and JAX/Flax functionality. The target audience is researchers and practitioners already well-versed in Bayesian optimization, who want to **define and train their own Gaussian Process surrogate models** for Bayesian optimization in OSS Vizier. Additional resources for TFP[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#additional-resources-for-tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------- If you’re new to TFP, a good place to start is [A tour of TensorFlow Probability](https://www.tensorflow.org/probability/examples/A_Tour_of_TensorFlow_Probability) . TFP began as a TensorFlow-only library, but now has a [JAX backend](https://www.tensorflow.org/probability/examples/TensorFlow_Probability_on_JAX) that is entirely independent of TensorFlow (such that “Tensor-Friendly Probability” might be a better backronym). This Colab uses TFP’s JAX backend (see the “Imports” cell for how to import it). Additional resources for Flax[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#additional-resources-for-flax "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- OSS Vizier’s Bayesian Optimization models are defined as [Flax](https://flax.readthedocs.io/en/latest/) modules. ### Imports[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#imports "Link to this heading") import chex import jax from jax import numpy as jnp, random, tree\_util import numpy as np import optax from mpl\_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt from tensorflow\_probability.substrates import jax as tfp from typing import Any \# Vizier models can freely access modules from vizier.\_src from vizier.\_src.benchmarks.experimenters.synthetic import bbob from vizier.jax import optimizers from vizier.\_src.jax import stochastic\_process\_model as spm tfd \= tfp.distributions tfb \= tfp.bijectors tfpk \= tfp.math.psd\_kernels Defining a GP surrogate model and hyperparameters[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#defining-a-gp-surrogate-model-and-hyperparameters "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To write a GP surrogate model, first write a coroutine that yields parameter specifications (`ModelParameter`) and returns a GP distribution. Downstream, the parameter specifications are used to define Flax module parameters. The inputs to the coroutine function represent the index points of the GP (in the remainder of this Colab, we refer to “inputs” and “index points” interchangeably). The rationale for the coroutine design is that it lets us automate the application of the parameter constraint and initialization functions (corresponding to hyperpriors, e.g.), and enables simultaneous specification of the model parameters and how their values are used to instantiate a GP. ### Coroutine example[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#coroutine-example "Link to this heading") The following cell shows a coroutine defining a GP with a squared exponential kernel and two parameters: the length scale of the kernel and the observation noise variance of the GP. def simple\_gp\_coroutine(inputs: chex.Array\=None): length\_scale \= yield spm.ModelParameter.from\_prior( tfd.Gamma(1., 1., name\='length\_scale')) amplitude \= 2. \# Non-trainable parameters may be defined as constants. kernel \= tfpk.ExponentiatedQuadratic( amplitude\=amplitude, length\_scale\=length\_scale) observation\_noise\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: jnp.exp(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: x\*\*2, name\='observation\_noise\_variance') return tfd.GaussianProcess( kernel, index\_points\=inputs, observation\_noise\_variance\=observation\_noise\_variance) ModelParameter[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#modelparameter "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------- `ModelParameter` may be used to define hyperpriors. ### Parameter specifications from priors[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#parameter-specifications-from-priors "Link to this heading") The length scale parameter has a Gamma prior. This is equivalent to defining a `ModelParameter` with a regularizer that computes the Gamma negative log likelihood and an initialization function that samples from the Gamma distribution. As the constraint was not specified, a default one is assigned which is the “default event space bijector” of the TFP distribution (each TFP distribution has a constraining bijector that maps the real line to the support of the distribution). ### Specifying parameters explicitly[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#specifying-parameters-explicitly "Link to this heading") Observation noise variance, which is passed to the Gaussian process and represents the scalar variance of zero-mean Gaussian noise in the observed labels, is not given a `tfd.Distribution` prior. Instead, it has its initialization, constraining, and regularization functions defined individually. Note that the initialization function is in the constrained space. Constraints[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#constraints "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- ModelParameter allows to define constraints on the model parameters using the ‘Constraint’ object which is initiated with a tuple of ‘bounds’ and ‘bijector’ function. Though the constraints are defined as part of the ModelParameter the Flax model itself does not use them, but rather it expects to receive parameter values already in the constrained space. This means that it’s the responsibility of the user/optimizer to pass the GP parameter values that are already in the constrained space. ### Exercise: Write a GP model[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#exercise-write-a-gp-model "Link to this heading") Write an ARD Gaussian Process model with three parameters: `signal_variance`, `length_scale`, and `observation_noise_variance`. (This is a slightly simplified version of the Vizier GP.) * `signal_variance` and `observation noise_variance` are both: * regularized by the function \\(f(x) = 0.01\\log(x)^2\\) * bounded to be positive. * `signal_variance` parameterizes a Matern 5/2 kernel, where the amplitude of the kernel is the square root of `signal_variance`. Use [`tfpk.MaternFiveHalves`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/matern.py#L414) . * `length_scale` has a \\(LogNormal(0, 1)\\) prior for each dimension. Assume there are 4 dimensions, and use [`tfd.Sample`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/distributions/sample.py) to build a 4-dimensional distribution consisting of IID LogNormal distributions. (Note that the `length_scale` parameter is a vector – all other parameters are scalars.) * In TFP, ARD kernels are implemented with [`tfpk.FeatureScaled`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/feature_scaled.py) , with `scale_diag` representing the length scale along each dimension. def vizier\_gp\_coroutine(inputs: chex.Array\=None): pass ### Solution[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#solution "Link to this heading") data\_dimensionality \= 2 def vizier\_gp\_coroutine(inputs: chex.Array\=None): """A coroutine that follows the \`ModelCoroutine\` protocol.""" signal\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: tfb.Softplus()(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: 0.01 \* jnp.log(x)\*\*2, name\='signal\_variance') length\_scale \= yield spm.ModelParameter.from\_prior( tfd.Sample( tfd.LogNormal(loc\=0., scale\=1.), sample\_shape\=\[data\_dimensionality\], name\='length\_scale'), constraint\=spm.Constraint(bounds\=(0.0, None))) kernel \= tfpk.MaternFiveHalves( amplitude\=jnp.sqrt(signal\_variance), validate\_args\=True) kernel \= tfpk.FeatureScaled( kernel, scale\_diag\=length\_scale, validate\_args\=True) observation\_noise\_variance \= yield spm.ModelParameter( init\_fn\=lambda x: jnp.exp(random.normal(x)), constraint\=spm.Constraint(bounds\=(0.0, 100.0), bijector\=tfb.Softplus()), regularizer\=lambda x: 0.01 \* jnp.log(x)\*\*2, name\='observation\_noise\_variance') return tfd.GaussianProcess( kernel\=kernel, index\_points\=inputs, observation\_noise\_variance\=observation\_noise\_variance, validate\_args\=True) To build a GP Flax module, instantiate a `StochasticProcessModel` with a GP coroutine as shown below. The module runs the coroutine in the `setup` and `__call__` methods to initialize the parameters and then instantiate the GP object with the given parameters. Recall that Flax modules have two primary methods: `init`, which initializes parameters, and `apply`, which computes the model’s forward pass given a set of parameters and input data. model \= spm.StochasticProcessModel(coroutine\=vizier\_gp\_coroutine) \# Sample some fake data. \# Assume we have \`num\_points\` observations, each with \`dim\` features. num\_points \= 12 \# Sample a set of index points. index\_points \= np.random.normal( size\=\[num\_points, data\_dimensionality\]).astype(np.float32) \# Sample function values observed at the index points observations \= np.random.normal(size\=\[num\_points\]).astype(np.float32) \# Call the Flax module's \`init\` method to obtain initial parameter values. init\_params \= model.init(random.PRNGKey(0), index\_points) We can observe the initial parameters values of the Flax model and see that they match with the ‘ModelParameter’ definitions in our coroutine. print(init\_params\['params'\]) To instantiate a GP with a set of parameters and index points, use the Flax module’s `apply` method. `apply` also returns the regularization `losses` for the parameters, in `mutables`. The regularization `losses` are treated as mutable state because they are recomputed internally with each forward pass of the model. For more on mutable state in Flax, see [this](https://flax.readthedocs.io/en/latest/guides/state_params.html) tutorial. gp, mutables \= model.apply( init\_params, index\_points, mutable\=\['losses'\]) assert isinstance(gp, tfd.GaussianProcess) Optimizing hyperparameters[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#optimizing-hyperparameters "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise: Loss function[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#exercise-loss-function "Link to this heading") Write down a loss function that takes a parameters dict and returns the loss value, using `model.apply`. The function will close over the observed data. The loss should be the sum of the GP negative log likelihood and the regularization losses. The regularization loss values are computed when the module is called, using the `ModelParameter` regularization functions. They are stored in a mutable variable collection called `"losses"`, using the Flax method [`sow`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen.html?highlight=sow#flax.linen.Module.sow) . def loss\_fn(params): ... return loss, {} \# Return an empty dict as auxiliary state. ### Solution[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#id1 "Link to this heading") def loss\_fn(params): gp, mutables \= model.apply({'params': params}, index\_points, mutable\=\['losses'\]) loss \= (\-gp.log\_prob(observations) + jax.tree\_util.tree\_reduce(jnp.add, mutables\['losses'\])) \# add the regularization losses. return loss, {} The gradients of the loss have the same structure as the `params` dict. grads \= jax.grad(loss\_fn, has\_aux\=True)(init\_params\['params'\])\[0\] print(grads) We can use `jax.tree_util` to take a step along the gradient (though in practice, with Optax, we can use `update` and `apply_updates` to update the parameters at each train step). learning\_rate \= 1e-3 updated\_params \= jax.tree\_util.tree\_map( lambda p, g: p \- learning\_rate \* g, init\_params\['params'\], grads) print(updated\_params) Optimize hyperparameters with Vizier optimizers[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#optimize-hyperparameters-with-vizier-optimizers "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Flax modules are often optimized using Optax which requires the developer to write a routine that initializes parameter values and then repeatedly computes the loss function gradients and updates the parameter values accordingly. [Vizier Optimizers](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#TODO) is a library of optimizers that automate the process of finding the optimal Flax parameter values and wrap optimizers from libraries such as Optax and Jaxopt in a common interface. To use a Vizier Optimizer you have to specify the following: * `setup` function which is used to generate the initial parameter values. * `loss_fn` function which is used for computing the loss function value and gradients. For example, the loss function of a GP model would be a marginal likelihood plus the parameters regularizations. * `rng` PRNGKey for controlling pseudo randomization. * `constraints` on the parameters (optional). Below we use the Vizier `JaxoptLbfgsB` optimizer to run a constrained L-BFGS-B algorithm. Unconstrainted optimizers (e.g. Adam) use a bijector function to map between the unconstrained space where the search is performed, and the constrained space where the loss function is evaluated. On the contrary, constrained optimizers (e.g. L-BGFS-B) use the constraint bounds directly in the search process. To pass the constraints bounds to the `JaxoptLbfgsB` optimizer we use the `spm.get_constraints` function that traverse the parameters defined in the module coroutine and extract their bounds. setup \= lambda rng: model.init(rng, index\_points)\['params'\] model\_optimizer \= optimizers.JaxoptLbfgsB( random\_restarts\=20, best\_n\=None ) constraints \= spm.get\_constraints(model) optimal\_params, \_ \= model\_optimizer(setup, loss\_fn, random.PRNGKey(0), constraints\=constraints) Predict on new inputs, conditional on observations[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#predict-on-new-inputs-conditional-on-observations "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To compute the posterior predictive GP on unseen points, conditioned on observed data, use the `precompute_predictive` and `posterior_predictive` methods of the Flax module. `precompute_predictive` must be called first; it runs and stores the Cholesky decomposition of the kernel matrix for the observed data. `posterior_predictive` then returns a posterior predictive GP at new index points, avoiding recomputation of the Cholesky. \# Precompute the Cholesky. \_, pp\_state \= model.apply( {'params': optimal\_params}, index\_points, observations, mutable\=\['predictive'\], method\=model.precompute\_predictive) \# Predict on new index points. predictive\_index\_points \= np.random.normal( size\=\[5, data\_dimensionality\]).astype(np.float32) pp\_dist \= model.apply( {'params': optimal\_params, \*\*pp\_state}, predictive\_index\_points, index\_points, observations, method\=model.posterior\_predictive) \# \`posterior\_predictive\` returns a TFP distribution, whose mean, variance, and \# samples we can use to compute an acquisition function. assert pp\_dist.mean().shape \== (5,) Optimize a black-box function[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#optimize-a-black-box-function "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- For an end-to-end example of Bayesian optimization, we’ll use the GP surrogate model defined above along with an Upper Confidence Bound acquisition function to try to find the maximum of the Weierstrass function. First, visualize the function surface. \# Use the Weierstrass function from Vizier's Black-Box Optimization Benchmarking \# (BBOB) library. bb\_fun \= bbob.Weierstrass \# Sample a set of index points in a 2D space. num\_points \= 6 max\_x \= np.array(2.).astype(np.float32) index\_points \= random.uniform( random.PRNGKey(3), shape\=\[num\_points, data\_dimensionality\], dtype\=jnp.float32) \* max\_x \# Compute function values observed at the index points. observations \= np.apply\_along\_axis( bb\_fun, axis\=1, arr\=index\_points).astype(np.float32) \# Define a grid of points in the function domain for plotting. n\_grid \= 100 x \= y \= np.linspace(0, max\_x, n\_grid, dtype\=np.float32) X, Y \= np.meshgrid(x, y) x\_grid \= np.vstack(\[X.ravel(), Y.ravel()\]).T y\_grid \= np.apply\_along\_axis(bb\_fun, axis\=1, arr\=x\_grid) Z \= y\_grid.reshape(X.shape) \# Plot the black-box function values. fig \= plt.figure(figsize\=(8, 8)) ax \= fig.add\_subplot(111, projection\='3d') ax.plot\_surface(X, Y, Z, alpha\=0.5) ax.scatter(index\_points\[:, 0\], index\_points\[:, 1\], observations, color\='r', label\='Initial observed data') plt.title('Black-box (Weierstrass) function values and observed data') plt.legend() plt.show() Next, run a few iterations of Bayesian optimization to maximize the black-box function given the observed data. A single iteration consists of the following steps: 1. Optimize the GP hyperparameters. 2. Find a suggestion that maximizes an Upper Confidence Bound acquisition function. In this example, we use grid search for the optimization. 3. Evaluate the black-box function on the suggestion and append it to the set of observed data. (Note that this simple Bayesopt algorithm is for educational purposes and that we’d expect Vizier’s GP bandit algorithm to give better results.) num\_bayesopt\_iter \= 5 \# At each iteration, redefine the loss function given the current observed data. def build\_loss\_fn(index\_points, observations): def loss\_fn(params): gp, mutables \= model.apply({'params': params}, index\_points, mutable\=\['losses'\]) loss \= (\-gp.log\_prob(observations) + jax.tree\_util.tree\_reduce(jnp.add, mutables\['losses'\])) \# add the regularization losses. return loss, {} return loss\_fn for i in range(num\_bayesopt\_iter): \# Update the loss function to condition on all observed data. loss\_fn \= build\_loss\_fn(index\_points, observations) \# Optimize the GP hyperparameters. optimal\_params, \_ \= model\_optimizer(setup, loss\_fn, random.PRNGKey(0), constraints\=constraints) \# Compute the posterior predictive distribution over a grid of points in the \# function domain (x\_grid). \_, pp\_state \= model.apply( {'params': optimal\_params}, index\_points, observations, mutable\=\['predictive'\], method\=model.precompute\_predictive) pp\_dist \= model.apply( {'params': optimal\_params, \*\*pp\_state}, x\_grid, index\_points, observations, method\=model.posterior\_predictive) \# Compute the acquisition function value at each point in the grid. pred\_mean \= pp\_dist.mean() ucb\_vec \= pred\_mean + 2. \* pp\_dist.stddev() \# Find the grid point with the highest acquisition function value. ind \= np.argmax(ucb\_vec) \# Evaluate the black box function at the selected point. f\_val \= bb\_fun(x\_grid\[ind\]) \# Visualize the surrogate model mean and acquisition function surface at this \# iteration. fig \= plt.figure(figsize\=(10, 10)) ax \= fig.add\_subplot(121, projection\='3d') W \= pred\_mean.reshape(X.shape) ax.plot\_surface(X, Y, W, alpha\=0.5) ax.scatter(index\_points\[:, 0\], index\_points\[:, 1\], observations, color\='r', label\='Observed data') ax.set\_title('Observed data and posterior predictive GP mean') ax.legend() ax \= fig.add\_subplot(122, projection\='3d') ucb \= ucb\_vec.reshape(X.shape) ax.plot\_surface(X, Y, ucb, alpha\=0.5) ax.scatter(\*x\_grid\[ind\], ucb\_vec\[ind\], color\='r', label\='New suggestion') ax.set\_title('Acquisition function') ax.legend() plt.show() \# Append the new suggestion and function value to the set of observations. index\_points \= np.concatenate(\[index\_points, x\_grid\[ind\]\[np.newaxis\]\]) observations \= np.concatenate( \[observations, np.array(f\_val).astype(np.float32)\[np.newaxis\]\]) print(f'Iteration: {i}') print(f'Acquisition function value at suggestion: {ucb\_vec\[ind\]}') print(f'Black-box function value at suggestion: {f\_val}') Deeper dive on selected topics in TFP[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#deeper-dive-on-selected-topics-in-tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As shown above, the Flax GP model makes use of a number of TFP components: * Distributions specify parameter priors (e.g. `tfd.Gamma`). The stochastic process model itself is also a TFP distribution, `tfd.GaussianProcess`. * Bijectors (e.g. `tfb.Softplus`) are used to constrain parameters for optimization, and may also be used for input/output warping. * PSD kernels (e.g. `tfpk.ExponentiatedQuadratic`) specify the kernel function for the stochastic process. The next sections of this Colab introduce these and how they’re used in Bayesopt modeling. ### `tfd.GaussianProcess` and friends[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#tfd-gaussianprocess-and-friends "Link to this heading") The stochastic process Flax modules return a TFP distribution in the [Gaussian Process](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/distributions/gaussian_process.py#L93) family (an instance of `tfd.GaussianProcess`, `tfd.StudentTProcess`, or `tfde.MultiTaskGaussianProcess`). This Colab doesn’t go into detail on TFP distributions, since advanced usage and implementation of distributions is rarely required for Bayesopt modeling with Vizier. For an overview of TFP distributions, see [TensorFlow Distributions: A Gentle Introduction](https://www.tensorflow.org/probability/examples/TensorFlow_Distributions_Tutorial) . Some of the methods of the Gaussian Process distribution are demonstrated below. [Gaussian Process Regression in TFP](https://www.tensorflow.org/probability/examples/Gaussian_Process_Regression_In_TFP) is also worth reading. \# Build a kernel function (see "PSD kernels" section below) and GP. num\_points \= 6 index\_points \= random.uniform( random.PRNGKey(3), shape\=\[num\_points, data\_dimensionality\], dtype\=jnp.float32) observations \= random.uniform( random.PRNGKey(4), shape\=\[num\_points\], dtype\=jnp.float32) kernel \= tfpk.MaternFiveHalves( amplitude\=2., length\_scale\=0.3, validate\_args\=True \# Run additional runtime checks; possibly expensive. ) observation\_noise\_variance \= jnp.ones(\[\], dtype\=observations.dtype) gp \= tfd.GaussianProcess( kernel, index\_points\=index\_points, observation\_noise\_variance\=observation\_noise\_variance, cholesky\_fn\=lambda x: tfp.experimental.distributions.marginal\_fns.retrying\_cholesky(x)\[0\], \# See commentary below. validate\_args\=True) \# Take 4 samples from the GP at the index points. s \= gp.sample(4, seed\=random.PRNGKey(0)) assert s.shape \== (4, num\_points) \# Compute the log likelihood of the sampled values. lp \= gp.log\_prob(s) assert lp.shape \== (4,) \# GPs can also be instantiated without index points, in which case the index \# points must be passed to method calls. gp\_no\_index\_pts \= tfd.GaussianProcess( kernel, observation\_noise\_variance\=observation\_noise\_variance) s \= gp\_no\_index\_pts.sample(index\_points\=index\_points, seed\=random.PRNGKey(0)) \# Predictive GPs conditioned on observations can be built with \# \`GaussianProcess.posterior\_predictive\`. The Flax module's \# \`precompute\_predictive\` and \`posterior\_predictive\` methods call this GP method. gprm \= gp.posterior\_predictive( observations\=observations, predictive\_index\_points\=predictive\_index\_points) \# \`gprm\` is an instance of \`tfd.GaussianProcessRegressionModel\`. This class can \# also be instantiated directly (as a side note -- this isn't necessary for \# modeling with Vizier). same\_gprm \= tfd.GaussianProcessRegressionModel( kernel, index\_points\=predictive\_index\_points, observation\_index\_points\=index\_points, observations\=observations, observation\_noise\_variance\=observation\_noise\_variance) Aside from the kernel, index points, and noise variance, note the `cholesky_fn` arg to the `GaussianProcess` constructor. `cholesky_fn` is a callable that takes a matrix and returns a Cholesky-like lower triangular factor. The default function adds a jitter of 1e-6 to the diagonal and then calls `jnp.linalg.cholesky`. An alternative, used in the Vizier GP, is `tfp.experimental.distributions.marginal_fns.retrying_cholesky`, which adds progressively larger jitter until the Cholesky decomposition succeeds. ### A side note on batch shape in TFP[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/gp.html#a-side-note-on-batch-shape-in-tfp "Link to this heading") tl;dr: Don’t worry about batch shape. TFP objects have a notion of batch shape, which is useful for vectorized computations. For more on this, see [Understanding TensorFlow Distributions Shapes](https://www.tensorflow.org/probability/examples/Understanding_TensorFlow_Distributions_Shapes) . For the purposes of Bayesopt in Vizier, JAX’s `vmap` means that our TFP objects can have a single parameterization with empty batch shape. For example, in the following loss function takes a scalar `amplitude`, and the kernel and GP both have empty batch shape. def loss\_fn(amplitude): \# \`a\` is a scalar. k \= tfpk.ExponentiatedQuadratic(amplitude\=amplitude) \# batch shape \[\] gp \= tfd.GaussianProcess(k, index\_points\=index\_points) \# batch shape \[\] return \-gp.log\_prob(observations) initial\_amplitude \= np.random.uniform(size\=\[50\]) losses \= jax.vmap(loss\_fn)(initial\_amplitude) assert losses.shape \== (50,) We could also vectorize the loss computation by using a batched GP. In this simple case, the code is identical except that `vmap` is removed. Now, the kernel and GP represent a “batch” of kernels and GPs, each with different parameter values. Working with batch shape requires additional accounting on the part of the user to ensure that parameter shapes broadcast correctly, the correct dimensions are reduced over, etc. For Vizier’s use case, we find it simpler to rely on `vmap`. def loss\_fn(amplitude): \# \`a\` has shape \[50\]. k \= tfpk.ExponentiatedQuadratic(amplitude\=amplitude) \# batch shape \[50\] gp \= tfd.GaussianProcess(k, index\_points\=index\_points) \# batch shape \[50\] return \-gp.log\_prob(observations) initial\_amplitude \= np.random.uniform(size\=\[50\]) \# No vmap. losses \= loss\_fn(initial\_amplitude) assert losses.shape \== (50,) --- # Vizier Basics — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Vizier Basics * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/user/running_vizier.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/running_vizier.ipynb) Vizier Basics[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#vizier-basics "Link to this heading") =================================================================================================================================== Below, we provide examples of how to: * Define a problem statement and study configuration. * Start a client. * (Optionally) Connect the client to a server. * Perform a typical tuning loop. * Use other client APIs. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] from vizier import service from vizier.service import clients from vizier.service import pyvizier as vz from vizier.service import servers Setting up the problem statement[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#setting-up-the-problem-statement "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here we setup the problem statement, which contains information about the search space and the metrics to optimize. problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', 0.0, 1.0) problem.search\_space.root.add\_float\_param('y', 0.0, 1.0) problem.metric\_information.append(vz.MetricInformation(name\='maximize\_metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) def evaluate(x: float, y: float) \-> float: return x\*\*2 \- y\*\*2 Setting up the study configuration[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#setting-up-the-study-configuration "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The study configuration contains additional information, such as the algorithm to use and level of noise that we think the objective will have. study\_config \= vz.StudyConfig.from\_problem(problem) study\_config.algorithm \= 'DEFAULT' Setting up the client[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#setting-up-the-client "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Starts a `study_client`, which will implicitly create a local Vizier Service which will be shared across other clients in the same Python process. Studies will then be stored locally in a SQL database file located at `service.VIZIER_DB_PATH`. study\_client \= clients.Study.from\_study\_config(study\_config, owner\='owner', study\_id\='example\_study\_id') print('Local SQL database file located at: ', service.VIZIER\_DB\_PATH) Obtaining suggestions[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#obtaining-suggestions "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Start requesting suggestions from the server, for evaluating objectives. Suggestions can be made sequentially (`count=1`) or in batches (`count>1`). for i in range(10): suggestions \= study\_client.suggest(count\=1) for suggestion in suggestions: x \= suggestion.parameters\['x'\] y \= suggestion.parameters\['y'\] objective \= evaluate(x, y) print(f'Iteration {i}, suggestion ({x},{y}) led to objective value {objective}.') final\_measurement \= vz.Measurement({'maximize\_metric': objective}) suggestion.complete(final\_measurement) Find optimal trial[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#find-optimal-trial "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Find the best objective so far, with corresponding suggestion value. For multiobjective cases, there may be multiple outputs of `optimal_trials()`, all corresponding to a Pareto-optimal curve. for optimal\_trial in study\_client.optimal\_trials(): optimal\_trial \= optimal\_trial.materialize() print("Optimal Trial Suggestion and Objective:", optimal\_trial.parameters, optimal\_trial.final\_measurement) Other client commands[](https://oss-vizier.readthedocs.io/en/stable/guides/user/running_vizier.html#other-client-commands "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- The `study_client` can also send other requests, such as the following: study\_client.get\_trial(1) \# Get the first trial. study\_client.trials() \# Get all trials so far. \# Obtain only the completed trials. trial\_filter \= vz.TrialFilter(status\=\[vz.TrialStatus.COMPLETED\]) study\_client.trials(trial\_filter\=trial\_filter) --- # Bijectors — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * Bijectors * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/tfp/bijectors.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/bijectors.ipynb) Bijectors[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/bijectors.html#bijectors "Link to this heading") ============================================================================================================================== TFP [bijectors](https://github.com/tensorflow/probability/tree/main/tensorflow_probability/python/bijectors) represent (mostly) invertible, smooth functions. For Bayesopt modeling in Vizier, they are used to: * to constrain parameter values for optimization in an unconstrained space. * For input warping or output warping (e.g. the [Yeo Johnson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html) bijector). Each bijector implements at least 3 methods: * `forward`, * `inverse`, and * (at least) one of `forward_log_det_jacobian` and `inverse_log_det_jacobian`. When bijectors are used to transform distributions (with `tfd.TransformedDistribution`), the log det Jacobian ensures that the transformation is volume-preserving and the distribution’s PDF still integrates to 1. Bijectors also cache the forward and inverse computations, and log-det-Jacobians. This has two purposes: * Avoid repeating potentially expensive computations (as with the `CholeskyOuterProduct` bijector). * Maintain numerical precision so that `b.inverse(b.forward(x)) == x`. Below is an illustration of preservation of numerical precision. Although TFP library bijectors are written in TensorFlow (and automatically converted to JAX with TFP’s rewrite machinery), user-defined bijectors can be written in JAX directly. For example, a complete JAX reimplementation of the `Exp` bijector is below. TFP’s library already contains an `Exp` bijector and it’s JAX supported, so it isn’t actually necessary to implement this. While it’s rare that Vizier users will have to implement new TFP components, we include this as an example to show how it would be done using TFP’s JAX backend, since all TFP library bijectors are written in TensorFlow. Imports[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/bijectors.html#imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------- from jax import numpy as jnp import numpy as np from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfb \= tfp.bijectors tfpk \= tfp.math.psd\_kernels class Exp(tfb.AutoCompositeTensorBijector): def \_\_init\_\_(self, validate\_args\=False, name\='exp'): """Instantiates the \`Exp\` bijector.""" parameters \= dict(locals()) super(Exp, self).\_\_init\_\_( forward\_min\_event\_ndims\=0, validate\_args\=validate\_args, parameters\=parameters, \# TODO(emilyaf): explain why this is necessary. name\=name) @classmethod def \_parameter\_properties(cls, dtype): return dict() @classmethod def \_is\_increasing(cls): return True def \_forward(self, x): return jnp.exp(x) def \_inverse(self, y): return jnp.log(y) def \_inverse\_log\_det\_jacobian(self, y): return \-jnp.log(y) \# Make sure it gives the same results as the TFP library bijector. x \= np.random.normal(size\=\[5\]) tfp\_exp \= tfb.Exp() my\_exp \= Exp() np.testing.assert\_allclose(tfp\_exp.forward(x), my\_exp.forward(x)) np.testing.assert\_allclose(tfp\_exp.forward\_log\_det\_jacobian(x), my\_exp.forward\_log\_det\_jacobian(x), rtol\=1e-6) TFP’s bijector library includes: * Simple bijectors (for example, there are many more): * `Scale(k)` multiplies its input by `k`. * `Shift(k)` adds `k` to its input. * `Sigmoid()` computes the sigmoid function. * `FillScaleTriL()` packs its input, a vector, into a lower-triangular matrix. * … * `Invert` wraps any bijector instance and swaps its forward and inverse methods, e.g. `inv_sigmoid = tfb.Invert(tfb.Sigmoid())`. * `Chain` composes a series of bijectors. The function \\(f(x) = 3 + 2x\\) can be expressed as `tfb.Chain([tfb.Shift(3.), tfb.Scale(2.)])`. Note that the bijectors in the list are applied from right to left. * `JointMap` applies a nested structure of bijectors to an identical nested structure of inputs. `build_constraining_bijector`, shown above, returns a `JointMap` which applies a nested structure of bijectors to an identical nested structure of inputs. Vizier `get_constraints` function could be used to generate a `JointMap` based on the `Constraint`s of the `ModelParameter`s defined in the coroutine. * `Restructure` packs the elements of one nested structure (e.g. a list) into a different structure (e.g. a dict). `spm.build_restructure_bijector`, for example, is a `Chain` bijector that takes a vector of parameters, splits it into a list, and packs the elements of the list into a dictionary with the same structure as the Flax parameters dict. Exercise: Bijectors[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/bijectors.html#exercise-bijectors "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------- Write a bijector (with `Chain`) that computes the function \\(f(x) = e^{x^2 + 1}\\). b \= tfb.Chain(\[...\]) f \= lambda x: jnp.exp(x\*\*2 + 1) x \= np.random.normal(size\=\[5\]) np.testing.assert\_allclose(f(x), b.forward(x)) Solution[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/bijectors.html#solution "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------- b \= tfb.Chain(\[tfb.Exp(), tfb.Shift(1.), tfb.Square()\]) --- # Frequently Used Import Targets — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [API Reference](https://oss-vizier.readthedocs.io/en/stable/api_reference/index.html) * Frequently Used Import Targets * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/api_reference/faq_imports.rst.txt) * * * Frequently Used Import Targets[](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html#frequently-used-import-targets "Link to this heading") ==================================================================================================================================================================== Includes a brief summary of important symbols and modules. Service Users[](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html#service-users "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------- If you write client code interacting with the OSS Vizier service, use these import targets: * **from vizier.service import pyvizier as vz**: Exposes the same set of symbol names as `vizier.pyvizier`. `vizier.service.pyvizier.Foo` is a subclass or an alias of `vizier.pyvizier.Foo`, and can be converted into protobufs. * **from vizier.service import …**: Include binaries and internal utilities. Algorithm Developers[](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html#algorithm-developers "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------ If you write algorithm code (Designers or Pythia policies) in OSS Vizier, use these import targets: * **from vizier import pyvizier as vz**: Pure python building blocks of OSS Vizier. Cross-platform code, including Pythia policies, must use this `pyvizier` instance. * `Trial` and `ProblemStatement` are important classes. * **from vizier.pyvizier import converters**: Convert between `pyvizier` objects and numpy arrays. * `TrialToNumpyDict`: Converts parameters (and metrics) into a dict of numpy arrays. Preferred conversion method if you intended to train an embedding of categorical/discrete parameters, or data includes missing parameters or metrics. * `TrialToArrayConverter`: Converts parameters (and metrics) into an array. * **from vizier.interfaces import serializable**: Abstractions for serializable objects. * `PartiallySerializable`, `Serializable` ### Algorithm Abstractions[](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html#algorithm-abstractions "Link to this heading") * **from vizier import pythia**: Abstractions for Pythia policies. * `Policy`, `PolicySupporter`: Key abstractions. * `LocalPolicyRunner`: Use it for running a `Policy` in RAM. * **from vizier import algorithms**: Abstractions for algorithms. * `Designer`: Stateful algorithm abstraction. * `DesignerPolicy`: Wraps `Designer` into a Pythia Policy. * `GradientFreeMaximizer`: For optimizing acquisition functions. * `(Partially)SerializableDesigner`: Designers who wish to optimize performance by saving states. ### Tensorflow Modules[](https://oss-vizier.readthedocs.io/en/stable/api_reference/faq_imports.html#tensorflow-modules "Link to this heading") * **from vizier import tfp**: Tensorflow-Probability utilities. * `acquisitions`: Acquisition functions module. * `AcquisitionFunction`: Abstraction. * `UpperConfidenceBound`, `ExpectedImprovement`, etc. * `bijectors`: Bijectors module. * `YeoJohnson`: Implements both Yeo-Johnson and Box-Cox transformations. * `optimal_power_transformation`: Returns the optimal power transformation. * `flip_sign`: returns a sign-flip bijector. * **from vizier import keras as vzk**: * `vzk.layers`: Layers usually wrapping `tfp` classes. * `variable_from_prior`: Utility layer for handling regularized variables. * `vzk.models`: Most of the useful models don’t easily fit into Keras’s `Model` abstraction, but we may add some for display. * `vzk.optim`: Wrappers around optimizers in `tfp` or `keras`. --- # Distributed Vizier — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Distributed Vizier * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/user/distributed.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/distributed.ipynb) Distributed Vizier[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#distributed-vizier "Link to this heading") ========================================================================================================================================== This documentation shows how to perform distributed optimization over multiple clients. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#installation-and-reference-imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] import multiprocessing from vizier import service from vizier.service import clients from vizier.service import pyvizier as vz from vizier.service import servers Regular setup[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#regular-setup "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------- We setup a regular study configuration below. study\_config \= vz.StudyConfig() study\_config.search\_space.root.add\_float\_param('x', 0.0, 1.0) study\_config.metric\_information.append(vz.MetricInformation(name\='metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) study\_config.algorithm \= 'DEFAULT' def evaluate(x: float) \-> float: return 2\*x \- x\*\*2 Server creation[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#server-creation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ Unlike the single-client case, in the distributed case, we require a single explicit server to accept requests from all other client processses. Details such as the `host`, `port`, `database_url`, `policy_factory`, etc. can be configured in the server’s initializer. server \= servers.DefaultVizierServer() \# Ideally created on a separate process such as a server machine. Client parallelization[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#client-parallelization "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- We may simultaneously create multiple clients to work on the same study, useful for parallelizing evaluation workload. All client processes (on a single machine or over multiple machines) will connect to this server via a globally specified `endpoint`. clients.environment\_variables.server\_endpoint \= server.endpoint \# Server address. study\_client \= clients.Study.from\_study\_config(study\_config, owner\='owner', study\_id \= 'example\_study\_id') \# Now connects to the explicitly created server. another\_study\_client \= clients.Study.from\_resource\_name(study\_client.resource\_name) \# Another way to fork clients. Distributed suggestions[](https://oss-vizier.readthedocs.io/en/stable/guides/user/distributed.html#distributed-suggestions "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------- We may now distribute our workflow, with each worker/client using the same loop below. Each client requires a unique `client_id` however, to ensure the server can identify client workers and distribute workloads properly. def tuning\_loop(client\_id: str): for i in range(10): suggestions \= study\_client.suggest(count\=1, client\_id\=client\_id) for suggestion in suggestions: objective \= evaluate(suggestion.parameters\['x'\]) final\_measurement \= vz.Measurement({'metric': objective}) suggestion.complete(final\_measurement) For example, we may perform a threadpool and construct multiple clients to parallelize evaluations on a single machine. NUM\_CLIENTS \= 10 NUM\_TRIALS\_PER\_CLIENT \= 50 pool \= multiprocessing.pool.ThreadPool(NUM\_CLIENTS) pool.map(tuning\_loop, range(NUM\_CLIENTS)) --- # Designers — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Designers * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/developer/designers.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/designers.ipynb) Designers[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html#designers "Link to this heading") =========================================================================================================================== This documentation will allow a developer to use the Designer API for typical algorithm design. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from typing import Optional, Sequence import numpy as np from vizier import algorithms as vza from vizier import pythia from vizier import pyvizier as vz from vizier.algorithms import designers Designers[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html#id1 "Link to this heading") --------------------------------------------------------------------------------------------------------------------- The `Designer` API is an intuitive abstraction for writing and _designing_ algorithms. It only requires two basic methods, `update()` and `suggest()`, shown below. The source of truth for `Designer` can be found [here](https://github.com/google/vizier/blob/main/vizier/algorithms/__init__.py) . class Designer(...): """Suggestion algorithm for sequential usage.""" @abc.abstractmethod def update(self, completed: CompletedTrials, all\_active: ActiveTrials) \-> None: """Updates recently completed and ALL active trials into the designer's state.""" @abc.abstractmethod def suggest(self, count: Optional\[int\] \= None) \-> Sequence\[vz.TrialSuggestion\]: """Make new suggestions.""" Every time `update()` is called, the `Designer` will get any newly `COMPLETED` trials since the last `update()` call, and will get all `ACTIVE` trials at the current moment in time. **Note:** Trials which may have been provided as `ACTIVE` in previous `update()` calls, can be provided as `COMPLETED` in subsequent `update()` calls. GP-Bandit Designer Example[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html#gp-bandit-designer-example "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------- The following example, using the default GP-Bandit algorithm, shows how to interact with Vizier designers. \# The problem statement (which parameters are being optimized) problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', 0.0, 1.0) problem.search\_space.root.add\_float\_param('y', 0.0, 1.0) problem.metric\_information.append( vz.MetricInformation( name\='maximize\_metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) \# Create a new designer object designer \= designers.VizierGPBandit(problem) \# Ask the designer for 2 suggestions suggestions \= designer.suggest(count\=2) In this case, since the designer was not update with any `COMPLETED` or `ACTIVE` trials, it will produce suggestions which will look like: \[TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.5, 'y': 0.5}), metadata\=Metadata((namespace:, items: {'seeded': 'center'}), current\_namespace\=)),\ TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.10274669379450661, 'y': 0.10191725529767912}), metadata\=Metadata((namespace:, items: {}), current\_namespace\=))\] Note that the first suggestion is seeded at the center of the search space, and the second suggestion is random. If we call `designer.suggest()` again before calling `update()`, the designer will produce an identical first suggestion at the center of the search space, and a second random suggestion. Only when we call `update()`, will the designer update its internal state and generate different suggestions: completed\_trials \= \[\] for suggestion in suggestions: metric\_value \= np.random.random() \# Make up a fake metric value. suggestion.to\_trial().complete( vz.Measurement(metrics\={'maximize\_metric': metric\_value}) ) \# Update the designer with the completed trials. designer.update(vza.CompletedTrials(completed\_trials), vza.ActiveTrials()) \# Ask for more suggestions. new\_suggestions \= designer.suggest(count\=2) Thus `COMPLETED` trials should be incrementally updated, while all `ACTIVE` trials are passed to the designer in every `update()` call. A `Designer` can also be seeded with pre-existing data. Consider the following example: \# Make a fresh designer. designer \= designers.VizierGPBandit(problem) \# Create completed trials representing pre-existing training data. trials \= \[vz.Trial(parameters\={'x': 0.5, 'y': 0.6}).complete(vz.Measurement(metrics\={'maximize\_metric': 0.3}))\] designer.update(vza.CompletedTrials(trials), vza.ActiveTrials()) \# As the designer for suggestions. suggestions \= designer.suggest(count\=2) In this case, the designer will **not** return a first trial seeded at the center of the search space, since it has been updated with completed trials. The new suggestions will look something like: \[TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.7199945005054509, 'y': 0.3800034493548722}), ...\] Additional References[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html#additional-references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- * Our [designers folder](https://github.com/google/vizier/tree/main/vizier/_src/algorithms/designers) contains examples of designers. * Our [evolution folder](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/evolution) contains examples of creating evolutionary designers, such as [NSGA2](https://ieeexplore.ieee.org/document/996017/) . * Our [designer testing routine](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/testing/test_runners.py) contains up-to-date examples on interacting with designers. --- # Bijectors — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * Bijectors * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/tfp/bijectors.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/bijectors.ipynb) Bijectors[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/bijectors.html#bijectors "Link to this heading") ============================================================================================================================== TFP [bijectors](https://github.com/tensorflow/probability/tree/main/tensorflow_probability/python/bijectors) represent (mostly) invertible, smooth functions. For Bayesopt modeling in Vizier, they are used to: * to constrain parameter values for optimization in an unconstrained space. * For input warping or output warping (e.g. the [Yeo Johnson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html) bijector). Each bijector implements at least 3 methods: * `forward`, * `inverse`, and * (at least) one of `forward_log_det_jacobian` and `inverse_log_det_jacobian`. When bijectors are used to transform distributions (with `tfd.TransformedDistribution`), the log det Jacobian ensures that the transformation is volume-preserving and the distribution’s PDF still integrates to 1. Bijectors also cache the forward and inverse computations, and log-det-Jacobians. This has two purposes: * Avoid repeating potentially expensive computations (as with the `CholeskyOuterProduct` bijector). * Maintain numerical precision so that `b.inverse(b.forward(x)) == x`. Below is an illustration of preservation of numerical precision. Although TFP library bijectors are written in TensorFlow (and automatically converted to JAX with TFP’s rewrite machinery), user-defined bijectors can be written in JAX directly. For example, a complete JAX reimplementation of the `Exp` bijector is below. TFP’s library already contains an `Exp` bijector and it’s JAX supported, so it isn’t actually necessary to implement this. While it’s rare that Vizier users will have to implement new TFP components, we include this as an example to show how it would be done using TFP’s JAX backend, since all TFP library bijectors are written in TensorFlow. Imports[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/bijectors.html#imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------- from jax import numpy as jnp import numpy as np from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfb \= tfp.bijectors tfpk \= tfp.math.psd\_kernels class Exp(tfb.AutoCompositeTensorBijector): def \_\_init\_\_(self, validate\_args\=False, name\='exp'): """Instantiates the \`Exp\` bijector.""" parameters \= dict(locals()) super(Exp, self).\_\_init\_\_( forward\_min\_event\_ndims\=0, validate\_args\=validate\_args, parameters\=parameters, \# TODO(emilyaf): explain why this is necessary. name\=name) @classmethod def \_parameter\_properties(cls, dtype): return dict() @classmethod def \_is\_increasing(cls): return True def \_forward(self, x): return jnp.exp(x) def \_inverse(self, y): return jnp.log(y) def \_inverse\_log\_det\_jacobian(self, y): return \-jnp.log(y) \# Make sure it gives the same results as the TFP library bijector. x \= np.random.normal(size\=\[5\]) tfp\_exp \= tfb.Exp() my\_exp \= Exp() np.testing.assert\_allclose(tfp\_exp.forward(x), my\_exp.forward(x)) np.testing.assert\_allclose(tfp\_exp.forward\_log\_det\_jacobian(x), my\_exp.forward\_log\_det\_jacobian(x), rtol\=1e-6) TFP’s bijector library includes: * Simple bijectors (for example, there are many more): * `Scale(k)` multiplies its input by `k`. * `Shift(k)` adds `k` to its input. * `Sigmoid()` computes the sigmoid function. * `FillScaleTriL()` packs its input, a vector, into a lower-triangular matrix. * … * `Invert` wraps any bijector instance and swaps its forward and inverse methods, e.g. `inv_sigmoid = tfb.Invert(tfb.Sigmoid())`. * `Chain` composes a series of bijectors. The function f(x)\=3+2x can be expressed as `tfb.Chain([tfb.Shift(3.), tfb.Scale(2.)])`. Note that the bijectors in the list are applied from right to left. * `JointMap` applies a nested structure of bijectors to an identical nested structure of inputs. `build_constraining_bijector`, shown above, returns a `JointMap` which applies a nested structure of bijectors to an identical nested structure of inputs. Vizier `get_constraints` function could be used to generate a `JointMap` based on the `Constraint`s of the `ModelParameter`s defined in the coroutine. * `Restructure` packs the elements of one nested structure (e.g. a list) into a different structure (e.g. a dict). `spm.build_restructure_bijector`, for example, is a `Chain` bijector that takes a vector of parameters, splits it into a list, and packs the elements of the list into a dictionary with the same structure as the Flax parameters dict. Exercise: Bijectors[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/bijectors.html#exercise-bijectors "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------- Write a bijector (with `Chain`) that computes the function f(x)\=ex2+1. b \= tfb.Chain(\[...\]) f \= lambda x: jnp.exp(x\*\*2 + 1) x \= np.random.normal(size\=\[5\]) np.testing.assert\_allclose(f(x), b.forward(x)) Solution[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/bijectors.html#solution "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------- b \= tfb.Chain(\[tfb.Exp(), tfb.Shift(1.), tfb.Square()\]) [**AI features? Redis belongs in your hot path.** Vector search, semantic cache, agent memory. **Get started**](https://server.ethicalads.io/proxy/click/10370/019da328-371d-78e2-85f2-3aa930f414ec/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10370/019da328-371d-78e2-85f2-3aa930f414ec/) --- # Search Spaces — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Search Spaces * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/user/search_spaces.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/search_spaces.ipynb) Search Spaces[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#search-spaces "Link to this heading") ================================================================================================================================== Below, we provide examples of how to: * Setup a flat search space consisting of all four parameter types and additional auxiliary parameter types. * Setup a conditional search space correctly. * Reparameterize search spaces, which is useful for combinatorial search spaces. * Use infeasibility to define shaped search spaces. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import math from typing import List from vizier import pyvizier as vz Flat search spaces[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#flat-search-spaces "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------- Below are the core primitive parameter types and their specifications: * `DOUBLE`: Continuous range of possible values in the closed interval \[a,b\] for some real numbers a≤b. * `INTEGER`: Integer range of possible values in \[a,b\]⊂Z for some integers a≤b. * `DISCRETE`: Finite, ordered set of values from R. * `CATEGORICAL`: Unordered list of strings. flat\_problem \= vz.ProblemStatement() flat\_problem\_root \= flat\_problem.search\_space.root flat\_problem\_root.add\_float\_param(name\='double', min\_value\=0.0, max\_value\=1.0) flat\_problem\_root.add\_int\_param(name\='int', min\_value\=1, max\_value\=10) flat\_problem\_root.add\_discrete\_param( name\='discrete', feasible\_values\=\[0.1, 0.3, 0.5\]) flat\_problem\_root.add\_categorical\_param( name\='categorical', feasible\_values\=\['a', 'b', 'c'\]) PyVizier also has a `BOOLEAN` parameter which under-the-hood, is a binary `CATEGORICAL` parameter with values `'True'` and `'False'`. flat\_problem\_root.add\_bool\_param(name\='bool') A default value for seeding the study may be used when constructing a parameter. flat\_problem\_root.add\_float\_param( name\='double\_with\_default', min\_value\=0.0, max\_value\=1.0, default\_value\=0.5) Scaling[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#scaling "Link to this heading") ---------------------------------------------------------------------------------------------------------------------- Each of the numerical parameter types (`DOUBLE`, `INTEGER`, `DISCRETE`) may also have a **scaling type**, which toggles whether optimization occurs over a transformed space. \# Default scaling used. flat\_problem\_root.add\_float\_param( name\='double\_uniform', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.LINEAR) \# Points near min\_value are more important. flat\_problem\_root.add\_float\_param( name\='double\_log', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) \# Points near the max\_value are more important. flat\_problem\_root.add\_float\_param( name\='double\_reverse\_log', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) \# Default scaling used for DISCRETE parameters. flat\_problem\_root.add\_discrete\_param( name\='discrete\_uniform', feasible\_values\=\[0.1, 0.3, 0.5\], scale\_type\=vz.ScaleType.UNIFORM\_DISCRETE) Conditional search spaces[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#conditional-search-spaces "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------- Sometimes, **child parameters** only exist in specific scenarios or _conditions_ when a **parent parameter** is equal to one or more specific values. Example: Momentum hyperparameters are used by the [Adam optimizer](https://arxiv.org/abs/1412.6980) , but not stochastic gradient descent (SGD). **Caveat:** Since the value of a “learning rate” depends strongly on the optimizer being used (e.g. a learning rate of 0.1 to SGD means completely differently to Adam), we must create two separate child parameters, rather than sharing a single one. conditional\_problem \= vz.ProblemStatement() conditional\_problem\_root \= conditional\_problem.search\_space.root optimizer \= conditional\_problem\_root.add\_categorical\_param( name\='optimizer', feasible\_values\=\['sgd', 'adam'\]) \# SGD child parameters optimizer.select\_values(\['sgd'\]).add\_float\_param( 'sgd\_learning\_rate', min\_value\=0.0001, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) \# Adam child parameters optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_learning\_rate', min\_value\=0.0001, max\_value\=1.0, scale\_type\=vz.ScaleType.LOG) optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_beta1', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) optimizer.select\_values(\['adam'\]).add\_float\_param( 'adam\_beta2', min\_value\=0.0, max\_value\=1.0, scale\_type\=vz.ScaleType.REVERSE\_LOG) Combinatorial Reparamterization[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#combinatorial-reparamterization "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- When dealing with a combinatorial search space X, one way to easily deal with such cases is to construct a reparameterization. Mathematically, this means finding a practical search space Z and surjective mapping Φ:Z→X. Below is an example over the space of permutations of size N, where our mapping utilizes the [Lehmer code](https://en.wikipedia.org/wiki/Lehmer_code) . N \= 10 \# Setup search space. permutation\_problem \= vz.ProblemStatement() for n in range(N): permutation\_problem.search\_space.root.add\_int\_param( name\=str(n), min\_value\=0, max\_value\=n) def compute\_index(trial: vz.Trial) \-> int: """Computes index from Lehmer code.""" index \= 0 for n in range(N): index += trial.parameters.get\_value(str(n)) \* math.factorial(n) return index def compute\_permutation(index: int) \-> List\[int\]: """Outputs a N-permutation as a list of indices.""" all\_indices \= list(range(N)) temp\_index \= index output \= \[\] for k in range(1, N + 1): factorial\_value \= math.factorial(N \- k) value \= all\_indices\[temp\_index // factorial\_value\] output.append(value) all\_indices.remove(value) temp\_index \= temp\_index % factorial\_value return output def phi(trial: vz.Trial) \-> List\[int\]: """Maps a suggestion to a permutation.""" return compute\_permutation(compute\_index(trial)) Infeasibility[](https://oss-vizier.readthedocs.io/en/stable/guides/user/search_spaces.html#infeasibility "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------- Consider an optimization problem where we only consider float parameters (x,y) from the unit disk x2+y2≤1. For such a scenario, we may denote any parameters outside of this area to be **infeasible**. disk\_problem \= vz.ProblemStatement() disk\_problem\_root \= disk\_problem.search\_space.root disk\_problem\_root.add\_float\_param(name\='x', min\_value\=-1.0, max\_value\=1.0) disk\_problem\_root.add\_float\_param(name\='y', min\_value\=-1.0, max\_value\=1.0) def evaluate(trial: vz.Trial) \-> vz.Trial: x \= trial.parameters\['x'\] y \= trial.parameters\['y'\] if x\*\*2 + y\*\*2 <= 1: trial.complete(vz.Measurement(metrics\={'sum': x + y})) else: trial.complete(vz.Measurement(), infeasibility\_reason\='Outside of range.') return trial [**AI features? Redis belongs in your hot path.** Vector search, semantic cache, agent memory. **Get started**](https://server.ethicalads.io/proxy/click/10370/019da328-371d-78e2-85f2-3aa930f414ec/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10370/019da328-371d-78e2-85f2-3aa930f414ec/) --- # PSD kernels — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * PSD kernels * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/tfp/kernels.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/kernels.ipynb) PSD kernels[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/kernels.html#psd-kernels "Link to this heading") ================================================================================================================================ TFP’s [PSD kernels](https://github.com/tensorflow/probability/tree/main/tensorflow_probability/python/math/psd_kernels) compute positive semidefinite kernel functions. A PSD kernel instance is a required arg to TFP’s Gaussian Process distribution, so specifying a GP model coroutine will generally involve defining a PSD kernel as an intermediate. PSD kernel subclasses take hyperparameters, such as amplitude and length scale, as constructor args. They have three primary public methods: `apply`, `matrix`, and `tensor`, each of which computes the kernel function pairwise on inputs in different ways: * `apply` computes the value of the kernel function at a pair of (batches of) input locations. It’s the only required method for subclasses: `matrix` and `tensor` are implemented in terms of `apply` (except when a more efficient method exists to compure pairwise kernel matrices). * `matrix` computes the value of the kernel _pairwise_ on two (batches of) lists of input examples. When the two collections are the same the result is called the [Gram matrix](https://en.wikipedia.org/wiki/Gramian_matrix) . `matrix` is the most important method for GPs. * `tensor` generalizes `matrix`, taking rank `k1` and `k2` collections of input examples to a rank `k1 + k2` collection of kernel values. (We mention `tensor` for completeness, but it isn’t relevant to GPs). PSD kernels have somewhat complex [shape semantics](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/positive_semidefinite_kernel.py#L97) , due to the need to define which input dimensions should be included in pairwise computations and which should be treated as batch dimensions (denoting independent sets of input points.) Imports[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/kernels.html#imports "Link to this heading") ------------------------------------------------------------------------------------------------------------------------ from jax import numpy as jnp import numpy as np from tensorflow\_probability.python.internal import dtype\_util from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfpk \= tfp.math.psd\_kernels Some examples of PSD kernel usage: \# Construct a MaternFiveHalves kernel (with empty batch shape). amplitude \= 2. length\_scale \= 0.5 k \= tfpk.MaternFiveHalves( amplitude\=amplitude, length\_scale\=length\_scale) \# Randomly sample some input data. num\_features \= 5 num\_observations \= 12 x \= np.random.normal(size\=\[num\_observations, num\_features\]) \# \`matrix\` computes pairwise kernel values for the Cartesian product over the \# second-to-rightmost dimension of the inputs. Following the terminology in the \# PSD kernel docstring, there is a single example dimension (and single feature \# dimension). assert k.matrix(x, x).shape \== (12, 12) \# Calling \`matrix\` on inputs of shape \[12, d\] and \[10, d\] results in a kernel \# matrix of shape (12, 10) y \= np.random.normal(size\=\[10, num\_features\]) assert k.matrix(x, y).shape \== (12, 10) ARD kernels in TFP are implemented with the `FeatureScaled` kernel. length\_scale \= np.random.uniform(size\=\[num\_features\]) ard\_kernel \= tfpk.FeatureScaled( tfpk.MaternFiveHalves(amplitude\=np.float64(0.3)), scale\_diag\=length\_scale) Sums and products of PSD kernels are easy to compute, via operator overloading. matern \= tfpk.MaternFiveHalves(amplitude\=2.) squared\_exponential \= tfpk.ExponentiatedQuadratic(length\_scale\=0.1) sum\_kernel \= matern + squared\_exponential np.testing.assert\_allclose( sum\_kernel.matrix(x, x), matern.matrix(x, x) + squared\_exponential.matrix(x, x)) Exercise: Implemented a squared exponential kernel[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/kernels.html#exercise-implemented-a-squared-exponential-kernel "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As an exercise, try implementing a squared exponential PSD kernel: k(x, y) \= amplitude\*\*2 \* exp(\-||x \- y||\*\*2 / (2 \* length\_scale\*\*2)) In TFP library kernels (see TFP’s [squared exponential kernel](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/exponentiated_quadratic.py) ), there are other details to consider, like handling of different dtypes, accepting either `length_scale` or `inverse_length_scale`, and ensuring that kernel batch shapes broadcast correctly with inputs. For the purpose of the exercise we can ignore these, and `apply` can be written as a straightforward implementation of the kernel function. (New PSD kernels added to TFP would have to treat this more carefully, and existing kernels serve as good guides). Try implementing `_apply` below (the solution is a couple cells down). class MyExponentiatedQuadratic(tfpk.AutoCompositeTensorPsdKernel): def \_\_init\_\_(self, amplitude, length\_scale): self.amplitude \= amplitude self.length\_scale \= length\_scale super(MyExponentiatedQuadratic, self).\_\_init\_\_( feature\_ndims\=1, dtype\=jnp.float32, name\='MyExponentiatedQuadratic', validate\_args\=False) @classmethod def \_parameter\_properties(cls, dtype): \# All TFP objects have parameter properties, which contain information on \# the shape and domain of the parameters. The Softplus bijector is \# associated with both the amplitude and length scale parameters, and may be \# used to constrain these parameters to be positive. These bijectors are NOT \# automatically applied when the kernel is called -- users may apply them \# explicitly when doing unconstrained parameter optimization, e.g. return dict( amplitude\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype)))), length\_scale\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype))))) def \_apply(self, x1, x2, example\_ndims\=0): del example\_ndims \# Can ignore this arg. pass Make sure this kernel gives the same output as `ExponentiatedQuadratic` in the TFP library. my\_kernel \= MyExponentiatedQuadratic(amplitude\=2., length\_scale\=0.5) tfp\_kernel \= tfpk.ExponentiatedQuadratic(amplitude\=2., length\_scale\=0.5) np.testing.assert\_allclose(my\_kernel.matrix(x, y), tfp\_kernel.matrix(x, y), rtol\=1e-5) Solution[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/kernels.html#solution "Link to this heading") -------------------------------------------------------------------------------------------------------------------------- class MyExponentiatedQuadratic(tfpk.AutoCompositeTensorPsdKernel): def \_\_init\_\_(self, amplitude, length\_scale): self.amplitude \= amplitude self.length\_scale \= length\_scale super(MyExponentiatedQuadratic, self).\_\_init\_\_( feature\_ndims\=1, dtype\=jnp.float32, name\='MyExponentiatedQuadratic', validate\_args\=False) @classmethod def \_parameter\_properties(cls, dtype): return dict( amplitude\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype)))), length\_scale\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype))))) def \_apply(self, x1, x2, example\_ndims\=0): del example\_ndims pairwise\_sq\_distance \= jnp.sum((x1 \- x2)\*\*2, axis\=-1) return jnp.exp(\-0.5 \* pairwise\_sq\_distance / self.length\_scale \*\* 2) \* self.amplitude \*\* 2 [**AI features? Redis belongs in your hot path.** Vector search, semantic cache, agent memory. **Get started**](https://server.ethicalads.io/proxy/click/10370/019da328-371d-78e2-85f2-3aa930f414ec/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10370/019da328-371d-78e2-85f2-3aa930f414ec/) --- # Unknown Advanced Topics =============== Tensorflow Probability ---------------------- .. toctree:: :maxdepth: 1 tfp/gp tfp/bijectors tfp/kernels tfp/debugging PyGlove ------- .. toctree:: :maxdepth: 1 pyglove/vizier\_as\_backend --- # Unknown Applications ============ OSS Vizier is used in the following: Codebases --------- - \`Vertex AI \`\_\_ - \`PyGlove \`\_\_ - \`OptFormer \`\_\_ - \`Init2winit \`\_\_ - \`Tensorflow Federated \`\_\_ - \`Tensorflow GNN \`\_\_ - \`CFU-Playground \`\_\_ - \`Architecture Gym (ArchGym) \`\_\_ - \`OpenML (Converter) \`\_\_ Guides + Courses ------ - \`Deep Learning Tuning Playbook \`\_\_ - \`Stanford STATS 285 (Fall 2023): Massive Computational Experiments, Painlessly \`\_\_ - \`Stanford XCS224U (Spring 2023): Natural Language Understanding \`\_\_ Papers ------ - \`Fishy: Layerwise Fisher Approximation for Higher-order Neural Network Optimization \`\_\_ - \`Massively Scaling Heteroscedastic Classifiers \`\_\_ - \`Towards Learning Universal Hyperparameter Optimizers with Transformers \`\_\_ - \`Task Selection for AutoML System Evaluation \`\_\_ --- # Unknown Media ===== OSS Vizier has been featured in: Articles -------- - \`Google Research, 2022 & Beyond: Algorithmic Advances \`\_\_ - \`MarkTechPost \`\_\_ - \`The Sequence \`\_\_ - \`ML News by Weights & Biases \`\_\_ - \`Analytics India Magazine \`\_\_ - \`This Week in AI by Lighting AI \`\_\_ - \`gHacks \`\_\_ - \`WebBigdata (Japanese) \`\_\_ - \`Random Access (Spanish) \`\_\_ - \`Electronic Smith \`\_\_ - \`Deep Learning Weekly \`\_\_ - \`China Z (Chinese) \`\_\_ - \`TuringPost \`\_\_ - \`Open Data Science \`\_\_ Videos/Talks ------------ - \`Beijing Academy of Artificial Intelligence (BAAI) \`\_\_ - \`AutoML Conference 2023 Tutorial \`\_\_ (\`Hands-on Colab \`\_\_) - \`AutoML Seminar 2023 Talk \`\_\_ - \`AutoML Conference 2022 Paper Presentation \`\_\_ - \`AutoML Conference 2022 AutoRL Tutorial \`\_\_ - \`ML News by Yannic Kilcher \`\_\_ --- # Unknown Guides ====== For Users --------- .. toctree:: :maxdepth: 1 user/running\_vizier user/distributed user/search\_spaces user/converters Switching to Vertex user/supported\_algorithms For Developers -------------- .. toctree:: :maxdepth: 1 developer/designers developer/pythia\_policies developer/early\_stopping developer/metadata developer/predict For Benchmarking ---------------- .. toctree:: :maxdepth: 1 benchmarks/creating\_benchmarks benchmarks/running\_benchmarks benchmarks/analyzing\_benchmarks benchmarks/ray\_benchmarks --- # Unknown Frequently Used Import Targets ============================== Includes a brief summary of important symbols and modules. Service Users ------------- If you write client code interacting with the OSS Vizier service, use these import targets: - \*\*from vizier.service import pyvizier as vz\*\*: Exposes the same set of symbol names as \`\`vizier.pyvizier\`\`. \`\`vizier.service.pyvizier.Foo\`\` is a subclass or an alias of \`\`vizier.pyvizier.Foo\`\`, and can be converted into protobufs. - \*\*from vizier.service import ...\*\*: Include binaries and internal utilities. Algorithm Developers -------------------- If you write algorithm code (Designers or Pythia policies) in OSS Vizier, use these import targets: - \*\*from vizier import pyvizier as vz\*\*: Pure python building blocks of OSS Vizier. Cross-platform code, including Pythia policies, must use this \`\`pyvizier\`\` instance. - \`\`Trial\`\` and \`\`ProblemStatement\`\` are important classes. - \*\*from vizier.pyvizier import converters\*\*: Convert between \`\`pyvizier\`\` objects and numpy arrays. - \`\`TrialToNumpyDict\`\`: Converts parameters (and metrics) into a dict of numpy arrays. Preferred conversion method if you intended to train an embedding of categorical/discrete parameters, or data includes missing parameters or metrics. - \`\`TrialToArrayConverter\`\`: Converts parameters (and metrics) into an array. - \*\*from vizier.interfaces import serializable\*\*: Abstractions for serializable objects. - \`\`PartiallySerializable\`\`, \`\`Serializable\`\` Algorithm Abstractions ~~~~~~~~~~~~~~~~~~~~~~ - \*\*from vizier import pythia\*\*: Abstractions for Pythia policies. - \`\`Policy\`\`, \`\`PolicySupporter\`\`: Key abstractions. - \`\`LocalPolicyRunner\`\`: Use it for running a \`\`Policy\`\` in RAM. - \*\*from vizier import algorithms\*\*: Abstractions for algorithms. - \`\`Designer\`\`: Stateful algorithm abstraction. - \`\`DesignerPolicy\`\`: Wraps \`\`Designer\`\` into a Pythia Policy. - \`\`GradientFreeMaximizer\`\`: For optimizing acquisition functions. - \`\`(Partially)SerializableDesigner\`\`: Designers who wish to optimize performance by saving states. Tensorflow Modules ~~~~~~~~~~~~~~~~~~ - \*\*from vizier import tfp\*\*: Tensorflow-Probability utilities. - \`\`acquisitions\`\`: Acquisition functions module. - \`\`AcquisitionFunction\`\`: Abstraction. - \`\`UpperConfidenceBound\`\`, \`\`ExpectedImprovement\`\`, etc. - \`\`bijectors\`\`: Bijectors module. - \`\`YeoJohnson\`\`: Implements both Yeo-Johnson and Box-Cox transformations. - \`\`optimal\_power\_transformation\`\`: Returns the optimal power transformation. - \`\`flip\_sign\`\`: returns a sign-flip bijector. - \*\*from vizier import keras as vzk\*\*: - \`\`vzk.layers\`\`: Layers usually wrapping \`\`tfp\`\` classes. - \`\`variable\_from\_prior\`\`: Utility layer for handling regularized variables. - \`\`vzk.models\`\`: Most of the useful models don’t easily fit into Keras's \`\`Model\`\` abstraction, but we may add some for display. - \`\`vzk.optim\`\`: Wrappers around optimizers in \`\`tfp\`\` or \`\`keras\`\`. --- # Unknown Frequently Used Import Targets ============================== Includes a brief summary of important symbols and modules. Service Users ------------- If you write client code interacting with the OSS Vizier service, use these import targets: - \*\*from vizier.service import pyvizier as vz\*\*: Exposes the same set of symbol names as \`\`vizier.pyvizier\`\`. \`\`vizier.service.pyvizier.Foo\`\` is a subclass or an alias of \`\`vizier.pyvizier.Foo\`\`, and can be converted into protobufs. - \*\*from vizier.service import ...\*\*: Include binaries and internal utilities. Algorithm Developers -------------------- If you write algorithm code (Designers or Pythia policies) in OSS Vizier, use these import targets: - \*\*from vizier import pyvizier as vz\*\*: Pure python building blocks of OSS Vizier. Cross-platform code, including Pythia policies, must use this \`\`pyvizier\`\` instance. - \`\`Trial\`\` and \`\`ProblemStatement\`\` are important classes. - \*\*from vizier.pyvizier import converters\*\*: Convert between \`\`pyvizier\`\` objects and numpy arrays. - \`\`TrialToNumpyDict\`\`: Converts parameters (and metrics) into a dict of numpy arrays. Preferred conversion method if you intended to train an embedding of categorical/discrete parameters, or data includes missing parameters or metrics. - \`\`TrialToArrayConverter\`\`: Converts parameters (and metrics) into an array. - \*\*from vizier.interfaces import serializable\*\*: Abstractions for serializable objects. - \`\`PartiallySerializable\`\`, \`\`Serializable\`\` Algorithm Abstractions ~~~~~~~~~~~~~~~~~~~~~~ - \*\*from vizier import pythia\*\*: Abstractions for Pythia policies. - \`\`Policy\`\`, \`\`PolicySupporter\`\`: Key abstractions. - \`\`LocalPolicyRunner\`\`: Use it for running a \`\`Policy\`\` in RAM. - \*\*from vizier import algorithms\*\*: Abstractions for algorithms. - \`\`Designer\`\`: Stateful algorithm abstraction. - \`\`DesignerPolicy\`\`: Wraps \`\`Designer\`\` into a Pythia Policy. - \`\`GradientFreeMaximizer\`\`: For optimizing acquisition functions. - \`\`(Partially)SerializableDesigner\`\`: Designers who wish to optimize performance by saving states. Tensorflow Modules ~~~~~~~~~~~~~~~~~~ - \*\*from vizier import tfp\*\*: Tensorflow-Probability utilities. - \`\`acquisitions\`\`: Acquisition functions module. - \`\`AcquisitionFunction\`\`: Abstraction. - \`\`UpperConfidenceBound\`\`, \`\`ExpectedImprovement\`\`, etc. - \`\`bijectors\`\`: Bijectors module. - \`\`YeoJohnson\`\`: Implements both Yeo-Johnson and Box-Cox transformations. - \`\`optimal\_power\_transformation\`\`: Returns the optimal power transformation. - \`\`flip\_sign\`\`: returns a sign-flip bijector. - \*\*from vizier import keras as vzk\*\*: - \`\`vzk.layers\`\`: Layers usually wrapping \`\`tfp\`\` classes. - \`\`variable\_from\_prior\`\`: Utility layer for handling regularized variables. - \`\`vzk.models\`\`: Most of the useful models don’t easily fit into Keras's \`\`Model\`\` abstraction, but we may add some for display. - \`\`vzk.optim\`\`: Wrappers around optimizers in \`\`tfp\`\` or \`\`keras\`\`. --- # OSS Vizier as a Backend — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * OSS Vizier as a Backend * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/pyglove/vizier_as_backend.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/pyglove/vizier_as_backend.ipynb) OSS Vizier as a Backend[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#oss-vizier-as-a-backend "Link to this heading") ====================================================================================================================================================================== We demonstrate how OSS Vizier can be used as a distributed backend for PyGlove-based tuning tasks. This assumes the user is already familiar with PyGlove primitives. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#installation-and-reference-imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier !pip install pyglove import multiprocessing import multiprocessing.pool import os import pyglove as pg from vizier import pyglove as pg\_vizier from vizier.service import servers Preliminaries[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#preliminaries "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- In the original PyGlove setting, one can normally perform evolutionary computation, for example: search\_space \= pg.Dict(x\=pg.floatv(0.0, 1.0), y\=pg.floatv(0.0, 1.0)) algorithm \= pg.evolution.regularized\_evolution() num\_trials \= 100 def evaluator(value: pg.Dict): return value.x\*\*2 \- value.y\*\*2 for value, feedback in pg.sample( search\_space, algorithm\=algorithm, num\_examples\=num\_trials, name\='basic\_run', ): reward \= evaluator(value) feedback(reward\=reward) However, in many real-world scenarios, the evaluator may be much more expensive. For example, in neural architecture search applications, `evaluator` may be the result of an entire neural network training pipeline. This leads to the need for a **backend**, in order to: 1. Distribute the evaluations over multiple workers. 2. Store the valuable results reliably and handle worker faults. Initializing the OSS Vizier backend[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#initializing-the-oss-vizier-backend "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The main initializer to call is `vizier.pyglove.init(...)`, **which should only be called once per process (not thread).** This function will edit global Python variables for determining values such as: 1. Prefix for study names. 2. Endpoint of the `VizierService` for storing data and handling requests. 3. Port for the `PythiaService` for computing suggestions. In the local case, this can be called as-is: pg\_vizier.init('my\_study') **Alternatively**, if using a remote server, the endpoint can be specified as well: server \= servers.DefaultVizierServer() \# Normally hosted on a remote machine. pg\_vizier.init('my\_study', vizier\_endpoint\=server.endpoint) Parallelization[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#parallelization "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------ Due to the OSS Vizier backend, all workers may conveniently use exactly the same evaluation loop to work on a study: NUM\_WORKERS \= 10 def work\_fn(worker\_id): print(f"Worker ID: {worker\_id}") for value, feedback in pg.sample( search\_space, algorithm\=algorithm, num\_examples\=num\_trials // NUM\_WORKERS, name\="worker\_run", ): reward \= evaluator(value) feedback(reward\=reward) There are three common forms of parallelization over the evaluation computation: 1. Multiple threads, single process. 2. Multiple processes, single machine. 3. Multiple machines. Each of these cases defines the “worker”, which can be a thread, process or machine respectively. We demonstrate examples of every type of parallelization below. ### Multiple threads, single process[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#multiple-threads-single-process "Link to this heading") with multiprocessing.pool.ThreadPool(num\_workers) as pool: pool.map(work\_fn, range(NUM\_WORKERS)) ### Multiple processes, single machine[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#multiple-processes-single-machine "Link to this heading") processes \= \[\] for worker\_id in range(NUM\_WORKERS): p \= multiprocessing.Process(target\=work\_fn, args\=(worker\_id,)) p.start() processes.append(p) for p in processes: p.join() ### Multiple machines[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/pyglove/vizier_as_backend.html#multiple-machines "Link to this heading") \# Server Machine server \= servers.DefaultVizierServer() \# Worker Machine worker\_id \= os.uname()\[1\] pg\_vizier.init('my\_study', vizier\_endpoint\=server.endpoint) work\_fn(worker\_id) [**Build and run** apps in over 115 regions with MongoDB Atlas, the database for every enterprise.](https://server.ethicalads.io/proxy/click/10124/019da328-43a5-7df1-af8b-f1d1d08231d8/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10124/019da328-43a5-7df1-af8b-f1d1d08231d8/) --- # PSD kernels — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * PSD kernels * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/tfp/kernels.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/kernels.ipynb) PSD kernels[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/kernels.html#psd-kernels "Link to this heading") ================================================================================================================================ TFP’s [PSD kernels](https://github.com/tensorflow/probability/tree/main/tensorflow_probability/python/math/psd_kernels) compute positive semidefinite kernel functions. A PSD kernel instance is a required arg to TFP’s Gaussian Process distribution, so specifying a GP model coroutine will generally involve defining a PSD kernel as an intermediate. PSD kernel subclasses take hyperparameters, such as amplitude and length scale, as constructor args. They have three primary public methods: `apply`, `matrix`, and `tensor`, each of which computes the kernel function pairwise on inputs in different ways: * `apply` computes the value of the kernel function at a pair of (batches of) input locations. It’s the only required method for subclasses: `matrix` and `tensor` are implemented in terms of `apply` (except when a more efficient method exists to compure pairwise kernel matrices). * `matrix` computes the value of the kernel _pairwise_ on two (batches of) lists of input examples. When the two collections are the same the result is called the [Gram matrix](https://en.wikipedia.org/wiki/Gramian_matrix) . `matrix` is the most important method for GPs. * `tensor` generalizes `matrix`, taking rank `k1` and `k2` collections of input examples to a rank `k1 + k2` collection of kernel values. (We mention `tensor` for completeness, but it isn’t relevant to GPs). PSD kernels have somewhat complex [shape semantics](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/positive_semidefinite_kernel.py#L97) , due to the need to define which input dimensions should be included in pairwise computations and which should be treated as batch dimensions (denoting independent sets of input points.) Imports[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/kernels.html#imports "Link to this heading") ------------------------------------------------------------------------------------------------------------------------ from jax import numpy as jnp import numpy as np from tensorflow\_probability.python.internal import dtype\_util from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfpk \= tfp.math.psd\_kernels Some examples of PSD kernel usage: \# Construct a MaternFiveHalves kernel (with empty batch shape). amplitude \= 2. length\_scale \= 0.5 k \= tfpk.MaternFiveHalves( amplitude\=amplitude, length\_scale\=length\_scale) \# Randomly sample some input data. num\_features \= 5 num\_observations \= 12 x \= np.random.normal(size\=\[num\_observations, num\_features\]) \# \`matrix\` computes pairwise kernel values for the Cartesian product over the \# second-to-rightmost dimension of the inputs. Following the terminology in the \# PSD kernel docstring, there is a single example dimension (and single feature \# dimension). assert k.matrix(x, x).shape \== (12, 12) \# Calling \`matrix\` on inputs of shape \[12, d\] and \[10, d\] results in a kernel \# matrix of shape (12, 10) y \= np.random.normal(size\=\[10, num\_features\]) assert k.matrix(x, y).shape \== (12, 10) ARD kernels in TFP are implemented with the `FeatureScaled` kernel. length\_scale \= np.random.uniform(size\=\[num\_features\]) ard\_kernel \= tfpk.FeatureScaled( tfpk.MaternFiveHalves(amplitude\=np.float64(0.3)), scale\_diag\=length\_scale) Sums and products of PSD kernels are easy to compute, via operator overloading. matern \= tfpk.MaternFiveHalves(amplitude\=2.) squared\_exponential \= tfpk.ExponentiatedQuadratic(length\_scale\=0.1) sum\_kernel \= matern + squared\_exponential np.testing.assert\_allclose( sum\_kernel.matrix(x, x), matern.matrix(x, x) + squared\_exponential.matrix(x, x)) Exercise: Implemented a squared exponential kernel[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/kernels.html#exercise-implemented-a-squared-exponential-kernel "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As an exercise, try implementing a squared exponential PSD kernel: k(x, y) \= amplitude\*\*2 \* exp(\-||x \- y||\*\*2 / (2 \* length\_scale\*\*2)) In TFP library kernels (see TFP’s [squared exponential kernel](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/math/psd_kernels/exponentiated_quadratic.py) ), there are other details to consider, like handling of different dtypes, accepting either `length_scale` or `inverse_length_scale`, and ensuring that kernel batch shapes broadcast correctly with inputs. For the purpose of the exercise we can ignore these, and `apply` can be written as a straightforward implementation of the kernel function. (New PSD kernels added to TFP would have to treat this more carefully, and existing kernels serve as good guides). Try implementing `_apply` below (the solution is a couple cells down). class MyExponentiatedQuadratic(tfpk.AutoCompositeTensorPsdKernel): def \_\_init\_\_(self, amplitude, length\_scale): self.amplitude \= amplitude self.length\_scale \= length\_scale super(MyExponentiatedQuadratic, self).\_\_init\_\_( feature\_ndims\=1, dtype\=jnp.float32, name\='MyExponentiatedQuadratic', validate\_args\=False) @classmethod def \_parameter\_properties(cls, dtype): \# All TFP objects have parameter properties, which contain information on \# the shape and domain of the parameters. The Softplus bijector is \# associated with both the amplitude and length scale parameters, and may be \# used to constrain these parameters to be positive. These bijectors are NOT \# automatically applied when the kernel is called -- users may apply them \# explicitly when doing unconstrained parameter optimization, e.g. return dict( amplitude\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype)))), length\_scale\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype))))) def \_apply(self, x1, x2, example\_ndims\=0): del example\_ndims \# Can ignore this arg. pass Make sure this kernel gives the same output as `ExponentiatedQuadratic` in the TFP library. my\_kernel \= MyExponentiatedQuadratic(amplitude\=2., length\_scale\=0.5) tfp\_kernel \= tfpk.ExponentiatedQuadratic(amplitude\=2., length\_scale\=0.5) np.testing.assert\_allclose(my\_kernel.matrix(x, y), tfp\_kernel.matrix(x, y), rtol\=1e-5) Solution[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/kernels.html#solution "Link to this heading") -------------------------------------------------------------------------------------------------------------------------- class MyExponentiatedQuadratic(tfpk.AutoCompositeTensorPsdKernel): def \_\_init\_\_(self, amplitude, length\_scale): self.amplitude \= amplitude self.length\_scale \= length\_scale super(MyExponentiatedQuadratic, self).\_\_init\_\_( feature\_ndims\=1, dtype\=jnp.float32, name\='MyExponentiatedQuadratic', validate\_args\=False) @classmethod def \_parameter\_properties(cls, dtype): return dict( amplitude\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype)))), length\_scale\=parameter\_properties.ParameterProperties( default\_constraining\_bijector\_fn\=( lambda: tfb.Softplus(low\=dtype\_util.eps(dtype))))) def \_apply(self, x1, x2, example\_ndims\=0): del example\_ndims pairwise\_sq\_distance \= jnp.sum((x1 \- x2)\*\*2, axis\=-1) return jnp.exp(\-0.5 \* pairwise\_sq\_distance / self.length\_scale \*\* 2) \* self.amplitude \*\* 2 [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad --- # Metadata — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Metadata * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/developer/metadata.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/metadata.ipynb) Metadata[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#metadata "Link to this heading") ======================================================================================================================== We provide a guide below on common developer uses of the `Metadata` primitive. OSS Vizier can store `Metadata` in both the `ProblemStatement` and each `TrialSuggestion`/`Trial`, with common use cases: * Containing additional information outside of standard parameter types. * Allowing user code to store small amounts of state information inside OSS Vizier, attached to the OSS Vizier study. * Wrapping search spaces and corresponding algorithms which are naturally incompatible with OSS Vizier’s default API, to still allow a distributed backend service. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from vizier import pyvizier as vz from google.protobuf import any\_pb2 Metadata basics[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#metadata-basics "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------- The [`Metadata`](https://github.com/google/vizier/blob/main/vizier/pyvizier/__init__.py) is a key-value store, where: * Keys are UTF-8 strings. * Values can be strings or protocol buffers. While values of type `int`, `float`, and more complex objects can also be used, **the developer is responsible for serializing / unserializing said objects.** metadata \= vz.Metadata() metadata\['proto'\] \= any\_pb2.Any(...) metadata\['string'\] \= 'hello' Additionally, `Metadata` can act as a “dictionary of dictionaries”, i.e. a hierarchy of dictionaries, via its `Namespace` functionality via calling `.ns()`, which creates another `Metadata` which shares data with the original. child\_metadata \= metadata.ns('child') grandchild\_metadata \= child\_metadata.ns('child') grandchild\_metadata\['string'\] \= 'goodbye' assert metadata.ns('child').ns('child')\['string'\] \== 'goodbye' ProblemStatement Metadata[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#problemstatement-metadata "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------- The `ProblemStatement` object contains a `metadata` attribute, ideally for storing global metadata related to the study. Note that `Metadata` will not be used in the optimization process, UNLESS there is a custom algorithm configured to use it. Below is a usage example when training an image classifier, where one may wish to store training-related attributes in `Metadata`. problem\_statement \= vz.ProblemStatement() problem\_statement.metadata\['dataset'\] \= 'cifar10' problem\_statement.metadata\['architecture'\] \= 'resnet\_18' Trial Metadata[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#trial-metadata "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ `TrialSuggestion` and subclass `Trial` also contain a `metadata` attribute. This in contrast, should be used to store metadata related to the specific Trial. In the image classification case, examples would be the type of GPU used for training and if the training worker has been preempted. trial \= vz.Trial() trial.metadata\['gpu\_used'\] \= 'P100' trial.metadata\['preempted'\] \= 'True' OSS Vizier as a backend via `Metadata`[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/metadata.html#oss-vizier-as-a-backend-via-metadata "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As an advanced developer use case, one may extend OSS Vizier’s search space capabilities using `Metadata`. Custom algorithms can provide full freedom in expressing more complex search spaces (e.g. graphs) using `Metadata`. Example use cases: * Combinatorial optimization, where the search space may consist of graphs or multiple selection (e.g. (NK)) primitives. Algorithms commonly include evolutionary methods, which also require custom mutation operations. * Free-form textual data used for suggestions (and maybe even evaluation metrics!), as common with language-based applications. \# Setup combinatorial search space. choose\_problem \= vz.ProblemStatement() choose\_problem.metadata \= vz.Metadata({'N': '10', 'K': '3'}) \# Example of a suggestion proposed by a custom algorithm. suggestion \= vz.TrialSuggestion() suggestion.metadata\['chosen\_indices'\] \= '\[0, 3, 7\]' The algorithm behavior can even be changed mid-optimization with `Metadata` using a client! This is in fact used extensively in our integrations with [PyGlove](https://github.com/google/pyglove) to allow a running Pythia policy to change search spaces or mutations online. \# Original mutation rate. mutation\_problem \= vz.ProblemStatement() mutation\_problem.metadata \= vz.Metadata({'mutation\_rate': '0.1'}) \# ... \# Assume algorithm started running in the Pythia service. \# ... \# Set new mutation rate. study\_metadata \= vz.Metadata({'mutation\_rate': '0.2'}) \# Prevent this trial from being used in the population. trial\_metadata \= vz.Metadata({'use\_in\_population' \= 'False'}) trial\_id \= 1 \# Create unit of metadata update. metadata\_delta \= vz.MetadataDelta( on\_study\=study\_metadata, on\_trials\={trial\_id: trial\_metadata}) Once we have a client, we can commit the metadata update: client.update\_metadata(metadata\_delta) [**Build and run** apps in over 115 regions with MongoDB Atlas, the database for every enterprise.](https://server.ethicalads.io/proxy/click/10124/019da328-43a5-7df1-af8b-f1d1d08231d8/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10124/019da328-43a5-7df1-af8b-f1d1d08231d8/) --- # OSS Vizier as a Backend — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * OSS Vizier as a Backend * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/pyglove/vizier_as_backend.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/pyglove/vizier_as_backend.ipynb) OSS Vizier as a Backend[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#oss-vizier-as-a-backend "Link to this heading") ====================================================================================================================================================================== We demonstrate how OSS Vizier can be used as a distributed backend for PyGlove-based tuning tasks. This assumes the user is already familiar with PyGlove primitives. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#installation-and-reference-imports "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier !pip install pyglove import multiprocessing import multiprocessing.pool import os import pyglove as pg from vizier import pyglove as pg\_vizier from vizier.service import servers Preliminaries[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#preliminaries "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------------------- In the original PyGlove setting, one can normally perform evolutionary computation, for example: search\_space \= pg.Dict(x\=pg.floatv(0.0, 1.0), y\=pg.floatv(0.0, 1.0)) algorithm \= pg.evolution.regularized\_evolution() num\_trials \= 100 def evaluator(value: pg.Dict): return value.x\*\*2 \- value.y\*\*2 for value, feedback in pg.sample( search\_space, algorithm\=algorithm, num\_examples\=num\_trials, name\='basic\_run', ): reward \= evaluator(value) feedback(reward\=reward) However, in many real-world scenarios, the evaluator may be much more expensive. For example, in neural architecture search applications, `evaluator` may be the result of an entire neural network training pipeline. This leads to the need for a **backend**, in order to: 1. Distribute the evaluations over multiple workers. 2. Store the valuable results reliably and handle worker faults. Initializing the OSS Vizier backend[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#initializing-the-oss-vizier-backend "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The main initializer to call is `vizier.pyglove.init(...)`, **which should only be called once per process (not thread).** This function will edit global Python variables for determining values such as: 1. Prefix for study names. 2. Endpoint of the `VizierService` for storing data and handling requests. 3. Port for the `PythiaService` for computing suggestions. In the local case, this can be called as-is: pg\_vizier.init('my\_study') **Alternatively**, if using a remote server, the endpoint can be specified as well: server \= servers.DefaultVizierServer() \# Normally hosted on a remote machine. pg\_vizier.init('my\_study', vizier\_endpoint\=server.endpoint) Parallelization[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#parallelization "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------ Due to the OSS Vizier backend, all workers may conveniently use exactly the same evaluation loop to work on a study: NUM\_WORKERS \= 10 def work\_fn(worker\_id): print(f"Worker ID: {worker\_id}") for value, feedback in pg.sample( search\_space, algorithm\=algorithm, num\_examples\=num\_trials // NUM\_WORKERS, name\="worker\_run", ): reward \= evaluator(value) feedback(reward\=reward) There are three common forms of parallelization over the evaluation computation: 1. Multiple threads, single process. 2. Multiple processes, single machine. 3. Multiple machines. Each of these cases defines the “worker”, which can be a thread, process or machine respectively. We demonstrate examples of every type of parallelization below. ### Multiple threads, single process[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#multiple-threads-single-process "Link to this heading") with multiprocessing.pool.ThreadPool(num\_workers) as pool: pool.map(work\_fn, range(NUM\_WORKERS)) ### Multiple processes, single machine[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#multiple-processes-single-machine "Link to this heading") processes \= \[\] for worker\_id in range(NUM\_WORKERS): p \= multiprocessing.Process(target\=work\_fn, args\=(worker\_id,)) p.start() processes.append(p) for p in processes: p.join() ### Multiple machines[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/pyglove/vizier_as_backend.html#multiple-machines "Link to this heading") \# Server Machine server \= servers.DefaultVizierServer() \# Worker Machine worker\_id \= os.uname()\[1\] pg\_vizier.init('my\_study', vizier\_endpoint\=server.endpoint) work\_fn(worker\_id) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Pythia Policies and Hosting Designers — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Pythia Policies and Hosting Designers * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/developer/pythia_policies.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/pythia_policies.ipynb) Pythia Policies and Hosting Designers[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#pythia-policies-and-hosting-designers "Link to this heading") ========================================================================================================================================================================================= This documentation will allow a developer to: * Understand the basic structure of a Pythia Policy. * Host Designers in the service. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from typing import Optional, Sequence from vizier import pythia from vizier import algorithms from vizier.service import pyvizier as vz from vizier.\_src.algorithms.policies import designer\_policy from vizier.\_src.algorithms.evolution import nsga2 Pythia Policies[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#pythia-policies "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- The Pythia Service maps algorithm names to `Policy` objects. All algorithms which need to be hosted on the server must eventually be wrapped into a `Policy`. Every `Policy` is injected with a `PolicySupporter`, which is a client used for fetching data from the datastore. This design choice serves two core purposes: 1. The `Policy` is effectively stateless, and thus can be deleted and recovered at any time (e.g. due to a server preemption or failure). 2. Consequently, this avoids needing to save an explicit and potentially complicated algorithm state. Instead, the “algorithm state” can be recovered purely from the entire study containing (`metadata`, `study_config`, `trials`). We show the `Policy` abstract class explicitly below. Exact class entrypoint can be found [here](https://github.com/google/vizier/blob/main/vizier/pythia.py) . class Policy(abc.ABC): """Interface for Pythia Policy subclasses.""" @abc.abstractmethod def suggest(self, request: SuggestRequest) \-> SuggestDecision: """Compute suggestions that Vizier will eventually hand to the user.""" @abc.abstractmethod def early\_stop(self, request: EarlyStopRequest) \-> EarlyStopDecisions: """Decide which Trials Vizier should stop.""" @property def should\_be\_cached(self) \-> bool: """Returns True if it's safe & worthwhile to cache this Policy in RAM.""" return False ### Fundamental Rule of Service Pythia Policies[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#fundamental-rule-of-service-pythia-policies "Link to this heading") For algorithms used in the Pythia Service, the fundamental rule is to assume that a Pythia policy class instance will only call once per user interaction: * `__init__` * `suggest()` and be immediately deleted afterwards. Thus a typical policy will use a `stateless_algorithm` and roughly look like: class TypicalPolicy(Policy): def \_\_init\_\_(self, policy\_supporter: PolicySupporter): self.\_policy\_supporter \= policy\_supporter def suggest(self, request: SuggestRequest) \-> SuggestDecision: all\_completed \= policy\_supporter.GetTrials(status\_matches\=COMPLETED) all\_active \= policy\_supporter.GetTrials(status\_matches\=ACTIVE) suggestions \= stateless\_algorithm(all\_completed, all\_active) return SuggestDecision(suggestions) Example Pythia Policy[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#example-pythia-policy "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- Here, we write a toy policy, where we only act on `CATEGORICAL` parameters for simplicity. The `make_parameters` function will simply for-loop over every category and then cycle back. def make\_parameters( search\_space: vz.SearchSpace, index: int ) \-> vz.ParameterDict: parameter\_dict \= vz.ParameterDict() for parameter\_config in search\_space.parameters: if parameter\_config.type != vz.ParamterType.CATEGORICAL: raise ValueError("This function only supports CATEGORICAL parameters.") feasible\_values \= parameter\_config.feasible\_values parameter\_dict\[parameter\_config.name\] \= vz.ParameterValue( value\=feasible\_values\[index % len(feasible\_values)\] ) return parameter\_dict To collect the `index` from the database, we will use the `PolicySupporter` to obtain the maximum trial ID based on completed and active trials. def get\_next\_index(policy\_supporter: pythia.PolicySupporter): """Returns current trial index.""" completed \= policy\_supporter.GetTrials(status\_matches\=vz.TrialStatus.COMPLETED) active \= policy\_supporter.GetTrials(status\_matches\=vz.TrialStatus.ACTIVE) trial\_ids \= \[t.id for t in completed + active\] if trial\_ids: return max(trial\_ids) return 0 We can now put it all together into our Pythia Policy. class MyPolicy(pythia.Policy): def \_\_init\_\_(self, policy\_supporter: pythia.PolicySupporter): self.\_policy\_supporter \= policy\_supporter def suggest(self, request: pythia.SuggestRequest) \-> pythia.SuggestDecision: """Gets number of Trials to propose, and produces Trials.""" suggest\_decision\_list \= \[\] for \_ in range(request.count): index \= get\_next\_index(self.\_policy\_supporter) parameters \= make\_parameters(request.study\_config.search\_space, index) suggest\_decision\_list.append(vz.TrialSuggestion(parameters\=parameters)) return pythia.SuggestDecision( suggestions\=suggest\_decision\_list, metadata\=vz.MetadataDelta() ) Wrapping Designers as Pythia Policies[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#wrapping-designers-as-pythia-policies "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider if your algorithm code fits in the simpler [Designer](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html) abstraction, which avoids needing to deal with distributed systems logic. For example, the same exact behavior above can be re-written as a `Designer`: class MyDesigner(algorithms.Designer): def \_\_init\_\_(self, study\_config: vz.StudyConfig): self.\_study\_config \= study\_config self.\_completed\_trials \= \[\] self.\_active\_trials \= \[\] def update( self, completed: algorithms.CompletedTrials, all\_active: algorithms.ActiveTrials, ) \-> None: self.\_completed\_trials.extend(completed.trials) self.\_active\_trials \= all\_active.trials def suggest( self, count: Optional\[int\] \= None ) \-> Sequence\[vz.TrialSuggestion\]: if count is None: return \[\] trial\_ids \= \[t.id for t in self.\_completed\_trials + self.\_active\_trials\] current\_index \= max(trial\_ids) return \[\ make\_parameters(self.\_study\_config.search\_space, current\_index + i)\ for i in range(count)\ \] The entire designer (if deleted or preempted) can conveniently be recovered in just a **single** call of `update()` after `__init__`. Thus we may immediately wrap `MyDesigner` into a Pythia Policy with the following Pythia `suggest()` implementation: * Create the designer temporarily. * Update the temporary designer with **all** previously completed trials and active trials. * Obtain suggestions from the temporary designer. This is done conveniently with the `DesignerPolicy` wrapper ([code](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/policies/designer_policy.py) ): class DesignerPolicy(Policy): """Wraps a Designer into a Pythia Policy.""" def \_\_init\_\_(self, supporter: PolicySupporter, designer\_factory: Factory\[Designer\]): self.\_supporter \= supporter self.\_designer\_factory \= designer\_factory def suggest(self, request: SuggestRequest) \-> SuggestDecision: completed \= self.\_supporter.GetTrials(status\_matches\=COMPLETED) active \= self.\_supporter.GetTrials(status\_matches\=ACTIVE) designer \= self.\_designer\_factory(...) designer.update(CompletedTrials(completed), ActiveTrials(active)) return SuggestDecision(designer.suggest(request.count)) Below is the actual act of wrapping: designer\_factory \= lambda study\_config: MyDesigner(study\_config) supporter: PolicySupporter \= ... \# Assume PolicySupporter was created. pythia\_policy \= DesignerPolicy(supporter, designer\_factory) Serializing Designer States[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#serializing-designer-states "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- The above method can gradually become slower as the number of completed trials in the study increases. Thus we may consider storing a compressed representation of the algorithm state instead. Examples include: * The coordinate position in a grid search algorithm. * The population for evolutionary algorithms such as NSGA2. * Directory location for stored neural network weights. As a simple example, consider the case if our designer stores a `_counter` of **all** suggestions it has made: class CounterDesigner(Designer): def \_\_init\_\_(self, ...): ... self.\_counter \= 0 def suggest(self, count: Optional\[int\] \= None) \-> Sequence\[TrialSuggestion\]: ... self.\_counter += len(suggestions) return suggestions Vizier offers [two Designer subclasses](https://github.com/google/vizier/blob/main/vizier/interfaces/serializable.py) , both of which will use the `Metadata` primitive to store algorithm state data: * `SerializableDesigner` will use additional `recover`/`dump` methods and should be used if the entire algorithm state can be easily serialized and can be saved and restored in full. * `PartiallySerializableDesigner` will use additional `load`/`dump` methods and be used if the algorithm has subcomponents that are not easily serializable. State recovery will be handled by calling the Designer’s `__init__` (with same arguments as before) and then `load`. They can also be converted into Pythia Policies using `SerializableDesignerPolicy` and `PartiallySerializableDesignerPolicy` respectively. Below is an example modifying our `CounterDesigner` into `CounterSerialDesigner` and `CounterPartialDesigner` respectively: class CounterSerialDesigner(algorithms.SerializableDesigner): def \_\_init\_\_(self, counter: int): self.\_counter \= counter @classmethod def recover(cls, metadata: vz.Metadata) \-> CounterSerialDesigner: return cls(metadata\['counter'\]) def dump(self) \-> vz.Metadata: metadata \= vz.Metadata() metadata\['counter'\] \= str(self.\_counter) return metadata class CounterPartialDesigner(algorithms.PartiallySerializableDesigner): def load(self, metadata: vz.Metadata) \-> None: self.\_counter \= int(metadata\['counter'\]) def dump(self) \-> vz.Metadata: metadata \= vz.Metadata() metadata\['counter'\] \= str(self.\_counter) return metadata Additional References[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/pythia_policies.html#additional-references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- * Our [policies folder](https://github.com/google/vizier/tree/main/vizier/_src/algorithms/policies) contains examples of Pythia policies. [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Early Stopping — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/latest/guides/index.html) * Early Stopping * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/guides/developer/early_stopping.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/early_stopping.ipynb) Early Stopping[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html#early-stopping "Link to this heading") ========================================================================================================================================== This notebook will allow a developer to: * Understand the Early Stopping API. * Write Pythia policies for early stopping. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import numpy as np from vizier import pythia Early Stopping[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html#id1 "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- In hyperparameter optimization, early stopping is a useful mechanism to prevent wasted resources by stopping unpromising trials. Two main considerations for determining whether to stop an active trial are: * **At a macro level, how a trial’s performance compares to the rest of the trials globally.** For example, we may stop a trial if it is predicted to significantly underperform compared to the history of trials so far in the study. * **At a micro level, how a trial’s intermediate measurements are changing over time.** For example, in a classification task, overfitting may be happening when test accuracy starts to decrease. API[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html#api "Link to this heading") -------------------------------------------------------------------------------------------------------------------- Based on the above considerations, to allow full flexibility to consider when to stop a trial, we thus use the following abridged API below. Exact class entrypoint can be found [here](https://github.com/google/vizier/blob/main/vizier/pythia.py) . The `EarlyStopRequest` takes in a set of trial ID’s for early stopping consideration. However, note that trials outside of this set can also be stopped. class EarlyStopRequest: """Early stopping request.""" trial\_ids: Optional\[FrozenSet\[int\]\] In addition, we have the `EarlyStopDecision` to denote a single trial’s stopping condition and the plural `EarlyStopDecisions` for a set of trials: class EarlyStopDecision: """Stopping decision on a single trial.""" id: int should\_stop: bool class EarlyStopDecisions: """This is the output of the Policy.early\_stop() method.""" decisions: list\[EarlyStopDecision\] metadata: vz.MetadataDelta They will be used in the Pythia policy’s `early_stop` method: class Policy(abc.ABC): """Interface for Pythia2 Policy subclasses.""" @abc.abstractmethod def early\_stop(self, request: EarlyStopRequest) \-> EarlyStopDecisions: """Decide which Trials Vizier should stop.""" Example usage[](https://oss-vizier.readthedocs.io/en/latest/guides/developer/early_stopping.html#example-usage "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------- As an example, suppose our rule is to stop all requested trials whose 50th intermediate measurement is too low, e.g. bottom 10% of all trials so far. class MyEarlyStoppingPolicy(pythia.Policy): """Stops requested trial if its 50th measurement is too low.""" def \_\_init\_\_(self, policy\_supporter: pythia.PolicySupporter, index: int \= 50): self.\_policy\_supporter \= policy\_supporter self.\_index \= index def early\_stop( self, request: pythia.EarlyStopRequest ) \-> pythia.EarlyStopDecisions: metric\_name \= request.study\_config.metric\_information.item().name \# Obtain cutoff for 10th percentile. all\_trials \= self.\_policy\_supporter.GetTrials(study\_guid\=request.study\_guid) all\_metrics \= \[\] for trial in all\_trials: if len(trial.measurements) \> self.\_index: all\_metrics.append(trial.measurements\[self.\_index\].metrics\[metric\_name\]) cutoff \= np.percentile(all\_metrics, 10) \# Filter requested trials by cutoff. considered\_trials \= \[\ trial for trial in all\_trials if trial.id in request.trial\_ids\ \] stopping\_decisions \= \[\] for trial in considered\_trials: if trial.measurements\[self.\_index\].metrics\[metric\_name\] < cutoff: decision \= pythia.EarlyStopDecision( trial.id, reason\='Below cutoff', should\_stop\=True ) else: decision \= pythia.EarlyStopDecision( trial.id, reason\='Above cutoff', should\_stop\=False ) stopping\_decisions.append(decision) return pythia.EarlyStopDecisions(decisions\=stopping\_decisions) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Supported Algorithms — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Supported Algorithms * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/user/supported_algorithms.ipynb.txt) * * * Supported Algorithms[](https://oss-vizier.readthedocs.io/en/stable/guides/user/supported_algorithms.html#supported-algorithms "Link to this heading") ======================================================================================================================================================= While we service all algorithms to the user in our [policy factory](https://github.com/google/vizier/blob/main/vizier/_src/service/policy_factory.py) , many can be organized by what level of support we provide to them, in terms of: * Search space (**Flat**, **Continuous-Only**, **Boolean-Only**) * Allowing batched suggestions (**+Batch**) * Allowing multiple objectives (**+MO**) Official[](https://oss-vizier.readthedocs.io/en/stable/guides/user/supported_algorithms.html#official "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- The following algorithms can be considered “official” and production-quality: 1. [**GP-UCB-PE**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/gp_ucb_pe.py) (`GP_UCB_PE`) \[**Flat**, **+Batch**, **+MO**\] 2. [**GP-Bandit**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/gp_bandit.py) (`GAUSSIAN_PROCESS_BANDIT`) \[**Flat**, **+MO**\] 3. [**Random Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/random.py) (`RANDOM_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 4. [**Quasi-Random Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/quasi_random.py) (`QUASI_RANDOM_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 5. [**Grid Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/grid.py) (`GRID_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 6. [**Shuffled Grid Search**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/grid.py) (`SHUFFLED_GRID_SEARCH`) \[**Flat**, **+Batch**, **+MO**\] 7. [**Eagle Strategy**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/eagle_strategy/eagle_strategy.py) (`EAGLE_STRATEGY`) \[**Flat**, **+Batch**\] External + Imported[](https://oss-vizier.readthedocs.io/en/stable/guides/user/supported_algorithms.html#external-imported "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- These algorithms are imported and wrapped from external packages (requiring additional installations via `pip install google-vizier[algorithms]`), and thus we cannot fully control their performance: 1. [**CMA-ES**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/cmaes.py) (`CMA_ES`): \[**Continuous-Only**, **+Batch**\] Reproduced[](https://oss-vizier.readthedocs.io/en/stable/guides/user/supported_algorithms.html#reproduced "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------- These algorithms are attempted reproductions of their original papers, sometimes using the authors’ original implementations as inspiration (but not as direct imports). While we try our best to ensure their quality, we cannot guarantee exact performance: 1. [**NSGA-II**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/evolution/nsga2.py) (`NSGA2`) \[**Flat**, **+Batch**, **+MO**\] 2. [**Bayesian Optimization of Combinatorial Structures**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/bocs.py) (`BOCS`) \[**Boolean Only**\] 3. [**Harmonica**](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/designers/harmonica.py) (`HARMONICA`) \[**Boolean Only**\] [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Analyzing Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Analyzing Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/benchmarks/analyzing_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/analyzing_benchmarks.ipynb) Analyzing Benchmarks[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#analyzing-benchmarks "Link to this heading") ============================================================================================================================================================= We will demonstrate below how to dstribute our benchmark runner pipeline over multiple benchmarks in conjunction with our suite of benchmark analysis tools to easily compare and visualize the performance of different algorithms over all benchmark problems. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from vizier import benchmarks as vzb from vizier.algorithms import designers from vizier.benchmarks import experimenters from vizier.benchmarks import analyzers import itertools import numpy as np import pandas as pd Algorithm and Experimenter Factories[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#algorithm-and-experimenter-factories "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To compare algorithms across multiple benchmarks, we want to first create a bunch of relevant benchmark experimenters. To do so, we use `SerializableExperimenterFactory` from our [Experimenters API](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) to modularize the construction of multiple benchmark components. For example, here we can create a diverse set of BBOB functions with different dimensions via the `BBOBExperimenterFactory`. Then, we can print out the full serialization of the benchmarks that we have created. function\_names \= \['Sphere', 'Discus'\] dimensions \= \[4, 8\] product\_list \= list(itertools.product(function\_names, dimensions)) experimenter\_factories \= \[\] for product in product\_list: name, dim \= product bbob\_factory \= experimenters.BBOBExperimenterFactory(name\=name, dim\=dim) experimenter\_factories.append(bbob\_factory) print(bbob\_factory.dump()) As mentioned in our previous tutorial, we can create a `BenchmarkState` from our algorithm and experimenter factories and apply a `BenchmarkRunner` benchmarking protocol to run the algorithm. We end up with a list of `BenchmarkState` objects, each representing a different benchmark run, possibly with repeats. Conveniently, we provide analysis utility functions in our [Analyzers API](https://github.com/google/vizier/blob/main/vizier/benchmarks/analyzers.py) that convert our `BenchmarkState` into summarized curves stored compactly in `BenchmarkRecord`, which also holds the algorithm name and experimenter factory serialization. We can visualize and later analyze our results using a dataframe. NUM\_REPEATS \= 5 \# @param NUM\_ITERATIONS \= 150 \# @param runner \= vzb.BenchmarkRunner( benchmark\_subroutines\=\[\ vzb.GenerateSuggestions(),\ vzb.EvaluateActiveTrials(),\ \], num\_repeats\=NUM\_ITERATIONS, ) algorithms \= { 'grid': designers.GridSearchDesigner.from\_problem, 'random': designers.RandomDesigner.from\_problem, 'eagle': designers.EagleStrategyDesigner, } records \= \[\] for experimenter\_factory in experimenter\_factories: for algo\_name, algo\_factory in algorithms.items(): benchmark\_state\_factory \= vzb.ExperimenterDesignerBenchmarkStateFactory( experimenter\_factory\=experimenter\_factory, designer\_factory\=algo\_factory ) states \= \[\] for \_ in range(NUM\_REPEATS): benchmark\_state \= benchmark\_state\_factory() runner.run(benchmark\_state) states.append(benchmark\_state) record \= analyzers.BenchmarkStateAnalyzer.to\_record( algorithm\=algo\_name, experimenter\_factory\=experimenter\_factory, states\=states, ) records.append(record) records\_list \= \[\ (rec.algorithm, dict(rec.experimenter\_metadata), rec) for rec in records\ \] df \= pd.DataFrame(records\_list, columns\=\['algorithm', 'experimenter', 'record'\]) df Visualization from Records[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#visualization-from-records "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Given a sequence of `BenchmarkRecords`, we provide utility plotting functions via the `matplotlib.pyplot` library to plot and visualize the relative performance of each algorithm on each benchmark. Currently, for single-objective optimization, we extract and plot the `objective` metric, which represents the objective of the best Trial seen so far as a function of Trial id/count (default). **Note**: this `objective` curve is monotonic and is computing upon converting to `BenchmarkRecord`. analyzers.plot\_from\_records(records) Observe that `plot_from_records` is a general plotting utility function that generates a grid of algorithm comparison plots. Specifically, it generates one plot for each Experimenter x Metrics in records, where each row represents an Experimenter and each column is a Metric represented in the record’s elements dictionary. Each plot has a curve for each algorithm. Adding Analysis[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#adding-analysis "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- Oftentimes, further analysis is needed to normalize metrics across multiple benchmarks or to visualize more context-dependent metrics, such as visualizing the Pareto frontier as a scatter plot. We focus on the former case, where objective curves require some form of normalization for each comparison across benchmarks. Many success metrics have been proposed: win rates, relative convergence, normalized objective score, [NeurIPS competition scores](https://arxiv.org/pdf/2012.03826.pdf) . To broadly cover such analysis scores, our [API](https://github.com/google/vizier/blob/main/vizier/benchmarks/__init__.py) introduces the `ConvergenceComparator` abstraction that compares two `ConvergenceCurve` at specified quantiles: @attr.define class ConvergenceComparator(abc.ABC): """(Simplified) Base class for convergence curve comparators. Attributes: baseline\_curve: The baseline ConvergenceCurve. compared\_curve: The compared ConvergenceCurve. """ \_baseline\_curve: ConvergenceCurve \= attr.field() \_compared\_curve: ConvergenceCurve \= attr.field() @abc.abstractmethod def score(self) \-> float: """Returns a summary score for the comparison between base and compared. Usually, higher positive numbers mean the compared curve is better than the baseline and vice versa. """ pass @abc.abstractmethod def curve(self) \-> ConvergenceCurve: """Returns a score curve for each xs.""" pass Generally, a higher score by convention should indicate that the compared curve is better than the baseline. Furthermore, a score of 0.0 indicates that the performance is similar and it would make sense of these scores to be symmetric. However, there is no such restrictions imposed on the API. As an example, we can add the `LogEfficiencyScore`, which is based off of [performance profiles](https://arxiv.org/pdf/cs/0102001.pdf) , a gold standard in optimization benchmarking. The LogEfficiencyScore essentially measures the percentage of Trials needed for the compared algorithm to match the baseline performance. If score = 1, then the compared algorithm uses e−1∗T Trials to reach the same objective as the baseline algorithm in T trials. analyzed\_records \= analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=records, baseline\_algo\='random' ) analyzers.plot\_from\_records(analyzed\_records) Custom Comparators[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#custom-comparators "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- To write a custom `ConvergenceComparator`, simply follow the abstract class defined above and form a `ConvergenceComparatorFactory`, which can then be passed into `add_comparison_metrics`. Note that we are constantly adding more benchmarking scores into our analyzers base and welcome submissions. Let us try to write a custom `WinRateComparator` that looks at the simple metric of comparing whether one curve is better than the other, for each `xs`. **NOTE:** You may always assume in a `Comparator` that both curves are either `INCREASING` (sign = 1) or `DECREASING` (signand that the sign of the curves is stored in `self._sign`. class WinRateComparator(analyzers.ConvergenceComparator): """Comparator method based on simple win rate comparison.""" def score(self) \-> float: return np.nanmedian(self.curve().ys) def curve(self) \-> analyzers.ConvergenceCurve: baseline\_ys \= self.\_sign \* self.\_baseline\_curve.ys compared\_ys \= self.\_sign \* self.\_compared\_curve.ys \# Compares all pairs of compared to baseline curve. all\_comparisons \= np.apply\_along\_axis( lambda base: np.mean(compared\_ys \> base, axis\=0), axis\=1, arr\=baseline\_ys, ) return analyzers.ConvergenceCurve( xs\=self.\_baseline\_curve.xs, ys\=np.mean(all\_comparisons, axis\=0, keepdims\=True), ) Now, we add a simple ComparatorFactory and inject the factory into `add_comparison_metrics` to create our new scoring plots. Note that one can also manually add customized `PlotElements` that can be in histogram, or scatter form. class WinRateComparatorFactory(analyzers.ConvergenceComparatorFactory): """Factory class for WinRateComparatorFactory.""" def \_\_call\_\_( self, baseline\_curve: analyzers.ConvergenceCurve, compared\_curve: analyzers.ConvergenceCurve, baseline\_quantile: float \= 0.5, compared\_quantile: float \= 0.5, ) \-> analyzers.ConvergenceComparator: return WinRateComparator( baseline\_curve\=baseline\_curve, compared\_curve\=compared\_curve, baseline\_quantile\=baseline\_quantile, compared\_quantile\=compared\_quantile, name\='win\_rate', ) \# Add WinRateComparator plots and visualize them. analyzed\_records\_with\_winrate \= ( analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=analyzed\_records, baseline\_algo\='random', comparator\_factory\=WinRateComparatorFactory(), ) ) analyzers.plot\_from\_records(analyzed\_records\_with\_winrate) Summarizing with Comparators[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#summarizing-with-comparators "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- One of the main benefits of using `ConvergenceComparator` is that these metrics are usually already normalized across different optimization problems and contexts. We provide utility functions to perform summarization procedures across `BenchmarkRecords`, which can used with the `summarize` feature of the `BenchmarkRecordAnalyzer`. Note that we can write a custom score and summary function, but we use our default summarizer that simply plots the distribution of scores across problems as histograms. import json \# Summarized across experimenters by giving reduced keys. def record\_to\_reduced\_keys(record): bbob\_dict \= json.loads(record.experimenter\_metadata\['bbob\_factory'\]) experimenter \= bbob\_dict.pop('name') return experimenter, json.dumps(bbob\_dict) summarized\_records \= analyzers.BenchmarkRecordAnalyzer.summarize(analyzed\_records\_with\_winrate, record\_to\_reduced\_keys) analyzers.plot\_from\_records(summarized\_records) References[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/analyzing_benchmarks.html#references "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------- * Benchmark analysis tools can be found [here](https://github.com/google/vizier/tree/main/vizier/benchmarks/analyzers.py) . * Convergence curve utils and comparators can be found [here](https://github.com/google/vizier/tree/main/vizier/_src/benchmarks/analyzers/convergence_curve.py) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Debugging tips — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/latest/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/index.html) * Debugging tips * [View page source](https://oss-vizier.readthedocs.io/en/latest/_sources/advanced_topics/tfp/debugging.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/debugging.ipynb) Debugging tips[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/debugging.html#debugging-tips "Link to this heading") ======================================================================================================================================== JAX[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/debugging.html#jax "Link to this heading") ------------------------------------------------------------------------------------------------------------------ JAX’s has a number of useful [debugging tools](https://jax.readthedocs.io/en/latest/debugging/index.html) including: * `jax.debug.print` to print values, even inside of jit-compiled code. * jit-able runtime error checking with `jax.experimental.checkify`. * `jax_debug_nans` flag to automatically detect when NaNs are produced in jit-compiled code. * [`disable_jit`](https://jax.readthedocs.io/en/latest/_autosummary/jax.disable_jit.html) , a context manager that disables `jit()` behavior. TFP[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/debugging.html#tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * TFP objects (bijectors, distributions, PSD kernels) have a `validate_args` boolean arg to `__init__`. If `True`, it runs additional (possibly expensive) runtime checks, e.g. to verify that parameters like `length_scale` are nonnegative. In TFP, we enable `validate_args` in unit tests, and use it as a debugging tool. * Reproducibility: All functions and methods in TFP rely on random number generation, such as the `sample` method of distributions, take a `seed` arg, which in JAX is an instance of `jax.random.PRNGKey`. This arg is mandatory in TFP-on-JAX, and ensures reproducible random number generation. See the `jax.random` [documentation](https://jax.readthedocs.io/en/latest/jax.random.html) for more details. * Tests of sample statistics: TFP’s internal `test_util` module includes [`assertAllMeansClose`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/internal/test_util.py#L349) , which asserts that the mean of a sample is as expected, and diagnoses the statistical significance of failures. #@title Imports from jax import numpy as jnp, tree\_util from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfpk \= tfp.math.psd\_kernels \# Demo of \`validate\_args\`. print('Without runtime arg validation, the kernel with negative amplitude happily builds.') k \= tfpk.MaternFiveHalves(amplitude\=-1., validate\_args\=False) print('With runtime arg validation:') k \= tfpk.MaternFiveHalves(amplitude\=-1., validate\_args\=True) What is “AutoCompositeTensor”?[](https://oss-vizier.readthedocs.io/en/latest/advanced_topics/tfp/debugging.html#what-is-autocompositetensor "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might have noticed that the base classes of the bijectors and PSD kernels are `AutoCompositeTensorBijector` and `AutoCompositeTensorPSDKernel`. In TensorFlow, objects that inherit from `CompositeTensor` have a recipe that allows them to be flattened into collections of Tensors and rebuilt, so that they can cross `tf.function` boundaries and interact with TF control flow similarly to Tensors (e.g., be passed in a `while_loop`’s carried state). JAX has a similar notion called [Pytree](https://jax.readthedocs.io/en/latest/pytrees.html) . Subclassing the `AutoCompositeTensor*` versions of TFP base classes means that the class will be registered as a Pytree node (making use of shared CompositeTensor/Pytree machinery in TFP). For the Flax model to return a GP in JIT-compiled code, it’s necessary for the GP and its PSD kernel to be Pytrees. gp \= tfd.GaussianProcess( tfpk.MaternFiveHalves(length\_scale\=jnp.ones(\[5\])), observation\_noise\_variance\=jnp.array(\[0.5\])) gp\_flat, gp\_tree \= tree\_util.tree\_flatten(gp) print(f'GP flattened into arrays: {gp\_flat}') rebuilt\_gp \= tree\_util.tree\_unflatten(gp\_tree, gp\_flat) assert isinstance(rebuilt\_gp, tfd.GaussianProcess) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Debugging tips — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Advanced Topics](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/index.html) * Debugging tips * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/advanced_topics/tfp/debugging.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/advanced_topics/tfp/debugging.ipynb) Debugging tips[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/debugging.html#debugging-tips "Link to this heading") ======================================================================================================================================== JAX[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/debugging.html#jax "Link to this heading") ------------------------------------------------------------------------------------------------------------------ JAX’s has a number of useful [debugging tools](https://jax.readthedocs.io/en/latest/debugging/index.html) including: * `jax.debug.print` to print values, even inside of jit-compiled code. * jit-able runtime error checking with `jax.experimental.checkify`. * `jax_debug_nans` flag to automatically detect when NaNs are produced in jit-compiled code. * [`disable_jit`](https://jax.readthedocs.io/en/latest/_autosummary/jax.disable_jit.html) , a context manager that disables `jit()` behavior. TFP[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/debugging.html#tfp "Link to this heading") ------------------------------------------------------------------------------------------------------------------ * TFP objects (bijectors, distributions, PSD kernels) have a `validate_args` boolean arg to `__init__`. If `True`, it runs additional (possibly expensive) runtime checks, e.g. to verify that parameters like `length_scale` are nonnegative. In TFP, we enable `validate_args` in unit tests, and use it as a debugging tool. * Reproducibility: All functions and methods in TFP rely on random number generation, such as the `sample` method of distributions, take a `seed` arg, which in JAX is an instance of `jax.random.PRNGKey`. This arg is mandatory in TFP-on-JAX, and ensures reproducible random number generation. See the `jax.random` [documentation](https://jax.readthedocs.io/en/latest/jax.random.html) for more details. * Tests of sample statistics: TFP’s internal `test_util` module includes [`assertAllMeansClose`](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/python/internal/test_util.py#L349) , which asserts that the mean of a sample is as expected, and diagnoses the statistical significance of failures. #@title Imports from jax import numpy as jnp, tree\_util from tensorflow\_probability.substrates import jax as tfp tfd \= tfp.distributions tfpk \= tfp.math.psd\_kernels \# Demo of \`validate\_args\`. print('Without runtime arg validation, the kernel with negative amplitude happily builds.') k \= tfpk.MaternFiveHalves(amplitude\=-1., validate\_args\=False) print('With runtime arg validation:') k \= tfpk.MaternFiveHalves(amplitude\=-1., validate\_args\=True) What is “AutoCompositeTensor”?[](https://oss-vizier.readthedocs.io/en/stable/advanced_topics/tfp/debugging.html#what-is-autocompositetensor "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might have noticed that the base classes of the bijectors and PSD kernels are `AutoCompositeTensorBijector` and `AutoCompositeTensorPSDKernel`. In TensorFlow, objects that inherit from `CompositeTensor` have a recipe that allows them to be flattened into collections of Tensors and rebuilt, so that they can cross `tf.function` boundaries and interact with TF control flow similarly to Tensors (e.g., be passed in a `while_loop`’s carried state). JAX has a similar notion called [Pytree](https://jax.readthedocs.io/en/latest/pytrees.html) . Subclassing the `AutoCompositeTensor*` versions of TFP base classes means that the class will be registered as a Pytree node (making use of shared CompositeTensor/Pytree machinery in TFP). For the Flax model to return a GP in JIT-compiled code, it’s necessary for the GP and its PSD kernel to be Pytrees. gp \= tfd.GaussianProcess( tfpk.MaternFiveHalves(length\_scale\=jnp.ones(\[5\])), observation\_noise\_variance\=jnp.array(\[0.5\])) gp\_flat, gp\_tree \= tree\_util.tree\_flatten(gp) print(f'GP flattened into arrays: {gp\_flat}') rebuilt\_gp \= tree\_util.tree\_unflatten(gp\_tree, gp\_flat) assert isinstance(rebuilt\_gp, tfd.GaussianProcess) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Benchmarking with Ray — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Benchmarking with Ray * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/benchmarks/ray_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/ray_benchmarks.ipynb) Benchmarking with Ray[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html#benchmarking-with-ray "Link to this heading") ========================================================================================================================================================= We provide a brief guide below on the Vizier + Ray integration, and how to benchmark with all publicly available algorithms on [Ray Tune](https://docs.ray.io/en/latest/tune/) . Notably, Tune integrates with a wide range of additional hyperparameter optimization tools, including Ax, BayesOpt, BOHB, Dragonfly, FLAML, HEBO, Hyperopt, Nevergrad, Optuna, SigOpt, skopt, and ZOOpt. ![alt-text](https://docs.ray.io/en/latest/_images/tune_overview.png) Initial Installation[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html#initial-installation "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax\] !pip install \-U "ray\[default\]" Algorithm and Experimenter Factories[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html#algorithm-and-experimenter-factories "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As mentioned in previous guides, since we want to compare algorithms across multiple benchmarks, we first create a bunch of relevant benchmark experimenters. To do so, we use `SerializableExperimenterFactory` from our [Experimenters API](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) to modularize the construction of multiple benchmark components. For example, here we can create a diverse set of BBOB functions with different dimensions via the `BBOBExperimenterFactory`. Then, we can print out the full serialization of the benchmarks that we have created. import itertools import numpy as np from vizier.benchmarks import experimenters function\_names \= \[\ 'Sphere',\ 'BentCigar',\ 'Katsuura',\ \] dimensions \= \[4, 8\] product\_list \= list(itertools.product(function\_names, dimensions)) experimenter\_factories \= \[\] for product in product\_list: name, dim \= product bbob\_factory \= experimenters.BBOBExperimenterFactory(name\=name, dim\=dim) experimenter\_factory \= experimenters.SingleObjectiveExperimenterFactory( bbob\_factory, shift\=np.random.uniform(low\=-2, high\=2, size\=dim), noise\_type\='LIGHT\_ADDITIVE\_GAUSSIAN', ) experimenter\_factories.append(experimenter\_factory) print(experimenter\_factory.dump()) Next, we need to define our algorithms by installing the relevant packages and importing the relevant algorithms. For simplicity, we only compare against only a subset of the algorithms that Ray supports. **NOTE:** We provide the `VizierSearch` class in our own libaries that can directly use the `Searcher` API in Ray. The imports are given below. pip install ax\-platform scikit\-optimize hyperopt optuna bayesian\-optimization from ray import tune from ray.tune.search.ax import AxSearch from ray.tune.search.bayesopt import BayesOptSearch from ray.tune.search.hyperopt import HyperOptSearch from ray.tune.search.optuna import OptunaSearch from ray.tune.search.skopt import SkOptSearch from vizier import raytune as vzr from vizier.\_src.raytune.vizier\_search import VizierSearch algorithm\_factories \= { 'ray': lambda: None, 'vizier': VizierSearch, 'ax': AxSearch, 'bayesopt': BayesOptSearch, 'optuna': OptunaSearch, 'hyperopt': HyperOptSearch, 'skopt': SkOptSearch, } Running RayTune[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html#running-raytune "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Running RayTune using `ExperimenterFactory` is made easy using our utility libraries which takes in any factory with a `TuneConfig` to run the algorithm on the corresponding benchmark. Let us first run one algorithm on the first benchmark and see the results that we get. **NOTE:** This uses a local Ray instance. ALGORITHM\_NAME \= 'ray' \# @param str experimenter\_factory \= experimenter\_factories\[0\] factory \= algorithm\_factories\[ALGORITHM\_NAME\] tune\_config \= tune.TuneConfig( search\_alg\=factory(), num\_samples\=4, max\_concurrent\_trials\=1, ) vzr.run\_tune.run\_tune\_from\_factory(experimenter\_factory, tune\_config) Now, we repeat our runs for each `ExperimenterFactory` and each algorithm, converting the results into `PlotElements` for easy plotting and comparison. from vizier.benchmarks import analyzers NUM\_REPEATS \= 3 \# @param NUM\_ITERATIONS \= 50 \# @param def results\_to\_element(results\_list): curves \= \[\] for results in results\_list: raw\_ys \= np.array(results.get\_dataframe()\['bbob\_eval\_before\_noise'\]) ys \= np.minimum.accumulate(raw\_ys) curve \= analyzers.ConvergenceCurve( xs\=np.arange(1, len(ys) + 1), ys\=ys.reshape((1, len(ys))), trend\=analyzers.ConvergenceCurve.YTrend.DECREASING, ) curves.append(curve) all\_curves \= analyzers.ConvergenceCurve.align\_xs(curves) ele \= analyzers.PlotElement(curve\=all\_curves\[0\], yscale\='symlog') return ele all\_records \= \[\] for experimenter\_factory in experimenter\_factories: for algorithm, factory in algorithm\_factories.items(): results \= \[\] for \_ in range(NUM\_REPEATS): tune\_config \= tune.TuneConfig( search\_alg\=factory(), num\_samples\=NUM\_ITERATIONS, max\_concurrent\_trials\=1, ) results.append( vzr.run\_tune.run\_tune\_from\_factory(experimenter\_factory, tune\_config) ) ele \= results\_to\_element(results) record \= analyzers.BenchmarkRecord( algorithm\=algorithm, experimenter\_metadata\=experimenter\_factory.dump(), plot\_elements\={'objective': ele}, ) all\_records.append(record) analyzed\_records \= analyzers.BenchmarkRecordAnalyzer.add\_comparison\_metrics( records\=all\_records, baseline\_algo\='ray' ) analyzers.plot\_from\_records(analyzed\_records) Running Parallelized Ray[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/ray_benchmarks.html#running-parallelized-ray "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------- In the previous example, we are using Ray local instances and running each benchmark in sequential format, which can take minutes. When there are a large number of benchmarks or computationally intensive benchmark runs, using parallelism distributed across each (algorithm, benchmark) tuple is crucial for reasonable benchmarking turnaround. We recommend using the [Ray Jobs API](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html) to distribute work across clusters. [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Converters — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Converters * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/user/converters.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/user/converters.ipynb) Converters[](https://oss-vizier.readthedocs.io/en/stable/guides/user/converters.html#converters "Link to this heading") ========================================================================================================================= This documentation demonstrates how to use converters for representing PyVizier objects as NumPy arrays and vice-versa. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/user/converters.html#installation-and-reference-imports "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from vizier import pyvizier as vz from vizier.pyvizier import converters Suppose we had a problem statement and some trials associated to the study. \# Setup search space search\_space \= vz.SearchSpace() root \= search\_space.root root.add\_float\_param(name\='double', min\_value\=0.0, max\_value\=1.0) root.add\_int\_param(name\='int', min\_value\=1, max\_value\=10) root.add\_discrete\_param(name\='discrete', feasible\_values\=\[0.1, 0.3, 0.5\]) root.add\_categorical\_param(name\='categorical', feasible\_values\=\['a', 'b', 'c'\]) \# Setup metric configurations m1 \= vz.MetricInformation(name\='m1', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) m2 \= vz.MetricInformation(name\='m2', goal\=vz.ObjectiveMetricGoal.MINIMIZE) \# Final problem problem \= vz.ProblemStatement(search\_space, metric\_information\=\[m1, m2\]) \# Example trials trial1 \= vz.Trial( parameters\={'double': 0.6, 'int': 2, 'discrete': 0.1, 'categorical': 'a'}, final\_measurement\=vz.Measurement(metrics\={'m1': 0.1, 'm2': 0.2}), ) trial2 \= vz.Trial( parameters\={'double': 0.1, 'int': 6, 'discrete': 0.3, 'categorical': 'b'}, final\_measurement\=vz.Measurement(metrics\={'m1': \-1.0, 'm2': 0.8}), ) Quick Start[](https://oss-vizier.readthedocs.io/en/stable/guides/user/converters.html#quick-start "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- To use numerical models, both our `x` (parameters) and `y` (metrics) need to be formatted as numpy arrays. We can directly do so with `TrialToArrayConverter`: t2a\_converter \= converters.TrialToArrayConverter.from\_study\_config(problem) xs, ys \= t2a\_converter.to\_xy(\[trial1, trial2\]) We can also convert the `xs` back into PyVizier `ParameterDict`s: t2a\_converter.to\_parameters(xs) Behind the scenes, the `TrialToArrayConverter` actually uses a `DefaultTrialConverter` which first converts both trial parameters and metrics into `dict[str, np.ndarray]` and then concatenates the arrays together. converter \= converters.DefaultTrialConverter.from\_study\_config(problem) xs\_dict, ys\_dict \= converter.to\_xy(\[trial1, trial2\]) Trials can be recovered too: original\_trials \= converter.to\_trials(xs\_dict, ys\_dict) Customization[](https://oss-vizier.readthedocs.io/en/stable/guides/user/converters.html#customization "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- There are multiple ways to convert parameters of specific types. For example, some common methods to convert the `'categorical'` parameter (with feasible values `['a', 'b', 'c']`) can be: * Integer Index: `'b' -> 1` since `b` has index 1 among feasible values. * One-Hot: `'b' -> [0, 1, 0]` using one-hot encoding. Additional considerations can be, for example: * Whether to scale continuous parameter values into `[0,1]` * Whether to always sign-flip metrics to assume maximization only. These options can be specified when constructing both `TrialToArrayConverter` and `DefaultTrialConverter` ([source code](https://github.com/google/vizier/blob/main/vizier/pyvizier/converters/core.py) ): @classmethod def from\_study\_config( cls, study\_config: pyvizier.ProblemStatement, \*, scale: bool \= True, pad\_oovs: bool \= True, max\_discrete\_indices: int \= 0, flip\_sign\_for\_minimization\_metrics: bool \= True, dtype\=np.float64, ): For more fine-grained control over specific `ParameterConfig`s and `MetricInformation`s, a user can specify individual arguments to each `DefaultModelInputConverter` and `DefaultModelOutputConverter` respectively. \# Only considers the 'double' parameter values. double\_pc \= search\_space.get('double') double\_converter \= converters.DefaultModelInputConverter(double\_pc, scale\=True) double\_converter.convert(\[trial1, trial2\]) \# Only considers the 'categorical' parameter values. categorical\_pc \= search\_space.get('categorical') categorial\_converter \= converters.DefaultModelInputConverter(categorical\_pc, onehot\_embed\=True) categorial\_converter.convert(\[trial1, trial2\]) \# Only considers the 'm1' metric values. m1\_converter \= converters.DefaultModelOutputConverter(m1) m1\_converter.convert(\[trial1.final\_measurement, trial2.final\_measurement\]) These can be inserted into the `DefaultTrialConverter`: parameter\_converters \= \[double\_converter, categorial\_converter\] metric\_converters \= \[m1\_converter\] custom\_converter \= converters.DefaultTrialConverter(parameter\_converters, metric\_converters) custom\_converter.to\_xy(\[trial1, trial2\]) \# Same array outputs as above. For full customization, the user may create their own `ModelInputConverter`s and `ModelOutputConverter`s. class ModelInputConverter(metaclass\=abc.ABCMeta): """Interface for extracting inputs to the model.""" @abc.abstractmethod def convert(self, trials: Sequence\[vz.TrialSuggestion\]) \-> np.ndarray: """Returns an array of shape (number of trials, feature dimension).""" @property @abc.abstractmethod def output\_spec(self) \-> NumpyArraySpec: """Provides specification of the output from this converter.""" @property @abc.abstractmethod def parameter\_config(self): """Original ParameterConfig that this converter acts on.""" @abc.abstractmethod def to\_parameter\_values( self, array: np.ndarray ) \-> List\[Optional\[vz.ParameterValue\]\]: """Convert and clip to the nearest feasible parameter values.""" class ModelOutputConverter(metaclass\=abc.ABCMeta): """Metric converter interface.""" @abc.abstractmethod def convert(self, measurements: Sequence\[vz.Measurement\]) \-> np.ndarray: """Returns N x 1 array.""" pass @abc.abstractmethod def to\_metrics(self, labels: np.ndarray) \-> Sequence\[Optional\[vz.Metric\]\]: """Returns a list of pyvizier metrics.""" @property @abc.abstractmethod def metric\_information(self) \-> vz.MetricInformation: """Describes the semantics of the return value from convert() method.""" @property def output\_shape(self) \-> Tuple\[None, int\]: return (None, 1) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Designers — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Designers * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/developer/designers.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/designers.ipynb) Designers[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html#designers "Link to this heading") =========================================================================================================================== This documentation will allow a developer to use the Designer API for typical algorithm design. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from typing import Optional, Sequence import numpy as np from vizier import algorithms as vza from vizier import pythia from vizier import pyvizier as vz from vizier.algorithms import designers Designers[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html#id1 "Link to this heading") --------------------------------------------------------------------------------------------------------------------- The `Designer` API is an intuitive abstraction for writing and _designing_ algorithms. It only requires two basic methods, `update()` and `suggest()`, shown below. The source of truth for `Designer` can be found [here](https://github.com/google/vizier/blob/main/vizier/algorithms/__init__.py) . class Designer(...): """Suggestion algorithm for sequential usage.""" @abc.abstractmethod def update(self, completed: CompletedTrials, all\_active: ActiveTrials) \-> None: """Updates recently completed and ALL active trials into the designer's state.""" @abc.abstractmethod def suggest(self, count: Optional\[int\] \= None) \-> Sequence\[vz.TrialSuggestion\]: """Make new suggestions.""" Every time `update()` is called, the `Designer` will get any newly `COMPLETED` trials since the last `update()` call, and will get all `ACTIVE` trials at the current moment in time. **Note:** Trials which may have been provided as `ACTIVE` in previous `update()` calls, can be provided as `COMPLETED` in subsequent `update()` calls. GP-Bandit Designer Example[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html#gp-bandit-designer-example "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------------------------- The following example, using the default GP-Bandit algorithm, shows how to interact with Vizier designers. \# The problem statement (which parameters are being optimized) problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', 0.0, 1.0) problem.search\_space.root.add\_float\_param('y', 0.0, 1.0) problem.metric\_information.append( vz.MetricInformation( name\='maximize\_metric', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) \# Create a new designer object designer \= designers.VizierGPBandit(problem) \# Ask the designer for 2 suggestions suggestions \= designer.suggest(count\=2) In this case, since the designer was not update with any `COMPLETED` or `ACTIVE` trials, it will produce suggestions which will look like: \[TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.5, 'y': 0.5}), metadata\=Metadata((namespace:, items: {'seeded': 'center'}), current\_namespace\=)),\ TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.10274669379450661, 'y': 0.10191725529767912}), metadata\=Metadata((namespace:, items: {}), current\_namespace\=))\] Note that the first suggestion is seeded at the center of the search space, and the second suggestion is random. If we call `designer.suggest()` again before calling `update()`, the designer will produce an identical first suggestion at the center of the search space, and a second random suggestion. Only when we call `update()`, will the designer update its internal state and generate different suggestions: completed\_trials \= \[\] for suggestion in suggestions: metric\_value \= np.random.random() \# Make up a fake metric value. suggestion.to\_trial().complete( vz.Measurement(metrics\={'maximize\_metric': metric\_value}) ) \# Update the designer with the completed trials. designer.update(vza.CompletedTrials(completed\_trials), vza.ActiveTrials()) \# Ask for more suggestions. new\_suggestions \= designer.suggest(count\=2) Thus `COMPLETED` trials should be incrementally updated, while all `ACTIVE` trials are passed to the designer in every `update()` call. A `Designer` can also be seeded with pre-existing data. Consider the following example: \# Make a fresh designer. designer \= designers.VizierGPBandit(problem) \# Create completed trials representing pre-existing training data. trials \= \[vz.Trial(parameters\={'x': 0.5, 'y': 0.6}).complete(vz.Measurement(metrics\={'maximize\_metric': 0.3}))\] designer.update(vza.CompletedTrials(trials), vza.ActiveTrials()) \# As the designer for suggestions. suggestions \= designer.suggest(count\=2) In this case, the designer will **not** return a first trial seeded at the center of the search space, since it has been updated with completed trials. The new suggestions will look something like: \[TrialSuggestion(parameters\=ParameterDict(\_items\={'x': 0.7199945005054509, 'y': 0.3800034493548722}), ...\] Additional References[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/designers.html#additional-references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- * Our [designers folder](https://github.com/google/vizier/tree/main/vizier/_src/algorithms/designers) contains examples of designers. * Our [evolution folder](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/evolution) contains examples of creating evolutionary designers, such as [NSGA2](https://ieeexplore.ieee.org/document/996017/) . * Our [designer testing routine](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/testing/test_runners.py) contains up-to-date examples on interacting with designers. [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Early Stopping — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Early Stopping * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/developer/early_stopping.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/early_stopping.ipynb) Early Stopping[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html#early-stopping "Link to this heading") ========================================================================================================================================== This notebook will allow a developer to: * Understand the Early Stopping API. * Write Pythia policies for early stopping. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import numpy as np from vizier import pythia Early Stopping[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html#id1 "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------- In hyperparameter optimization, early stopping is a useful mechanism to prevent wasted resources by stopping unpromising trials. Two main considerations for determining whether to stop an active trial are: * **At a macro level, how a trial’s performance compares to the rest of the trials globally.** For example, we may stop a trial if it is predicted to significantly underperform compared to the history of trials so far in the study. * **At a micro level, how a trial’s intermediate measurements are changing over time.** For example, in a classification task, overfitting may be happening when test accuracy starts to decrease. API[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html#api "Link to this heading") -------------------------------------------------------------------------------------------------------------------- Based on the above considerations, to allow full flexibility to consider when to stop a trial, we thus use the following abridged API below. Exact class entrypoint can be found [here](https://github.com/google/vizier/blob/main/vizier/pythia.py) . The `EarlyStopRequest` takes in a set of trial ID’s for early stopping consideration. However, note that trials outside of this set can also be stopped. class EarlyStopRequest: """Early stopping request.""" trial\_ids: Optional\[FrozenSet\[int\]\] In addition, we have the `EarlyStopDecision` to denote a single trial’s stopping condition and the plural `EarlyStopDecisions` for a set of trials: class EarlyStopDecision: """Stopping decision on a single trial.""" id: int should\_stop: bool class EarlyStopDecisions: """This is the output of the Policy.early\_stop() method.""" decisions: list\[EarlyStopDecision\] metadata: vz.MetadataDelta They will be used in the Pythia policy’s `early_stop` method: class Policy(abc.ABC): """Interface for Pythia2 Policy subclasses.""" @abc.abstractmethod def early\_stop(self, request: EarlyStopRequest) \-> EarlyStopDecisions: """Decide which Trials Vizier should stop.""" Example usage[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/early_stopping.html#example-usage "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------- As an example, suppose our rule is to stop all requested trials whose 50th intermediate measurement is too low, e.g. bottom 10% of all trials so far. class MyEarlyStoppingPolicy(pythia.Policy): """Stops requested trial if its 50th measurement is too low.""" def \_\_init\_\_(self, policy\_supporter: pythia.PolicySupporter, index: int \= 50): self.\_policy\_supporter \= policy\_supporter self.\_index \= index def early\_stop( self, request: pythia.EarlyStopRequest ) \-> pythia.EarlyStopDecisions: metric\_name \= request.study\_config.metric\_information.item().name \# Obtain cutoff for 10th percentile. all\_trials \= self.\_policy\_supporter.GetTrials(study\_guid\=request.study\_guid) all\_metrics \= \[\] for trial in all\_trials: if len(trial.measurements) \> self.\_index: all\_metrics.append(trial.measurements\[self.\_index\].metrics\[metric\_name\]) cutoff \= np.percentile(all\_metrics, 10) \# Filter requested trials by cutoff. considered\_trials \= \[\ trial for trial in all\_trials if trial.id in request.trial\_ids\ \] stopping\_decisions \= \[\] for trial in considered\_trials: if trial.measurements\[self.\_index\].metrics\[metric\_name\] < cutoff: decision \= pythia.EarlyStopDecision( trial.id, reason\='Below cutoff', should\_stop\=True ) else: decision \= pythia.EarlyStopDecision( trial.id, reason\='Above cutoff', should\_stop\=False ) stopping\_decisions.append(decision) return pythia.EarlyStopDecisions(decisions\=stopping\_decisions) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Metadata — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Metadata * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/developer/metadata.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/metadata.ipynb) Metadata[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#metadata "Link to this heading") ======================================================================================================================== We provide a guide below on common developer uses of the `Metadata` primitive. OSS Vizier can store `Metadata` in both the `ProblemStatement` and each `TrialSuggestion`/`Trial`, with common use cases: * Containing additional information outside of standard parameter types. * Allowing user code to store small amounts of state information inside OSS Vizier, attached to the OSS Vizier study. * Wrapping search spaces and corresponding algorithms which are naturally incompatible with OSS Vizier’s default API, to still allow a distributed backend service. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from vizier import pyvizier as vz from google.protobuf import any\_pb2 Metadata basics[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#metadata-basics "Link to this heading") -------------------------------------------------------------------------------------------------------------------------------------- The [`Metadata`](https://github.com/google/vizier/blob/main/vizier/pyvizier/__init__.py) is a key-value store, where: * Keys are UTF-8 strings. * Values can be strings or protocol buffers. While values of type `int`, `float`, and more complex objects can also be used, **the developer is responsible for serializing / unserializing said objects.** metadata \= vz.Metadata() metadata\['proto'\] \= any\_pb2.Any(...) metadata\['string'\] \= 'hello' Additionally, `Metadata` can act as a “dictionary of dictionaries”, i.e. a hierarchy of dictionaries, via its `Namespace` functionality via calling `.ns()`, which creates another `Metadata` which shares data with the original. child\_metadata \= metadata.ns('child') grandchild\_metadata \= child\_metadata.ns('child') grandchild\_metadata\['string'\] \= 'goodbye' assert metadata.ns('child').ns('child')\['string'\] \== 'goodbye' ProblemStatement Metadata[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#problemstatement-metadata "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------- The `ProblemStatement` object contains a `metadata` attribute, ideally for storing global metadata related to the study. Note that `Metadata` will not be used in the optimization process, UNLESS there is a custom algorithm configured to use it. Below is a usage example when training an image classifier, where one may wish to store training-related attributes in `Metadata`. problem\_statement \= vz.ProblemStatement() problem\_statement.metadata\['dataset'\] \= 'cifar10' problem\_statement.metadata\['architecture'\] \= 'resnet\_18' Trial Metadata[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#trial-metadata "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------ `TrialSuggestion` and subclass `Trial` also contain a `metadata` attribute. This in contrast, should be used to store metadata related to the specific Trial. In the image classification case, examples would be the type of GPU used for training and if the training worker has been preempted. trial \= vz.Trial() trial.metadata\['gpu\_used'\] \= 'P100' trial.metadata\['preempted'\] \= 'True' OSS Vizier as a backend via `Metadata`[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/metadata.html#oss-vizier-as-a-backend-via-metadata "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As an advanced developer use case, one may extend OSS Vizier’s search space capabilities using `Metadata`. Custom algorithms can provide full freedom in expressing more complex search spaces (e.g. graphs) using `Metadata`. Example use cases: * Combinatorial optimization, where the search space may consist of graphs or multiple selection (e.g. (NK)) primitives. Algorithms commonly include evolutionary methods, which also require custom mutation operations. * Free-form textual data used for suggestions (and maybe even evaluation metrics!), as common with language-based applications. \# Setup combinatorial search space. choose\_problem \= vz.ProblemStatement() choose\_problem.metadata \= vz.Metadata({'N': '10', 'K': '3'}) \# Example of a suggestion proposed by a custom algorithm. suggestion \= vz.TrialSuggestion() suggestion.metadata\['chosen\_indices'\] \= '\[0, 3, 7\]' The algorithm behavior can even be changed mid-optimization with `Metadata` using a client! This is in fact used extensively in our integrations with [PyGlove](https://github.com/google/pyglove) to allow a running Pythia policy to change search spaces or mutations online. \# Original mutation rate. mutation\_problem \= vz.ProblemStatement() mutation\_problem.metadata \= vz.Metadata({'mutation\_rate': '0.1'}) \# ... \# Assume algorithm started running in the Pythia service. \# ... \# Set new mutation rate. study\_metadata \= vz.Metadata({'mutation\_rate': '0.2'}) \# Prevent this trial from being used in the population. trial\_metadata \= vz.Metadata({'use\_in\_population' \= 'False'}) trial\_id \= 1 \# Create unit of metadata update. metadata\_delta \= vz.MetadataDelta( on\_study\=study\_metadata, on\_trials\={trial\_id: trial\_metadata}) Once we have a client, we can commit the metadata update: client.update\_metadata(metadata\_delta) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Running Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Running Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/benchmarks/running_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/running_benchmarks.ipynb) Running Benchmarks[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#running-benchmarks "Link to this heading") ======================================================================================================================================================= We will demonstrate below how to use our benchmark runner pipeline. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#installation-and-reference-imports "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] from vizier import algorithms as vza from vizier import benchmarks as vzb from vizier.algorithms import designers from vizier.benchmarks import experimenters Example experimenter and designer factory which we will use later. experimenter \= experimenters.NumpyExperimenter( experimenters.bbob.Sphere, experimenters.bbob.DefaultBBOBProblemStatement(5) ) designer\_factory \= designers.GridSearchDesigner.from\_problem Algorithms and Experimenters[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#algorithms-and-experimenters "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every study can be seen conceptually as a simple loop between an algorithm and objective. In terms of code, the algorithm corresponds to a `Designer`/`Policy` and objective to an `Experimenter`. Below is a simple sequential loop. designer \= designer\_factory(experimenter.problem\_statement()) for \_ in range(100): suggestion \= designer.suggest()\[0\] trial \= suggestion.to\_trial() experimenter.evaluate(\[trial\]) completed\_trials \= vza.CompletedTrials(\[trial\]) designer.update(completed\_trials, vza.ActiveTrials()) As seen above however, one modification we can make is to use variable batch sizes, rather than only suggesting and evaluating one-by-one. More generally, certain implementation details may arise: * How many parallel suggestions should the algorithm generate? * How many suggestions can be evaluated at once? * Should we use early stopping on certain unpromising trials? * Should we use a custom stopping condition instead of a fixed for-loop? * Can we swap in a different algorithm mid-loop? * Can we swap in a different objective mid-loop? API[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#api "Link to this heading") ------------------------------------------------------------------------------------------------------------------------- The code flexibility needed to simulate these real-life scenarios may cause complications as the evaluation benchmark may no longer be stateless. In order to broadly cover such scenarios, our [API](https://github.com/google/vizier/blob/main/vizier/benchmarks/__init__.py) introduces the `BenchmarkSubroutine`: class BenchmarkSubroutine(Protocol): """Abstraction for core benchmark routines. Benchmark protocols are modular alterations of BenchmarkState by reference. """ def run(self, state: BenchmarkState) \-> None: """Abstraction to alter BenchmarkState by reference.""" All routines use and potentially modify a `BenchmarkState`, which holds information about the objective via an `Experimenter` and the algorithm itself wrapped by a `PolicySuggester`. class BenchmarkState: """State of a benchmark run. It is altered via benchmark protocols.""" experimenter: Experimenter algorithm: PolicySuggester To wrap multiple `BenchmarkSubRoutines` together, we can use the `BenchmarkRunner`: class BenchmarkRunner(BenchmarkSubroutine): """Run a sequence of subroutines, all repeated for a few iterations.""" \# A sequence of benchmark subroutines that alter BenchmarkState. benchmark\_subroutines: Sequence\[BenchmarkSubroutine\] \# Number of times to repeat applying benchmark\_subroutines. num\_repeats: int def run(self, state: BenchmarkState) \-> None: """Run algorithm with benchmark subroutines with repetitions.""" Example usage[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#example-usage "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- Below is a typical example of simple suggestion and evaluation: runner \= vzb.BenchmarkRunner( benchmark\_subroutines\=\[\ vzb.GenerateSuggestions(),\ vzb.EvaluateActiveTrials(),\ \], num\_repeats\=100, ) benchmark\_state\_factory \= vzb.DesignerBenchmarkStateFactory( experimenter\=experimenter, designer\_factory\=designer\_factory ) benchmark\_state \= benchmark\_state\_factory() runner.run(benchmark\_state) We may obtain the evaluated trials via the `benchmark_state`, which contains a `PolicySupporter` via its `algorithm` field: all\_trials \= benchmark\_state.algorithm.supporter.trials print(all\_trials) Note that this design is maximally informative on everything that has happened so far in the study. For instance, we may also query incomplete/unused suggestions using the `PolicySupporter`. References[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/running_benchmarks.html#references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------- * Benchmark Runners can be found [here](https://github.com/google/vizier/tree/main/vizier/_src/benchmarks/runners) . [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Predictors — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Predictors * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/developer/predict.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/predict.ipynb) Predictors[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/predict.html#predictors "Link to this heading") =========================================================================================================================== This documentation will allow a developer to understand and use the `Predictor` API. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/predict.html#installation-and-reference-imports "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier\[jax,algorithms\] import numpy as np from vizier.\_src.benchmarks.experimenters.synthetic import bbob from vizier.algorithms import designers from vizier import algorithms as vza from vizier import pyvizier as vz import matplotlib.pyplot as plt Predictors[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/predict.html#id1 "Link to this heading") -------------------------------------------------------------------------------------------------------------------- The `Predictor` exposes a `predict()` method which takes `TrialSuggestion`s as inputs and returns their corresponding objective value predictions, represented by a `Prediction` class. The source of truth for predictors can be found [here](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/core/abstractions.py) . class Prediction: """Container to hold predictions.""" mean: chex.Array stddev: chex.Array metadata: Optional\[Metadata\] \= None class Predictor(abc.ABC): """Mixin for algorithms to expose prediction API.""" @abc.abstractmethod def predict( self, trials: Sequence\[TrialSuggestion\], ... ) \-> Prediction: In some cases involving a underlying probabilistic model, there’s a need to sample the posterior distribution in order to obtain the mean and standard deviation. In this case, the API allows specifying the the random key and number of samples (hidden arguments as `...` in `predict()` above). GP-Bandit Predict Example[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/predict.html#gp-bandit-predict-example "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- The `VizierGPBandit` class acts as both a `Designer` and a `Predictor` and allows users to obtain the underlying GP model’s mean and standard deviation for a given set of points. The example below demonstrates this capability. Setup problem statement and objective: \# The problem statement (which parameters are being optimized) problem \= vz.ProblemStatement() problem.search\_space.root.add\_float\_param('x', \-5.0, 5.0) problem.metric\_information.append( vz.MetricInformation( name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE)) \# The real objective function used for generating observations. f \= lambda x: bbob.Weierstrass(np.array(\[x\])) Create observations (i.e. completed trials) of the objective function. \# Generate suggestions. observations \= designers.QuasiRandomDesigner(problem.search\_space).suggest(30) \# Compute the real objective value and complete the trials. trials \= \[\] for idx, obs in enumerate(observations): trials.append( obs.to\_trial(idx).complete( vz.Measurement(metrics\={'obj': f(obs.parameters\['x'\].value)}) ) ) Create a `VizierGPBandit` designer and update it with the observations. **Note:** When two `VizierGPBandit` designers are updated with identical trials, they may still produce slightly different models and predictions due to inherent stochasticity during the training process. \# Create the GPBandit designer. gp\_designer \= designers.VizierGPBandit(problem) \# Update the GP-Bandit designer with completed trials. gp\_designer.update(vza.CompletedTrials(trials), vza.ActiveTrials()) Generate predictions in arbitrary points. \# Generate predictions. suggestions \= designers.GridSearchDesigner(problem.search\_space, double\_grid\_resolution\=500).suggest(500) predictions \= gp\_designer.predict(suggestions) Plot the predictions. plt.figure(figsize\=(8, 6)) \# Visualize the real objective function. xs \= np.linspace(\-5, 5, num\=1000) ys \= \[f(x) for x in xs\] plt.plot(xs, ys, label\='actual', color\='blue', alpha\=0.6) \# Visualize the observation points. obs\_x \= \[obs.parameters\['x'\].value for obs in observations\] obs\_y \= \[f(x) for x in obs\_x\] plt.scatter(obs\_x, obs\_y, label\='observations', marker\='o', color\='red') \# Visualize the predictions and confidence bounds. pred\_x \= \[suggestion.parameters\['x'\].value for suggestion in suggestions\] plt.plot(pred\_x, predictions.mean, label\='prediction', color\='green') lower \= predictions.mean \- predictions.stddev upper \= predictions.mean + predictions.stddev plt.fill\_between(pred\_x, lower, upper, color\='grey', alpha\=0.2) \# Add legend and title. plt.legend(loc\='best') plt.title(f'GPBandit Prediction vs. Actual') plt.xlabel('x') plt.show() ![../../_images/d226edbf2883cd7ee6d06ce9ff199547cdb96d390380923be3362396204ba9e4.png](https://oss-vizier.readthedocs.io/en/stable/_images/d226edbf2883cd7ee6d06ce9ff199547cdb96d390380923be3362396204ba9e4.png) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Pythia Policies and Hosting Designers — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Pythia Policies and Hosting Designers * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/developer/pythia_policies.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/developer/pythia_policies.ipynb) Pythia Policies and Hosting Designers[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#pythia-policies-and-hosting-designers "Link to this heading") ========================================================================================================================================================================================= This documentation will allow a developer to: * Understand the basic structure of a Pythia Policy. * Host Designers in the service. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#installation-and-reference-imports "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier from typing import Optional, Sequence from vizier import pythia from vizier import algorithms from vizier.service import pyvizier as vz from vizier.\_src.algorithms.policies import designer\_policy from vizier.\_src.algorithms.evolution import nsga2 Pythia Policies[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#pythia-policies "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------- The Pythia Service maps algorithm names to `Policy` objects. All algorithms which need to be hosted on the server must eventually be wrapped into a `Policy`. Every `Policy` is injected with a `PolicySupporter`, which is a client used for fetching data from the datastore. This design choice serves two core purposes: 1. The `Policy` is effectively stateless, and thus can be deleted and recovered at any time (e.g. due to a server preemption or failure). 2. Consequently, this avoids needing to save an explicit and potentially complicated algorithm state. Instead, the “algorithm state” can be recovered purely from the entire study containing (`metadata`, `study_config`, `trials`). We show the `Policy` abstract class explicitly below. Exact class entrypoint can be found [here](https://github.com/google/vizier/blob/main/vizier/pythia.py) . class Policy(abc.ABC): """Interface for Pythia Policy subclasses.""" @abc.abstractmethod def suggest(self, request: SuggestRequest) \-> SuggestDecision: """Compute suggestions that Vizier will eventually hand to the user.""" @abc.abstractmethod def early\_stop(self, request: EarlyStopRequest) \-> EarlyStopDecisions: """Decide which Trials Vizier should stop.""" @property def should\_be\_cached(self) \-> bool: """Returns True if it's safe & worthwhile to cache this Policy in RAM.""" return False ### Fundamental Rule of Service Pythia Policies[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#fundamental-rule-of-service-pythia-policies "Link to this heading") For algorithms used in the Pythia Service, the fundamental rule is to assume that a Pythia policy class instance will only call once per user interaction: * `__init__` * `suggest()` and be immediately deleted afterwards. Thus a typical policy will use a `stateless_algorithm` and roughly look like: class TypicalPolicy(Policy): def \_\_init\_\_(self, policy\_supporter: PolicySupporter): self.\_policy\_supporter \= policy\_supporter def suggest(self, request: SuggestRequest) \-> SuggestDecision: all\_completed \= policy\_supporter.GetTrials(status\_matches\=COMPLETED) all\_active \= policy\_supporter.GetTrials(status\_matches\=ACTIVE) suggestions \= stateless\_algorithm(all\_completed, all\_active) return SuggestDecision(suggestions) Example Pythia Policy[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#example-pythia-policy "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- Here, we write a toy policy, where we only act on `CATEGORICAL` parameters for simplicity. The `make_parameters` function will simply for-loop over every category and then cycle back. def make\_parameters( search\_space: vz.SearchSpace, index: int ) \-> vz.ParameterDict: parameter\_dict \= vz.ParameterDict() for parameter\_config in search\_space.parameters: if parameter\_config.type != vz.ParamterType.CATEGORICAL: raise ValueError("This function only supports CATEGORICAL parameters.") feasible\_values \= parameter\_config.feasible\_values parameter\_dict\[parameter\_config.name\] \= vz.ParameterValue( value\=feasible\_values\[index % len(feasible\_values)\] ) return parameter\_dict To collect the `index` from the database, we will use the `PolicySupporter` to obtain the maximum trial ID based on completed and active trials. def get\_next\_index(policy\_supporter: pythia.PolicySupporter): """Returns current trial index.""" completed \= policy\_supporter.GetTrials(status\_matches\=vz.TrialStatus.COMPLETED) active \= policy\_supporter.GetTrials(status\_matches\=vz.TrialStatus.ACTIVE) trial\_ids \= \[t.id for t in completed + active\] if trial\_ids: return max(trial\_ids) return 0 We can now put it all together into our Pythia Policy. class MyPolicy(pythia.Policy): def \_\_init\_\_(self, policy\_supporter: pythia.PolicySupporter): self.\_policy\_supporter \= policy\_supporter def suggest(self, request: pythia.SuggestRequest) \-> pythia.SuggestDecision: """Gets number of Trials to propose, and produces Trials.""" suggest\_decision\_list \= \[\] for \_ in range(request.count): index \= get\_next\_index(self.\_policy\_supporter) parameters \= make\_parameters(request.study\_config.search\_space, index) suggest\_decision\_list.append(vz.TrialSuggestion(parameters\=parameters)) return pythia.SuggestDecision( suggestions\=suggest\_decision\_list, metadata\=vz.MetadataDelta() ) Wrapping Designers as Pythia Policies[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#wrapping-designers-as-pythia-policies "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider if your algorithm code fits in the simpler [Designer](https://oss-vizier.readthedocs.io/en/latest/guides/developer/designers.html) abstraction, which avoids needing to deal with distributed systems logic. For example, the same exact behavior above can be re-written as a `Designer`: class MyDesigner(algorithms.Designer): def \_\_init\_\_(self, study\_config: vz.StudyConfig): self.\_study\_config \= study\_config self.\_completed\_trials \= \[\] self.\_active\_trials \= \[\] def update( self, completed: algorithms.CompletedTrials, all\_active: algorithms.ActiveTrials, ) \-> None: self.\_completed\_trials.extend(completed.trials) self.\_active\_trials \= all\_active.trials def suggest( self, count: Optional\[int\] \= None ) \-> Sequence\[vz.TrialSuggestion\]: if count is None: return \[\] trial\_ids \= \[t.id for t in self.\_completed\_trials + self.\_active\_trials\] current\_index \= max(trial\_ids) return \[\ make\_parameters(self.\_study\_config.search\_space, current\_index + i)\ for i in range(count)\ \] The entire designer (if deleted or preempted) can conveniently be recovered in just a **single** call of `update()` after `__init__`. Thus we may immediately wrap `MyDesigner` into a Pythia Policy with the following Pythia `suggest()` implementation: * Create the designer temporarily. * Update the temporary designer with **all** previously completed trials and active trials. * Obtain suggestions from the temporary designer. This is done conveniently with the `DesignerPolicy` wrapper ([code](https://github.com/google/vizier/blob/main/vizier/_src/algorithms/policies/designer_policy.py) ): class DesignerPolicy(Policy): """Wraps a Designer into a Pythia Policy.""" def \_\_init\_\_(self, supporter: PolicySupporter, designer\_factory: Factory\[Designer\]): self.\_supporter \= supporter self.\_designer\_factory \= designer\_factory def suggest(self, request: SuggestRequest) \-> SuggestDecision: completed \= self.\_supporter.GetTrials(status\_matches\=COMPLETED) active \= self.\_supporter.GetTrials(status\_matches\=ACTIVE) designer \= self.\_designer\_factory(...) designer.update(CompletedTrials(completed), ActiveTrials(active)) return SuggestDecision(designer.suggest(request.count)) Below is the actual act of wrapping: designer\_factory \= lambda study\_config: MyDesigner(study\_config) supporter: PolicySupporter \= ... \# Assume PolicySupporter was created. pythia\_policy \= DesignerPolicy(supporter, designer\_factory) Serializing Designer States[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#serializing-designer-states "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- The above method can gradually become slower as the number of completed trials in the study increases. Thus we may consider storing a compressed representation of the algorithm state instead. Examples include: * The coordinate position in a grid search algorithm. * The population for evolutionary algorithms such as NSGA2. * Directory location for stored neural network weights. As a simple example, consider the case if our designer stores a `_counter` of **all** suggestions it has made: class CounterDesigner(Designer): def \_\_init\_\_(self, ...): ... self.\_counter \= 0 def suggest(self, count: Optional\[int\] \= None) \-> Sequence\[TrialSuggestion\]: ... self.\_counter += len(suggestions) return suggestions Vizier offers [two Designer subclasses](https://github.com/google/vizier/blob/main/vizier/interfaces/serializable.py) , both of which will use the `Metadata` primitive to store algorithm state data: * `SerializableDesigner` will use additional `recover`/`dump` methods and should be used if the entire algorithm state can be easily serialized and can be saved and restored in full. * `PartiallySerializableDesigner` will use additional `load`/`dump` methods and be used if the algorithm has subcomponents that are not easily serializable. State recovery will be handled by calling the Designer’s `__init__` (with same arguments as before) and then `load`. They can also be converted into Pythia Policies using `SerializableDesignerPolicy` and `PartiallySerializableDesignerPolicy` respectively. Below is an example modifying our `CounterDesigner` into `CounterSerialDesigner` and `CounterPartialDesigner` respectively: class CounterSerialDesigner(algorithms.SerializableDesigner): def \_\_init\_\_(self, counter: int): self.\_counter \= counter @classmethod def recover(cls, metadata: vz.Metadata) \-> CounterSerialDesigner: return cls(metadata\['counter'\]) def dump(self) \-> vz.Metadata: metadata \= vz.Metadata() metadata\['counter'\] \= str(self.\_counter) return metadata class CounterPartialDesigner(algorithms.PartiallySerializableDesigner): def load(self, metadata: vz.Metadata) \-> None: self.\_counter \= int(metadata\['counter'\]) def dump(self) \-> vz.Metadata: metadata \= vz.Metadata() metadata\['counter'\] \= str(self.\_counter) return metadata Additional References[](https://oss-vizier.readthedocs.io/en/stable/guides/developer/pythia_policies.html#additional-references "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------------- * Our [policies folder](https://github.com/google/vizier/tree/main/vizier/_src/algorithms/policies) contains examples of Pythia policies. [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) --- # Creating Benchmarks — Open Source Vizier documentation * [](https://oss-vizier.readthedocs.io/en/stable/index.html) * [Guides](https://oss-vizier.readthedocs.io/en/stable/guides/index.html) * Creating Benchmarks * [View page source](https://oss-vizier.readthedocs.io/en/stable/_sources/guides/benchmarks/creating_benchmarks.ipynb.txt) * * * [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/vizier/blob/main/docs/guides/benchmarks/creating_benchmarks.ipynb) Creating Benchmarks[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/creating_benchmarks.html#creating-benchmarks "Link to this heading") ========================================================================================================================================================== We provide a guide below on creating benchmarks, through the use of either: * Standard search space primitives. * Metadata for complex search spaces. Installation and reference imports[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/creating_benchmarks.html#installation-and-reference-imports "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- !pip install google\-vizier import abc import random from typing import Sequence from vizier import pyvizier as vz from vizier.benchmarks import experimenters Experimenters[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/creating_benchmarks.html#experimenters "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------- The core base class of any objective function is the `Experimenter` class, which simply contains a method to evaluate a `Trial` and a `ProblemStatement` to describe its search space and metrics. The exact entry into the class can be found [here](https://github.com/google/vizier/blob/main/vizier/benchmarks/experimenters/__init__.py) . class Experimenter(metaclass\=abc.ABCMeta): """Abstract base class for Experimenters.""" @abc.abstractmethod def evaluate(self, suggestions: Sequence\[vz.Trial\]) \-> None: """Evaluates and mutates the Trials in-place.""" @abc.abstractmethod def problem\_statement(self) \-> vz.ProblemStatement: """The search configuration generated by this experimenter.""" Below is an example of a basic 1D objective function f(x)\=x2. class Basic1DExperimenter(experimenters.Experimenter): def evaluate(self, suggestions: Sequence\[vz.Trial\]) \-> None: for suggestion in suggestions: x \= suggestion.parameters\['x'\].value objective \= x\*\*2 measurement \= pyvizier.Measurement(metrics\={'obj': objective}) suggestion.complete(measurement) def problem\_statement(self) \-> vz.ProblemStatement: problem\_statement \= vz.ProblemStatement() root \= problem\_statement.search\_space.root root.add\_float\_param(name\='x', min\_value\=-1.0, max\_value\=1.0) metric \= vz.MetricInformation(name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) problem\_statement.metric\_information.append(metric) return problem\_statement We may thus evaluate a suggestion. Note that such suggestions are actually `Trial`s, to allow maximum flexibility. basic\_experimenter \= Basic1DExperimenter() trial \= vz.Trial() trial.parameters\['x'\] \= 0.1 basic\_experimenter.evaluate(\[trial\]) assert trial.final\_measurement.metrics\['obj'\].value \== 0.1 \*\* 2 Metadata-based Experimenters[](https://oss-vizier.readthedocs.io/en/stable/guides/benchmarks/creating_benchmarks.html#metadata-based-experimenters "Link to this heading") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Similar to using the `Metadata` primitive to create custom algorithms and complex search spaces, creating custom `Experimenter`s provides the freedom to define custom objective functions. As an example, suppose our search space consisted of unbounded-length sequences consisting of some vocabulary (e.g. the letters ‘A’ to ‘Z’ if considering the space of English words), and we wish to maximize the sequence’s average ASCII value. class VocabularyExperimenter(experimenters.Experimenter): def evaluate(self, suggestions: Sequence\[vz.Trial\]): for suggestion in suggestions: x \= suggestion.metadata\['word'\] objective \= float(sum(\[ord(c) for c in x\])) / len(x) measurement \= vz.Measurement(metrics\={'obj': objective}) suggestion.complete(measurement) def problem\_statement(self) \-> pyvizier.ProblemStatement: problem\_statement \= vz.ProblemStatement() problem\_statement.metadata\['vocab'\] \= 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' metric \= vz.MetricInformation(name\='obj', goal\=vz.ObjectiveMetricGoal.MAXIMIZE) problem\_statement.metric\_information.append(metric) return problem\_statement Below is an example of constructing a valid suggestion and evaluating it. vocab\_experimenter \= VocabularyExperimenter() vocabulary \= vocab\_experimenter.problem\_statement().metadata\['vocab'\] trial \= vz.Trial() trial.metadata\['word'\] \= str( \[random.randint(0, len(vocabulary)) for \_ in range(10)\] ) vocab\_experimenter.evaluate(\[trial\]) print('Average ASCII value is:', trial.final\_measurement.metrics\['obj'\].value) [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/10123/019da328-43b2-7c51-b344-e25ded4b3559/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/topics/data-science/?ref=ea-text) Close Ad ![](https://server.ethicalads.io/proxy/view/10123/019da328-43b2-7c51-b344-e25ded4b3559/) ---