{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "789a785d",
   "metadata": {},
   "source": [
    "# Random Variable Transition Guide"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46afc50c",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86010e09",
   "metadata": {},
   "source": [
    "Prior to SciPy 1.15, all of SciPy's continuous probability distributions (e.g. `scipy.stats.norm`) have been instances of subclasses of `scipy.stats.rv_continuous`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f009703",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "dist = stats.norm\n",
    "type(dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17142625",
   "metadata": {},
   "outputs": [],
   "source": [
    "isinstance(dist, stats.rv_continuous)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df96c31d",
   "metadata": {},
   "source": [
    "There were two obvious ways to work these objects.\n",
    "\n",
    "According to the more common way, both \"arguments\" (e.g. `x`) and \"distribution parameters\" (e.g. `loc`, `scale`) were provided as inputs to methods of the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "addfd816",
   "metadata": {},
   "outputs": [],
   "source": [
    "x, loc, scale = 1., 0., 1.\n",
    "dist.pdf(x, loc, scale)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b748d46",
   "metadata": {},
   "source": [
    "The less common approach was to invoke the `__call__` method of the distribution object, which returned an instance of `rv_continuous_frozen`, regardless of the original class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24dd1690",
   "metadata": {},
   "outputs": [],
   "source": [
    "frozen = stats.norm()\n",
    "type(frozen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f9a8736",
   "metadata": {},
   "source": [
    "Methods of this new object accept only arguments, not the distribution parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc6902c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "frozen.pdf(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4f49de6",
   "metadata": {},
   "source": [
    "In a sense, the instances of `rv_continuous` like `norm` represented \"distribution families\", which require parameters to identify a particular probability distribution, and an instance of `rv_continuous_frozen` was akin to a \"random variable\" - a mathematical object that follows a particular probability distribution.\n",
    "\n",
    "Both approaches are valid and have advantages in certain situations. For instance, `stats.norm.pdf(x)` appears more natural than `stats.norm().pdf(x)` for such a simple invocation or when methods are viewed as functions of distribution parameters rather than the usual arguments (e.g., likelihood). However, the former approach has a few inherent disadvantages; e.g., all of SciPy's 125 continuous distributions have to be instantiated at import time, distribution parameters must be validated every time a method is called, and documentation of methods must either a) be generated separately for every method of every distribution or b) omit the shape parameters that are unique for each distribution.\n",
    "To address these and other shortcomings, [gh-15928](https://github.com/scipy/scipy/issues/15928) proposed a new, separate infrastructure based on the latter (random variable) approach. This transition guide documents how users of `rv_continuous` and `rv_continuous_frozen` can migrate to the new infrastructure.\n",
    "\n",
    "*Note: The new infrastructure may not yet be as convenient for some use cases, especially fitting distribution parameters to data. Users are welcome to continue to use the old infrastructure if it suits their needs; it is not deprecated at this time.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbbdf164",
   "metadata": {},
   "source": [
    "## Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a421791",
   "metadata": {},
   "source": [
    "In the new infrastructure, distributions families are classes named according to `CamelCase` conventions. They must be instantiated before use, with parameters passed as keyword-only arguments.\n",
    "*Instances* of the distribution family classes can be thought of as random variables, which are commonly denoted in mathematics using capital letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b98fedd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "mu, sigma = 0, 1\n",
    "X = stats.Normal(mu=mu, sigma=sigma)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "079eebe6",
   "metadata": {},
   "source": [
    "Once instantiated, shape parameters can be read (but not written) as attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44b6f7b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.mu, X.sigma"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9554fda",
   "metadata": {},
   "source": [
    "The documentation of [`scipy.stats.Normal`](https://scipy.github.io/devdocs/reference/generated/scipy.stats.Normal.html#scipy.stats.Normal) contains links to detailed documentation of each of its methods. (Compare, for instance, against the documentation of [`scipy.stats.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html).)\n",
    "\n",
    "*Note: Although only [`Normal`](https://scipy.github.io/devdocs/reference/generated/scipy.stats.Normal.html) and [`Uniform`](https://scipy.github.io/devdocs/reference/generated/scipy.stats.Uniform.html) have rendered documentation and can be imported directly from `stats` as of SciPy 1.15, almost all other old distributions can be used with the new interface via [`make_distribution`](https://scipy.github.io/devdocs/reference/generated/scipy.stats.make_distribution.html) (discussed shortly).*\n",
    "\n",
    "Methods are then called by passing only the argument(s), if there are any, not distribution parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e75d31",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92106525",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.pdf(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8804196",
   "metadata": {},
   "source": [
    "For simple calls like this (e.g. the argument is a valid float), calls to methods of the new random variables will typically be faster than comparable calls to the old distribution methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d700828f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit X.pdf(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c7b32cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit frozen.pdf(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce34c0e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit dist.pdf(x, loc=mu, scale=sigma)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7f1a07b",
   "metadata": {},
   "source": [
    "Note that the way `X.pdf` and `frozen.pdf` are called is identical, and the call to `dist.pdf` is very similar - the only difference is that the call to `dist.pdf` includes the shape parameters, whereas in the new infrastructure, shape parameters are only provided when the random variable is instantiated.\n",
    "\n",
    "Besides `pdf` and `mean`, several other methods of the new infrastructure are essentially the same as the old methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b76289",
   "metadata": {},
   "source": [
    "- `logpdf` (logarithm of probability density function)\n",
    "- `cdf` (cumulative distribution function)\n",
    "- `logcdf` (logarithm of cumulative distribution function)\n",
    "- `entropy` (differential entropy)\n",
    "- `median`\n",
    "- `support`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669e5414",
   "metadata": {},
   "source": [
    "Others methods have new names, but are otherwise drop-in replacements.\n",
    "- `sf` (survival function) $\\rightarrow$ `ccdf` (complementary cumulative distribution function)\n",
    "- `logsf` $\\rightarrow$ `logccdf`\n",
    "- `ppf` (percent point function) $\\rightarrow$ `icdf` (inverse cumulative distribution function)\n",
    "- `isf` (inverse survive function) $\\rightarrow$ `iccdf` (inverse complementary cumulative distribution function)\n",
    "- `std` $\\rightarrow$ `standard_deviation`\n",
    "- `var` $\\rightarrow$ `variance`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07b8a768",
   "metadata": {},
   "source": [
    "The new infrastructure has several *new* methods in the same vein as those above.\n",
    "\n",
    "- `ilogcdf` (inverse of the logarithm of the cumulative distribution function)\n",
    "- `ilogccdf` (inverse of the logarithm of the complementary cumulative distribution function)\n",
    "- `logentropy` (logarithm of the entropy)\n",
    "- `mode` (mode of the distribution)\n",
    "- `skewness`\n",
    "- `kurtosis` (*non-excess* kurtosis; see \"Standardized Moments\" below)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a787e6",
   "metadata": {},
   "source": [
    "And it has a new `plot` method for convenience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11c4d5a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "X.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f3915ab",
   "metadata": {},
   "source": [
    "Most of the remaining methods of the old infrastructure (`rvs`, `moment`, `stats`, `interval`, `fit`, `nnlf`, `fit_loc_scale`, and `expect`) have analogous functionality, but some care is required. Before describing the replacements, we briefly mention how to work with random variables that are not normally distributed: almost all old distribution objects can be converted into a new distribution class with `scipy.stats.make_distribution`, and the new distribution class can be instantiated by passing the shape parameters as keyword arguments. For instance, consider the [Weibull distribution](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.weibull_min.html#scipy.stats.weibull_min). We can create a new class that is an abstraction of the distribution family like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f1353c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "Weibull = stats.make_distribution(stats.weibull_min)\n",
    "print(Weibull.__doc__[:288])  # help(Weibull) prints this and (too much) more"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "698aac70",
   "metadata": {},
   "source": [
    "According to the documentation of [`weibull_min`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.weibull_min.html#scipy.stats.weibull_min), the shape parameter is denoted `c`, so we can instantiate a random variable by passing `c` as a keyword argument to the new `Weibull` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5998abb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "c = 2.\n",
    "X = Weibull(c=c)\n",
    "X.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc1b5ec",
   "metadata": {},
   "source": [
    "Previously, all distributions inherited `loc` and `scale` distribution parameters.\n",
    "*These are no longer accepted.* Instead, random variables can be shifted and scaled with arithmetic operators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d4f47fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = 2*X + 1\n",
    "Y.plot()  # note the change in the abscissae\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4760cfa7",
   "metadata": {},
   "source": [
    "A separate distribution, `weibull_max`, was provided as the reflection of `weibull_min` about the origin. Now, this is just `-X`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e0b081b",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = -X\n",
    "Y.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1420b31",
   "metadata": {},
   "source": [
    "### Moments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc581967",
   "metadata": {},
   "source": [
    "The old infrastructure offers a `moments` method for raw moments. When only the order is specified, the new `moment` method is a drop-in replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90214a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.moment(1, c), X.moment(1)  # first raw moment of the Weibull distribution with shape c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac700ac4",
   "metadata": {},
   "source": [
    "However, the old infrastructure also has a `stats` method, which provided various statistics of the distribution. The following statistics were associated with the indicated characters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "540f2c6a",
   "metadata": {},
   "source": [
    "- Mean (first raw moment about the origin, `'m'`)\n",
    "- Variance (second central moment, `'v'`)\n",
    "- Skewness (third standarized moment, `'s'`)\n",
    "- Excess kurtosis (fourh standarized moment minus 3, `'k'`)\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0776a49a",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.stats(c, moments='mvsk')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dc5ff40",
   "metadata": {},
   "source": [
    "Now, moments of any `order` and `kind` (raw, central, and standardized) can be computed by passing the appropriate arguments to the new `moment` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e2c42a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(X.moment(order=1, kind='raw'), \n",
    " X.moment(order=2, kind='central'), \n",
    " X.moment(order=3, kind='standardized'), \n",
    " X.moment(order=4, kind='standardized'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23c57a36",
   "metadata": {},
   "source": [
    "Note the difference in definition of [kurtosis](https://en.wikipedia.org/wiki/Kurtosis). Previously, the \"excess\" (AKA \"Fisher's\") kurtosis was provided. As a matter of convention (rather than the natural mathematical definition), this is the standardized fourth moment shifted by a constant value (`3`) to give a value of `0.0` for the normal distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba2413db",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.norm.stats(moments='k')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f608d016",
   "metadata": {},
   "source": [
    "The new `moment` and `kurtosis` methods do not observe this convention by default; they report the standardized fourth moment according to the mathematical definition. This is also known as the \"non-excess\" (or \"Pearson's\") kurtosis, and for the standard normal, it has a value of `3.0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "989dcdcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.Normal().moment(4, kind='standardized') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68261b26",
   "metadata": {},
   "source": [
    "For convenience, the `kurtosis` method offers a `convention` argument to select between the two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d81af86",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.Normal().kurtosis(convention='excess'), stats.Normal().kurtosis(convention='non-excess')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0050a9af",
   "metadata": {},
   "source": [
    "### Random Variates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1164e3ab",
   "metadata": {},
   "source": [
    "The old infrastructure's `rvs` method has been replaced with `sample`, but there are a two differences that should be noted when either 1) control of the random state is required or 2) arrays are used for shape, location, or scale parameters.\n",
    "\n",
    "First, argument `random_state` is replaced by `rng` per [SPEC 7](https://scientific-python.org/specs/spec-0007/). A pattern to control the random state in the past has been to use `numpy.random.seed` or to pass integer seeds to the `random_state` argument. The integer seeds were converted to instances of `numpy.random.RandomState`, so behavior for a given integer seed would be identical in these two cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77f97540",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(1)\n",
    "dist = stats.norm\n",
    "dist.rvs(), dist.rvs(random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315b162b",
   "metadata": {},
   "source": [
    "In the new infrastructure, pass instances of `numpy.Generator` to the `rng` argument of `sample` to control the random state. Note that integer seeds are now converted to instances of `numpy.random.Generator`, not instances of `numpy.random.RandomState`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de5a551a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stats.Normal()\n",
    "rng = np.random.default_rng(1)  # instantiate a numpy.random.Generator\n",
    "X.sample(rng=rng), X.sample(rng=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1611a48e",
   "metadata": {},
   "source": [
    "Second, the argument `shape` replaces argument `size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b8a9689",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist.rvs(size=(3, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e70fc46",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.sample(shape=(3, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c363934b",
   "metadata": {},
   "source": [
    "Besides the difference in name, there is a difference in behavior when array shape parameters are involved. Previously, the shape of distribution parameter arrays had to be included in `size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc61e609",
   "metadata": {},
   "outputs": [],
   "source": [
    "dist.rvs(size=(3, 4, 2), loc=[0, 1]).shape  # `loc` has shape (2,)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10231fc3",
   "metadata": {},
   "source": [
    "Now, the shape of the parameter arrays is considered to be a property of the random variable object itself. Specifying the shape of array shape parameters would be redundant, so it is not included when specifying the `shape` of the sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a08a5b02",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = stats.Normal(mu = [0, 1])\n",
    "Y.sample(shape=(3, 4)).shape  # the sample has shape (3, 4); each element is of shape (2,)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "605a164e",
   "metadata": {},
   "source": [
    "### Probability Mass (AKA \"Confidence\") Intervals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb6102a0",
   "metadata": {},
   "source": [
    "The old infrastructure offered an `interval` method which, by default, would return a symmetric (about the median) interval containing a specified percentage of the distribution's probability mass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a2ff42",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 0.95\n",
    "dist.interval(confidence=a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff20d096",
   "metadata": {},
   "source": [
    "Now, call the inverse CDF and complementary CDF methods with the desired probability mass in each tail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d682a3b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = 1 - a\n",
    "X.icdf(p/2), X.iccdf(p/2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87940265",
   "metadata": {},
   "source": [
    "#### Likelihood Function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a2c94b",
   "metadata": {},
   "source": [
    "The old infrastructure offers a function to compute the **n**egative **l**og-[**l**ikelihood **f**unction](https://en.wikipedia.org/wiki/Likelihood_function), erroneously named `nnlf` (instead of `nllf`). It accepts the parameters of the distribution as a tuple and observations as an array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c99b7b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "mu = 0\n",
    "sigma = 1\n",
    "data = stats.norm.rvs(size=100, loc=mu, scale=sigma)\n",
    "stats.norm.nnlf((mu, sigma), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179d6989",
   "metadata": {},
   "source": [
    "Now, simply compute the negative log-likehood according to its mathematical definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3936ba71",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stats.Normal(mu=mu, sigma=sigma)\n",
    "-X.logpdf(data).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6902290d",
   "metadata": {},
   "source": [
    "### Expected Values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74f857f",
   "metadata": {},
   "source": [
    "The `expect` method of the old infrastructure estimates a definite integral of an arbitrary function weighted by the PDF of the distribution. For instance, the fourth moment of the distribution is given by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b9a1d48",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x): return x**4\n",
    "stats.norm.expect(f, lb=-np.inf, ub=np.inf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "442f2ea9",
   "metadata": {},
   "source": [
    "This provides little added convencience over what the source code does: use `scipy.integrate.quad` to perform the integration numerically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cd00b7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import integrate\n",
    "def f(x): return x**4 * stats.norm.pdf(x)\n",
    "integrate.quad(f, a=-np.inf, b=np.inf)  # integral estimate, estimate of the error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10d0cb72",
   "metadata": {},
   "source": [
    "Of course, this can just as easily be done with the new infrastructure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82f389d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x): return x**4 * X.pdf(x)\n",
    "integrate.quad(f, a=-np.inf, b=np.inf)  # integral estimate, estimate of the error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db4dc43",
   "metadata": {},
   "source": [
    "The `conditional` argument simply scales the result by inverse of the probability mass contained within the interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d96a686",
   "metadata": {},
   "outputs": [],
   "source": [
    "a, b = -1, 3\n",
    "def f(x): return x**4\n",
    "stats.norm.expect(f, lb=a, ub=b, conditional=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fcc8cff",
   "metadata": {},
   "source": [
    "Using `quad` directly, that is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5badabaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x): return x**4 * stats.norm.pdf(x)\n",
    "prob = stats.norm.cdf(b) - stats.norm.cdf(a)\n",
    "integrate.quad(f, a=a, b=b)[0] / prob"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccfef5a8",
   "metadata": {},
   "source": [
    "And with the new random variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "715e4342",
   "metadata": {},
   "outputs": [],
   "source": [
    "integrate.quad(f, a=a, b=b)[0] / X.cdf(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee47ec7",
   "metadata": {},
   "source": [
    "Note that the `cdf` method of the new random variables accepts two arguments to compute the probability mass within the specified interval. In many cases, this can avoid subtractive error (\"catastrophic cancellation\") when the probability mass is very small."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc53129",
   "metadata": {},
   "source": [
    "### Fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb09e164",
   "metadata": {},
   "source": [
    "#### Location / Scale Estimation\n",
    "The old infrastructure offered a function to estimate location and scale parameters of the distribution from data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54c67320",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(91392794601852341588152500276639671056)\n",
    "dist = stats.weibull_min\n",
    "c, loc, scale = 0.5, 0, 4.\n",
    "data = dist.rvs(size=100, c=c, loc=loc, scale=scale, random_state=rng)\n",
    "dist.fit_loc_scale(data, c)\n",
    "# compare against 0 (default location) and 4 (specified scale)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532fb65b",
   "metadata": {},
   "source": [
    "Based on the source code, it is easy to replicate the method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b68570da",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = Weibull(c=c)\n",
    "def fit_loc_scale(X, data):\n",
    "    m, v = X.mean(), X.variance()\n",
    "    m_, v_ = data.mean(), data.var()\n",
    "    scale = np.sqrt(v_ / v)\n",
    "    loc = m_ - scale*m\n",
    "    return loc, scale\n",
    "\n",
    "fit_loc_scale(X, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7df18b9",
   "metadata": {},
   "source": [
    "Note that the estimates are quite poor in this example, and poor performance of the heuristic factored into the decision not to provide a replacement.\n",
    "\n",
    "#### Maximum Likelihood Estimation\n",
    "\n",
    "The last method of the old infrastructure is `fit`, which estimates the location, scale, and shape parameters of an underlying distribution given a sample from the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbb3623f",
   "metadata": {},
   "outputs": [],
   "source": [
    "c_, loc_, scale_ = stats.weibull_min.fit(data)\n",
    "c_, loc_, scale_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ff548d",
   "metadata": {},
   "source": [
    "The convenience of this method is a blessing and a curse. \n",
    "\n",
    "When it works, the simplicity is much appreciated. For some distributions, analytical expressions for the [maximum likelihood estimates](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) of the parameters have been programmed manually, and for these distributions, the `fit` method is quite reliable. \n",
    "\n",
    "For most distributions, however, fitting is performed by numerical minimization of the negative log-likelihood function. This is not guaranteed to be successful - both because of inherent limitations of MLE and the state of the art of numerical optimization - and in these cases, the user of `fit` is stuck. However, a modicum of understanding goes a long way toward ensuring success, so here we present a mini-tutorial of fitting using the new infrastructure. \n",
    "\n",
    "First, note that MLE is one of several potential strategies for fitting distributions to data. It does nothing more than find the values of the distribution parameters for which the negative log-likelihood (NLLF) is minimized. Note that for the distribution from which the data were generated, the NLLF is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3322d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.nnlf((c, loc, scale), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245ababd",
   "metadata": {},
   "source": [
    "The NLLF of the parameters estimated using `fit` are lower:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48099cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.nnlf((c_, loc_, scale_), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bead3c1",
   "metadata": {},
   "source": [
    "Therefore, `fit` considers its job done, whether or not this satisfies the user's notions of a \"good fit\" or whether the parameters are within reasonable bounds. Besides allowing the user to specify a guess for distribution parameters, it gives little control over the process. (*Note: technically, the `optimizer` argument can be used for greater control, but this is not well documented, and its use negates any convenience offered by the `fit` method.*)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2a9a400",
   "metadata": {},
   "source": [
    "Another problem is that MLE is inherently poorly behaved for some distributions. In this example, for instance, we can make the NLLF as low as we please simply by setting the location to the minimum of the data - even for bogus values of the shape and scale parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f068a61",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.nnlf((0.1, np.min(data), 10), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7131cc73",
   "metadata": {},
   "source": [
    "Compounding with these issues is that `fit` estimates all distribution parameters by default, including the location. In the example above, we are probably only interested in the more common two-parameter Weibull distribution because the actual location is zero. `fit` *can* accommodate this by specifing that the location is fixed parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "129ad467",
   "metadata": {},
   "outputs": [],
   "source": [
    "# c_, loc_, scale_ = stats.weibull_min.fit(data, loc=0)  # careful! this provides loc=0 as a *guess*\n",
    "c_, loc_, scale_ = stats.weibull_min.fit(data, floc=0)\n",
    "c_, loc_, scale_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ae973a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.weibull_min.nnlf((c_, loc_, scale_), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a2e027d",
   "metadata": {},
   "source": [
    "But by providing a `fit` method that purportedly \"just works\", users are encouraged to neglect some of these important details.\n",
    "\n",
    "Instead, we suggest that it is almost as simple to fit distributions to data using the generic optimization tools provided by SciPy. However, this allows the user to perform the fit according to their preferred objective function, provides options when things don't work as expected, and helps the user to avoid the most common pitfalls.\n",
    "\n",
    "For instance, suppose our desired objective function is indeed the negative log likelihood. It is simple to define as a function of the shape parameter `c` and scale only, leaving the location at its default of zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebf72fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def nllf(x):\n",
    "    c, scale = x  # pass (only) `c` and `scale` as elements of `x`\n",
    "    X = Weibull(c=c) * scale\n",
    "    return -X.logpdf(data).sum()\n",
    "\n",
    "nllf((c_, scale_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c9fd67",
   "metadata": {},
   "source": [
    "To perform the minimization, we can use `scipy.optimize.minimize`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03e1e0fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import optimize\n",
    "eps = 1e-10  # numerical tolerance to avoid invalid parameters\n",
    "lb = [eps, eps]  # lower bound on `c` and `scale`\n",
    "ub = [10, 10]  # upper bound on `c` and `scale`\n",
    "x0 = [1, 1]  # guess to get optimization started\n",
    "bounds = optimize.Bounds(lb, ub)\n",
    "res_mle = optimize.minimize(nllf, x0, bounds=bounds)\n",
    "res_mle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e95b5c",
   "metadata": {},
   "source": [
    "The value of the objective function is essentially identical, and the PDFs are indistinguishable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b0a5031",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x = np.linspace(eps, 15, 300)\n",
    "\n",
    "c_, scale_ = res_mle.x\n",
    "X = Weibull(c=c_)*scale_\n",
    "plt.plot(x, X.pdf(x), '-', label='numerical optimization')\n",
    "\n",
    "c_, _, scale_ = stats.weibull_min.fit(data, floc=0)\n",
    "Y = Weibull(c=c_)*scale_\n",
    "plt.plot(x, Y.pdf(x), '--', label='`weibull_min.fit`')\n",
    "\n",
    "plt.hist(data, bins=np.linspace(0, 20, 30), density=True, alpha=0.1)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('pdf(x)')\n",
    "plt.legend()\n",
    "plt.ylim(0, 0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "036d31cb",
   "metadata": {},
   "source": [
    "However, with this approach, it is trivial to make changes to the fitting procedure to suit our needs, enabling estimation procedures other than those provided by [`rv_continuous.fit`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_continuous.fit.html#scipy.stats.rv_continuous.fit). For instance, we can perform [maximum spacing estimation](https://en.wikipedia.org/wiki/Maximum_spacing_estimation) (MSE) by changing the objective function to the **n**egative **l**og **p**roduct **s**pacing as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef555a81",
   "metadata": {},
   "outputs": [],
   "source": [
    "def nlps(x):\n",
    "    c, scale = x\n",
    "    X = Weibull(c=c) * scale\n",
    "    x = np.sort(np.concatenate((data, X.support())))  # Append the endpoints of the support to the data\n",
    "    return -X.logcdf(x[:-1], x[1:]).sum().real  # Minimize the sum of the logs the probability mass between points\n",
    "\n",
    "res_mps = optimize.minimize(nlps, x0, bounds=bounds)\n",
    "res_mps.x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a50bd61a",
   "metadata": {},
   "source": [
    "For method of [L-moments](https://en.wikipedia.org/wiki/L-moment) (which attempts to match the distribution and sample L-moments):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6ade819",
   "metadata": {},
   "outputs": [],
   "source": [
    "def lmoment_residual(x):\n",
    "    c, scale = x\n",
    "    X = Weibull(c=c) * scale\n",
    "    E11 = stats.order_statistic(X, r=1, n=1).mean()\n",
    "    E12, E22 = stats.order_statistic(X, r=[1, 2], n=2).mean()\n",
    "    lmoments_X = [E11, 0.5*(E22 - E12)]  # the first two l-moments of the distribution\n",
    "    lmoments_x = stats.lmoment(data, order=[1, 2])  # first two l-moments of the data\n",
    "    return np.linalg.norm(lmoments_x - lmoments_X)  # Minimize the norm of the difference\n",
    "\n",
    "x0 = [0.4, 3]  # This method is a bit sensitive to the initial guess\n",
    "res_lmom = optimize.minimize(lmoment_residual, x0, bounds=bounds)\n",
    "res_lmom.x, res_lmom.fun  # residual should be ~0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10c8e21c",
   "metadata": {},
   "source": [
    "The plots for all three methods look similar and seem to describe the data, giving us some confidence that we have a good fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59abd63c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x = np.linspace(eps, 10, 300)\n",
    "\n",
    "for res, label in [(res_mle, \"MLE\"), (res_mps, \"MPS\"), (res_lmom, \"L-moments\")]:\n",
    "    c_, scale_ = res.x\n",
    "    X = Weibull(c=c_)*scale_\n",
    "    plt.plot(x, X.pdf(x), '-', label=label)\n",
    "\n",
    "plt.hist(data, bins=np.linspace(0, 10, 30), density=True, alpha=0.1)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('pdf(x)')\n",
    "plt.legend()\n",
    "plt.ylim(0, 0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79088a67",
   "metadata": {},
   "source": [
    "Estimation does not always involve fitting distributions to data. For instance, in [gh-12134](https://github.com/scipy/scipy/issues/12134), a user needed to calculate the shape and scale parameters of the Weibull distribution to achieve a given mean and standard deviation. This fits easily into the same framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47ae3445",
   "metadata": {},
   "outputs": [],
   "source": [
    "moments_0 = np.asarray([8, 20])  # desired mean and standard deviation\n",
    "def f(x):\n",
    "    c, scale = x\n",
    "    X = Weibull(c=c) * scale\n",
    "    moments_X = X.mean(), X.standard_deviation()\n",
    "    return np.linalg.norm(moments_X - moments_0)\n",
    "\n",
    "bounds.lb = np.asarray([0.1, 0.1])  # the Weibull distribution is problematic for very small c\n",
    "res = optimize.minimize(f, x0, bounds=bounds, method='slsqp')  # easily change the minimization method\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7584e270",
   "metadata": {},
   "source": [
    "## New and Advanced Features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2417d611",
   "metadata": {},
   "source": [
    "<a id='Transformations'></a>\n",
    "### Transformations\n",
    "\n",
    "Mathematical transformations can be applied to random variables. For instance, many elementary arithmetic operations (`+`, `-`, `*`, `/`, `**`) between random variables and scalars work.\n",
    "\n",
    "For instance, we can see that the reciprocal of a Weibull-distributed RV is distributed according to `scipy.stats.invweibull`. (An \"inverse\" distribution is typically the distribution underlying the reciprocal of a random variable.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8c0964",
   "metadata": {},
   "outputs": [],
   "source": [
    "c = 10.6\n",
    "\n",
    "X = Weibull(c=10.6)  \n",
    "Y = 1 / X  # compare to `invweibull`\n",
    "Y.plot()\n",
    "\n",
    "x = np.linspace(0.8, 2.05, 300)\n",
    "plt.plot(x, stats.invweibull(c=c).pdf(x), '--')\n",
    "plt.legend(['`1 / X`', '`invweibull`'])\n",
    "plt.title(\"Inverse Weibull PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "816456fe",
   "metadata": {},
   "source": [
    "`scipy.stats.chi2` describes a sum of the squares of normally-distributed random variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20f7278f",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stats.Normal()\n",
    "Y = X**2  # compare to chi2\n",
    "Y.plot(t=('x', eps, 5));\n",
    "\n",
    "x = np.linspace(eps, 5, 300)\n",
    "plt.plot(x, stats.chi2(df=1).pdf(x), '--')\n",
    "plt.ylim(0, 3)\n",
    "plt.legend(['`X**2`', '`chi2`'])\n",
    "plt.title(\"Chi-squared PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85381e85",
   "metadata": {},
   "source": [
    "`scipy.stats.foldcauchy` describes the absolute value of a Cauchy-distributed random variable. (A \"folded\" distribution is the distribution underlying the absolute value of a random variable.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d5d1ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Cauchy = stats.make_distribution(stats.cauchy)\n",
    "c = 4.72\n",
    "\n",
    "X = Cauchy() + c  \n",
    "Y = abs(X)  # compare to `foldcauchy`\n",
    "Y.plot(t=('x', 0, 60))\n",
    "\n",
    "x = np.linspace(0, 60, 300)\n",
    "plt.plot(x, stats.foldcauchy(c=c).pdf(x), '--')\n",
    "plt.legend(['`abs(X)`', '`foldcauchy`'])\n",
    "plt.title(\"Folded Cauchy PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ad17b7",
   "metadata": {},
   "source": [
    "`scipy.stats.lognormal` describes the exponential of a normally distributed random variable. It is so-named because the logarithm of the resulting random variable is normally distributed. (In general, a \"log\" distribution is typically the distribution underlying the *exponential* of a random variable.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15486739",
   "metadata": {},
   "outputs": [],
   "source": [
    "u, s = 1, 0.5\n",
    "\n",
    "X = stats.Normal()*s + u\n",
    "Y = stats.exp(X)  # compare to `lognorm`\n",
    "Y.plot(t=('x', eps, 9))\n",
    "\n",
    "x = np.linspace(eps, 9, 300)\n",
    "plt.plot(x, stats.lognorm(s=s, scale=np.exp(u)).pdf(x), '--')\n",
    "plt.legend(['`exp(X)`', '`lognorm`'])\n",
    "plt.title(\"Log-Normal PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c009f2d2",
   "metadata": {},
   "source": [
    "`scipy.stats.loggamma` describes the logarithm of of a Gamma-distributed random variable. Note that according to typical conventions, the appropriate name of this distribution would be [exp-gamma](https://reference.wolfram.com/language/ref/ExpGammaDistribution.html.en#:~:text=The%20term%20exp%2Dgamma%20is,of%20qualitatively%20similar%20probability%20distributions.) because the exponential of the RV is gamma-distributed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d9e5119",
   "metadata": {},
   "outputs": [],
   "source": [
    "Gamma = stats.make_distribution(stats.gamma)\n",
    "a = 0.414\n",
    "\n",
    "X = Gamma(a=a)  \n",
    "Y = stats.log(X)  # compare to `loggamma`\n",
    "Y.plot()\n",
    "\n",
    "x = np.linspace(-17.5, 2, 300)\n",
    "plt.plot(x, stats.loggamma(c=a).pdf(x), '--')\n",
    "plt.legend(['`log(X)`', '`loggamma`'])\n",
    "plt.title(\"Exp-Gamma PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ed08d2",
   "metadata": {},
   "source": [
    "`scipy.stats.truncnorm` is the distribution underlying a truncated normal random variable. Note that the truncation transformation can be applied either before or after shifting and scaling the normally-distributed random variable, which can make it much easier to achieve the desired result than `scipy.stats.truncnorm` (which inherently shifts and scales *after* truncation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b21f520",
   "metadata": {},
   "outputs": [],
   "source": [
    "a, b = 0.1, 2\n",
    "\n",
    "X = stats.Normal()  \n",
    "Y = stats.truncate(X, a, b)  # compare to `truncnorm`\n",
    "Y.plot()\n",
    "\n",
    "x = np.linspace(a, b, 300)\n",
    "plt.plot(x, stats.truncnorm(a, b).pdf(x), '--')\n",
    "plt.legend(['`truncate(X, a, b)`', '`truncnorm`'])\n",
    "plt.title(\"Truncated Normal PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c79a4795",
   "metadata": {},
   "source": [
    "`scipy.stats.dgamma` is a mixture of two gamma-distributed RVs, one of which is reflected about the origin. (Typically, a \"double\" distribution is the distribution underlying a mixture of RVs, one of which is reflected.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a93b3da",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 1.1\n",
    "X = Gamma(a=a)\n",
    "Y = stats.Mixture((X, -X), weights=[0.5, 0.5])\n",
    "# plot method not available for mixtures\n",
    "\n",
    "x = np.linspace(-4, 4, 300)\n",
    "plt.plot(x, Y.pdf(x))\n",
    "plt.plot(x, stats.dgamma(a=a).pdf(x), '--')\n",
    "plt.legend(['`Mixture(X, -X)`', '`dgamma`'])\n",
    "plt.title(\"Double Gammma PDF\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7202aa3a",
   "metadata": {},
   "source": [
    "#### Limitations\n",
    "\n",
    "While most arithmetic transformations between random variables and Python scalars or NumPy arrays are supported, there are a few restrictions.\n",
    "\n",
    "- Raising a random variable to the power of an argument is only implemented when the argument is a positive integer.\n",
    "- Raising an argument to the power of a random variable is only implemented when the argument is a positive scalar other than 1.\n",
    "- Division by a random variable is only implemented when the support is either entirely non-negative or non-positive.\n",
    "- The logarithm of a random variable is only implemented when the support is non-negative. (The logarithm of negative values is imaginary.)\n",
    "\n",
    "Arithmetic operations between two random variables are *not* yet supported. Note that such operations are much more complex mathematically; for instance, the PDF of the sum of the two random variables involves convolution of the two PDFs.\n",
    "\n",
    "Also, while transformations are composable, a) truncation and b) shifting/scaling can each be done only once. For instance, `Y = 2 * stats.exp(X + 1)` will raise an error because this would require shifting before exponentiation and scaling after exponentiation; this is treated as \"shifting/scaling\" twice. However, a mathematical simplification is possible here (and in many cases) to avoid the problem: `Y = (2*math.exp(2)) * stats.exp(X)` is equivalent and requires only one scaling operation.\n",
    "\n",
    "Although the transformations are fairly robust, they all rely on generic implementations which may cause numerical difficulties. If you are concerned about the accuracy of the results of a transformation, consider comparing the resulting PDF against a histogram of a random sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c0fe914",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stats.Normal()\n",
    "Y = X**3\n",
    "x = np.linspace(-5, 5, 300)\n",
    "plt.plot(x, Y.pdf(x), label='pdf')\n",
    "plt.hist(X.sample(100000)**3, density=True, bins=np.linspace(-5, 5, 100), alpha=0.5);\n",
    "plt.ylim(0, 2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "708d4c92",
   "metadata": {},
   "source": [
    "### Quasi-Monte Carlo Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb49380f",
   "metadata": {},
   "source": [
    "Random variables enable generation of quasi-random, low-discrepancy samples from statistical distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60f30e98",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import stats\n",
    "X = stats.Normal()\n",
    "\n",
    "rng = np.random.default_rng(7824387278234)\n",
    "qrng = stats.qmc.Sobol(1, rng=rng)  # instantiate a QMCEngine\n",
    "\n",
    "bins = np.linspace(-3.5, 3.5, 31)\n",
    "plt.hist(X.sample(512, rng=qrng), bins, alpha=0.5, label='quasi-random')\n",
    "plt.hist(X.sample(512, rng=rng), bins, alpha=0.5, label='pseudo-random')\n",
    "plt.title('Histogram of normally-distributed sample')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4b0f12",
   "metadata": {},
   "source": [
    "Note that when generating multiple samples (e.g. `len(shape) > 1`),  each slice along the zeroth axis is an independent, low-discrepancy sequency. The following generates two independent QMC samples, each of length 512."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcfaf98d",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = X.sample((512, 2), rng=qrng)\n",
    "plt.hist(samples[:, 0], bins, alpha=0.5, label='sample 0')\n",
    "plt.hist(samples[:, 1], bins, alpha=0.5, label='sample 1')\n",
    "plt.title('Histograms of normally-distributed samples')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b1a595a",
   "metadata": {},
   "source": [
    "The result is quite different if we generate 512 independent QMC sequences, each of length 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98e6d8e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = X.sample((2, 512), rng=qrng)\n",
    "plt.hist(samples[0], bins, alpha=0.5, label='sample 0')\n",
    "plt.hist(samples[1], bins, alpha=0.5, label='sample 1')\n",
    "plt.title('Histograms of normally-distributed samples')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00325e00",
   "metadata": {},
   "source": [
    "### Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d2a8f7a",
   "metadata": {},
   "source": [
    "For some distributions, like `scipy.stats.norm`, almost all methods of the distribution are customized to ensure accurate computation. For others, like `scipy.stats.gausshyper`, little more than the PDF is defined, and the other methods must be computed numerically based on the PDF. For example, the survival function (complementary CDF) of the Gauss hypergeometric distribution is calculated by numerically integrating the PDF from `0` (the left end of the support) to `x` to get the CDF, then the complement is taken."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f25471b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "a, b, c, z = 1.5, 2.5, 2, 0\n",
    "x = 0.5\n",
    "frozen = stats.gausshyper(a=a, b=b, c=c, z=z)\n",
    "frozen.sf(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2237ec00",
   "metadata": {},
   "outputs": [],
   "source": [
    "frozen.sf(x) == 1 - integrate.quad(frozen.pdf, 0, x)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34c810b1",
   "metadata": {},
   "source": [
    "However, another aproach would be to numerically integrate the PDF from `x` to `1` (the right end of the support)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a22deb9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "integrate.quad(frozen.pdf, x, 1)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5fc8acf",
   "metadata": {},
   "source": [
    "These are distinct but equally valid approaches, so assuming the PDF is accurate, it is unlikely that the two results are inaccurate in the same way. Therefore, we can estimate the accuracy of the results by comparing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b341aed",
   "metadata": {},
   "outputs": [],
   "source": [
    "res1 = frozen.sf(x)\n",
    "res2 = integrate.quad(frozen.pdf, x, 1)[0]\n",
    "abs((res1 - res2) / res1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "775188a1",
   "metadata": {},
   "source": [
    "The new infrastructure is aware of several different ways for computing most quantities. For example, this diagram illustrates the relationships between various distribution functions.\n",
    "\n",
    "![image.png](rv_infrastructure_1.png)\n",
    "\n",
    "It follows a decision tree to choose what is expected to be the most accurate way of estimating the quantity. These specific decision trees for the \"best\" method are subject to change between versions of SciPy, but an example for computing the complementary CDF might look something like:\n",
    "\n",
    "![image.png](rv_infrastructure_2.png)\n",
    "\n",
    "and for calculating a moment:\n",
    "\n",
    "![image.png](rv_infrastructure_3.png)\n",
    "\n",
    "However, you can override the method it uses with the `method` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef2009e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "GaussHyper = stats.make_distribution(stats.gausshyper)\n",
    "X = GaussHyper(a=a, b=b, c=c, z=z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5486fc77",
   "metadata": {},
   "source": [
    "For `ccdf`, `method='quadrature'` integrates the PDF in right tail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b44b29",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.ccdf(x, method='quadrature') == integrate.tanhsinh(X.pdf, x, 1).integral"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b95ee4f",
   "metadata": {},
   "source": [
    "`method='complement'` takes the complement of the CDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99bfb60e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.ccdf(x, method='complement') == (1 - integrate.tanhsinh(X.pdf, 0, x).integral)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cb2e5e8",
   "metadata": {},
   "source": [
    "`method='logexp'` takes the exponential of the logarithm of the CCDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e43a6b32",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.ccdf(x, method='logexp') == np.exp(integrate.tanhsinh(lambda x: X.logpdf(x), x, 1, log=True).integral)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad6942ee",
   "metadata": {},
   "source": [
    "When in doubt, consider trying different `method`s to compute a given quantity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ba39f82",
   "metadata": {},
   "source": [
    "### Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "723e7cb0",
   "metadata": {},
   "source": [
    "Consider the performance of calculations involving the Gauss hypergeometric distribution. For scalar shapes and argument, performance of the new and old infrastructures are comparable. The new infrastructure is not expected to be much faster; although it reduces the \"overhead\" of parameter validation, it is not significantly faster at numerical integration of scalar quantities, and the latter dominates the execution time here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9175841b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit X.cdf(x)  # new infrastructure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "513a0e11",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit stats.gausshyper.cdf(x, a, b, c, z)  # old infrastructure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc052882",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.isclose(X.cdf(x), stats.gausshyper.cdf(x, a, b, c, z))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9e376da",
   "metadata": {},
   "source": [
    "But the new infrastructure is much faster when arrays are involved. This is because the underlying integrator ([`scipy.integrate.tanhsinh`](https://scipy.github.io/devdocs/reference/generated/scipy.integrate.tanhsinh.html#scipy.integrate.tanhsinh)) and root finder ([`scipy.optimize.elementwise.find_root`](https://scipy.github.io/devdocs/reference/generated/scipy.optimize.elementwise.find_root.html)) developed for the new infrastructure are natively vectorized, whereas the legacy routines used by the old infrastructure (`scipy.integrate.quad` and `scipy.optimize.brentq`) are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "345ebf25",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 1, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0f1cdf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit X.cdf(x)  # new infrastructure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1255e690",
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit stats.gausshyper.cdf(x, a, b, c, z)  # old infrastructure"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11880416",
   "metadata": {},
   "source": [
    "Comparable performance improvements can be expected for array calculations whenever the underlying functions are computed by either numerical quadrature or inversion (root-finding)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a239db6",
   "metadata": {},
   "source": [
    "### Visualization\n",
    "\n",
    "We can readily visualize functions of the distribution underlying random variables using the convenience method, `plot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "306ffcc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "ax = X.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4489b0c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.plot(y='cdf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abbe8e90",
   "metadata": {},
   "source": [
    "The `plot` method is quite flexible, with a signature inspired by [The Grammar of Graphics](https://books.google.com/books?id=_kRX4LoFfGQC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false). For instance, with the argument $x$ on $[-10, 10]$, plot the PDF against the CDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "713d8c13",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.plot('cdf', 'pdf', t=('x', -10, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49658bb2",
   "metadata": {},
   "source": [
    "### Order Statistics\n",
    "As seen in the \"Fit\" section above, there is now support for distributions of [order statistics](https://en.wikipedia.org/wiki/Order_statistic) of random samples from distribution. For example, we can plot the probability density functions of the order statistics of a normal distribution with sample size 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ad9dfaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 4\n",
    "r = np.arange(1, n+1)\n",
    "X = stats.Normal()\n",
    "Y = stats.order_statistic(X, r=r, n=n)\n",
    "Y.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "674f440d",
   "metadata": {},
   "source": [
    "Compute the expected values of these order statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c92184e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323170a7",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7403e8",
   "metadata": {},
   "source": [
    "Although change can be uncomfortable, the new infrastructure paves the road to an improved experience for users of SciPy's probability distributions. \n",
    "It is not complete, of course! At the time this guide was written, the following features are planned for future releases:\n",
    "- Additional built-in distributions (besides `Normal` and `Uniform`)\n",
    "- Discrete distributions\n",
    "- Circular distributions, including the ability to wrap arbitrary continuous distributions around the circle\n",
    "- Support for other Python Array API Compatible backends beyond NumPy (e.g. CuPy, PyTorch, JAX)\n",
    "\n",
    "As always, development of SciPy is done for the SciPy community *by* the SciPy community. Suggestions for other features and help implementing them are always appreciated.\n",
    "\n",
    "We hope this guide motivates the advantages of trying the new infrastructure and simplifies the process."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "default_lexer": "ipython3",
   "text_representation": {
    "extension": ".md",
    "format_name": "myst",
    "format_version": 0.13,
    "jupytext_version": "1.16.4"
   }
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
