{
    "nbformat": 4,
    "nbformat_minor": 5,
    "metadata": {
        "kernelspec": {
            "display_name": "Python",
            "language": "python",
            "name": "python"
        },
        "language_info": {
            "name": "python"
        }
    },
    "cells": [
        {
            "id": "ba17a6ea",
            "cell_type": "markdown",
            "source": "<div class='alert alert-warning'>\n\nSciPy's interactive examples with Jupyterlite are experimental and may not always work as expected. Execution of cells containing imports may result in large downloads (up to 60MB of content for the first import from SciPy). Load times when importing from SciPy may take roughly 10-20 seconds. If you notice any problems, feel free to open an [issue](https://github.com/scipy/scipy/issues/new/choose).\n\n</div>",
            "metadata": {}
        },
        {
            "id": "0ef280ff",
            "cell_type": "markdown",
            "source": "Let us consider the problem of minimizing the Rosenbrock function. This\nfunction (and its respective derivatives) is implemented in `rosen`\n(resp. `rosen_der`, `rosen_hess`) in the `scipy.optimize`.\n",
            "metadata": {}
        },
        {
            "id": "367f7024",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "from scipy.optimize import minimize, rosen, rosen_der",
            "outputs": []
        },
        {
            "id": "767db001",
            "cell_type": "markdown",
            "source": "A simple application of the *Nelder-Mead* method is:\n",
            "metadata": {}
        },
        {
            "id": "4dff435f",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "x0 = [1.3, 0.7, 0.8, 1.9, 1.2]\nres = minimize(rosen, x0, method='Nelder-Mead', tol=1e-6)\nres.x",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "array([ 1.,  1.,  1.,  1.,  1.])"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "14b05d43",
            "cell_type": "markdown",
            "source": "Now using the *BFGS* algorithm, using the first derivative and a few\noptions:\n",
            "metadata": {}
        },
        {
            "id": "6adcd42d",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "res = minimize(rosen, x0, method='BFGS', jac=rosen_der,\n               options={'gtol': 1e-6, 'disp': True})",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "Optimization terminated successfully.\n         Current function value: 0.000000\n         Iterations: 26\n         Function evaluations: 31\n         Gradient evaluations: 31"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "69f0c9b3",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "res.x",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "array([ 1.,  1.,  1.,  1.,  1.])"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "ac2fe5d5",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "print(res.message)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "Optimization terminated successfully."
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "935e6fd7",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "res.hess_inv",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "array([\n    [ 0.00749589,  0.01255155,  0.02396251,  0.04750988,  0.09495377],  # may vary\n    [ 0.01255155,  0.02510441,  0.04794055,  0.09502834,  0.18996269],\n    [ 0.02396251,  0.04794055,  0.09631614,  0.19092151,  0.38165151],\n    [ 0.04750988,  0.09502834,  0.19092151,  0.38341252,  0.7664427 ],\n    [ 0.09495377,  0.18996269,  0.38165151,  0.7664427,   1.53713523]\n])"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "d9043d51",
            "cell_type": "markdown",
            "source": "Next, consider a minimization problem with several constraints (namely\nExample 16.4 from [5]). The objective function is:\n",
            "metadata": {}
        },
        {
            "id": "431d9d8a",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "fun = lambda x: (x[0] - 1)**2 + (x[1] - 2.5)**2",
            "outputs": []
        },
        {
            "id": "1f04d2a3",
            "cell_type": "markdown",
            "source": "There are three constraints defined as:\n",
            "metadata": {}
        },
        {
            "id": "5e1e1289",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "cons = ({'type': 'ineq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},\n        {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},\n        {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})",
            "outputs": []
        },
        {
            "id": "ac0de1f5",
            "cell_type": "markdown",
            "source": "And variables must be positive, hence the following bounds:\n",
            "metadata": {}
        },
        {
            "id": "4fefa87d",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "bnds = ((0, None), (0, None))",
            "outputs": []
        },
        {
            "id": "8e0fb68b",
            "cell_type": "markdown",
            "source": "The optimization problem is solved using the SLSQP method as:\n",
            "metadata": {}
        },
        {
            "id": "33e293ff",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "res = minimize(fun, (2, 0), method='SLSQP', bounds=bnds, constraints=cons)",
            "outputs": []
        },
        {
            "id": "a08e915c",
            "cell_type": "markdown",
            "source": "It should converge to the theoretical solution ``[1.4 ,1.7]``. *SLSQP* also\nreturns the multipliers that are used in the solution of the problem. These\nmultipliers, when the problem constraints are linear, can be thought of as the\nKarush-Kuhn-Tucker (KKT) multipliers, which are a generalization\nof Lagrange multipliers to inequality-constrained optimization problems ([20]).\n\nNotice that at the solution, the first constraint is active. Let's evaluate the\nfunction at solution:\n",
            "metadata": {}
        },
        {
            "id": "98226878",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "cons[0]['fun'](res.x)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "np.float64(1.4901224698604665e-09)"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "02d9cc77",
            "cell_type": "markdown",
            "source": "Also, notice that at optimality there is a non-zero multiplier:\n",
            "metadata": {}
        },
        {
            "id": "b7278f20",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "res.multipliers",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "array([0.8, 0. , 0. ])"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "c097e2b6",
            "cell_type": "markdown",
            "source": "This can be understood as the local sensitivity of the optimal value of the\nobjective function with respect to changes in the first constraint. If we\ntighten the constraint by a small amount ``eps``:\n",
            "metadata": {}
        },
        {
            "id": "6ac8c890",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "eps = 0.01\ncons[0]['fun'] = lambda x: x[0] - 2 * x[1] + 2 - eps",
            "outputs": []
        },
        {
            "id": "15d3a315",
            "cell_type": "markdown",
            "source": "we expect the optimal value of the objective function to increase by\napproximately ``eps * res.multipliers[0]``:\n",
            "metadata": {}
        },
        {
            "id": "7e758c8d",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "eps * res.multipliers[0]  # Expected change in f0",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "np.float64(0.008000000027153205)"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "d329e158",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "f0 = res.fun  # Keep track of the previous optimal value\nres = minimize(fun, (2, 0), method='SLSQP', bounds=bnds, constraints=cons)\nf1 = res.fun  # New optimal value\nf1 - f0",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "np.float64(0.008019998807885509)"
                    },
                    "execution_count": null
                }
            ]
        }
    ]
}