{
    "nbformat": 4,
    "nbformat_minor": 5,
    "metadata": {
        "kernelspec": {
            "display_name": "Python",
            "language": "python",
            "name": "python"
        },
        "language_info": {
            "name": "python"
        }
    },
    "cells": [
        {
            "id": "6bc45092",
            "cell_type": "markdown",
            "source": "<div class='alert alert-warning'>\n\nSciPy's interactive examples with Jupyterlite are experimental and may not always work as expected. Execution of cells containing imports may result in large downloads (up to 60MB of content for the first import from SciPy). Load times when importing from SciPy may take roughly 10-20 seconds. If you notice any problems, feel free to open an [issue](https://github.com/scipy/scipy/issues/new/choose).\n\n</div>",
            "metadata": {}
        },
        {
            "id": "e0673363",
            "cell_type": "markdown",
            "source": "The outcome of a fair coin is the most uncertain:\n",
            "metadata": {}
        },
        {
            "id": "3255da3a",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "import numpy as np\nfrom scipy.stats import entropy\nbase = 2  # work in units of bits\npk = np.array([1/2, 1/2])  # fair coin\nH = entropy(pk, base=base)\nH",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "1.0"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "40ef6021",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "H == -np.sum(pk * np.log(pk)) / np.log(base)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "True"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "3822617b",
            "cell_type": "markdown",
            "source": "The outcome of a biased coin is less uncertain:\n",
            "metadata": {}
        },
        {
            "id": "4f558b9e",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "qk = np.array([9/10, 1/10])  # biased coin\nentropy(qk, base=base)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "0.46899559358928117"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "04aaf5ba",
            "cell_type": "markdown",
            "source": "The relative entropy between the fair coin and biased coin is calculated\nas:\n",
            "metadata": {}
        },
        {
            "id": "f147e034",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "D = entropy(pk, qk, base=base)\nD",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "0.7369655941662062"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "8034495d",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "np.isclose(D, np.sum(pk * np.log(pk/qk)) / np.log(base), rtol=4e-16, atol=0)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "True"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "bfe888a9",
            "cell_type": "markdown",
            "source": "The cross entropy can be calculated as the sum of the entropy and\nrelative entropy`:\n",
            "metadata": {}
        },
        {
            "id": "f60e93a6",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "CE = entropy(pk, base=base) + entropy(pk, qk, base=base)\nCE",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "1.736965594166206"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "d55fed6a",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "CE == -np.sum(pk * np.log(qk)) / np.log(base)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "True"
                    },
                    "execution_count": null
                }
            ]
        }
    ]
}