{
    "nbformat": 4,
    "nbformat_minor": 5,
    "metadata": {
        "kernelspec": {
            "display_name": "Python",
            "language": "python",
            "name": "python"
        },
        "language_info": {
            "name": "python"
        }
    },
    "cells": [
        {
            "id": "5eb594a4",
            "cell_type": "markdown",
            "source": "<div class='alert alert-warning'>\n\nSciPy's interactive examples with Jupyterlite are experimental and may not always work as expected. Execution of cells containing imports may result in large downloads (up to 60MB of content for the first import from SciPy). Load times when importing from SciPy may take roughly 10-20 seconds. If you notice any problems, feel free to open an [issue](https://github.com/scipy/scipy/issues/new/choose).\n\n</div>",
            "metadata": {}
        },
        {
            "id": "4c4b76ef",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "import numpy as np\nfrom scipy import stats\nfrom scipy.stats import mstats\nmstats.pearsonr([1, 2, 3, 4, 5], [10, 9, 2.5, 6, 4])",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "(-0.7426106572325057, 0.1505558088534455)"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "6ba52c9f",
            "cell_type": "markdown",
            "source": "There is a linear dependence between x and y if y = a + b*x + e, where\na,b are constants and e is a random error term, assumed to be independent\nof x. For simplicity, assume that x is standard normal, a=0, b=1 and let\ne follow a normal distribution with mean zero and standard deviation s>0.\n",
            "metadata": {}
        },
        {
            "id": "680dcdc2",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "s = 0.5\nx = stats.norm.rvs(size=500)\ne = stats.norm.rvs(scale=s, size=500)\ny = x + e\nmstats.pearsonr(x, y)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "(0.9029601878969703, 8.428978827629898e-185) # may vary"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "c4ff448f",
            "cell_type": "markdown",
            "source": "This should be close to the exact value given by\n",
            "metadata": {}
        },
        {
            "id": "a43883d9",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "1/np.sqrt(1 + s**2)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "0.8944271909999159"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "e09bc187",
            "cell_type": "markdown",
            "source": "For s=0.5, we observe a high level of correlation. In general, a large\nvariance of the noise reduces the correlation, while the correlation\napproaches one as the variance of the error goes to zero.\n\nIt is important to keep in mind that no correlation does not imply\nindependence unless (x, y) is jointly normal. Correlation can even be zero\nwhen there is a very simple dependence structure: if X follows a\nstandard normal distribution, let y = abs(x). Note that the correlation\nbetween x and y is zero. Indeed, since the expectation of x is zero,\ncov(x, y) = E[x*y]. By definition, this equals E[x*abs(x)] which is zero\nby symmetry. The following lines of code illustrate this observation:\n",
            "metadata": {}
        },
        {
            "id": "6d1bd2f3",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "y = np.abs(x)\nmstats.pearsonr(x, y)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "(-0.016172891856853524, 0.7182823678751942) # may vary"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "c91f655e",
            "cell_type": "markdown",
            "source": "A non-zero correlation coefficient can be misleading. For example, if X has\na standard normal distribution, define y = x if x < 0 and y = 0 otherwise.\nA simple calculation shows that corr(x, y) = sqrt(2/Pi) = 0.797...,\nimplying a high level of correlation:\n",
            "metadata": {}
        },
        {
            "id": "f220a41c",
            "cell_type": "code",
            "metadata": {},
            "execution_count": null,
            "source": "y = np.where(x < 0, x, 0)\nmstats.pearsonr(x, y)",
            "outputs": [
                {
                    "output_type": "execute_result",
                    "metadata": {},
                    "data": {
                        "text/plain": "(0.8537091583771509, 3.183461621422181e-143) # may vary"
                    },
                    "execution_count": null
                }
            ]
        },
        {
            "id": "51a34a40",
            "cell_type": "markdown",
            "source": "This is unintuitive since there is no dependence of x and y if x is larger\nthan zero which happens in about half of the cases if we sample x and y.",
            "metadata": {}
        }
    ]
}