让你拥有超能力：程序员应该掌握的统计学公式

阅读量：6274 次

发布时间：2019-06-22

本文共 9589 字，大约阅读时间需要 31 分钟。

让你拥有超能力：程序员应该掌握的统计学公式

作者：

出处：

Evan Miller收集了程序员应该掌握的统计学公式：

非常有用。

但他在文章前面的这段话更有启发意义：

能够运用统计学，就像拥有了秘密的超能力。

大多数人只会看看平均数，而你会看置信区间。

别人说“7比5大”，你会说，生成数据的过程是无法辨识的。

在众声喧哗的噪音里，你会听到关键的求救。

糟糕的是，并不是很多程序员具有这种超能力。这真是悲剧，因为运用统计学几乎总是能改善数据的展示和阐释。为此，我收集了自认为最有用的统计学公式。

Statistical Formulas For Programmers

DRAFT: May 19, 2013

Being able to apply statistics is like having a secret superpower.

Where most people see averages, you see confidence intervals.

When someone says “7 is greater than 5,” you declare that they're really the same.

In a cacophony of noise, you hear a cry for help.

Unfortunately, not enough programmers have this superpower. That's a shame, because the application of statistics can almost always enhance the display and interpretation of data.

As my modest contribution to developer-kind, I've collected together the statistical formulas that I find to be most useful; this page presents them all in one place, a sort of statistical cheat-sheet for the practicing programmer.

Most of these formulas can be found in Wikipedia, but others are buried in journal articles or in professors' web pages. They are all classical (not Bayesian), and to motivate them I have added concise commentary. I've also added links and references, so that even if you're unfamiliar with the underlying concepts, you can go out and learn more. Wearing a red cape is optional.

Send suggestions and corrections to emmiller@gmail.com

1. Formulas For Reporting Averages

One of the first programming lessons in any language is to compute an average. But rarely does anyone stop to ask: what does the average actually tell us about the underlying data?

1.1 Corrected Standard Deviation

The standard deviation is a single number that reflects how spread out the data actually is. It should be reported alongside the average (unless the user will be confused).

s = 1 N - 1 \sum i = 1 N (x i - x ¯) 2 ----------------⎷

Where:

N is the number of observations

xi is the value of the i^th observation

x¯ is the average value of xi

Reference:

1.2 Standard Error of the Mean

From a statistical point of view, the "average" is really just an estimate of an underlying population mean. That estimate has uncertainty that is summarized by the standard error.

S E = s N --\sqrt

Reference:

1.3 Confidence Interval Around the Mean

A confidence interval reflects the set of statistical hypotheses that won't be rejected at a given significance level. So the confidence interval around the mean reflects all possible values of the mean that can't be rejected by the data. It is a multiple of the standard error added to and subtracted from the mean.

C I = x ¯ \pm t α / 2 S E

Where:

α is the significance level, typically 5% (one minus the confidence level)

tα/2 is the 1−α/2 quantile of a t-distribution with N−1 degrees of freedom

Reference:

1.4 Two-Sample T-Test

A two-sample t-test can tell whether two groups of observations differ in their mean.

The test statistic is given by:

t = x 1 ¯ - x 2 ¯ s 2 1 / n 1 + s 2 2 / n 2 ------------\sqrt

The hypothesis of equal means is rejected if |t| exceeds the (1−α/2) quantile of a t distribution with degrees of freedom equal to:

d f = ( s 2 1 / n 1 + s 2 2 / n 2 ) 2 ( s 2 1 / n 1 ) 2 / ( n 1 - 1 ) + ( s 2 2 / n 2 ) 2 / ( n 2 - 1 )

You can see a demonstration of these concepts in .

Reference:

2. Formulas For Reporting Proportions

It's common to report the relative proportions of binary outcomes or categorical data, but in general these are meaningless without confidence intervals and tests of independence.

2.1 Confidence Interval of a Bernoulli Parameter

A Bernoulli parameter is the proportion underlying a binary-outcome event (for example, the percent of the time a coin comes up heads). The confidence interval is given by:

C I = ⎛⎝ p + z 2 α / 2 2 N \pm z α / 2 [p (1 - p) + z 2 α / 2 / 4 N] / N -------------------\sqrt ⎞⎠ / (1 + z 2 α / 2)

Where:

p is the observed proportion of interest

zα/2 is the (1−α/2) quantile of a normal distribution

This formula can also be used as a .

Reference:

2.2 Multinomial Confidence Intervals

If you have more than two categories, a multinomial confidence interval supplies upper and lower confidence limits on all of the category proportions at once. The formula is nearly identical to the preceding one.

C I = ⎛⎝ p j + z 2 α / 2 2 N \pm z α / 2 [p j (1 - p j) + z 2 α / 2 / 4 N] / N ---------------------\sqrt ⎞⎠ / (1 + z 2 α / 2)

Where:

pj is the observed proportion of the jth category

Reference:

2.3 Chi-Squared Test

Pearson's chi-squared test can detect whether the distribution of row counts seem to differ across columns (or vice versa). It is useful when comparing two or more sets of category proportions.

The test statistic, called X2, is computed as:

X 2 = \sum i = 1 n \sum j = 1 m ( O i , j - E i , j ) 2 E i , j

Where:

n is the number of rows

m is the number of columns

Oi,j is the observed count in row i and column j

Ei,j is the expected count in row i and column j

The expected count is given by:

E i, j = \sum n k = 1 O k , j \sum m l = 1 O i , l N

A statistical dependence exists if X2 is greater than the (1−α) quantile of a χ2 distribution with (m−1)×(n−1) degrees of freedom.

You can see a 2x2 demonstration of these concepts in .

Reference:

3. Formulas For Reporting Count Data

If the incoming events are independent, their counts are well-described by a Poisson distribution. A Poisson distribution takes a parameter λ, which is the distribution's mean — that is, the average arrival rate of events per unit time.

3.1. Standard Deviation of a Poisson Distribution

The standard deviation of Poisson data usually doesn't need to be explicitly calculated. Instead it can be inferred from the Poisson parameter:

σ = λ \sqrt

This fact can be used to read an , for example.

Reference:

3.2. Confidence Interval Around the Poisson Parameter

The confidence interval around the Poisson parameter represents the set of arrival rates that can't be rejected by the data. It can be inferred from a single data point of c events observed over t time periods with the following formula:

C I = (γ - 1 ( α / 2 , c ) t, γ - 1 ( 1 - α / 2 , c + 1 ) t)

Where:

γ−1(p,c) is the inverse of the lower incomplete gamma function

Reference:

3.3. Conditional Test of Two Poisson Parameters

Please never do this:

From a statistical point of view, 5 events is indistinguishable from 7 events. Before reporting in bright red text that one count is greater than another, it's best to perform a test of the two Poisson means.

The p-value is given by:

p = 2 \times c ! t c \times min ⎧⎩⎨ \sum i = 0 c 1 t i 1 t c - i 2 i ! ( c - i ) !, \sum i = c 1 c t i 1 t c - i 2 i ! ( c - i ) ! ⎫⎭⎬

Where:

Observation 1 consists of c1 events over t1 time periods

Observation 2 consists of c2 events over t2 time periods

c=c1+c2 and t=t1+t2

You can see a demonstration of these concepts in .

Reference:

4. Formulas For Comparing Distributions

If you want to test whether groups of observations come from the same (unknown) distribution, or if a single group of observations comes from a known distribution, you'll need a Kolmogorov-Smirnov test. A K-S test will test the entire distribution for equality, not just the distribution mean.

4.1. Comparing An Empirical Distribution to a Known Distribution

The simplest version is a one-sample K-S test, which compares a sample of n points having an observed cumulative distribution function F to a known distribution function having a c.d.f. of G. The test statistic is:

D n = sup x | F (x) - G (x) |

In plain English, Dn is the absolute value of the largest difference in the two c.d.f.s for any value of x.

The critical value of Dn is given by Kα/n√, where Kα is the value of x that solves:

1 - α = 2 π ---\sqrt x \sum k = 1 \infty exp (- (2 k - 1) 2 π 2 / (8 x 2))

The critical must be solved iteratively, e.g. by Newton's method. If only the p-value is needed, it can be computed directly by solving the above for α.

Reference:

4.2. Comparing Two Empirical Distributions

The two-sample version is similar, except the test statistic is given by:

D n 1, n 2 = sup x | F 1 (x) - F 2 (x) |

Where F1 and F2 are the empirical c.d.f.s of the two samples, having n1 and n2 observations, respectively. The critical value of the test statistic is Kα/n1n2/(n1+n2)−−−−−−−−−−−−√ with the same value of Kα above.

Reference:

4.3. Comparing Three or More Empirical Distributions

A k-sample extension of Kolmogorov-Smirnov was described by J. Kiefer in a . The test statistic is:

T = sup x \sum j = 1 k n j | F j (x) - F ¯ (x) |

Where F¯ is the c.d.f. of the combined samples. The critical value of T is a2 where a solves:

1 - α = 4 Γ ( h 2 ) 2 h / 2 a h \sum n = 1 \infty ( γ ( h - 2 ) / 2 , n ) h - 2 exp [ - ( γ ( h - 2 ) / 2 , n ) 2 / 2 a 2 ] [ J h / 2 ( γ ( h - 2 ) / 2 , n ) ] 2

Where:

h=k−1

Jh/2 is a Bessel function of the first kind with order h/2

γ(h−2)/2,n is the n^th zero of J(h−2)/2

To compute the critical value, this equation must also be solved iteratively. When k=2, the equation reduces to a two-sample Kolmogorov-Smirnov test. The case of k=4 can also be reduced to a simpler form, but for other values of k, the equation cannot be reduced.

Reference: