Statistics Glossary

A controlled experiment that randomly assigns users to two versions and compares an outcome. It is the standard tool for testing website and product changes.

P(A or B) equals P(A) + P(B) − P(A and B). The subtraction removes the overlap that would otherwise be counted twice.

The claim that there is an effect or a difference, accepted only if the data provide sufficient evidence against the null. It is written H₁ or Hₐ.

Analysis of variance, a test comparing the means of three or more groups by comparing variation between groups with variation within groups. A significant F-test says at least one group differs.

The area under the ROC curve, equal to the probability that a randomly chosen positive case scores higher than a randomly chosen negative one. It is 0.5 for a useless score and 1.0 for a perfect one.

The correlation of a time series with a delayed copy of itself. It shows how strongly present values depend on past values.

How common a condition is in the population before any test or evidence is considered. Ignoring it is called the base rate fallacy and leads to badly overestimating the meaning of positive results.

A rule for reversing conditional probabilities: P(A | B) = P(B | A) × P(A) ÷ P(B). It updates a prior belief in light of new evidence.

An approach to statistics that updates probability distributions for unknown parameters using Bayes' theorem. Results are expressed as probabilities about parameters given the data.

A single random experiment with exactly two outcomes, conventionally called success and failure, with a fixed success probability. A binomial variable counts successes across independent Bernoulli trials.

The difference between an estimator's average value over all possible samples and the true parameter. An estimator with zero bias is called unbiased.

The balance between a model that is too simple, with high bias, and one that is too flexible, with high variance. Prediction error is lowest at an intermediate complexity.

The distribution of the number of successes in a fixed number of independent trials with the same success probability. Its mean is np and its variance is np(1 − p).

The result that in a group of only 23 people, the chance that two share a birthday exceeds 50%. It shows how quickly the number of possible pairs grows.

A simple way to control the family-wise error rate by dividing the significance level by the number of tests. It is safe but conservative when there are many tests.

A resampling method that draws many samples with replacement from the data to estimate the variability of a statistic. It produces standard errors and confidence intervals without formulas.

A chart of the five-number summary, with a box from Q1 to Q3, a line at the median, and whiskers reaching toward the extremes. Points beyond the whiskers are often flagged as outliers.

A variable that places observations into groups or labels, either unordered (nominal) or ordered (ordinal). Also called qualitative data.

A situation in which the exact time of an event is unknown for some subjects, only that it is later than their last observation. Handling censored data correctly is central to survival analysis.

The result that the distribution of sample means approaches a normal shape as the sample size grows, whatever the shape of the underlying population. It justifies many confidence intervals and tests.

A guarantee that for any distribution, at least 1 − 1/k² of values lie within k standard deviations of the mean. It holds regardless of shape but is conservative.

A right-skewed distribution of a sum of squared standard normal values, determined by its degrees of freedom. It underlies chi-square tests of independence and goodness of fit.

A test that compares observed counts with the counts expected under a hypothesis. It is used for independence between categorical variables and for goodness of fit.

A design that randomly selects whole groups, such as schools or villages, and surveys all or some members within them. It is cheaper for dispersed populations but usually less precise.

The standard deviation divided by the mean, usually shown as a percentage. It compares variability between datasets measured in different units or on different scales.

The difference between two group means divided by their pooled standard deviation. Values near 0.2, 0.5, and 0.8 are conventionally called small, medium, and large.

A measure of agreement between two raters that corrects for the agreement expected by chance. A value of 1 is perfect agreement and 0 means no better than chance.

A selection of items in which order does not matter. The number of ways to choose r items from n is n! ÷ (r! × (n − r)!).

The probability that an event does not happen equals one minus the probability that it does. It often makes "at least one" problems far easier.

The probability of an event given that another event has occurred, written P(A | B) and equal to P(A and B) ÷ P(B). It restricts attention to the cases where B holds.

A range of plausible values for a parameter, built so that the procedure captures the true value in a stated percentage of repeated samples. A 95% interval does not mean a 95% chance for this particular interval.

The long-run proportion of intervals from a procedure that would contain the true parameter, commonly 95%. Higher confidence gives a wider interval.

A variable related to both the predictor and the outcome that creates a misleading association between them. It is a main reason correlation does not establish causation.

A table of true positives, false positives, false negatives, and true negatives for a classifier. Sensitivity, specificity, and predictive values are all read from it.

A variable that can take any value within a range, such as height or time. Measured values are limited only by the precision of the instrument.

A chart of a process measurement over time with limits showing expected variation. Points outside the limits signal a process change worth investigating.

A measure of how much a regression's fitted values change when one observation is removed. It combines the point's residual and its leverage.

A number from −1 to 1 measuring the strength and direction of a linear relationship between two variables. Pearson's r is the standard version for numeric data.

The principle that two variables moving together does not show that one causes the other. Confounders, reverse causation, and coincidence can all produce correlation.

A measure of how two variables vary together: positive when they tend to rise together, negative when one rises as the other falls. Its size depends on units, so correlation is used to judge strength.

A Bayesian interval that contains the parameter with a stated posterior probability. Unlike a confidence interval, it can be read directly as a probability statement about the parameter.

The cutoff on the test statistic's distribution beyond which the null hypothesis is rejected. It is determined by the significance level and whether the test is one- or two-tailed.

A measure of internal consistency reliability for a scale made of several items. Values near 1 indicate items move together strongly, and 0.70 to 0.80 is often considered acceptable.

A method of estimating how well a model predicts new data by repeatedly training on part of the data and testing on the held-out remainder. In k-fold cross-validation the data are split into k parts.

A function giving the probability that a random variable is less than or equal to a value. It rises from 0 to 1 and works for both discrete and continuous variables.

A running total of frequencies up to and including each class. Plotting it against upper class boundaries gives an ogive, which can be used to estimate the median and percentiles.

The graphical presentation of data. Effective charts choose the display type to match the question, avoid distortion, and remove clutter.

The number of values in a calculation that are free to vary once constraints are imposed. For a sample variance it is n − 1, because the deviations from the sample mean must sum to zero.

A variable that takes countable separate values, such as the number of children in a family. Gaps exist between possible values.

A 0/1 variable that represents membership in a category so it can be used in a regression. A variable with k categories needs k − 1 dummies to avoid perfect multicollinearity.

A test for first-order autocorrelation in regression residuals, comparing how much consecutive residuals differ to how large they are overall. Values near 2 suggest no autocorrelation; near 0 suggests positive, near 4 suggests negative.

A measure of the magnitude of an effect that does not depend on sample size, such as Cohen's d or Cramér's V. It shows how big a difference is, whereas a p-value only shows whether it is detectable.

For a normal distribution, about 68% of values lie within one standard deviation of the mean, 95% within two, and 99.7% within three. It is also called the 68-95-99.7 rule.

A rule for calculating an estimate of a population parameter from sample data. Estimators are judged by properties such as bias, variance, and consistency.

A set of outcomes from the sample space, such as rolling an even number. The probability of an event is the sum of the probabilities of the outcomes it contains.

The long-run average of a random variable, found by weighting each possible value by its probability. It is the theoretical mean of the distribution.

The practice of examining data with summaries and graphs before formal modelling. It reveals shape, outliers, errors, and relationships worth investigating.

A continuous distribution for the waiting time until the next event in a Poisson process. It is memoryless, so the time already waited does not affect the time remaining.

Using a fitted model to predict outside the range of the data used to build it. Predictions there are unreliable because the relationship may change.

The distribution of a ratio of two independent scaled chi-square variables. It is the reference distribution for ANOVA and for F-tests in regression.

A test of whether a regression model, or a set of added predictors, explains significantly more variation than a simpler model. It compares explained variance with unexplained variance.

The product of all positive whole numbers up to n, written n!. It counts the number of ways to arrange n distinct items in a row.

The expected proportion of rejected hypotheses that are false discoveries. The Benjamini–Hochberg procedure controls it and is more powerful than Bonferroni.

A factor that reduces the standard error when a large share of a small population is sampled without replacement. It can be ignored when the sample is a small fraction of the population.

The minimum, first quartile, median, third quartile, and maximum. Together these five values describe the centre, spread, and range of a dataset and form the basis of a box plot.

A table showing how many observations fall into each value or class interval. Relative and cumulative versions express the counts as proportions and running totals.

An approach in which probability describes long-run frequencies of events under repeated sampling. Parameters are treated as fixed, and results are stated through p-values and confidence intervals.

The non-parametric counterpart to repeated-measures ANOVA, comparing three or more related conditions measured on the same subjects by ranking the conditions within each subject.

The mistaken belief that past random outcomes make a different outcome more likely next time. Independent events such as coin flips have no memory.

The distribution of the number of trials needed to get the first success in repeated independent trials. Its mean is 1 divided by the success probability.

The nth root of the product of n positive values. It is the right average for growth rates and ratios, and it is always less than or equal to the arithmetic mean.

The reciprocal of the average of the reciprocals of the values. It is the appropriate average for rates such as speeds over equal distances.

A situation in which the spread of the residuals changes across the range of the predictor. It violates the constant-variance assumption and can make standard errors misleading.

A bar chart of a numeric variable in which touching bars show how many values fall in each class interval. The choice of class width strongly affects how the shape appears.

The distribution of successes in a sample drawn without replacement from a finite population. It differs from the binomial because each draw changes the composition of what remains.

A method for the probability or count of a union of overlapping events: add the singles, subtract the pairs, add the triples, and so on. It ensures every outcome is counted exactly once.

Events where the occurrence of one does not change the probability of the other. For independent events, P(A and B) equals P(A) × P(B).

An observation whose removal noticeably changes a fitted model. Influence requires both unusual predictor values and a poor fit to the trend.

A situation in which the effect of one predictor depends on the level of another. It is modelled by adding a product term of the two predictors.

The predicted value of the outcome when the predictor equals zero. It may have no practical meaning if zero lies outside the range of the data.

The distance between the third quartile (Q3) and the first quartile (Q1), covering the middle 50% of the data. It is a spread measure that outliers barely affect.

A numeric scale with equal spacing between values but no true zero, such as temperature in Celsius. Differences are meaningful, but ratios are not.

A resampling method that recomputes a statistic n times, leaving out one observation each time. It estimates bias and standard error.

The probability that two events both occur, written P(A and B). In a two-way table it is a single cell divided by the grand total.

A method for estimating a survival curve from time-to-event data that includes censored observations. The estimate multiplies survival probabilities at each event time.

A test comparing a sample's empirical distribution function with a reference distribution, or two samples with each other. Its statistic is the largest gap between the two cumulative curves.

The non-parametric counterpart to one-way ANOVA, comparing three or more groups by rank rather than assuming normally distributed data. A significant result means the groups' distributions differ somewhere, without pinpointing which pairs.

A measure of how heavy the tails of a distribution are relative to a normal distribution. High kurtosis means extreme values occur more often than a normal curve predicts.

A regression that penalizes the sum of absolute coefficient values, which can shrink some coefficients exactly to zero. It performs variable selection as well as shrinkage.

The principle that as the number of independent trials grows, the sample average converges to the expected value. It explains why long-run frequencies settle near true probabilities.

A way to find an overall probability by splitting it across a partition of cases: P(B) = Σ P(B | Aᵢ) × P(Aᵢ). It forms the denominator in Bayes' theorem.

The method of choosing regression coefficients that minimize the sum of the squared residuals. It gives the slope Sxy ÷ Sxx and an intercept passing through the point of means.

A measure of how far a point's predictor values lie from the centre of the data, showing its potential to pull the fitted line toward itself. Average leverage equals the number of parameters divided by n.

The probability of the observed data viewed as a function of the parameter values. It measures how well each candidate parameter value explains the data.

How much a test result multiplies the odds of a condition. The positive likelihood ratio equals sensitivity ÷ (1 − specificity).

A test comparing two nested models by the ratio of their likelihoods. Twice the log of the ratio approximately follows a chi-square distribution.

The rule that the expected value of a sum equals the sum of expected values, whether or not the variables are independent. It makes many counting-expectation problems easy through indicator variables.

A distribution of a positive variable whose logarithm is normally distributed. It is right-skewed, with the mode below the median and the median below the mean.

A regression model for a binary outcome that predicts the log-odds of the event as a linear function of the predictors. Its exponentiated coefficients are odds ratios.

The half-width of a confidence interval, equal to a critical value multiplied by the standard error. It states how far the estimate could plausibly be from the truth.

The probability of one event ignoring the other variable, found from a row or column total of a two-way table. It is the sum of joint probabilities across the other variable.

A method that chooses the parameter values that make the observed data most probable. It gives estimators with strong large-sample properties.

A test for paired yes/no outcomes, such as before-and-after answers from the same people. It uses only the pairs whose answer changed.

The sum of all values divided by how many there are, also called the arithmetic average. It uses every observation, which makes it sensitive to extreme values.

The average squared distance between an estimator and the true value, equal to variance plus squared bias. It combines accuracy and precision in one number.

The middle value of a sorted dataset, or the average of the two middle values when the count is even. It is resistant to outliers, so it often describes skewed data better than the mean.

The median of the absolute differences between each value and the median. It is a robust measure of spread that a few extreme values cannot distort.

The most frequently occurring value in a dataset. A dataset can have one mode, several, or none if every value appears equally often.

A probability puzzle in which switching doors after a host reveals a losing door wins two-thirds of the time. It illustrates how new information changes conditional probabilities.

The average of the most recent k observations, recomputed as time advances. It smooths short-term fluctuations to reveal the underlying trend.

Strong correlation among predictor variables, which makes coefficient estimates unstable and hard to interpret. The variance inflation factor is a common diagnostic.

A generalization of the binomial to more than two outcome categories. It gives the probability of a particular set of counts across categories in a fixed number of independent trials.

An extension of logistic regression to outcomes with more than two unordered categories. One category serves as the baseline against which the others are compared.

The growth in the chance of at least one false positive as more tests are run. Ten independent tests at the 5% level give about a 40% chance of a false alarm.

A regression model with two or more predictors. Each coefficient shows the change in the outcome for a one-unit increase in that predictor, holding the others constant.

P(A and B) equals P(A) × P(B | A). For independent events it simplifies to P(A) × P(B).

Events that cannot happen at the same time, so their intersection has probability zero. Mutually exclusive events with nonzero probabilities are never independent.

The distribution of the number of trials needed to reach a fixed number of successes. It generalizes the geometric distribution and models overdispersed counts.

A level of measurement for categories with no natural order, such as eye colour or country. Only counts and the mode are meaningful summaries.

A test that makes few assumptions about the shape of the population distribution, often working with ranks. Examples include the Mann-Whitney, Wilcoxon, and Kruskal-Wallis tests.

A symmetric, bell-shaped distribution defined by its mean and standard deviation. Many natural measurements and sample means follow it approximately.

The default claim that there is no effect or no difference, which a test tries to find evidence against. It is written H₀.

A study that records outcomes without assigning treatments. It can show association but is vulnerable to confounding, so it cannot easily establish causation.

The ratio of the probability that an event happens to the probability that it does not, p ÷ (1 − p). A probability of 0.75 corresponds to odds of 3 to 1.

The ratio of the odds of an outcome in one group to the odds in another. A value of 1 means no association, and it is the natural effect measure in logistic regression.

A test that looks for an effect in one specified direction only. It has more power in that direction but cannot detect an effect in the other.

A level of measurement for categories with a meaningful order but unequal or unknown gaps, such as satisfaction ratings from poor to excellent. The median and percentiles are appropriate summaries.

An observation that lies unusually far from the rest of the data. A common rule flags values more than 1.5 times the IQR beyond the quartiles.

When a model captures noise in the training data instead of the underlying pattern, so it fits the sample well but predicts new data poorly. More flexible models overfit more easily.

Trying many analyses or subsets until something reaches significance, then reporting only that result. It inflates the false positive rate far above the stated level.

The probability, assuming the null hypothesis is true, of obtaining results at least as extreme as those observed. It is not the probability that the null hypothesis is true.

A fixed but usually unknown numerical characteristic of a population, such as its true mean. Statistics computed from samples are used to estimate it.

A bar chart of categories sorted from largest to smallest with a cumulative percentage line. It highlights the few causes responsible for most of a problem.

The correlation between two variables after removing the linear influence of one or more other variables. It helps separate direct relationships from those driven by a third variable.

The covariance of two variables divided by the product of their standard deviations. It measures linear association only and is sensitive to outliers.

A value below which a stated percentage of the observations fall. Being at the 90th percentile means 90% of values are at or below yours.

An arrangement of items in which order matters. The number of ways to arrange r items chosen from n is n! ÷ (n − r)!.

A test that builds its reference distribution by repeatedly shuffling group labels and recomputing the statistic. The p-value is the proportion of shuffles at least as extreme as the observed result.

A single number used as the best guess for a population parameter, such as a sample mean. It carries no information about its own precision.

A distribution for the number of events occurring in a fixed interval when events happen independently at a constant average rate. Its mean and variance are both equal to that rate.

A regression model for count outcomes that links the log of the expected count to the predictors. It assumes the variance equals the mean, and overdispersion breaks that assumption.

A regression that adds powers of a predictor, such as x², to capture curved relationships. It is still linear in its coefficients, but high degrees risk overfitting.

The complete group about which conclusions are wanted, such as all voters in a country. Because measuring everyone is usually impractical, a sample is studied instead.

The probability that a person with a positive test result truly has the condition. Unlike sensitivity, it depends heavily on how common the condition is.

A follow-up comparison run after a significant ANOVA to find which specific groups differ. Tukey's HSD is a common choice because it controls the family-wise error rate.

The updated probability of a hypothesis after taking new evidence into account. It results from multiplying the prior by the likelihood and normalizing.

Whether an effect is large enough to matter in the real world, as opposed to merely being statistically detectable. With large samples, trivial differences can be highly significant.

A range expected to contain a single new observation at a given predictor value. It is always wider than a confidence interval for the mean response at the same point.

A technique that rotates the variables into uncorrelated components ordered by how much variance they capture. It is used to compress data and reveal structure.

The probability assigned to a hypothesis before seeing new evidence. In Bayes' theorem it is combined with the likelihood to give the posterior.

A number from 0 to 1 measuring how likely an event is, where 0 means impossible and 1 means certain. For equally likely outcomes it equals favourable outcomes divided by total outcomes.

A function describing a continuous distribution, where the area under the curve between two values gives the probability of falling in that range. The total area under the curve is 1.

A description of all possible values of a random variable and how likely each is. Discrete variables use a probability mass function; continuous variables use a density.

A function giving the probability that a discrete random variable equals each specific value. The probabilities across all values sum to 1.

A diagram whose branches show the stages of a multi-step process, with probabilities along each branch. Multiplying along a path gives the probability of that sequence.

A plot comparing the quantiles of a dataset with those of a theoretical distribution. Points close to a straight line suggest the data follow that distribution.

One of three values that split sorted data into four groups of equal size. Q1 is the 25th percentile, Q2 is the median, and Q3 is the 75th percentile.

The proportion of the outcome's variance explained by the model, from 0 to 1. A high value does not prove the model is correct or that the relationship is causal.

Selecting subjects by chance so that each has a known probability of inclusion. It protects against selection bias and makes probability-based inference possible.

A numerical quantity whose value depends on the outcome of a random process, such as the sum of two dice. It can be discrete or continuous.

A study in which subjects are randomly assigned to conditions. Randomization balances confounders on average, which is what allows causal conclusions.

The difference between the largest and smallest values. It is quick to compute but depends entirely on the two most extreme observations.

A numeric scale with equal spacing and a true zero, such as height, weight, or income. Both differences and ratios are meaningful.

The tendency for extreme measurements to be followed by measurements closer to average. It can create the illusion of an effect after an intervention chosen because of an extreme result.

Adding a penalty to a model's fitting criterion to discourage large coefficients and complexity. It trades a little bias for a larger reduction in variance.

The difference between an observed value and the value the model predicts. Residual plots are used to check whether a model's assumptions hold.

A regression that adds a penalty on the sum of squared coefficients, shrinking them toward zero. It reduces variance and helps when predictors are correlated.

A plot of the true positive rate against the false positive rate across every possible classification threshold. A curve hugging the top-left corner indicates a strong classifier.

A test of whether a sequence of two outcomes (pass/fail, above/below the median) is genuinely random, based on the number of unbroken streaks (runs) it contains. Too few runs suggests clumping; too many suggests suspiciously frequent alternation.

A subset of the population that is actually observed. Its usefulness depends on how it was selected, not only on its size.

The number of observations in a sample. Larger samples shrink the standard error, but the gain slows as the size grows, with precision improving in proportion to the square root of n.

The set of all possible outcomes of a random process. Rolling a die has the sample space {1, 2, 3, 4, 5, 6}.

A flaw in how a sample is selected that makes some members of the population more likely to be included than others. It produces estimates that are systematically off, regardless of sample size.

The distribution of a statistic across all possible samples of a given size. Its spread is described by the standard error.

A graph plotting pairs of values as points, showing the form, direction, and strength of a relationship. It should accompany any correlation or regression.

Systematic error arising from how subjects come to be included in a study or analysis. It makes the studied group unrepresentative of the population of interest.

The proportion of people who truly have a condition that a test correctly identifies as positive. It is also called the true positive rate or recall.

A test of whether a sample comes from a normal distribution. A small p-value is evidence against normality.

The threshold for rejecting the null hypothesis, chosen before the test and commonly 0.05. It equals the probability of a Type I error when the null is true.

A model that fits a straight line to predict one variable from another. The line is chosen to minimize the sum of squared vertical errors.

A sample in which every member of the population and every possible group of the chosen size has an equal chance of selection. It is the standard against which other designs are compared.

A reversal in which a trend present within every subgroup disappears or reverses when the groups are combined. It arises when a hidden variable affects group sizes.

A measure of asymmetry in a distribution. Positive skew means a longer right tail with the mean above the median; negative skew means a longer left tail.

The predicted change in the outcome for a one-unit increase in the predictor. In simple regression it equals the covariance divided by the predictor's variance.

The correlation calculated on the ranks of the data rather than the raw values. It captures any monotonic relationship and is less affected by outliers than Pearson's r.

The proportion of people without a condition that a test correctly identifies as negative. One minus specificity is the false positive rate.

The square root of the variance, expressed in the same units as the data. It describes the typical distance of observations from the mean.

The standard deviation of a statistic's sampling distribution. For a sample mean it equals the standard deviation divided by the square root of the sample size.

The normal distribution with mean 0 and standard deviation 1. Converting values to z-scores lets any normal probability be read from it.

A number computed from sample data, such as the sample mean. Because different samples give different values, a statistic has its own sampling distribution.

The use of statistical models to learn patterns from data for prediction or understanding. It covers regression, classification, and methods for judging model performance.

The probability that a test correctly rejects a false null hypothesis. It rises with larger samples, larger true effects, and higher significance levels.

A display that splits each value into a stem (leading digits) and a leaf (final digit). It shows the distribution's shape while keeping every original value visible.

A design that divides the population into subgroups and samples from each. It guarantees representation of every subgroup and often improves precision.

The error of studying only the cases that survived a selection process while overlooking those that dropped out. It makes outcomes look better than they truly are.

Selecting every kth member from an ordered list after a random start. It is simple but can be biased if the list has a hidden periodic pattern.

A bell-shaped distribution with heavier tails than the normal, used when the population standard deviation is estimated from a small sample. It approaches the normal as degrees of freedom increase.

A test comparing means using the t-distribution. Variants compare one sample to a value, two independent samples, or paired measurements.

A number computed from sample data that measures how far the results are from what the null hypothesis predicts. Examples include the t, z, chi-square, and F statistics.

Data recorded at regular intervals over time. Analysis separates trend, seasonality, and random variation, and accounts for the fact that neighbouring values are usually related.

A mean calculated after discarding a fixed percentage of the smallest and largest values. It keeps most of the mean's efficiency while limiting the influence of outliers.

A test that looks for a difference in either direction. It splits the significance level between both tails of the distribution.

Rejecting a null hypothesis that is actually true, also called a false positive. Its probability is the significance level.

Failing to reject a null hypothesis that is actually false, also called a false negative. Its probability is denoted β, and power equals 1 − β.

A distribution in which every value in a range is equally likely. The continuous version has a flat density between its minimum and maximum.

The average squared distance of the values from their mean. The sample version divides by n − 1 rather than n so that it is an unbiased estimate of the population variance.

A flexible distribution for times to failure whose hazard rate can rise, fall, or stay constant depending on its shape parameter. It is widely used in reliability engineering.

An average in which each value counts in proportion to a weight, such as credit hours in a grade point average. Weights must be chosen to reflect each value's importance.

A confidence interval for a proportion that performs better than the simple normal-approximation interval, especially for small samples or proportions near 0 or 1.

The number of standard deviations a value lies above or below the mean, calculated as (x − mean) ÷ standard deviation. It puts values from different scales on a common footing.