Random Variables

So far, we have learned to describe and calculate the probability of events. But in practice, we often want to quantify the outcomes of random experiments, not just talk about whether an event happens or not. For example, we might care about “the number of heads” in a sequence of coin tosses, or “the profit” in a gambling game.

This is where random variables come in: they provide a systematic way to assign numbers to random outcomes, allowing us to model, analyze, and compute probabilities for quantitative questions. More formally we define a random variable (r.v. or RV) \(X\) as a function as follows:

where \(\Omega\) is the sample space of the random experiment and \(a\) is a real number. Importantly, for the random variable to be well defined, the function \(X\) must be measurable. In other words, for all \(a \in \mathbb{R}\), the following set must be measurable:

where measurable means that the set is in the sigma-algebra \(\mathcal{F}\) of the sample space. Because it is in the sigma-algebra, it means that we can assign a probability to the event that the random variable takes on a value less than or equal to \(a\). This is important because it allows us to use the random variable to model the outcomes of the random experiment in a probabilistic way. Here as always we need to be careful if our set of outcomes is not countable as then the sigma-algebra can get complicated and the definition of the random variable can be more complex.

Depending on the values of the random variable we can define different types of random variables. For example, if the experiment maps to a finite or countable set of values, we can define a discrete random variable. If the experiment maps to an uncountable set of values such as a real interval, we can define a continuous random variable. We will see more about this later.

Probability of Random Variables

So far, we have defined what a random variable is. Next, we’ll see how we use random variables to define probabilities and distributions. Specifically we have seen that a random variable is a function that assigns a numerical value to each outcome of a random experiment and we have seen that we can assign a probability to an event. In this section we want to combine these two concepts and assign a probability to a random variable. We have already seen that for all values \(a \in \mathbb{R}\) of a random variable \(X\) we can define the following set:

This set contains all the outcomes of the random experiment for which the random variable \(X\) takes on a value less than or equal to \(a\). Because it is a set of outcomes, i.e. an event it means we can assign a probability to it. We can do this by using the probability measure \(P\) of the sigma-algebra \(\mathcal{F}\). More specifically the probability of the random variable \(X\) taking on a value less than or equal to \(a\) is defined as follows:

To simplify the notation we usually omit the dependence on the outcome \(\omega\) and the brackets and just simply write:

This is the probability of the random variable \(X\) taking on a value less than or equal to \(a\). We can also extend this notation to the case where the random variable \(X\) takes on a value in an interval \((a, b]\):

Or also more rarely used to the case where the random variable \(X\) takes on a value in some set \(A\):

Almost Sure Events

So far we have said that an event \(A\) is a sure event if it contains all the outcomes of the sample space \(\Omega\) and therefore has the probability \(\P(A) = 1\). However, we can also define a different type of sure event called an almost sure event, commonly abbreviated as “a.s.”. An event \(A\) is called almost surely if the following holds:

Therefore an equivalent definition of an almost sure event \(A\) is also if the complementary event \(A^c\) has the following probability:

Now you might be thinking well what is the difference between a sure event and an almost sure event? The difference is that a sure event contains all the outcomes of the sample space \(\Omega\) and therefore has the probability \(\P(A) = 1\). An almost sure event on the other hand does not contain all the outcomes of the sample space \(\Omega\) but has the probability \(\P(A) = 1\). This means that an almost sure event can be a subset of the sample space \(\Omega\) but still have the probability \(\P(A) = 1\). All the outcomes that are not in the event \(A\) then just have a probability of 0. Another way you can think about it is that in an infinite sample spaces, events of probability zero can still be possible, but just extremely unlikely (e.g., picking a single real number at random from [0,1]: the chance of hitting any particular number is zero, but it’s not impossible). So picking a single outcome from an infinite sample space is almost surely not going to happen, but it is still possible.

Because of this definition on some event we can also extend the idea to random variables. We can for example say that \(X \leq a\) almost surely if the following holds:

This means that the set of outcomes for which the random variable \(X\) takes on a value less than or equal to \(a\) is almost surely. In other words, the set of outcomes for which the random variable \(X\) takes on a value less than or equal to \(a\) has the probability \(\P(X \leq a) = 1\).

The same can be done for two random variables \(X\) and \(Y\). We can say that \(X \leq Y\) almost surely if the following holds:

The definition of this set of outcomes is a bit more complex as we have two random variables. However, an example of this would be if we defined the random variable \(X\) as the outcome of a die throw and the random variable \(Y\) boolean value that is true if the die throw is even and false if the die throw is odd.

Discrete Random Variables

We have already seen that if a random variable \(X\) takes on a finite or countable set of values, we can define a discrete random variable. Common examples of discrete random variables are throwing a die, flipping a coin, or counting the number of something happening. We will go into more detail about some special discrete random variables later that have earned their own names.

Probability Mass Function (PMF)

If the random variable is discrete, so in other words the range of the random variable are countable, we can define the probability mass function short PMF, or sometimes also called the density function or just simply discrete distribution of \(X\). The PMF is a function that assigns a probability to each value of the random variable. This means that the PMF maps a value of the range of the random variable to a probability between 0 and 1. More formally we the PMF maps the values as follows:

The PMF is defined for all values in the countable range \(a \in W\) of the random variable \(X\) and is defined as follows:

where \(a\) is a value of the random variable \(X\). Sometimes if the random variable is clear from the context we just write \(p(a)\). Notice that we orignally defined the random variable \(X\) to have a range \(a \in \mathbb{R}\), but now we are defining the PMF to only assign probabilities to values in finite or countable sets \(W\). This is because the PMF is only defined for discrete random variables, so it only assigns probabilities to values that the random variable can take on. You can think of the PMF as a function that is also defiend for all values in the range of the random variable \(X\), but it only assigns a nonzero probability to values that are actually in the range of \(X\). In other words, if \(a\) is not in the range of \(X\), then \(p_X(a) = 0\).

Because the PMF is a function that assigns probabilities, it has very similar properties to a probability measure, one of which is that the sum of all probabilities is equal to 1 so we have:

Because it is discrete the graph of the PMF is a series of discrete points, this is commonly shown using a table or a bar graph.

Cumulative Distribution Function (CDF)

For any random variable \(X\) we can define the cumulative distribution function, short CDF or sometimes also called the distribution function. Again the CDF is a function that assigns a probability to the random variable \(X\) taking on a value less than or equal to \(a\). The CDF is defined for all values of \(a \in \mathbb{R}\) and is defined as follows:

Importantly the difference between the PMF and the CDF is that the CDF assigns a probability to the random variable \(X\) taking on a value less than or equal to \(a\) and not just equal to \(a\). This also means that it assigns a probability to the entire range of the random variable \(X\) and not just to a single value. More formally we can define the CDF as follows:

We will first focus on the case where the random variable is discrete. In this case we can define the CDF as follows:

Where we sum the probabilities over all values of the random variable \(X\) that are less than or equal to \(a\). Intuitively this makes sense.

We can see that the CDF is a step function that increases at the points where the random variable \(X\) takes on a value. This is because the CDF is defined as the sum of the probabilities of all values less than or equal to \(a\). You can think of the CDF as the accumulated height of the PMF.

Because it is a step function the CDF is a non-decreasing function, meaning that it can only stay the same or increase as \(a\) increases.This comes from the fact that if \(a \leq b\) then \(\{\omega \in \Omega | X(\omega) \leq a\} \subseteq \{\omega \in \Omega | X(\omega) \leq b\}\), so we have:

If we have \(a < b\) then we can also combine this to get the probability of the random variable \(X\) taking on a value in the interval \((a, b]\):

This is the so called basic identity and can directly be proven by reordering the disjoint union:

From the example above and the plot we notice that for a function to be a CDF it fulfills three properties:

1. \(F_X(a)\) is a non-decreasing function.
2. \(F_X(a)\) is a right-continuous function.
3. As the variable value \(a\) goes to infinity the CDF goes to 1 and as \(a\) goes to minus infinity the CDF goes to 0.

The first property is rather trivial and comes from the fact that probabilities are always between 0 and 1 and that we are looking at values less than or equal to \(a\).

The second property is important because it means that the CDF is continuous from the right. This means it can have jumps but that the limits from the right exist and are equal to the value of the function at that point, more formally we can say that for all \(a \in \mathbb{R}\) and \(b \geq a\) we have:

The last property in a way is like with the PMF, to guarantee that the CDF is a valid probability measure. For this to be the case the values of the CDF must be bounded between 0 and 1. This means that the CDF must go to 1 as \(a\) goes to infinity and to 0 as \(a\) goes to minus infinity. This is important so it can be used to assign probabilities to events.

Properties of Random Variables

Conditional Random Variables

Just as we can talk about the probability of an event given another event (i.e., conditional probability), we can also consider the probability distribution of a random variable given that some event has occurred. This leads us to the idea of a conditional random variable. Think of this as “restricting our attention” to a subset of the sample space: only those outcomes that make \(B\) true. We then ask: “within that world, how does \(X\) behave?”

More formally we can define a conditional random variable \(X\) given an event \(B\). The idea is that we are looking at the distribution of \(X\) but only considering the outcomes where \(B\) occurs. So given an event \(B\) with \(\P(B) > 0\), we can define the conditional probability of \(X\) taking a value \(a\), given \(B\):

Alternatively, in terms of the PMF, the conditional PMF of \(X\) given \(B\) is:

This describes the “new” probability distribution of \(X\) if we know that \(B\) occurred. You can think of this as a new random variable, but with the probability measure changed to reflect the conditioning on \(B\).

Transforming Random Variables

Often we want to study functions of random variables. This might be to rescale, shift, combine, or otherwise manipulate random variables. The resulting object is always another random variable. This is very useful for example to model functions of measurements (e.g., profit as a function of quantity sold), to construct new statistics (e.g., the sample mean is a function of several \(X\_i\) or to analyze joint behavior (e.g., sums of random variables). This becomes especially clear in the estimator section.

So suppose \(X: \Omega \to W\) is a random variable and \(g:W \to \mathbb{R}\) is a (measurable) function. Then \(Y = g(X)\) is a new random variable defined by:

The range of \(Y\) is \(g\) applied to the range of \(X\). So in other words the possible values of \(Y\) are \(g(a)\) for all \(a\) in the range of \(X\). The probability distribution of \(Y\) can be derived from the distribution of \(X\) by summing over all values of \(X\) that map to a given value of \(Y\). So for a discrete random variable \(X\) and \(Y = g(X)\), for any \(b\) in the range of \(Y\) we can define the probability mass function (PMF) of \(Y\) as follows:

The idea is that we sum over all values \(a\) in the range of \(X\) that map to the value \(b\) under the function \(g\). This gives us the total probability of \(Y\) taking on the value \(b\).

Just as we can apply a function to a single random variable to create a new one, we can also apply a function of several random variables to create a new random variable.

Suppose we have two (or more) random variables \(X\) and \(Y\) on the same probability space, and we have a function \(g: \mathbb{W}^2 \to \mathbb{R}\) (or in general \(g: \mathbb{R}^n \to \mathbb{R}\)). We define a new random variable \(Z\) by:

The possible values of \(Z\) are given by all values \(g(a, b)\) where \(a\) is in the range of \(X\) and \(b\) in the range of \(Y\). \(Z\) is a random variable because, for any \(c \in \mathbb{R}\), the set \({\omega \mid Z(\omega) \leq c} = {\omega \mid g(X(\omega), Y(\omega)) \leq c}\) can be written in terms of the events associated with \(X\) and \(Y\).

So for example, we could take:

• \(g(x, y) = x + y\) (sum)
• \(g(x, y) = \max(x, y)\) (maximum)
• \(g(x, y) = x \cdot y\) (product)
• or any other function that takes two numbers and produces another.

This construction allows us to create new random variables out of existing ones, and analyze things like the sum, difference, or maximum of two measurements.

Independent Random Variables

Just like with events, we can also define the concept of independence for random variables. We say that \(X_1, X_2, \ldots, X_n\) are independent random variables if the following holds:

This is very similar to the definition of independence for events. The only difference is that we are looking at the joint probability of the random variables taking on a value less than or equal to their respective values \(a_i\). We discuss joint distributions here in more detail.

It actually turns out that the random variables \(X_1, X_2, \ldots, X_n\) are independent if and only if for any choice of Intervals \(I_1, I_2, \ldots, I_n\) the probability that each random variable \(X_i\) takes on a value in the interval \(I_i\) is equal to the product of the probabilities of each random variable taking on a value in their respective intervals. More formally we can say that:

When reading literature on probability and statistics you will often come across the abbreviation “i.i.d.”. This stands for independent and identically distributed. This is a stricter version of independence and means that the random variables or events are independent of each other and have the same distribution. The i.i.d. assumption is often used in statistics and machine learning to simplify the analysis of data and models. More formally we can say that two random variables \(X_1\) and \(X_2\) are i.i.d. if they are independent and the following holds:

An example of this would be if we have two coins that are thrown independently of each other. We can define the random variables \(X_1\) and \(X_2\) as the outcome of the first and second coin respectively. The two coins are then i.i.d. as they are independent of each other and have the same distribution, so they are identically distributed, so they have the same probability of showing heads or tails. Having the i.i.d condition would also allow us to make the coins unfair, so for example the coin has a 60% chance of showing heads and a 40% chance of showing tails. However, both coins would need to have the same probability measure, so they would both need to have a 60% chance of showing heads and a 40% chance of showing tails, hence they are identically distributed.

Grouping Random Variables

Up to now, we’ve considered the independence of individual random variables. But sometimes, we want to combine variables into groups, apply functions to each group, and ask: are the results still independent? For instance, imagine we split a collection of independent random variables into several disjoint subgroups and define new random variables by applying (possibly different) functions to each group. Intuitively, the answer is yes and this is a powerful and frequently used property in probability theory. This property is particularly useful because it allows us to break problems into smaller, independent parts which can simplify analysis in statistics or make algorithms more efficient.

Let \(X_1, X_2, \ldots, X_n\) be independent random variables. Suppose we partition the indices \({1, \ldots, n}\) into \(k\) disjoint, non-overlapping groups:

so that every index appears in exactly one group, and the groups are disjoint. Now, let \(g\_1, \ldots, g\_k\) be arbitrary (measurable) functions, and define new random variables:

Then the new random variables \(Y_1, Y_2, \ldots, Y_k\) are independent. That is, functions of disjoint groups of independent random variables are themselves independent, no matter which (measurable) functions you use. The key intuition is that independence is preserved under grouping: If variables are independent, then anything you do to a group of them (as long as you don’t “mix” the groups) can’t introduce dependence between the groups. You can think of each group as an “island”. Whatever happens within a group is independent of what happens in the other groups. Even if you apply complicated, nonlinear, or even random functions inside the group, as long as the groups don’t overlap, the results remain independent.

Examples of Discrete Random Variables

Bernoulli Distribution

We have already seen bernoulli experiments as a special case of a random experiment. A Bernoulli experiment is an experiment that has only two possible outcomes, usually called success and failure. We can also define a random variable and therefore distribution for a Bernoulli experiment. We denote a random variable \(X\) as a Bernoulli random variable with parameter \(p\) as \(X \sim \text{Ber}(p)\), where \(p\) is the probability of success. The Bernoulli random variable takes the value 1 with probability \(p\) and the value 0 with probability \(1 - p\). In other words, we can say that a Bernoulli random variable is a discrete random variable that takes the values in \(W=\{0,1\}\) with the following probabilities:

So the PMF of a Bernoulli random variable is given by:

and the CDF is given by:

The expectation and variance of a Bernoulli random variable are given by:

The derivation of the expectation and variance can be seen in the section on expectation, variance and covariance.

Binomial Distribution

We have already seen Bernoulli experiments and the Bernoulli random variable \(X \sim \text{Ber}(p)\), which models the outcome of a single experiment with two possible results: “success” (with probability \(p\)) and “failure” (with probability \(1-p\)).

The binomial distribution extends this idea to a sequence of \(n\) independent Bernoulli experiments, each with the same success probability \(p\). The binomial random variable counts the total number of successes in these \(n\) trials.

Let \(X\) be the random variable representing the number of successes in \(n\) independent Bernoulli trials, each with success probability \(p\). We write:

where \(n\) = number of trials (fixed, known in advance) and \(p\) = probability of success on each trial. The possible values of \(X\) are \(W = {0, 1, 2, ..., n}\): from zero successes (all failures) up to \(n\) successes (all successes). So you can also think of a Bernoulli random variable as just a special case of the binomial with \(n=1\):

The probability of observing exactly \(k\) successes (\(k \in {0, 1, ..., n}\)) in \(n\) independent trials is given by:

where \(\binom{n}{k}\) is the binomial coefficient, which counts the number of different ways to choose \(k\) trials (out of \(n\)) to be successes. The intuition behind this formula is to have exactly \(k\) we need to know the number of ways to choose which \(k\) trials are successes, multiplied by the probability of those trials being successes, and the remaining trials being failures. So we get:

• \(\binom{n}{k}\) counts the number of different ways \(k\) successes could be distributed among the \(n\) trials.
• \(p^k\) is the probability that \(k\) specific trials are all successes,
• \((1-p)^{n-k}\) is the probability that the remaining \(n-k\) trials are all failures,

So in a way we choose which \(k\) trials out of \(n\) will be successes out of the \(\binom{n}{k}\) ways. Then the probability of one specific arrangement of successes and failures is given by \(p^k (1-p)^{n-k}\) (since each trial is independent) and we need to get \(k\) successes and \(n-k\) failures.

The CDF of the binomial is then:

The expectation and variance of the binomial random variable \(X \sim \text{Bin}(n, p)\) are:

This matches the intuition that for a fair coin in 10 coin flips, you expect 5 heads. The derivation of these formulas can be found in the section on expectation, variance and covariance.

The binomial distribution is also additive. So if you have two independent binomial random variables with the same success probability \(p\), you can add them together to get another binomial random variable with the sum of the number of trials. Intuitively this makes sense as all the trials are independent and the success probability is the same, so you can just combine them into one larger set of trials. So more formally if \(X \sim \text{Bin}(n_1, p)\) and \(Y \sim \text{Bin}(n_2, p)\) are independent random variables, then their sum \(Z = X + Y\) is also a binomial random variable:

Geometric Distribution

We’ve seen how the Bernoulli and Binomial distributions model the number of successes in a fixed number of independent trials. But what if we flip the question to “how many trials will it take to get the first success?” This leads us to the geometric distribution

Suppose we let \(X_1, X_2, \ldots\) be independent random variables with \(X_i \sim \mathrm{Ber}(p)\). Then let \(X\) be the random variable representing the trial number on which the first success occurs. So we define \(X\) as:

We say \(X\) is a geometric random variable with parameter \(p\):

The PMF of \(X\) is given by:

for \(k \in \mathbb{N}^+\). The intuition behind this is that think of flipping a coin over and over: what is the probability that you get your first heads exactly on the \(k\)th flip? To do this, you must have gotten \(k-1\) tails (failures) in a row, followed by a heads (success) on the \(k\)th flip. Each outcome is independent, so the probability multiplies. This is why the geometric distribution is sometimes called the “waiting time” distribution as it tells you how long you’ll wait for the first success.

You can also think of the event as follows:

where each \(X_i \sim \mathrm{Ber}(p)\) is the indicator of success on trial \(i\). Technically, it’s possible (though with probability zero) that we never see a success, i.e., \(X=+\infty\). But

so for all practical purposes, \(X\) takes finite values with probability 1. So we say it is almost surely finite.

The CDF of \(X\) is the probability that the first success happens by trial \(k\):

notice again that this uses the formula for the sum of a geometric series, this is also the reason why the geometric distribution is called “geometric” as it is related to the geometric series. We can alternatively also get the probability that you have not yet succeeded after \(n\) trials is, i.e. the survival function

This makes sense because you need \(k\) consecutive failures.

The geometric random variable \(X\) has the following expectation and variance:

Intuitively: if the chance of success is small, you expect to wait longer. If \(p = 0.2\), you expect to wait \(1/0.2 = 5\) trials for a success. If \(p = 0.5\), you expect to wait \(1/0.5 = 2\) trials. The proof and derivation for these can be found in the expectation and variance section.

One of the key properties of the geometric distribution is that it is memoryless, also known as the markov property. This means that the probability of waiting for \(m\) more trials for a success does not depend on how many trials have already been conducted. In other words, if you have already waited \(n\) trials without success, the probability of needing \(m\) more trials to get a success is the same as if you had just started fresh. So in a way as soon as a trial is conducted it “forgets” it and therefore does not change the probability of success in the next trial.Formally for all \(m, n \in \mathbb{N}\):

Poisson Distribution

So far, we’ve seen discrete random variables such as the Bernoulli, Binomial, and Geometric distributions, which model processes based on repeated, independent trials. But what if we want to model the number of times an event happens in a fixed interval of time or space, especially when the probability of an individual event is very small so it is a rare event but there are many opportunities for it to occur, so the interval is large? This leads us to the Poisson distribution.

Think about how many typos you find on a single printed page, the number of emails you receive in one hour, or how many buses arrive at a stop in 10 minutes. Each possible typo, email, or bus is a “rare” event (unlikely per opportunity), but there are lots of chances for it to occur.

The Poisson distribution answers the question: If events happen at an average rate \(\lambda\) per interval (time, space, etc.), what is the probability that we observe exactly \(k\) events in that interval? Formally, we write:

where \(\lambda > 0\) is the rate parameter: the expected number of occurrences in the interval. The PMF of the Poisson distribution is given by:

where \(k \in \mathbb{N}_0 = {0, 1, 2, \ldots}\) is the number of occurrences.

This formula comes from the following idea. We use the poisson distribution as follows. Suppose we want to model how many events happen in a fixed interval (say 1 hour), where each possible opportunity for an event (say each minute) has a small probability \(p\) of an event. Then we have a large number \(n\) of such opportunities (n=60 for 1 hour). The average total number of events we expect in the interval is the rate parameter \(\lambda = np\) (so as \(n\) increases, \(p\) decreases). Notice this is exactly the same as the binomial distribution, where we have \(n\) independent trials with success probability \(p\) which is \(\lambda/n\) in this case. The idea is now that we can think of the Poisson distribution as a limit of the binomial distribution as we split the interval to infinitely many small intervals. Because as \(n\) increases, the probability of an event in each small interval becomes very small, but the total number of intervals increases, so we can still expect a fixed number of events \(\lambda\) in the whole interval.

So let \(X_n \sim \mathrm{Bin}(n, \frac{\lambda}{n})\) be the binomial random variable counting the number of successes in \(n\) independent trials, each with success probability \(\frac{\lambda}{n}\) such that over the whole interval we expect \(\lambda\) successes. Then as \(n \to \infty\) we get the PMF of the Poisson distribution:

So we have:

As mentioned you can think of the Poisson distribution as a limit of the binomial distribution as \(n \to \infty\) and \(p = \frac{\lambda}{n} \to 0\), while keeping \(\lambda = np\) constant. The binomial distribution was additive, meaning that if you have two independent binomial random variables with the same success probability \(p\), you can add them together to get another binomial random variable with the sum of the number of trials. The Poisson distribution is also additive, but in a different way. Suppose we observe two independent Poisson processes such as the number of emails from Alice (\(X \sim \mathrm{Poi}(\lambda_1)\)) and from Bob (\(Y \sim \mathrm{Poi}(\lambda_2)\)) in an hour. If we then wanted to know the total number of emails received in that hour, we can add the two independent Poisson random variables together:

Intuitively this makes sense as over the interval we have two independent sources of events, each with their own rate, and the total number of events is just the sum of the two independent sources.

Negative Binomial Distribution

Hypergeometric Distribution

Die Hypergeometrische Verteilung der Zufallsvariable \(X\) ist die Verteilung, die beim \(n\)-maligen Ziehen ohne Zurücklegen und ohne Reihenfolge aus einer Urne mit \(N\) Kugeln, von denen \(M\) eine spezielle Eigenschaft haben und wo die Anzahl der gezogenen Kugeln mit dieser speziellen Eigenschaft gezählt werden.

• Wir schreiben dann \(X \sim Hyp(N,M,n)\)
• Die Dichtefunktion von \(X\) ist \(f(k)=\binom{M}{k} \cdot \frac{\binom{N-M}{n-k}}{ \binom{N}{n}}\) wobei \(N\) die Gesamtanzahl der Kugeln ist, \(M\) die Anzahl mit der speziellen Eigenschaft. \(n\) ist dann der Umfang der Stichprobe also die Anzahl der entnommenen Kugeln und \(k\) die Anzahl angestrebte Kugeln mit der speziellen Eigenschaft.
• \(E(X) = n \cdot \frac{M}{N}\)
• \(V(X) = n \cdot \frac{M}{N} \cdot (1 - \frac{M}{N}) \cdot \frac{N-n}{N-1}\)

In Matlab haben wir die Funktionen:

• Dichtefunktion: \(hygepdf(k,N,M,n)\)
• Verteilungsfunktion \(hygecdf(k,N,M,n)\)

mehr dazu findest du hier .

Coupon Collector Problem

Continuous Random Variables

Recall that a random variable is a function that assigns a real number to each outcome in a sample space. We can have two types of random variables: discrete and continuous. In the discrete case the image of the random variable is a countable set, while in the continuous case it is an uncountable set (like an interval on the real line).

We also defined that a cumulative distribution function (CDF) \(F_X\) of a random variable \(X\) is defined as:

where \(a\) is a real number and the CDF \(F_X\) is left continuous. The probabilites of a discrete random variable are given by the probability mass function (PMF) \(p_X(a)\) which are non-zero at the countable set of points where the cdf has jumps. For a continuous random variable the CDF is continous so not just left continuous but also right continuous. Because the CDF is continous there exists a PDF \(f\_X\) such that:

and the probabilities are given by the probability density function (PDF) \(f_X(a)\) which is non-negative and integrates to 1 over the whole space.

By the fundamental theorem of calculus, we can also go from the PDF to the CDF:

We can also define the probability of a single point for a random variable \(X\) at a point \(a\) as:

In the case of a continuous random variable, this probability is always zero because the CDF is continuous at every point, meaning there are no jumps. In the case of a discrete random variable, this probability can be non-zero if there is a jump in the CDF at that point. This also becomes clear when you look at the following integral:

The intuition behind this is imagine we have the interval between \(a\) and \(a + h\), then the probability of the random variable being exactly at \(a\) is 0 because inbetween \(a\) and \(a + h\) there are infinitely many points, so the probability of hitting exactly one point is 0. This is why we say that for continuous random variables, the probability of a single point is always 0.

If we are given just the PDF \(f_X(a)\), we can also define the CDF by taking the integral of the PDF:

We can also define the survival function \(S_X(a)\) which is the probability that the random variable \(X\) is greater than \(a\):

Alternatively, we can also go from the CDF to the PDF by taking the derivative in general:

Examples of Continuous Random Variables

Uniform Distribution

We have already seen the discrete uniform distribution as a model where all outcomes are equally likely among a finite number of values, such as when throwing a fair die. We call this a laplace space.

We now extend this idea to the continuous case, where the random variable takes values in a real interval. For example, a time chosen uniformly at random in the interval \([a, b]\). This gives us the continuous uniform distribution.

The intuition is the same, each subinterval of equal length within \([a, b]\) is equally likely to contain the outcome. So, we are modeling complete uncertainty over an interval. We know the outcome lies somewhere in \([a, b]\), but we assume no preference for any subinterval over another. The resulting distribution is flat, or uniform, over the interval.

Let \(X\) be a continuous random variable defined on the interval \([a, b]\). Then \(X\) is said to have a uniform distribution on \([a, b]\) if its probability density function (PDF) is given by:

We write:

The PDF is zero outside \([a, b]\), meaning the random variable cannot take values outside this range.

The PDF is constant over the interval \([a, b]\), reflecting the fact that all values in this range are equally likely (in terms of density). However, since the distribution is continuous, the probability of hitting any exact point \(x\) is still zero:

So instead we compute probabilities over intervals using the cumulative distribution function. The CDF \(F_X(x)\) for a uniform distribution on \([a, b]\) is obtained by integrating the PDF:

This function smoothly increases from 0 to 1 as \(x\) goes from \(a\) to \(b\), and is continuous and strictly increasing on \([a, b]\). It is flat (zero) before \(a\), and reaches 1 after \(b\).

For continuous random variables, we compute probabilities by integrating the PDF or using the CDF. For the uniform distribution, we get fo any interval \([c, d]\) within \([a, b]\):

So the probability is proportional to the length of the interval \([c, d]\) with respect to the total length of \([a, b]\). This reflects the uniform nature of the distribution: longer intervals have higher probabilities. We can also use the CDF to compute intervals:

The expectation of a uniform random variable \(X \sim \text{Uniform}(a, b)\) is simply the midpoint:

and the variance is given by:

Exponential Distribution

We’ve seen how the geometric distribution models the number of Bernoulli trials until the first success, and how the Poisson distribution models the number of events in a fixed interval, given a constant event rate. Now what if, instead of counting the number of events, we want to model how long we wait until the first event occurs in a continuous-time process? This leads us to the exponential distribution.

Suppose we have a Poisson process, so events happen randomly, independently, and at a constant average rate in time (for example, arrivals of buses, radioactive decays, phone calls, or failures of light bulbs). If we fix a starting point, the time we must wait until the next event is an exponentially distributed random variable. So the exponential is to the geometric what the Poisson is to the binomial.

Let \(X\) be a continuous random variable modeling the waiting time until the first event in a process where events occur at a constant average rate \(\lambda > 0\) per unit time. So for example if \(\lambda = 2\), we expect 2 events per interval. We write:

The probability density function (PDF) is:

Where the rate parameter \(\lambda > 0\) controls the average waiting time until the first event occurs. And the support of \(X\) is \(x \in [0, \infty)\) as we can’t have a negative waiting time but we can have a waiting time of 0 (the event happens immediately) or infinite (the event never happens).

Intuitively, the chance you’re still waiting after some time \(t\) decays exponentially fast as \(t\) increases. This is the origin of the name. This follows from the poisson distribution where the PDF for a poisson random variable \(X\) is given by:

Where \(k\) is the number of events that have happened in the interval. Then the probability that no event has happened in the interval is given by:

This leads us to the PDF of the exponential distribution as the first event happening after some interval is the same as no events happening in that interval, so we can think of the exponential distribution as the time until the first event in a Poisson process.

The CDF gives the probability that the event has occurred by time \(x\). Importantly for \(x < 0\), the CDF is \(F_X(x) = 0\) since we can’t have a negative waiting time. For \(x \geq 0\), we compute the CDF as follows:

As \(x \to \infty\), \(F_X(x) \to 1\) the event will eventually occur almost surely, with probability 1:

For any \(0 \leq a < b\),

So, the probability the waiting time is between \(a\) and \(b\) is simply the difference between the survival probabilities at those two times.

Just like the geometric distribution, the exponential distribution is memoryless. This means that the probability of waiting an additional \(t\) units of time, given that you have already waited \(s\) units, is the same as if you had just started waiting. In other words, the process does not “remember” how long you have already waited. Formally, for all \(s, t \geq 0\):

Intuitively you can think of this as saying that the waiting time until the next event is independent of how long you have already waited as you don’t know when the last event happened, so you can only expect to wait the same amount of time as you would if you just started waiting.

The expectation and variance of an exponentially distributed random variable \(X \sim \mathrm{Exp}(\lambda)\) are given by:

Normal Distribution

One of the most important continuous random variables in probability theory and statistics is the normal distribution (also called the Gaussian distribution, named after Carl Friedrich Gauss who defined the standardized form). Because of the shape of the function graph it is often also referred to as the bell curve.

The normal distribution plays a central role in statistics and probability. It is the chosen model for a huge variety of real-world phenomena, especially when these phenomena result from the accumulation of many small, independent effects such as physical measurements, noise, heights, test scores, and many natural quantities. The reason why it is so prevalent is due to the Central Limit Theorem, which states that the sum (or average) of a large number of independent and identically distributed random variables will be approximately normally distributed, regardless of the original distribution of the variables. This is a fundamental result in probability theory and underpins many statistical methods.

Let’s now formally define the normal distribution. The random variable \(X\) is said to have a normal distribution with mean \(\mu\) and variance \(\sigma^2\) denoted as:

if its probability density function (PDF) is given by:

The expectation and variance of a normal random variable are the parameters \(\mu\) and \(\sigma^2\) respectively:

So the distribution is completely characterized by its mean and variance. From the graph and definition we can see that the normal distribution is symmetric around the mean \(\mu\), and the spread of the distribution is controlled by the standard deviation \(\sigma\). The PDF is bell-shaped, with the highest point at \(x = \mu\) and tails that approach zero as \(x\) moves away from \(\mu\). Unlike other distributions, the normal distribution is defined for all real numbers, meaning it has infinite support. The interpretation of the graph is if you measure the same physical quantity repeatedly, the errors you get will, under mild conditions, look normal—most values are close to the “true” mean, with fewer large deviations, and very rare extreme deviations. Intuitively as the variance \(\sigma^2\) increases, the distribution becomes wider and flatter, while as \(\sigma^2\) decreases, it becomes narrower and taller as the values cluster more closely around the mean \(\mu\) by the definition of variance.

The cumulative distribution function (CDF) of a normal random variable \(X \sim \mathcal{N}(\mu, \sigma^2)\) is given by:

There is no closed form for this integral in terms of elementary functions, but it is tabulated and implemented in all statistical packages.

Another reason why the normal distribution is so common is that it is closed under linear transformations. This means that if you apply a linear transformation to a normally distributed random variable, the result is still normally distributed. So if \(X \sim \mathcal{N}(\mu, \sigma^2)\), then for any constant \(c\):

So adding a constant shifts the mean, but does not affect the variance. This is intuitive if you think of just shifting the entire distribution left or right by \(c\) as the PDF is just the PDF of \(X\) shifted by \(c\).

We can also scale the normal distribution. This means multiplying the random variable by a constant \(a\). If \(X \sim \mathcal{N}(\mu, \sigma^2)\) and \(a \in \mathbb{R}\):

That is, scaling by \(a\) multiplies the mean by \(a\) and the variance by \(a^2\).

We have seen for the binomial and poisson distributions that we can add independent random variables together, and the result is also a random variable of the same type. The same holds for the normal distribution. Specifically if \(X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2)\) and \(X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2)\) are independent, then:

That is, the sum of independent normals is again normal, with the mean and variances adding.

Combining the above properties we can see that any linear combination of independent normal random variables is again normal:

If \(X_1, \ldots, X_n\) are independent, and \(Y = c_0 + \lambda_1 X_1 + \cdots + \lambda_n X_n\):

This is the stability property of the normal distribution, and is one of the reasons for its fundamental importance.

Standard Normal Distribution

Because we can scale and shift the normal distribution, we can always reduce any normal distribution to the standard normal distribution. This is a special case of the normal distribution where the mean is 0 and the variance is 1, often denoted as \(Z\):

We standardize it using the so-called z-score transformation:

This transformation centers the distribution at 0 and rescales the variance to 1. By removing the mean we center the distribution around 0, and by dividing by the standard deviation we ensure that the spread of the distribution is 1 because the variance of \(Z\) is:

The result is the standard normal distribution. The probability density function (PDF) of the standard normal distribution is denoted as \(\varphi(z)\) and is given by:

The cumulative distribution function (CDF) is denoted as \(\Phi(z)\):

In practice, any normal probability can be reduced to a standard normal calculation using this standardization. That is,

This is actually the key to using the normal distribution in practice: we can always convert any normal random variable to a standard normal variable, compute probabilities using the standard normal CDF, and then interpret those results in terms of the original variable.

Quantiles of the Normal Distribution

Using the CDF, we can find the probability that a random variable \(X\) takes on a value less than or equal to \(a\). The quantile function (or inverse CDF) allows us to do the opposite: given a probability \(a\), it gives us the value \(c\) such that:

This is written as:

What is the derivation of this? Because it is symmetric?

where \(\Phi^{-1}\) is the inverse of the standard normal CDF. For the standard normal distribution with mean \(\mu = 0\) and standard deviation \(\sigma = 1\), the quantile function is simply:

For example for the median (50th percentile), we have:

Quantiles are important for all distributions, but the normal distribution is so central to statistics where we often want to find a threshold for some sort of percentage such as for hypothesis tests or confidence intervals. They are also useful for setting cutoffs, such as in outlier detection or acceptance intervals.

Three Sigma Rule

One of the most important practical properties of the normal distribution is the three sigma rule:

• About 68% of values lie within 1 standard deviation (\(\mu \pm \sigma\))
• About 95% within 2 standard deviations (\(\mu \pm 2\sigma\))
• About 99.7% within 3 standard deviations (\(\mu \pm 3\sigma\))

Formally,

for \(k = 1, 2, 3\).

For example, for \(k = 3\):

So, the probability of being “more than three sigmas away” is about 0.27%, which is why “three sigma events” are considered extremely rare. This fact is used, for example, in quality control, where measurements outside \(\mu \pm 3\sigma\) are flagged as anomalies.

Surprisingly, this rule does not depend on the mean \(\mu\) or variance \(\sigma^2\) as can be seen from the formula above. This is a consequence of the symmetry and shape of the normal distribution, which is why it is so widely used in statistics.

Constructing Random Variables

Up to now, we have talked about random variables as functions defined on some probability space and described their distribution via the cumulative distribution function (CDF), \(F_X(a) = \P(X \leq a)\). But you might wonder: given any function \(F: \mathbb{R} \to [0,1]\) that looks like a “CDF”, does there actually exist a random variable \(X\) (possibly on some probability space) whose distribution function is exactly \(F\)? And how could we construct such an \(X\) in practice or in simulations?

This question lies at the heart of probability theory: it tells us that we can focus on the distribution of random variables—meaning their PMF, PDF, or CDF, without needing to worry about the detailed structure of the underlying probability space.

Recall that for a function \(F: \mathbb{R} \to [0,1]\) to be the CDF of some random variable, it must satisfy three properties:

1. Non-decreasing: \(F(a_1) \leq F(a_2)\) whenever \(a_1 < a_2\).
2. Right-continuity: For all \(a\), \(\lim_{x \to a^+} F(x) = F(a)\).
3. Limits at infinity: \(\lim_{a \to -\infty} F(a) = 0\) and \(\lim_{a \to +\infty} F(a) = 1\).

These are exactly the properties that arise from the definitions and basic results about probability measures.

The question now is for any function \(F\) with these properties, does there exist a random variable \(X\) such that \(F_X(a) = F(a)\) for all \(a\)? The answer is yes. For any function \(F\) with these properties, there exists a probability space \((\Omega, \mathcal{F}, \P)\) and a random variable \(X: \Omega \to \mathbb{R}\) with \(F_X(a) = \P(X \leq a) = F(a)\) for all \(a\).

This is often called the Kolmogorov existence theorem. In practice, this means that when defining or analyzing a random variable, we can simply specify its distribution function (or PMF or PDF), and be assured that a corresponding random variable does exist.

This is useful because in applications (especially in statistics and simulation), we then do not need to describe the probability space \((\Omega, \mathcal{F}, \P)\) explicitly. Instead, we work directly with the distributions of random variables.

The idea of kolmogorov’s theorem is as follows. Consider the “canonical” probability space: the infinite product space \(\Omega = {0,1}^{\mathbb{N}}\), i.e., all infinite sequences of independent coin flips (where 0 = tails, 1 = heads) and the probability measure \(\P\) is defined so that the coordinates are i.i.d. Bernoulli(1/2) random variables. We can now use such sequences to construct a wide variety of random variables, in particular the uniform distribution on \([0,1]\) which, in turn, can be used to generate any other desired distribution.

Let’s generate a uniform random variable on \([0,1]\) using this framework, and then show how to use it to construct any other distribution. Suppose \(X_1, X_2, \dots\) are i.i.d. Bernoulli(1/2) random variables (i.e., independent fair coin flips):

You can think of this as a binary expansion of a number in the interval \([0,1]\). Each \(X_n\) contributes either \(0\) or \(\frac{1}{2^n}\) to the sum, depending on whether the \(n\)-th coin flip is tails (0) or heads (1) which corresponds to the binary digit at position \(n\) in the expansion. So for each \(\omega \in \Omega\), this sum converges to a real number in \([0,1]\), whose binary expansion is given by the sequence \(X_1, X_2, \dots\). So each binary digit is determined by a fair coin, and the resulting distribution is uniform.

Now, suppose we want to construct a random variable \(X\) with any given distribution function \(F\) (satisfying the three properties above). The key idea is to use inverse transform sampling.

Let \(U\) be a uniform random variable on \([0,1]\). Define

where \(F^{-1}\) is the generalized inverse (sometimes called the quantile function). We know this inverse exists because \(F\) is non-decreasing and continuous. The generalized inverse is defined as:

Then \(X\) has distribution function \(F\):

This construction works for any CDF, whether \(F\) is continuous or has jumps (discrete points). If \(F\) is strictly increasing and continuous, \(F^{-1}\) is the usual inverse function. The idea is that we are transforming a uniform random variable into one with the desired distribution by mapping the uniform probabilities to the quantiles of the target distribution.