Table of Contents
- Introduction
- What Is Probability?
- Sample Spaces and Events
- Axioms of Probability
- Conditional Probability
- Bayes’ Theorem
- Independent and Dependent Events
- Random Variables: Discrete and Continuous
- Probability Distributions
- Expectation and Variance
- Common Discrete Distributions
- Common Continuous Distributions
- Law of Large Numbers
- Central Limit Theorem
- Applications in Science and Engineering
- Conclusion
1. Introduction
Probability theory provides a rigorous mathematical framework to model uncertainty and randomness. It plays a central role in physics, computer science, statistics, and engineering, allowing us to describe systems where outcomes are not deterministic.
2. What Is Probability?
Probability measures the likelihood of a given event occurring, ranging between 0 (impossible) and 1 (certain). In practice, it is used to model uncertain outcomes in experiments, physical systems, and information processes.
3. Sample Spaces and Events
- Sample space \( \Omega \): the set of all possible outcomes
- Event: a subset of the sample space
Example:
- Tossing a coin: \( \Omega = \{\text{Heads}, \text{Tails}\} \)
- Rolling a die: \( \Omega = \{1, 2, 3, 4, 5, 6\} \)
4. Axioms of Probability
- Non-negativity: \( P(A) \ge 0 \)
- Normalization: \( P(\Omega) = 1 \)
- Additivity: For disjoint events \( A \cap B = \emptyset \),
\[
P(A \cup B) = P(A) + P(B)
\]
5. Conditional Probability
The probability of \( A \) given \( B \):
\[
P(A | B) = \frac{P(A \cap B)}{P(B)}, \quad \text{if } P(B) > 0
\]
Describes dependent probabilities based on partial information.
6. Bayes’ Theorem
A fundamental result relating conditional probabilities:
\[
P(A | B) = \frac{P(B | A) P(A)}{P(B)}
\]
Used in:
- Statistical inference
- Machine learning
- Decision theory
7. Independent and Dependent Events
Events \( A \) and \( B \) are independent if:
\[
P(A \cap B) = P(A) P(B)
\]
Otherwise, they are dependent — knowing one affects the probability of the other.
8. Random Variables: Discrete and Continuous
A random variable is a function assigning numbers to outcomes in a sample space.
- Discrete: countable outcomes (e.g., coin toss)
- Continuous: uncountably many outcomes (e.g., height, time)
9. Probability Distributions
- Discrete: probability mass function (PMF): \( P(X = x) \)
- Continuous: probability density function (PDF): \( f(x) \)
Cumulative distribution function (CDF):
\[
F(x) = P(X \le x)
\]
10. Expectation and Variance
- Expected value (mean):
\[
\mathbb{E}[X] = \sum x_i P(x_i) \quad \text{(discrete)}, \quad \mathbb{E}[X] = \int x f(x) dx \quad \text{(continuous)}
\]
- Variance:
\[
\text{Var}(X) = \mathbb{E}[(X – \mu)^2] = \mathbb{E}[X^2] – (\mathbb{E}[X])^2
\]
11. Common Discrete Distributions
- Bernoulli: two outcomes (0 or 1)
- Binomial: \( n \) independent Bernoulli trials
- Geometric: trials until first success
- Poisson: rare events per unit time or space
12. Common Continuous Distributions
- Uniform: equal probability over interval
- Normal (Gaussian):
\[
f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-(x – \mu)^2 / 2\sigma^2}
\] - Exponential: time between events in Poisson process
- Gamma, Beta, Chi-square: advanced distributions in physics/statistics
13. Law of Large Numbers
As the number of trials increases:
\[
\frac{1}{n} \sum_{i=1}^n X_i \to \mathbb{E}[X] \quad \text{(in probability)}
\]
Describes long-run stability of averages.
14. Central Limit Theorem
If \( X_1, …, X_n \) are i.i.d. with mean \( \mu \) and variance \( \sigma^2 \), then:
\[
\frac{\sum X_i – n\mu}{\sqrt{n}\sigma} \to \mathcal{N}(0,1)
\]
Important result justifying normal approximations in statistics.
15. Applications in Science and Engineering
- Quantum mechanics: probabilistic nature of measurement
- Thermodynamics: statistical interpretation of entropy
- Signal processing: noise modeling
- Finance: option pricing and risk analysis
- Machine learning: probabilistic models and inference
16. Conclusion
Probability theory equips us with tools to reason about randomness, uncertainty, and complex systems. From simple experiments to stochastic processes in physics and data science, understanding these fundamentals is crucial for advanced theoretical and practical applications.
