Epistemological features of statistical inference

1.4.1 Inductive inference

A central question in scientifi c inference is how is it possible to draw conclu-sions of something that we are not capable of observing directly or completely.

In order to obtain knowledge and understanding about the surrounding world it is necessary to have both methods to acquire data and methods to reveal particulars and relations between the observed facts to establish generalisations and theories. A central part of scientifi c activity, or the pursuit of knowledge in general, is the logic by which investigators end up with conclusions from obser-vations, experiments, and initial premises.

The two main methods of scientifi c inference are called deduction and induc-tion. In some respect, they can be regarded as opposites: deduction goes from general to specifi c, and induction goes from specifi c to general. Induction is an ar-gument or theory starting with empirical observations and leading to a conclusion, while deduction goes in the opposite direction, from theory to observation.

Deduction is an old method to draw conclusions from given premises, pos-tulated already by Aristotle. The power of deductive inference comes from the fact that from true premises, correctly deduced conclusions are necessarily true.

A classic example of the competence of deduction is Euclidian geometry, where the whole system is deduced from a few axioms. The growth of mathematical theories in general, including the probability theory, is an example of the capa-bility of deductive reasoning.

In scientifi c experimentation, the so-called hypothetico-deductive method is frequently applied. Schematically, the method works as follows: From a general hypothesis and particular statements of initial conditions, a particular predictive statement is deduced. The statements of initial conditions, at least for the time, are accepted as true; the hypothesis is the statement whose truth is at issue.

By observation or experiment, we determine whether the predictive statement turned out to be true. If the predictive consequence is false, the hypothesis is disconfi rmed. If the observation reveals that the predictive statement is true, we say that the hypothesis is confi rmed. The design of a scientifi c experiment aims at creating such an experimental setup that the deductive procedure could be applied to draw conclusions.

In empirical research, deductive inference is not suffi cient. Francis Bacon⁴ recognized that the scientifi c method embodies a logic essentially different from that of Aristotle. Bacon commended the method of careful observation and experimentation. He put forward that scientifi c knowledge must somehow be built on inductive generalisation from experience.

A simple example of inductive inference is the following: if we draw balls from an urn and we only have white balls, we tend to infer that all balls in the urn are white. Every new observation of a white ball strengthens our conviction on the rule (that all balls in the urn are white), but we can never be absolutely sure. On the other hand, a single observation of a black ball ruins the rule. In-duction is said to be ampliative and undemonstrative. That is, it expands the observations to a wider domain than what was originally observed, but inductive inference cannot demonstrate that a rule is true.

More than a century after Bacon’s works, David Hume⁵ published a book in which he criticised the principle of inductive inference. His critique began with a simple question: How do we acquire knowledge about the unobserved? (Hume 1739 and 1748) Hume’s basic problem can be described as follows: Given that all the balls that were drawn from an urn have been white so far, and given that the conclusion has been entertained that the unobserved balls are also white, do the observed facts make up sound evidence for that conclusion? Basically, the problem of induction is a problem of explaining the concept of evidence.

Hume’s answer was sceptical. It is out of the scope of this study to deal comprehensively with this question, but several authors, for example, Salmon (1967), have analysed it thoroughly. Hacking (1975) treated Hume’s philosophy in the context of probability theory. In addition, Chatterjee (2003) has analysed profoundly Hume’s philosophy in relation to statistical inference.

Hume’s critique essentially rested on his attack on the principle of the uniformi-ty of nature. It is obvious that inductive inferences cannot be expected to yield cor-rect predictions if nature is not uniform. For example, if we do not know whether the future will be like the past, it is not possible know which facts will hold. Like-wise, if it is not believed that a population under study is uniform or stable in all of its parts, it is not feasible to generalize the results obtained from a sample.

4 Francis Bacon (1561–1626) was an English politician and philosopher. He put forth the view that only through reason are people able to understand and have control over the laws of nature. His famous adage, ‘Knowledge is power’, refl ects this conception. Francis Bacon’s infl uence on empirical research has been so strong that he has been called “the Father of Modern Science”.

5 David Hume (1711–1776) was a Scottish philosopher and historian who has been regar-ded as the founder of the sceptical, or agnostic, school of philosophy. He had a profound infl uence on European intellectual life.

Hume’s problem has been approached from many points of view. An ex-ample is the so-called induction by enumeration. Suppose that a coin has been thrown a large number of times. Given that m/n of observed throws has been heads, we infer that the “long run” relative frequency of heads is m/n. It is obvi-ous that induction by enumeration is closely related to the long-run frequency interpretation of probability.

Another, slightly different, example was given by Laplace at the end of the 18^th century. He posed the question: how certain can we be that the sun will rise tomorrow, given that we know that it has risen every day for the past 5,000 years (1,825,000 days). One can be pretty sure that it will rise, but we cannot be absolutely sure. In response to this question, Laplace proposed the Law of Suc-cession. In its simplest form, it means the following: If we have had x successes in n trials and ask what is the probability of success in the next trial, we add one to the numerator and two to the denominator ((x + 1)/(n + 2)) (see Chapter 4, Formula 4.7). Applying this procedure, one could be 99.999945% sure that the sun will rise tomorrow.

Induction by enumeration and hypothetico-deductive method are inherently different approaches. Induction by enumeration actually consists in simple in-ductive generalisations from instances, and the hypothetico-dein-ductive method is in contrast to it. The hypothetico-deductive method aims at confi rming or disconfi rming hypotheses derived from previous knowledge, while induction by enumeration aims at deriving scientifi c hypotheses.

An answer to Hume’s critique is that inductive conclusions are probabilistic, not absolutely certain. An inductive inference with true premises only establish-es its conclusions as probable. At the time when Hume published his critique, mathematicians dealt only with the problems of direct probability. The critique gradually initiated development of the methods for the calculation of inverse probability to address the problems of induction.

Inverse probability and statistical inference can be seen as a formal approach to apply induction in empirical research. Inverse probability in statistical science involves two problems: the problems of direct probability are mathematical and hence involve deductive inference; in inverse probability, known probability dis-tributions are applied to make inferences about the unobserved part of nature and it is inherently inductive. Statistical inference in the modern sense can be seen as an outgrowth of inverse probability.

The American mathematician, C.S. Peirce, defi ned induction to be “reason-ing from sample taken at random to the whole lot sampled” (see Stigler 1978, p. 247).

The famous Theorem of Thomas Bayes (Bayes 1763) is often regarded as the fi rst method to calculate inverse probability (see Chapter 3). However, Laplace gave the fi rst precise formulation of inverse probability in a careful scientifi c context in a mémoire in 1774. Laplace’s contributions on inverse probability are analysed in Chapter 4.

1.4.2 The inference model

It is obvious that probability and probability models play a central part in induc-tive inference. A probability model can be seen as an abstract description of mass events in the real world by which one is able to predict the frequency of future events and to analyze observations from such events, but probability models cannot be applied directly in inductive inference.

In direct probability, it is generally conceded that knowing the value of a sto-chastic probability factor, say s, the probability for an arbitrary or ‘random’ oc-currence of a chance event can be determined, like in coin-fl ipping, dice-rolling, or selecting balls from an urn.

In inverse probability, the question is reversed: Given the outcome of an ex-periment or observations, what can be concluded about the underlying causes of outcomes and their stochastic characteristics? Obtaining an answer to the ques-tion requires the use of direct probability in one way or another.

Thought experiment

Abstract probability models cannot be applied directly in real world phenomena because the situations to be analysed are much too diverse and usually too com-plex. The inference model requires an intermediate model, a thought model, which links an abstract probability model to the real-world phenomenon. Char-acteristic of a thought experiment is that it involves such a setup that can be (or could be) tested experimentally if necessary.

One of the oldest thought experiments is the so-called urn problem or urn trial. The urn problems have been a part of probability theory since at least the publication of the Ars conjectandi by Jakob Bernoulli in the beginning of the 18^th century (see later). Bernoulli considered the problem of determining from a number of pebbles drawn from an urn the proportions of different colours. The urn trial is often called a Bernoulli trial or Bernoulli experiment.

In an urn trial, an urn is thought to contain n balls (or tickets), x white and y black. One ball is drawn randomly from the urn and its colour is observed. It is then placed back in the urn⁶, and the selection process is repeated. Occasion-ally, the two possible outcomes are called “success” and “failure”. For example, a white ball may be called “success” and a black ball “failure”. The urn trial induces a Binomial Distribution.

Another example of an inference model is the one which Thomas Bayes’ ap-plied in formulating his theorem: the model was based on the positions of balls on a (billiard) table after they were rolled on it (see Chapter 3).

R.A. Fisher introduced a new inference model in the 1920s. Its central idea is to repeatedly draw samples from the same known probability distribution (see Chapter 9). Fisher’s thought model is still the predominating one in statistical inference. In the 1930s, Jerzy Neyman adapted it in a modifi ed form to fi nite

6 In the setup with the replacement of balls, the subsequent drawings are independent. In ano-ther setup, the urn is assumed to contain an infi nite number of balls and then the drawings can also be regarded as independent.

population inference (see Chapter 10). Neyman’s idea of drawing samples re-peatedly from the fi nite population is the core of modern sampling theory.

Thinking models are also applied in a wider scope. A common thinking model up to the 20^th century originated from the planetary system, which was also incorporated into social research. A parameter describing a state of popula-tion was paralleled with a planet and its posipopula-tion. Measurements gave varying results so that observations had a distribution around the true value. In addi-tion, the planet was moving all the time, and therefore its position could not be considered to be constant. The resulting uncertainty was described by a priori probability. In social research, this idea led to thinking that a society should be approached as a constantly changing universe, a superpopulation, and every observable population was a realization of some phase of the superpopulation.

The world view behind these thought models was mechanistic, comprising of distinct units, and often a Greater Cause was assumed to act behind the events.

This originated from Newton’s philosophy, and it dominated thinking until the beginning of the 21^st century.