A quantitative study of hypothesis selection

2 The Hypothesis Selection Problem
Suppose we are facing k alternative hypotheses. Each could be tested against a xed environment. Every such test returns a numerical measure of how `good' the hypothesis is. We call this numerical measure the utility of the hypothesis. Since the environment could be uncertain, the numerical utility of a hypothesis returned by the environment is in fact a random variable. A learning agent is free to sample the utility of each hypothesis in whatever order it prefers. Our goal is to decide which hypothesis has the highest average utility ( gure 1). This so called k-armed bandit problem has fascinated researchers of statistics (Berry & Fristedt 1985) and reinforcement learning (Kaelbling 1993) for years. Readers will nd no di culty imagining the usefulness of k-armed bandit problem in factory quality control, in experimental design, and in clinical treatment research. In fact, this problem is intimately related to many learning problems in arti cial intelligence, and any algorithmic solution to the problem
H1
H2
Байду номын сангаас
H3
Hk
Figure 1: The k-Armed Bandit Problem tation. We propose a novel family of sampling strategies called the -IE strategies, in which every unit of exploitation tendency is balanced out by unit of exploration tendency. The behaviour of a -IE strategy could be summarized by the formula: (exploitation) + (exploration) We establish the worst case sample complexity for the whole family of -IE strategy. Experimental results also suggest that no single member of the -IE family is cost-optimal in all cases. In fact, the best tradeo between exploration and exploitation is a function of the problem domain. It is observed that, due to the use of an exclusively exploration-oriented sampling strategy, utility analysis in the class of PALO speed-up learners (Greiner 1992) could be unnecessarily costly. Our sampling complexity results imply that more sophisticated strategies like -IE can actually be integrated into the PALO framework without sacri cing its theoretical properties.
performance history (Greiner & Jurisica 1992, Gratch & DeJong 1992). Inductive learning involves the selection of a hypothesis that best ts a given set of data (Musick et al 1993). Reinforcement learning selects a policy that returns best payo (Kaelbling 1993). Automatic improvement of heuristic estimation can be viewed as the selection of a competent heuristic function among a pool of alternatives (Yakowitz & Lugosi 1990). Even the study of representational shifts could be formulated as the selection of an optimal representation scheme (Greiner & Elkan 1991, Greiner 1992). Recent works in diagnosis (Benjamins & Jansweijer 1994) and probably approximate planning (Yang & Fong 1995) are also concerned with hypothesis selection. Here, we are not interested in how hypotheses are formed. Instead, we are interested in how a near optimal hypothesis can be selected from a pool of given alternatives. In addition, the only information that we can base our decision on is the actual experimentation of the hypotheses in an uncertain environment. This archetypal hypothesis selection task is called the k-armed bandit problem (Berry & Fristedt 1985). One is interested in knowing how the experimentation cost can be minimized while still guaranteeing the optimality of the selected hypothesis. This paper presents theoretical and empirical studies of a particular family of hypothesis selection algorithms under a probably approximately optimal (PAO) learning framework (Greiner & Orponen 1991). Hypothesis selection algorithms di er by their choice of sampling strategies. Sampling strategies could be exploration-oriented or exploitation-oriented. Exploration-oriented strategies are very eager to explore unfamiliar hypotheses. Exploitation-oriented strategies are more biased towards to the hypotheses with high utility. We argue that both the exploration and exploitation element of a hypothesis selection algorithm are important in reducing the cost of experimen-
Abstract
The hypothesis selection problem (or the karmed bandit problem) is central to the realization of many learning systems. This paper studies the minimization of sampling cost in hypothesis selection under a probably approximately optimal (PAO) learning framework. Hypothesis selection algorithms could be exploration-oriented or exploitationoriented. Exploration-oriented algorithms tend to explore unfamiliar alternatives eagerly, while exploitation-oriented algorithms focus their sampling e ort to the hypotheses which yield higher utility in the past. Both the exploration and exploitation element of a hypothesis selection algorithm could be useful in reducing sampling cost. We propose a novel family of learning algorithms, the -IE family, that explicitly trade o their exploration tendency with exploitation tendency. We establish the sample complexity for the entire -IE family. We empirically show that none of the algorithms in this family are costoptimal for all problems. In addition, our novel parameterization of the family allows users to select the instantiation that best ts his or her application. Our results also imply that the PALO class of speed-up learners can retain their theoretical properties even when a more sophisticated sampling strategy is used.