直接搜索算法

Administration under NASA Contract No.NAS1-97046.
2
This research was supported By the National Science Foundation under
Grant CCR-9734044 and by the National Aeronautics and Space Administration under NASA Contract No. NASI-97046, while the author was in residence at the Institute for Computer Applications in Science and Engineering (ICASE).
implies our preference, based on experience, for straightforward search strategies which employ no techniques of classical analysis except where there is a demonstrable advantage in doing so.
罗伯特·胡克和 T.A.捷吾斯首先在 1961 年的计算机械协会期刊上 的一篇论文上提出“直接搜索”[12].在他们的论文的引言中这样 描述直接搜索： Robert Hooke and T.A. Jeeves coined the phrase “direct search” in a paper that appeared in 1961 in the journal of the Association of Computing Machinery [12]. They Provided the following description of direct search in the introduction to their paper:
23681-2199. USA
b
Department of Computer Science, College of William & Mary187-8795, USA
c
Department of Mathematics, College of William & Mary, P.O. Box 8795,
Williamsburg, VA 23187-8795, USA Received 1 July 1999; received in revised form 23 February 2000 Abstract 摘要 我们讨论无约束优化的直接搜索法。我们从现在的观点来看这类与导 数无关的算法， 主要集中在 1960 到 1971 年的直接搜索法发展的黄金时期。我们首先 讨论在未构建目标模型的情况怎样使用直接搜索法。然后我们考虑一 些经典直接搜索法并揭示那些年这类算法的进展。特别地，当原始直 接搜索法开始直接利用启发式方法时，更多近来的分析表明，虽然不 是全部但大部分启发式方法实际上已经足可以保证迭代序列中至少有 一个子序列全局收敛到目标函数的一阶驻点。 关键词：求导无关优化；直接搜索法；模式搜索法 We discuss direct search methods for unconstrained optimization. We give a modern
Direct search methods:then and now 直接搜索法：历史和现状 Robert Michael Lewis a ,1 ,Virginia Torezon a,b,*,2 ,Michael W. Trosset c ,a
a
ICASE, Mail Stop 132C, NASA Langley Research Center, Hampton,VA
So in [96], this preference on the part of Hooke and Jeeves was now without justification. 四十年以后现在的问题是：为什么直接搜索法仍然在使用呢？的 确这些没有理论证明的基于启发式方法的杂乱的方法类应该被现代数 值优化方法取代。 Forty years later, the question we now ask is: why are direct search methods still in use? Surely this seemingly hodge-podge collection of methods based on heuristics, which generally appeared without any attempt at a theoretical justification, should have been superseded by more ”modern approaches to numerical optimization. 在更大的范围，直接搜索法已经被更多的复杂方法所取代。数值 优化已经成熟，软件技术已经使用户很容易利用更多的复杂数值技术。 许多用户已经例行使用许多全局拟牛顿法的变形。 To a large extent direct search methods have been replaced by more sophisticated techniques. At the field of numerical optimization has matured, and software has appeared which eases the ability of consumers to make use of these more sophisticated numerical techniques, many users now routinely rely on some variant of a globalized quasi-Newton method. 但是，直接搜索法仍然有它存在的理由。首先也是最重要的，直 接搜索法仍然流行的一个原因是它在实践中工作的很好。实际上，许 多直接搜索法的基础启发性最近被分析家证明有和全局拟牛顿法技术 类似的全局收敛性。直接搜索法的成功是因为他们中的许多，包括胡 克和吉夫斯的直接搜索法是依赖于经典分析技术，方法上外观不是显
0377-0427/00/$-see front matter © 2000 Elsevier Science B.V. All rights reserved. Pll:S0377-0427(00)00423-4
我们用“直接搜索”术语来描述试验解的有序检查，包括每个试验解 与当前“最好”解的比较和决定（作为以前结果的函数）下一个试验 解的策略。该术语暗示我们的优先选择是基于经验的。对于直接搜索 策略，一般的经典分析方法起不到任何作用。 We use the phrase “direct search” to describe sequential examination of trial solutions involving comparison of each trial solution with the “best” obtained up to that time together with a strategy for determining (as a function of earlier results) what the next trial solution will be. The phrase
perspective on this classical family of derivative-free algorithms, focusing on the development of direct search methods during their golden age from 1960 to 1971. We discuss how direct search methods are characterized by the absence of the construction of a model of the objective. We then consider a number of the classical direct search methods and discuss what research in the intervening years has uncovered about these algorithms. In particular, while the original direct search methods were consciously based on straightforward heuristics, more recent analysis has shown that in most – but not all – cases these heuristics actually suffice to ensure global convergence of at least one subsequence of the sequence of iterates to a first-order stationary point of the objective function. © 2000 Elsevier Science B.V. All rights reserved. Keywords: Derivative-free optimization; Direct search methods; Pattern search methods 1. Introduction 1． 引言
对于现代读者，这种“除非有特别优势”而回避一般经典分析技 术的偏好听起来好像十分奇怪。毕竟，拟牛顿法应用时的成功是不容 置疑的。但考虑胡克和捷吾斯的历史环境，我们现在所用的表示怎样 修正最速下降法以保证全局收敛的 Armijo-Goldstein-Wolfe 条件是在胡 克和捷吾斯论文发表 5 年以后才问世的。这篇论文仅在戴维森未发表 的用割线法推出拟牛顿法的报告 2 年后出现的，在计算机杂志上发表 的弗莱彻和鲍威尔的相似观点的论文的前两年[10]。因此，胡克和吉 夫斯的偏好现在没有证明。 To a modern reader, this preference for avoiding techniques of classical analysis “except when there is a demonstrable advantage in doing so” quite likely sounds odd. After all, the success quasi-Newton methods, when applicable, is now undisputed. But consider the historical context the remark by Hooke and Jeeves. Hooke and Jeeves’ paper appeared five years before what we now referred to as the Armijo-Goldstein-Wolfe conditions were introduced and used to show how the method of steepest descent could be modified to ensure global convergence [1,11,29]. The paper appeared only two years after Davidon’s unpublished report on using secant updates to deries quasi-Newton methods [8], and two years before Fletcher and Powell published a similar idea in The Computer Journal [10].