非线性最小二乘法LevenbergMarquardtmethod.docx
- 文档编号:30104723
- 上传时间:2023-08-05
- 格式:DOCX
- 页数:10
- 大小:108KB
非线性最小二乘法LevenbergMarquardtmethod.docx
《非线性最小二乘法LevenbergMarquardtmethod.docx》由会员分享,可在线阅读,更多相关《非线性最小二乘法LevenbergMarquardtmethod.docx(10页珍藏版)》请在冰豆网上搜索。
非线性最小二乘法LevenbergMarquardtmethod
Levenberg-MarquardtMethod(麦夸尔特法)
Levenberg-MarquardtisapopularalternativetotheGauss-Newtonmethodoffindingtheminimumofafunction
thatisasumofsquaresofnonlinearfunctions,
LettheJacobianof
bedenoted
thentheLevenberg-Marquardtmethodsearchesinthedirectiongivenbythesolution
totheequations
where
arenonnegativescalarsand
istheidentitymatrix.Themethodhasthenicepropertythat,forsomescalar
relatedto
thevector
isthesolutionoftheconstrainedsubproblemofminimizing
subjectto
(Gilletal.1981,p. 136).
ThemethodisusedbythecommandFindMinimum[f,
x,x0
]whengiventheMethod->LevenbergMarquardtoption.
窗体顶端
SEEALSO:
Minimum,Optimization
窗体底端
REFERENCES:
Bates,D. M.andWatts,D. G.NonlinearRegressionandItsApplications.NewYork:
Wiley,1988.
Gill,P. R.;Murray,W.;andWright,M. H."TheLevenberg-MarquardtMethod."§4.7.3inPracticalOptimization.London:
AcademicPress,pp. 136-137,1981.
Levenberg,K."AMethodfortheSolutionofCertainProblemsinLeastSquares."Quart.Appl.Math.2,164-168,1944.
Marquardt,D."AnAlgorithmforLeast-SquaresEstimationofNonlinearParameters."SIAMJ.Appl.Math.11,431-441,1963.
Levenberg–Marquardtalgorithm
FromWikipedia,thefreeencyclopedia
Jumpto:
navigation,search
Inmathematicsandcomputing,theLevenberg–Marquardtalgorithm(LMA)[1]providesanumericalsolutiontotheproblemofminimizingafunction,generallynonlinear,overaspaceofparametersofthefunction.Theseminimizationproblemsariseespeciallyinleastsquarescurvefittingandnonlinearprogramming.
TheLMAinterpolatesbetweentheGauss–Newtonalgorithm(GNA)andthemethodofgradientdescent.TheLMAismorerobustthantheGNA,whichmeansthatinmanycasesitfindsasolutionevenifitstartsveryfaroffthefinalminimum.Forwell-behavedfunctionsandreasonablestartingparameters,theLMAtendstobeabitslowerthantheGNA.LMAcanalsobeviewedasGauss–Newtonusingatrustregionapproach.
TheLMAisaverypopularcurve-fittingalgorithmusedinmanysoftwareapplicationsforsolvinggenericcurve-fittingproblems.However,theLMAfindsonlyalocalminimum,notaglobalminimum.
Contents
[hide]
∙1CaveatEmptor
∙2Theproblem
∙3Thesolution
o3.1Choiceofdampingparameter
∙4Example
∙5Notes
∙6Seealso
∙7References
∙8Externallinks
o8.1Descriptions
o8.2Implementations
[edit]CaveatEmptor
Oneimportantlimitationthatisveryoftenover-lookedisthatitonlyoptimisesforresidualerrorsinthedependantvariable(y).Ittherebyimplicitlyassumesthatanyerrorsintheindependentvariablearezerooratleastratioofthetwoissosmallastobenegligible.Thisisnotadefect,itisintentional,butitmustbetakenintoaccountwhendecidingwhethertousethistechniquetodoafit.Whilethismaybesuitableincontextofacontrolledexperimenttherearemanysituationswherethisassumptioncannotbemade.Insuchsituationseithernon-leastsquaresmethodsshouldbeusedortheleast-squaresfitshouldbedoneinproportiontotherelativeerrorsinthetwovariables,notsimplythevertical"y"error.Failingtorecognisethiscanleadtoafitwhichissignificantlyincorrectandfundamentallywrong.Itwillusuallyunderestimatetheslope.Thismayormaynotbeobvioustotheeye.
MicroSoftExcel'schartoffersatrendfitthathasthislimitationthatisundocumented.Usersoftenfallintothistrapassumingthefitiscorrectlycalculatedforallsituations.OpenOfficespreadsheetcopiedthisfeatureandpresentsthesameproblem.
[edit]Theproblem
TheprimaryapplicationoftheLevenberg–Marquardtalgorithmisintheleastsquarescurvefittingproblem:
givenasetofmempiricaldatumpairsofindependentanddependentvariables,(xi,yi),optimizetheparametersβofthemodelcurvef(x,β)sothatthesumofthesquaresofthedeviations
becomesminimal.
[edit]Thesolution
Likeothernumericminimizationalgorithms,theLevenberg–Marquardtalgorithmisaniterativeprocedure.Tostartaminimization,theuserhastoprovideaninitialguessfortheparametervector,β.Inmanycases,anuninformedstandardguesslikeβT=(1,1,...,1)willworkfine;inothercases,thealgorithmconvergesonlyiftheinitialguessisalreadysomewhatclosetothefinalsolution.
Ineachiterationstep,theparametervector,β,isreplacedbyanewestimate,β+δ.Todetermineδ,thefunctions
areapproximatedbytheirlinearizations
where
isthegradient(row-vectorinthiscase)offwithrespecttoβ.
Atitsminimum,thesumofsquares,S(β),thegradientofSwithrespecttoδwillbezero.Theabovefirst-orderapproximationof
gives
.
Orinvectornotation,
.
Takingthederivativewithrespecttoδandsettingtheresulttozerogives:
where
istheJacobianmatrixwhoseithrowequalsJi,andwhere
and
arevectorswithithcomponent
andyi,respectively.Thisisasetoflinearequationswhichcanbesolvedforδ.
Levenberg'scontributionistoreplacethisequationbya"dampedversion",
whereIistheidentitymatrix,givingastheincrement,δ,totheestimatedparametervector,β.
The(non-negative)dampingfactor,λ,isadjustedateachiteration.IfreductionofSisrapid,asmallervaluecanbeused,bringingthealgorithmclosertotheGauss–Newtonalgorithm,whereasifaniterationgivesinsufficientreductionintheresidual,λcanbeincreased,givingastepclosertothegradientdescentdirection.NotethatthegradientofSwithrespecttoβequals
.Therefore,forlargevaluesofλ,thestepwillbetakenapproximatelyinthedirectionofthegradient.Ifeitherthelengthofthecalculatedstep,δ,orthereductionofsumofsquaresfromthelatestparametervector,β+δ,fallbelowpredefinedlimits,iterationstopsandthelastparametervector,β,isconsideredtobethesolution.
Levenberg'salgorithmhasthedisadvantagethatifthevalueofdampingfactor,λ,islarge,invertingJTJ + λIisnotusedatall.Marquardtprovidedtheinsightthatwecanscaleeachcomponentofthegradientaccordingtothecurvaturesothatthereislargermovementalongthedirectionswherethegradientissmaller.Thisavoidsslowconvergenceinthedirectionofsmallgradient.Therefore,Marquardtreplacedtheidentitymatrix,I,withthediagonalmatrixconsistingofthediagonalelementsofJTJ,resultingintheLevenberg–Marquardtalgorithm:
.
AsimilardampingfactorappearsinTikhonovregularization,whichisusedtosolvelinearill-posedproblems,aswellasinridgeregression,anestimationtechniqueinstatistics.
[edit]Choiceofdampingparameter
Variousmore-or-lessheuristicargumentshavebeenputforwardforthebestchoiceforthedampingparameterλ.Theoreticalargumentsexistshowingwhysomeofthesechoicesguaranteedlocalconvergenceofthealgorithm;howeverthesechoicescanmaketheglobalconvergenceofthealgorithmsufferfromtheundesirablepropertiesofsteepest-descent,inparticularveryslowconvergenceclosetotheoptimum.
Theabsolutevaluesofanychoicedependsonhowwell-scaledtheinitialproblemis.Marquardtrecommendedstartingwithavalueλ0andafactorν>1.Initiallysettingλ=λ0andcomputingtheresidualsumofsquaresS(β)afteronestepfromthestartingpointwiththedampingfactorofλ=λ0andsecondlywithλ0/ν.Ifbothoftheseareworsethantheinitialpointthenthedampingisincreasedbysuccessivemultiplicationbyνuntilabetterpointisfoundwithanewdampingfactorofλ0νkforsomek.
Ifuseofthedampingfactorλ/νresultsinareductioninsquaredresidualthenthisistakenasthenewvalueofλ(andthenewoptimumlocationistakenasthatobtainedwiththisdampingfactor)andtheprocesscontinues;ifusingλ/νresultedinaworseresidual,butusingλresultedinabetterresidualthenλisleftunchangedandthenewoptimumistakenasthevalueobtainedwithλasdampingfactor.
[edit]Example
PoorFit
BetterFit
BestFit
Inthisexamplewetrytofitthefunctiony=acos(bX)+bsin(aX)usingtheLevenberg–MarquardtalgorithmimplementedinGNUOctaveastheleasqrfunction.The3graphsFig1,2,3showprogressivelybetterfittingfortheparametersa=100,b=102usedintheinitialcurve.OnlywhentheparametersinFig3arechosenclosesttotheoriginal,arethecurvesfittingexactly.ThisequationisanexampleofverysensitiveinitialconditionsfortheLevenberg–Marquardtalgorithm.Onereasonforthissensitivityistheexistenceofmultipleminima—thefunctioncos(βx)hasminimaatparametervalue
and
[edit]Notes
1.^ThealgorithmwasfirstpublishedbyKennethLevenberg,whileworkingattheFrankfordArmyArsenal.ItwasrediscoveredbyDonaldMarquardtwhoworkedasastatisticianatDuPontandindependentlybyGirard,WynnandMorrison.
[edit]Seealso
∙Trustregion
[edit]References
∙KennethLevenberg(1944)."AMethodfortheSolutionofCertainNon-LinearProblemsinLeastSquares".TheQuarterlyofAppliedMathematics2:
164–168.
∙A.Girard(1958).Rev.Opt37:
225,397.
∙C.G.Wynne(1959)."LensDesigningbyElectronicDigitalComputer:
I".Proc.Phys.Soc.London73(5):
777.doi:
10.1088/0370-1328/73/5/310.
∙JorjeJ.MoréandDanielC.Sorensen(1983)."ComputingaTrust-RegionStep".SIAMJ.Sci.Stat.Comput.(4):
553–572.
∙D.D.Morrison(1960).JetPropulsionLaboratorySeminarproceedings.
∙DonaldMarquardt(1963)."AnAlgorithmforLeast-SquaresEstimationofNonlinearParameters".SIAMJournalonAppliedMathematics11
(2):
431–441.doi:
10.1137/0111030.
∙PhilipE.GillandWalterMurray(1978).
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 非线性 最小二乘法 LevenbergMarquardtmethod
![提示](https://static.bdocx.com/images/bang_tan.gif)