A geometric interpretation of the covariance matrix.docx
- 文档编号:4420143
- 上传时间:2022-12-01
- 格式:DOCX
- 页数:14
- 大小:310.20KB
A geometric interpretation of the covariance matrix.docx
《A geometric interpretation of the covariance matrix.docx》由会员分享,可在线阅读,更多相关《A geometric interpretation of the covariance matrix.docx(14页珍藏版)》请在冰豆网上搜索。
Ageometricinterpretationofthecovariancematrix
Ageometricinterpretationofthecovariancematrix
Contents
∙1 Introduction
∙2 Eigendecompositionofacovariancematrix
∙3 Covariancematrixasalineartransformation
∙4 Conclusion
Introduction
Inthisarticle,weprovideanintuitive,geometricinterpretationofthecovariancematrix,byexploringtherelationbetweenlineartransformationsandtheresultingdatacovariance.Mosttextbooksexplaintheshapeofdatabasedontheconceptofcovariancematrices.Instead,wetakeabackwardsapproachandexplaintheconceptofcovariancematricesbasedontheshapeofdata.
Inapreviousarticle,wediscussedtheconceptof variance,andprovidedaderivationandproofofthewellknownformulatoestimatethesamplevariance.Figure1wasusedinthisarticletoshowthatthestandarddeviation,asthesquarerootofthevariance,providesameasureofhowmuchthedataisspreadacrossthefeaturespace.
Figure1. Gaussiandensityfunction.Fornormallydistributeddata,68%ofthesamplesfallwithintheintervaldefinedbythemeanplusandminusthestandarddeviation.
Weshowedthatanunbiasedestimatorofthesamplevariancecanbeobtainedby:
(1)
However,variancecanonlybeusedtoexplainthespreadofthedatainthedirectionsparalleltotheaxesofthefeaturespace.Considerthe2Dfeaturespaceshownbyfigure2:
Figure2. Thediagnoalspreadofthedataiscapturedbythecovariance.
Forthisdata,wecouldcalculatethevariance
inthex-directionandthevariance
inthey-direction.However,thehorizontalspreadandtheverticalspreadofthedatadoesnotexplainthecleardiagonalcorrelation.Figure2clearlyshowsthatonaverage,ifthex-valueofadatapointincreases,thenalsothey-valueincreases,resultinginapositivecorrelation.Thiscorrelationcanbecapturedbyextendingthenotionofvariancetowhatiscalledthe‘covariance’ofthedata:
(2)
For2Ddata,wethusobtain
and
.Thesefourvaluescanbesummarizedinamatrix,calledthecovariancematrix:
(3)
Ifxispositivelycorrelatedwithy,yisalsopositivelycorrelatedwithx.Inotherwords,wecanstatethat
.Therefore,thecovariancematrixisalwaysasymmetricmatrixwiththevariancesonitsdiagonalandthecovariancesoff-diagonal.Two-dimensionalnormallydistributeddataisexplainedcompletelybyitsmeanandits
covariancematrix.Similarly,a
covariancematrixisusedtocapturethespreadofthree-dimensionaldata,anda
covariancematrixcapturesthespreadofN-dimensionaldata.
Figure3illustrateshowtheoverallshapeofthedatadefinesthecovariancematrix:
Figure3. Thecovariancematrixdefinestheshapeofthedata.Diagonalspreadiscapturedbythecovariance,whileaxis-alignedspreadiscapturedbythevariance.
Eigendecompositionofacovariancematrix
Inthenextsection,wewilldiscusshowthecovariancematrixcanbeinterpretedasalinearoperatorthattransformswhitedataintothedataweobserved.However,beforedivingintothetechnicaldetails,itisimportanttogainanintuitiveunderstandingofhoweigenvectorsandeigenvaluesuniquelydefinethecovariancematrix,andthereforetheshapeofourdata.
Aswesawinfigure3,thecovariancematrixdefinesboththespread(variance),andtheorientation(covariance)ofourdata.So,ifwewouldliketorepresentthecovariancematrixwithavectoranditsmagnitude,weshouldsimplytrytofindthevectorthatpointsintothedirectionofthelargestspreadofthedata,andwhosemagnitudeequalsthespread(variance)inthisdirection.
Ifwedefinethisvectoras
thentheprojectionofourdata
ontothisvectorisobtainedas
andthevarianceoftheprojecteddatais
.Sincewearelookingforthevector
thatpointsintothedirectionofthelargestvariance,weshouldchooseitscomponentssuchthatthecovariancematrix
oftheprojecteddataisaslargeaspossible.Maximizinganyfunctionoftheform
withrespectto
where
isanormalizedunitvector,canbeformulatedasasocalled RayleighQuotient.ThemaximumofsuchaRayleighQuotientisobtainedbysetting
equaltothelargesteigenvectorofmatrix
.
Inotherwords,thelargesteigenvectorofthecovariancematrixalwayspointsintothedirectionofthelargestvarianceofthedata,andthemagnitudeofthisvectorequalsthecorrespondingeigenvalue.Thesecondlargesteigenvectorisalwaysorthogonaltothelargesteigenvector,andpointsintothedirectionofthesecondlargestspreadofthedata.
Nowlet’shavealookatsomeexamples.Inanearlierarticlewesawthatalineartransformationmatrix
iscompletelydefinedbyits eigenvectorsandeigenvalues.Appliedtothecovariancematrix,thismeansthat:
(4)
where
isaneigenvectorof
and
isthecorrespondingeigenvalue.
Ifthecovariancematrixofourdataisadiagonalmatrix,suchthatthecovariancesarezero,thenthismeansthatthevariancesmustbeequaltotheeigenvalues
.Thisisillustratedbyfigure4,wheretheeigenvectorsareshowningreenandmagenta,andwheretheeigenvaluesclearlyequalthevariancecomponentsofthecovariancematrix.
Figure4. Eigenvectorsofacovariancematrix
However,ifthecovariancematrixisnotdiagonal,suchthatthecovariancesarenotzero,thenthesituationisalittlemorecomplicated.Theeigenvaluesstillrepresentthevariancemagnitudeinthedirectionofthelargestspreadofthedata,andthevariancecomponentsofthecovariancematrixstillrepresentthevariancemagnitudeinthedirectionofthex-axisandy-axis.Butsincethedataisnotaxisaligned,thesevaluesarenotthesameanymoreasshownbyfigure5.
Figure5. Eigenvaluesversusvariance
Bycomparingfigure5withfigure4,itbecomesclearthattheeigenvaluesrepresentthevarianceofthedataalongtheeigenvectordirections,whereasthevariancecomponentsofthecovariancematrixrepresentthespreadalongtheaxes.Iftherearenocovariances,thenbothvaluesareequal.
Covariancematrixasalineartransformation
Nowlet’sforgetaboutcovariancematricesforamoment.Eachoftheexamplesinfigure3cansimplybeconsideredtobealinearlytransformedinstanceoffigure6:
Figure6. Datawithunitcovariancematrixiscalledwhitedata.
Letthedatashownbyfigure6be
theneachoftheexamplesshownbyfigure3canbeobtainedbylinearlytransforming
:
(5)
where
isatransformationmatrixconsistingofarotationmatrix
andascalingmatrix
:
(6)
Thesematricesaredefinedas:
(7)
where
istherotationangle,and:
(8)
where
and
arethescalingfactorsinthexdirectionandtheydirectionrespectively.
Inthefollowingparagraphs,wewilldiscusstherelationbetweenthecovariancematrix
andthelineartransformationmatrix
.
Let’sstartwithunscaled(scaleequals1)andunrotateddata.Instatisticsthisisoftenreferedtoas‘whitedata’becauseitssamplesaredrawnfromastandardnormaldistributionandthereforecorrespondtowhite(uncorrelated)noise:
Figure7. Whitedataisdatawithaunitcovariancematrix.
Thecovariancematrixofthis‘white’dataequalstheidentitymatrix,suchthatthevariancesandstandarddeviationsequal1andthecovarianceequalszero:
(9)
Nowlet’sscalethedatainthex-directionwithafactor4:
(10)
Thedata
nowlooksasfollows:
Figure8. Varianceinthex-directionresultsinahorizontalscaling.
Thecovariancematrix
of
isnow:
(11)
Thus,thecovariancematrix
oftheresultingdata
isrelatedtothelineartransformation
thatisappliedtotheoriginaldataasfollows:
where
(12)
However,althoughequation(12)holdswhenthedataisscaledinthexandydirection,thequestionrisesifitalsoholdswhenarotationisapplied.Toinvestigatetherelationbetweenthelineartransformationmatrix
andthecovariancematrix
inthegeneralcase,wewillthereforetrytodecomposethecovariancematrixintotheproductofrotationandscalingmatrices.
Aswesawearlier,wecanrepresentthecovariancematrixbyitseigenvectorsandeigenvalues:
(13)
where
isaneigenvectorof
and
isthecorrespondingeigenvalue.
Equation(13)holdsforeacheigenvector-eigenvaluepairofmatrix
.Inthe2Dcase,weobtaintwoeigenvectorsandtwoeigenvalues.Thesystemoftwoequationsdefinedbyequation(13)canberepresentedefficientlyusingmatrixnotation:
(14)
where
isthematrixwhosecolumnsaretheeigenvectorsof
and
isthediagonalmatrixwhosenon-zeroelementsarethecorrespondingeigenvalues.
Thismeansthatwecanrepresentthecovariancematrixasafunctionofitseigenvectorsandeigenvalues:
(15)
Equation(15)iscalledtheeigendecompositionofthecovariancematrixandcanbeobtainedusinga SingularValueDecompositionalgorithm.Whereastheeigenvectorsrepresentthedirectionsofthelargestvarianceofthedata,theeigenvaluesrepresentthemagnitudeofthisvarianceinthosedirections.Inotherwords,
representsarotationmatrix,while
representsascalingmatrix.Thecovariancematrixcanthusbedecomposedfurtheras:
(16)
where
isarotationmatrixand
isascalingmatrix.
Inequation(6)wedefinedalineartransformation
.Since
isadiagonalscalingmatrix,
.Furthermore,since
isanorthogonalmatrix,
.Therefore,
.Thecovariancematrixcanthusbewrittenas:
(17)
Inotherwords,ifweapplythelineartransformationdefinedby
totheoriginalwhitedata
shownbyfigure7,weobtaintherotatedandscaleddata
withcovariancematrix
.Thisisillustratedbyfigure10:
Figure10. Thecovariancematrixrepresentsalineartransformationof
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- geometric interpretation of the covariance matrix
![提示](https://static.bdocx.com/images/bang_tan.gif)
链接地址:https://www.bdocx.com/doc/4420143.html