ELM极限学习机黄广斌学术报告讲稿.docx
- 文档编号:25395234
- 上传时间:2023-06-08
- 格式:DOCX
- 页数:35
- 大小:37.33KB
ELM极限学习机黄广斌学术报告讲稿.docx
《ELM极限学习机黄广斌学术报告讲稿.docx》由会员分享,可在线阅读,更多相关《ELM极限学习机黄广斌学术报告讲稿.docx(35页珍藏版)》请在冰豆网上搜索。
ELM极限学习机黄广斌学术报告讲稿
>>ZhengyouZhang:
Okay.Soit'smypleasuretointroduceProfessorGuangBinHuangfromNanyangTechnologicalUniversityofSingapore,NTU.Soheistalkingabouthislonginterestresearchinextremelearningmachine.
HegraduatedfromNortheasternUniversityinChinainappliedmathematicsandhasbeenwithNTUforquiteafewyears.Andhe'sassociateeditorofNeurocomputingandalsoIEEETransactionsonSystems,ManandCybernetics.
Andhe'sorganizingourworkshop.Youcanadvertiseyourconferencelater.Soplease.
>>GuangBinHuang:
Thanks,Dr.Zhang,forinvitingmehere.It'smyhonortogiveatalktointroducetheextremelearningmachine.
Thisisactually theideawasinitiallyinitiatedin'9 actually2003.Thatwasthefirsttimewesubmittedpapers.Andthenitbecomerecognizedbymoreandmoreresearchersrecently.SowejusthadaworkshopinAustralialastDecember.SowearegoingtohaveaninternationalsymposiumonextremelearningmachinethatiscomingDecemberinChina.It'sDr. Zhang'shometown,ornearbyhishometown.Sowe'rehopingyoucanjoin.
Okay.Sowhatisextremelearningmachine?
Actually,thisisatechnicaltalkabout talkingaboutthekindoflearningmethod.Suchalearningmethodisdifferentfromthetraditionallearningmethod.Sotuningisnotrequired.
SoyouwillwishtoknowextremelearningmachineIthinkit'sbettertogobacktoreviewthetraditionalfeedforwardneuralnetworks,includingsupportvectormachines.
Iassumemanypeopleconsidersupportvectormachinestonotbeknowntoneuralnetworks.Butactuallyinmyopinion[inaudible]theyhavethesametarget,samearchitectureinsomeaspects,insomesense.
Soafterwereviewfeedforwardneuralnetworks,thenwecangotothe tointroducewhatiscalledextremelearningmachine.Actually,extremelearningmachineisvery,verysimple.Okay.DuringmytalkIalsowishtogivethecomparisonbetweentheyearendandthissquareSVM.SofinallyIwishtoshowlinkagebetweentheELMandalsothetraditionalSVM,soweknowwhat'sthedifference,what'stherelationship.Okay.
Sothefeedforwardneuralnetwork,Ihaveseveraltypesofframeworks,architectures.Oneofthepopularonesismultilayerfeedforwardneuralnetworks.Butthenintheoryandalsoinapplicationspeoplefoundasinglehiddenlayerisenoughforustohandlealltheapplicationsintheory.Sothatmeansgivenanyapplications,wecandesignasinglehiddenlayerfeedforwardnetwork.
Thissinglehiddenlayerfeedforwardnetworkcanbeusefulastoapproximateanycontinuoustargetfunction.Canbeusefulastoclassifyanydisjoinedregions.
Okay.Soforsinglehiddenlayerfeedforwardnetworks,usuallywealsohavethetwotypeofpopulararchitectures.Thefirstoneissocalledsigmoidtypeoffeedforwardnetwork.Sothatmeanthehiddenlayerhereusesigmoidoftypenetwork.Okay.SometimeIcalladditivehiddennodes.Thatmeanstheinputofeachhiddennodeisweightedsumoftheinput.
Okay.SoofcourseGhere,usuallypeopleareusingsigmoidtypeoffunction.Butyoucanalsowritetheoutputhiddenlayer outputhiddennodeasuppercaseG.SoAIPIandX.Xisinputofthenetwork.AIPIarethehiddenparametersforeach sayforhiddennodeI,theAIPIaretheparametersofthenodeI.
Allright.Sothisisthesigmoidtypeofhiddenfeedforwardnetwork.Ofcourseanotheroneisvery,verypopularisRBFnetwork.SoRBFnetworkhere,ahiddennodeoutputfunctionisRBFfunction.
Sowhatyourewritetheninthiscomputerformat,weactuallyhavethesamesocalledoutputfunctionofthesinglehiddenlayerfeedforwardnetworkassigmoidtypenetwork.SoherealwaysuppercaseG.
Sothesetwotypeofnetworksareveryinterestinginthepasttwotothreedecades.Twogroupswrotepapers,researchers,toworkonthesetwoareas,andtheyconsiderthemseparate.Andtheyuseadifferentlearningmethodforthesetwomethodnetworks.
Butgenerallyforbothtypeofnetworks,wehavethissocalledtheorem.Sogivenanytargetcontinuousfunction,FX,sotheoutputoffunctionofthissinglehiddenlayercanbeascloseastothistargetcontinuousfunctiongivenanyarrow.
Anddefinitelyinseries,wecanfindsuchanetwork.Sothatistheoutputofthisnetworkcanbeasclosetothistargetfunctioneffects.
Ofcourse,inrealapplications,wedonotknowthetargetfunction,FX.Allright.Soyoursigmoidprocessingwillhaveasampling.Weonlycansamplethediscretesamples.Andwewishtolearnthesediscretesamples,trainingsamples.Sowewishtoadjusttheparametersofthehiddenlayerandalsothewastebetweenthehiddenlayertotheoutputlayer.Trytofindsomealgorithmtolearntheseparameterstomakesuretheoutputofthenetworktoapproximateatargetfunction.
Okay.Fromlearningpointofview sogiven we'regiven,say,Ntrainingsamples,XI,TI,sowewishtohavethenetwork outputofthenetworkwithrespecttotheinputXJincurredtoyourtargetoutput,TJ.Ofcourse,inmostcases,youroutputnotexactlythesameasyourtarget.Sothereissomeerrorthere.SupposetheoutputofthenetworkisOJ,sowewishtominimizethiscostfunction.
Soinordertodothis,manypeople,right,actuallyspendtimefindingdifferentmethodsonhowtotunethehiddenlayerparameters,AI,BI,andalsothewastebetweenthehiddenlayertotheoutputlayer.ThatisbetaI,whichisIcalloutputweight.
Allright.Sothatisthesituationforthelearning.Sothatisforapproximation.Buthowaboutclassification?
Soinmytheorypublishedinyear2000wesayaslongasthiskindofhidden singlehiddenlayerfeedforwardnetworkcanapproximateanytargetfunction,thisnetworkintheorycanbeusedforustoclassifyanydisjointregions.Sothisisaclassificationcase.Yeah.
>>:
Sothere'saverywellknownresultinneuralnetworksthatyouprobablyknowwhichsaysthatinorderforthe[inaudible]tobevalid,thenumberofunitshadtobevery,verylargeandthatifyoucaninfinite[inaudible]what'scalledaGaussianprocess.
>>GuangBinHuang:
Yeah,you'reright.
>>:
Sothatgives[inaudible]indicationofprocessing[inaudible].
>>GuangBinHuang:
Ah.Okay.Thatisactuallyveryusefultheory.Thatoneactuallyusedforourfurtherdevelopment.Sothatiswhywecometoextremelearningmachine.Thatisaguide.
Butinfinitynumberofhiddennodesusuallynotrequiredinrealapplications.Butintheory,inordertoapproximateanytargetfunction,sayepsilon,thearrowreachingzero,theninthatsenseinfinitynumberofhiddennodesneedstobegiven.Butinrealapplicationwedonotneedthat.SoIwillmentionlater.
Butthattheoryisvery,veryimportant.Okay.Without umhmm.
>>:
Also,recentlymanypeopleobservethatifyouhavemanylayers,actuallyyoucan[inaudible]singlelayersystem,evenifyouhaveverylargenumberofhiddenunitsinasinglelayersystem.
>>GuangBinHuang:
You'reright.Sothiswillhaveanotherpaper actually,Ididn'tmentionhere thatisweproveinstanceof,say,threehiddenlayer.Justtalkabouttwohiddenlayer.Compareonehiddenlayerofarchitecture.Sotwohiddenlayerofnetworkusuallyneedmuchfewernumberofhiddennodesthansinglehiddenlayer.
>>:
[inaudible].
>>GuangBinHuang:
Yeah.Iprovedintheoreticalandalsoshowinginsimulations.Sothatmeansthatfromlearningcapabilitypointofview,myhiddenlayerlooksmorepowerful.Butthatisactuallynotshownhere.Okay.Thatonewecandiscusslater.
Okay.Sothenyouwanttolearnthesekindofnetworks,sotheninthepasttwodecadesmostpeopleusegradientbasedlearningmethod.Sooneofthemostpopularoneisbackpropagation.Ofcourseanditsvariants.Somanyvariants.Peopletalkaboutjustsomeparameter.Theygenerateanotherlearningmethod.Okay.
AnothermethodiscalledforRBFnetwork.Sotalkaboutleastsquaremethod.Butinthisleastsquaremethodissomekindof somethingdifferentfromELM,whichIwillintroducelater.
Sosinglehidden sosingleimpactfactorusedinallhiddennodes.Thatmeansinallhiddennodestheyusethe allhiddennodesusethesameimpactfactor[inaudible].Okay.Sometimeitcallsigma.Right?
Okay.Sowhat'sthedrawbacksofthosegradientbasedmethodor sousuallywewillfearverydifficultinresearch.Sodifferentgroupofresearcherswillcountdifferentnetwork.Actually,intuitivelyspeaking,theyhavethesimilararchitectures,butusuallyRBFnetworkresearcherworkonRBF.Sofeedforwardnetworkpeopleworkonfeedforwardnetwork.Sotheyconsiderthelittledifference.Soweactuallysometimewasteresources.
Andalsoinallthenetwork,users actually,you[inaudible]twousers.There'ssometimessomanyparameterforusertotunemanually.Right?
It'scasebycase.Okay.Soit'ssometimesinconvenientforlongexpertusers.Okay.
Sousuallywealsofaceoverfittingissues.Sothisiswhyeventoomanyusethenumberofhiddennodesuseitinhiddenlayer.Wewereafraidit'saproblem,overfittingproblem.Right?
AndalsoforRBFalocalminimum.Right?
Soyoukindofgettheoptimumsolution.Usuallygetalocalminimalsolution.Thatisbetterforthatlocalareabutnotforthegeneral theentireapplications.
Ofcoursetimeconsuming.Timeconsumingnotonlyonlearning[inaudible]andalsoactuallyonhumaneffort.Humanhastospendtimetofindthesocalledproperparameters,userspecifiedparameters.
Sowewishtoovercomealltheselimitationsconstraintsintheoriginallearningmethods.
Now,let'slookatsupportvectormachine.Isthereanyrelationshipbetweenthesupportvectormachineandthetraditionalfeedforwardnetwork.Ofcourse,whenSVMpeopletalkaboutSVM,theynevertalkaboutneuralnetw
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- ELM 极限 学习机 黄广斌 学术报告 讲稿