最新PRIORITIZED EXPERIENCE REPLAY.docx
- 文档编号:10616195
- 上传时间:2023-02-21
- 格式:DOCX
- 页数:50
- 大小:246.01KB
最新PRIORITIZED EXPERIENCE REPLAY.docx
《最新PRIORITIZED EXPERIENCE REPLAY.docx》由会员分享,可在线阅读,更多相关《最新PRIORITIZED EXPERIENCE REPLAY.docx(50页珍藏版)》请在冰豆网上搜索。
最新PRIORITIZEDEXPERIENCEREPLAY
C.属性只是对象所具有的外部特征
5.1.1选择题
A.2个字节B.4个字节C.8个字节D.10个字节
s=s+(val(substr(alltrim(str(i)),j,1)))^3
settalkon
【答案】A
【答案】DELETE;PACK
B.数据库系统中数据的一致性是指数据类型一致
C.SELECT职员号FROM订单
49.假定学号的第3、4位为专业代码。
要计算各专业学生选修课程号为”101”课程的平均成绩,正确的SQL语句是________。
PRIORITIZEDEXPERIENCEREPLAY
TomSchaul,JohnQuan,IoannisAntonoglouandDavidSilver
GoogleDeepMind
{schaul,johnquan,ioannisa,davidsilver}@
ABSTRACT
Experiencereplayletsonlinereinforcementlearningagentsrememberandreuseexperiencesfromthepast.Inpriorwork,experiencetransitionswereuniformlysampledfromareplaymemory.However,thisapproachsimplyreplaystransitionsatthesamefrequencythattheywereoriginallyexperienced,regardlessoftheirsignificance.Inthispaperwedevelopaframeworkforprioritizingexperience,soastoreplayimportanttransitionsmorefrequently,andthereforelearnmoreefficiently.WeuseprioritizedexperiencereplayinDeepQ-Networks(DQN),areinforcementlearningalgorithmthatachievedhuman-levelperformanceacrossmanyAtarigames.DQNwithprioritizedexperiencereplayachievesanewstate-of-the-art,outperformingDQNwithuniformreplayon41outof49games.
1INTRODUCTION
Onlinereinforcementlearning(RL)agentsincrementallyupdatetheirparameters(ofthepolicy,valuefunctionormodel)whiletheyobserveastreamofexperience.Intheirsimplestform,theydiscardincomingdataimmediately,afterasingleupdate.Twoissueswiththisare(a)stronglycorrelatedupdatesthatbreakthei.i.d.assumptionofmanypopularstochasticgradient-basedalgo-rithms,and(b)therapidforgettingofpossiblyrareexperiencesthatwouldbeusefullateron.
Experiencereplay(Lin,1992)addressesbothoftheseissues:
withexperiencestoredinareplaymemory,itbecomespossibletobreakthetemporalcorrelationsbymixingmoreandlessrecentexperiencefortheupdates,andrareexperiencewillbeusedformorethanjustasingleupdate.ThiswasdemonstratedintheDeepQ-Network(DQN)algorithm(Mnihetal.,2013;2015),whichstabilizedthetrainingofavaluefunction,representedbyadeepneuralnetwork,byusingexperiencereplay.Specifically,DQNusedalargeslidingwindowreplaymemory,sampledfromituniformlyatrandom,andrevisitedeachtransition1eighttimesonaverage.Ingeneral,experiencereplaycanreducetheamountofexperiencerequiredtolearn,andreplaceitwithmorecomputationandmorememory–whichareoftencheaperresourcesthantheRLagent’sinteractionswithitsenvironment.
Inthispaper,weinvestigatehowprioritizingwhichtransitionsarereplayedcanmakeexperiencereplaymoreefficientandeffectivethanifalltransitionsarereplayeduniformly.ThekeyideaisthatanRLagentcanlearnmoreeffectivelyfromsometransitionsthanfromothers.Transitionsmaybemoreorlesssurprising,redundant,ortask-relevant.Sometransitionsmaynotbeimmediatelyusefultotheagent,butmightbecomesowhentheagentcompetenceincreases(Schmidhuber,1991).Experiencereplayliberatesonlinelearningagentsfromprocessingtransitionsintheexactordertheyareexperienced.Prioritizedreplayfurtherliberatesagentsfromconsideringtransitionswiththesamefrequencythattheyareexperienced.
Inparticular,weproposetomorefrequentlyreplaytransitionswithhighexpectedlearningprogress,asmeasuredbythemagnitudeoftheirtemporal-difference(TD)error.Thisprioritizationcanleadtoalossofdiversity,whichwealleviatewithstochasticprioritization,andintroducebias,whichwecorrectwithimportancesampling.Ourresultingalgorithmsarerobustandscalable,whichwedemonstrateontheAtari2600benchmarksuite,whereweobtainfasterlearningandstate-of-the-artperformance.
−
1AtransitionistheatomicunitofinteractioninRL,inourcaseatupleof(stateSt
1,actionAt
−1,reward
arXiv:
1511.05952v4[cs.LG]25Feb2016
Rt,discountγt,nextstateSt).Wechoosethisforsimplicity,butmostoftheargumentsinthispaperalsoholdforacoarserwaysofchunkingexperience,e.g.intosequencesorepisodes.
2BACKGROUND
Numerousneurosciencestudieshaveidentifiedevidenceofexperiencereplayinthehippocampusofrodents,suggestingthatsequencesofpriorexperiencearereplayed,eitherduringawakerestingorsleep.Sequencesassociatedwithrewardsappeartobereplayedmorefrequently(Athertonetal.,2015;O´lafsdo´ttiretal.,2015;Foster&Wilson,2006).ExperienceswithhighmagnitudeTDerroralsoappeartobereplayedmoreoften(Singer&Frank,2009;McNamaraetal.,2014).
Itiswell-knownthatplanningalgorithmssuchasvalueiterationcanbemademoreefficientbyprioritizingupdatesinanappropriateorder.Prioritizedsweeping(Moore&Atkeson,1993;Andreetal.,1998)selectswhichstatetoupdatenext,prioritizedaccordingtothechangeinvalue,ifthatupdatewasexecuted.TheTDerrorprovidesonewaytomeasurethesepriorities(vanSeijen&Sutton,2013).Ourapproachusesasimilarprioritizationmethod,butformodel-freeRLratherthanmodel-basedplanning.Furthermore,weuseastochasticprioritizationthatismorerobustwhenlearningafunctionapproximatorfromsamples.
TD-errorshavealsobeenusedasaprioritizationmechanismfordeterminingwheretofocusre-sources,forexamplewhenchoosingwheretoexplore(Whiteetal.,2014)orwhichfeaturestoselect(Geramifardetal.,2011;Sunetal.,2011).
Insupervisedlearning,therearenumeroustechniquestodealwithimbalanceddatasetswhenclassidentitiesareknown,includingre-sampling,under-samplingandover-samplingtechniques,possiblycombinedwithensemblemethods(forareview,seeGalaretal.,2012).Arecentpaperintroducedaformofre-samplinginthecontextofdeepRLwithexperiencereplay(Narasimhanetal.,2015);themethodseparatesexperienceintotwobuckets,oneforpositiveandonefornegativerewards,andthenpicksafixedfractionfromeachtoreplay.Thisisonlyapplicableindomainsthat(unlikeours)haveanaturalnotionof‘positive/negative’experience.Furthermore,Hinton(2007)introducedaformofnon-uniformsamplingbasedonerror,withanimportancesamplingcorrection,whichledtoa3xspeed-uponMNISTdigitclassification.
TherehavebeenseveralproposedmethodsforplayingAtariwithdeepreinforcementlearning,in-cludingdeepQ-networks(DQN)(Mnihetal.,2013;2015;Guoetal.,2014;Stadieetal.,2015;Nairetal.,2015;Bellemareetal.,2016),andtheDoubleDQNalgorithm(vanHasseltetal.,2016),whichisthecurrentpublishedstate-of-the-art.Simultaneouslywithourwork,anarchitecturalinnovationthatseparatesadvantagesfromthevaluefunction(seetheco-submissionbyWangetal.,2015)hasalsoledtosubstantialimprovementsontheAtaribenchmark.
3PRIORITIZEDREPLAY
Usingareplaymemoryleadstodesignchoicesattwolevels:
whichexperiencestostore,andwhichexperiencestoreplay(andhowtodoso).Thispaperaddressesonlythelatter:
makingthemosteffectiveuseofthereplaymemoryforlearning,assumingthatitscontentsareoutsideofourcontrol(butseealsoSection6).
3.1AMOTIVATINGEXAMPLE
Tounderstandthepotentialgainsofprioritization,weintroduceanartificial‘BlindCliffwalk’en-vironment(describedinFigure1,left)thatexemplifiesthechallengeofexplorationwhenrewardsarerare.Withonlynstates,theenvironmentrequiresanexponentialnumberofrandomstepsuntilthefirstnon-zeroreward;tobeprecise,thechancethatarandomsequenceofactionswillleadtotherewardis2−n.Furthermore,themostrelevanttransitions(fromraresuccesses)arehiddeninamassofhighlyredundantfailurecases(similartoabipedalrobotfallingoverrepeatedly,beforeitdiscovershowtowalk).
Weusethisexampletohighlightthedifferencebetweenthelearningtimesoftwoagents.BothagentsperformQ-learningupdatesontransitionsdrawnfromthesamereplaymemory.Thefirstagentreplaystransitionsuniformlyatrandom,whilethesecondagentinvokesanoracletoprioritizetransitions.Thisoraclegreedilyselectsthetransitionthatmaximallyreducesthegloballossinitscurrentstate(inhindsight,aftertheparameterupdate).Forthedetailsofthesetup,seeAppendixB.1.Figure1(right)showsthatpickingthetransitionsinagoodordercanleadtoexponentialspeed-ups
12…
R=1
n-1n
Figure1:
Left:
Illustrationofthe‘BlindCliffwalk’exampledomain:
therearetwoactions,a‘right’anda‘wrong’one,andtheepisodeisterminatedwhenevertheagenttakesthe‘wrong’action(dashedredarrows).Takingthe‘right’actionprogressesthroughasequenceofnstates(blackarrows),attheendofwhichliesafinalrewardof1(greenarrow);rewardis0elsewhere.Wechosearepresentationsuchthatgeneralizingoverwhatactionis‘right’isnotpossible.Right:
Mediannumberoflearningstepsrequiredtolearnthevaluefunctionasafunctionofthesizeofthetotalnumberoftransitionsinthereplaymemory.Notethelog-logscale,whichhighlightstheexponentialspeed-upfromreplayingwithanoracle(brightblue),comparedtouniformreplay(black);faintlinesaremin/maxvaluesfrom10independentruns.
overuniformchoice.Suchanoracleisofcoursenotrealistic,yetthelargegapmotivatesoursearchforapracticalapproachthatimprovesonuniformrandomreplay.
3.2PRIORITIZINGWITHTD-ERROR
Thecentralcomponentofprioritizedreplayisthecriterionbywhichtheimportanceofeachtransi-tionismeasured.OneidealisedcriterionwouldbetheamounttheRLagentcanlearnfromatran-sitioninitscurrentstate(expectedlearningprogress).Whilethismeasureisnotdirectlyaccessible,a
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 最新PRIORITIZED EXPERIENCE REPLAY 最新 PRIORITIZED
![提示](https://static.bdocx.com/images/bang_tan.gif)