BigTableWord格式.docx
- 文档编号:22464214
- 上传时间:2023-02-04
- 格式:DOCX
- 页数:34
- 大小:89.15KB
BigTableWord格式.docx
《BigTableWord格式.docx》由会员分享,可在线阅读,更多相关《BigTableWord格式.docx(34页珍藏版)》请在冰豆网上搜索。
{fay,jeff,sanjay,wilsonh,kerr,m3b,tushar,fikes,gruber}@
Google,Inc.
Abstract:
Bigtableisadistributedstoragesystemformanagingstructureddatathatisdesignedtoscaletoaverylargesize:
petabytesofdataacrossthousandsofcommodityservers.ManyprojectsatGooglestoredatainBigtable,includingwebindexing,GoogleEarth,andGoogleFinance.TheseapplicationsplaceverydifferentdemandsonBigtable,bothintermsofdatasize(fromURLstowebpagestosatelliteimagery)andlatencyrequirements(frombackendbulkprocessingtoreal-timedataserving).Despitethesevarieddemands,Bigtablehassuccessfullyprovidedaflexible,high-performancesolutionforalloftheseGoogleproducts.InthispaperwedescribethesimpledatamodelprovidedbyBigtable,whichgivesclientsdynamiccontroloverdatalayoutandformat,andwedescribethedesignandimplementationofBigtable.
1Introduction
Overthelasttwoandahalfyearswehavedesigned,implemented,anddeployedadistributedstoragesystemformanagingstructureddataatGooglecalledBigtable.Bigtableisdesignedtoreliablyscaletopetabytesofdataandthousandsofmachines.Bigtablehasachievedseveralgoals:
wideapplicability,scalability,highperformance,andhighavailability.BigtableisusedbymorethansixtyGoogleproductsandprojects,includingGoogleAnalytics,GoogleFinance,Orkut,PersonalizedSearch,Writely,andGoogleEarth.TheseproductsuseBigtableforavarietyofdemandingworkloads,whichrangefromthroughput-orientedbatch-processingjobstolatency-sensitiveservingofdatatoendusers.TheBigtableclustersusedbytheseproductsspanawiderangeofconfigurations,fromahandfultothousandsofservers,andstoreuptoseveralhundredterabytesofdata.
Inmanyways,Bigtableresemblesadatabase:
itsharesmanyimplementationstrategieswithdatabases.Paralleldatabases
[14]andmain-memorydatabases
[13]haveachievedscalabilityandhighperformance,butBigtableprovidesadifferentinterfacethansuchsystems.Bigtabledoesnotsupportafullrelationaldatamodel;
instead,itprovidesclientswithasimpledatamodelthatsupportsdynamiccontroloverdatalayoutandformat,andallowsclientstoreasonaboutthelocalitypropertiesofthedatarepresentedintheunderlyingstorage.Dataisindexedusingrowandcolumnnamesthatcanbearbitrarystrings.Bigtablealsotreatsdataasuninterpretedstrings,althoughclientsoftenserializevariousformsofstructuredandsemi-structureddataintothesestrings.Clientscancontrolthelocalityoftheirdatathroughcarefulchoicesintheirschemas.Finally,Bigtableschemaparametersletclientsdynamicallycontrolwhethertoservedataoutofmemoryorfromdisk.
Section
2describesthedatamodelinmoredetail,andSection
3providesanoverviewoftheclientAPI.Section
4brieflydescribestheunderlyingGoogleinfrastructureonwhichBigtabledepends.Section
5describesthefundamentalsoftheBigtableimplementation,andSection
6describessomeoftherefinementsthatwemadetoimproveBigtable'
sperformance.Section
7providesmeasurementsofBigtable'
sperformance.WedescribeseveralexamplesofhowBigtableisusedatGoogleinSection
8,anddiscusssomelessonswelearnedindesigningandsupportingBigtableinSection
9.Finally,Section
10describesrelatedwork,andSection
11presentsourconclusions.
2DataModel
ABigtableisasparse,distributed,persistentmulti-dimensionalsortedmap.Themapisindexedbyarowkey,columnkey,andatimestamp;
eachvalueinthemapisanuninterpretedarrayofbytes.
WesettledonthisdatamodelafterexaminingavarietyofpotentialusesofaBigtable-likesystem.Asoneconcreteexamplethatdrovesomeofourdesigndecisions,supposewewanttokeepacopyofalargecollectionofwebpagesandrelatedinformationthatcouldbeusedbymanydifferentprojects;
letuscallthisparticulartabletheWebtable.InWebtable,wewoulduseURLsasrowkeys,variousaspectsofwebpagesascolumnnames,andstorethecontentsofthewebpagesinthecontents:
columnunderthetimestampswhentheywerefetched,asillustratedinFigure
1.
Figure1:
AsliceofanexampletablethatstoresWebpages.TherownameisareversedURL.Thecontentscolumnfamilycontainsthepagecontents,andtheanchorcolumnfamilycontainsthetextofanyanchorsthatreferencethepage.CNN'
shomepageisreferencedbyboththeSportsIllustratedandtheMY-lookhomepages,sotherowcontainscolumnsnamedanchor:
andanchor:
my.look.ca.Eachanchorcellhasoneversion;
thecontentscolumnhasthreeversions,attimestamps
and
.
Rows
Therowkeysinatablearearbitrarystrings(currentlyupto64KBinsize,although10-100bytesisatypicalsizeformostofourusers).Everyreadorwriteofdataunderasinglerowkeyisatomic(regardlessofthenumberofdifferentcolumnsbeingreadorwrittenintherow),adesigndecisionthatmakesiteasierforclientstoreasonaboutthesystem'
sbehaviorinthepresenceofconcurrentupdatestothesamerow.
Bigtablemaintainsdatainlexicographicorderbyrowkey.Therowrangeforatableisdynamicallypartitioned.Eachrowrangeiscalledatablet,whichistheunitofdistributionandloadbalancing.Asaresult,readsofshortrowrangesareefficientandtypicallyrequirecommunicationwithonlyasmallnumberofmachines.Clientscanexploitthispropertybyselectingtheirrowkeyssothattheygetgoodlocalityfortheirdataaccesses.Forexample,inWebtable,pagesinthesamedomainaregroupedtogetherintocontiguousrowsbyreversingthehostnamecomponentsoftheURLs.Forexample,westoredataforunderthekeycom.google.maps/index.html.Storingpagesfromthesamedomainneareachothermakessomehostanddomainanalysesmoreefficient.
ColumnFamilies
Columnkeysaregroupedintosetscalledcolumnfamilies,whichformthebasicunitofaccesscontrol.Alldatastoredinacolumnfamilyisusuallyofthesametype(wecompressdatainthesamecolumnfamilytogether).Acolumnfamilymustbecreatedbeforedatacanbestoredunderanycolumnkeyinthatfamily;
afterafamilyhasbeencreated,anycolumnkeywithinthefamilycanbeused.Itisourintentthatthenumberofdistinctcolumnfamiliesinatablebesmall(inthehundredsatmost),andthatfamiliesrarelychangeduringoperation.Incontrast,atablemayhaveanunboundednumberofcolumns.
Acolumnkeyisnamedusingthefollowingsyntax:
family:
qualifier.Columnfamilynamesmustbeprintable,butqualifiersmaybearbitrarystrings.AnexamplecolumnfamilyfortheWebtableislanguage,whichstoresthelanguageinwhichawebpagewaswritten.Weuseonlyonecolumnkeyinthelanguagefamily,anditstoreseachwebpage'
slanguageID.Anotherusefulcolumnfamilyforthistableisanchor;
eachcolumnkeyinthisfamilyrepresentsasingleanchor,asshowninFigure
1.Thequalifieristhenameofthereferringsite;
thecellcontentsisthelinktext.
Accesscontrolandbothdiskandmemoryaccountingareperformedatthecolumn-familylevel.InourWebtableexample,thesecontrolsallowustomanageseveraldifferenttypesofapplications:
somethataddnewbasedata,somethatreadthebasedataandcreatederivedcolumnfamilies,andsomethatareonlyallowedtoviewexistingdata(andpossiblynoteventoviewalloftheexistingfamiliesforprivacyreasons).
Timestamps
EachcellinaBigtablecancontainmultipleversionsofthesamedata;
theseversionsareindexedbytimestamp.Bigtabletimestampsare64-bitintegers.TheycanbeassignedbyBigtable,inwhichcasetheyrepresent"
realtime"
inmicroseconds,orbeexplicitlyassignedbyclientapplications.Applicationsthatneedtoavoidcollisionsmustgenerateuniquetimestampsthemselves.Differentversionsofacellarestoredindecreasingtimestamporder,sothatthemostrecentversionscanbereadfirst.
Tomakethemanagementofversioneddatalessonerous,wesupporttwoper-column-familysettingsthattellBigtabletogarbage-collectcellversionsautomatically.Theclientcanspecifyeitherthatonlythelastnversionsofacellbekept,orthatonlynew-enoughversionsbekept(e.g.,onlykeepvaluesthatwerewritteninthelastsevendays).
InourWebtableexample,wesetthetimestampsofthecrawledpagesstoredinthecontents:
columntothetimesatwhichthesepageversionswereactuallycrawled.Thegarbage-collectionmechanismdescribedaboveletsuskeeponlythemostrecentthreeversionsofeverypage.
3API
TheBigtableAPIprovidesfunctionsforcreatinganddeletingtablesandcolumnfamilies.Italsoprovidesfunctionsforchangingcluster,table,andcolumnfamilymetadata,suchasaccesscontrolrights.
ClientapplicationscanwriteordeletevaluesinBigtable,lookupvaluesfromindividualrows,oriterateoverasubsetofthedatainatable.Figure
2showsC++codethatusesaRowMutationabstractiontoperformaseriesofupdates.(Irrelevantdetailswereelidedtokeeptheexampleshort.)ThecalltoApplyperformsanatomicmutationtotheWebtable:
itaddsoneanchortoanddeletesadifferentanchor.
Figure2:
WritingtoBigtable.
Figure
3showsC++codethatusesaScannerabstractiontoiterateoverallanchorsinaparticularrow.
Figure3:
ReadingfromBigtable.
Clientscaniterateovermultiplecolumnfamilies,andthereareseveralmechanismsforlimitingt
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- BigTable
![提示](https://static.bdocx.com/images/bang_tan.gif)