Ambari离线安装HDPHadoop集群汇总.docx

Ambari离线安装HDPHadoop集群汇总.docx

《Ambari离线安装HDPHadoop集群汇总.docx》由会员分享，可在线阅读，更多相关《Ambari离线安装HDPHadoop集群汇总.docx（12页珍藏版）》请在冰豆网上搜索。

Ambari离线安装HDPHadoop集群汇总

简介

Hortonworks采用了100%完全开源策略，产品名称为HDP（HortonworksDataPlatform）。

所有软件产品开源，用户免费使用，Hortonworks提供商业的技术支持服务。

与CDH相比，管理软件使用开源Ambari，数据治理使用Atlas，安全组件使用Ranger而非Sentry，SQL继续紧抱Hive大腿。

ApacheAmbari是对Hadoop进行监控、管理和生命周期管理的开源项目。

它也是一个为Hor

tonworks数据平台选择管理组建的项目。

Ambari向HadoopMapReduce、HDFS、

HBase、Pig,Hive、HCatalog以及Zookeeper提供服务。

Step0:

搭建本地yum源

我选择的是hdp相关包放到了apache服务器上,yum源下载地址Setp1列出。

1

#tree-L1www/

www/

|--ARTIFACTS

|--HDP

|--HDP-UTILS-1.1.0.20

|--ambari->ambari-2.1.0

`--ambari-2.1.0

5directories,0files

#tree-L3www/

www/

|--ARTIFACTS

|--HDP

|`--centos6

|`--2.x

|--HDP-UTILS-1.1.0.20

|`--repos

|`--centos6

|--ambari->ambari-2.1.0

`--ambari-2.1.0

`--centos6

|--RPM-GPG-KEY

|--ambari

|--build.id

|--build_metadata.txt

|--changelog.txt

|--hdp_mon

`--repodata

14directories,3files

ssh？

echo‘UseDNSno’>/etc/ssh/sshd_config&&/etc/init.d/sshdrestart

Step1:

DownloadtheAmbarirepositoryontheAmbariServerhost

—hdpurl—

1

wgethttp:

//public-repo-

wgethttp:

//public-repo-

wgethttp:

//public-repo-

wgethttp:

//public-repo-

wgethttp:

//public-repo-

关于如果搭建apache服务器，制作离线安装源，这里就不做过多介绍...

1

updaterootpasswd

mysqladmin-urootpasswordadmin

--mysql--

createdatabaseambari;

grantallonambari.*to'root'@'%'identifiedby'admin';

grantallonambari.*to'ambari'@'%'identifiedby'admin';

grantallonambari.*to'root'@'server1'identifiedby'admin';

grantallonambari.*to'ambari'@'server1'identifiedby'admin';

flushprivileges;

省略mysql安装直接使用

1

--localyumresource--

sudoyumcleanall

sudoyumrepolist

Step2:

installambari-server

1

sudoyuminstallambari-server

Step3:

setupambari-server

1

ambari-serversetup

DownloadingJDKfromhttp:

//public-repo-to/var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz

1

wget

sudomvjdk-7u79-linux-x64.tar.gz/var/lib/ambari-server/resources/

sudomvjdk-7u79-linux-x64.tar.gzjdk-7u67-linux-x64.tar.gz

1

WARNING:

BeforestartingAmbariServer,youmustrunthefollowingDDLagainstthedatabasetocreatetheschema:

/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

$sudoyuminstallmysql-connector-java

$mysql-uroot-padmin

mysql>useambari;

Databasechanged

mysql>source/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

1

$sudoambari-serversetup

Usingpython/usr/bin/python2.6

Setupambari-server

CheckingSELinux...

SELinuxstatusis'disabled'

Customizeuseraccountforambari-serverdaemon[y/n]（n）?

y

Enteruseraccountforambari-serverdaemon（root）:

Adjustingambari-serverpermissionsandownership...

Checkingfirewallstatus...

CheckingJDK...

DoyouwanttochangeOracleJDK[y/n]（n）?

n

Completingsetup...

Configuringdatabase...

Enteradvanceddatabaseconfiguration[y/n]（n）?

y

Configuringdatabase...

==============================================================================

Chooseoneofthefollowingoptions:

[1]-PostgreSQL（Embedded）

[2]-Oracle

[3]-MySQL

[4]-PostgreSQL

[5]-MicrosoftSQLServer（TechPreview）

==============================================================================

Enterchoice

（1）:

3

Hostname（localhost）:

Port（3306）:

Databasename（ambari）:

Username（root）:

EnterDatabasePassword（bigdata）:

Re-enterpassword:

Configuringambaridatabase...

CopyingJDBCdriverstoserverresources...

Configuringremotedatabaseconnectionproperties...

WARNING:

BeforestartingAmbariServer,youmustrunthefollowingDDLagainstthedatabasetocreatetheschema:

/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceedwithconfiguringremotedatabaseconnectionproperties[y/n]（y）?

y

Extractingsystemviews...

...ambari-admin-2.1.0.1470.jar

...

Adjustingambari-serverpermissionsandownership...

AmbariServer'setup'completedsuccessfully.

Step3:

startAmbari

1

$sudoambari-serverstart

Usingpython/usr/bin/python2.6

Startingambari-server

AmbariServerrunningwithadministratorprivileges.

Organizingresourcefilesat/var/lib/ambari-server/resources...

ServerPIDat:

/var/run/ambari-server/ambari-server.pid

Serveroutat:

/var/log/ambari-server/ambari-server.out

Serverlogat:

/var/log/ambari-server/ambari-server.log

Waitingforserverstart....................

AmbariServer'start'completedsuccessfully.

Step4:

DeployClusterusingAmbariWebUI

http:

//server2:

8080/#/login

用户名：

admin

密码：

admin

Step5:

InstallHadoopcluster

省略，都是下一步操作...

报错：

Error:

1

INFO2015-08-2523:

52:

43,898main.py:

287-ConnectingtoAmbariserverathttps:

//server2:

8440（192.168.2.201）

INFO2015-08-2523:

52:

43,898NetUtil.py:

59-Connectingtohttps:

//server2:

8440/ca

ERROR2015-08-2523:

52:

43,968NetUtil.py:

77-[Errno1]_ssl.c:

492:

error:

100AE081:

ellipticcurveroutines:

EC_GROUP_new_by_curve_name:

unknowngroup

ERROR2015-08-2523:

52:

43,968NetUtil.py:

78-SSLError:

Failedtoconnect.Pleasecheck`openssl`libraryversions.

Referto:

formoredetails.

系统为centos6.5

$cat/etc/redhat-release

CentOSrelease6.5（Final）

此处为RHEL/CentOS6.5一个小问题，更新openssl更新版本！

即可解决！

1

Remedy:

ChecktheOpenSSLlibraryversioninstalledonyourhost（s）:

rpm-qa|grepopenssl

openssl-1.0.1e-15.el6.x86_64

Iftheoutputsaysopenssl-1.0.1e-15.x86_64（1.0.1build15）youwillneedtoupgradetheOpenSSLlibrarybyrunningthefollowingcommand:

yumupgradeopenssl

VerifyyouhavethenewerversionofOpenSSL（1.0.1build16）:

rpm-qa|grepopenssl

openssl-1.0.1e-16.el6.x86_64

RestartAmbariAgent（s）andClickRetryFailed?

ontheWizard.

升级脚本

1

#opensslversion

OpenSSL1.0.1g7Apr2014

#sudorpm-e--justdb--nodepsopenssl

#sudorpm-qa|grepopenssl

#sudorpm-ivh--forcehttp:

//mirror.centos.org/centos/6/os/x86_64/Packages/openssl-1.0.1e-42.el6.x86_64.rpm

#rpm-qa|grepopenssl

openssl-1.0.1e-42.el6.x86_64

#opensslversion

OpenSSL1.0.1e-fips11Feb2013

完美解决opensslbug...头疼了一阵...

如果出现乱码：

echo'LANG="en_US.UTF-8"'>/etc/sysconfig/i18n,修改字符集即可解决！

1

Error:

Package:

snappy-devel-1.0.5-1.el6.x86_64（HDP-UTILS-1.1.0.20）

Requires:

snappy（x86-64）=1.0.5-1.el6

Installed:

snappy-1.1.0-1.el6.x86_64（@anaconda-CentOS-201311272149.x86_64/6.5）

snappy（x86-64）=1.1.0-1.el6

Available:

snappy-1.0.5-1.el6.x86_64（HDP-UTILS-1.1.0.20）

snappy（x86-64）=1.0.5-1.el6

解决

sudorpm-esnappy-1.1.0-1.el6.x86_64or[sudorpm-esnappy-1.1.0-1.el6.x86_64--nodeps]

yum-yinstallsnappy*

1

错误1：

File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py",line291,in_call

raiseFail（err_msg）

resource_management.core.exceptions.Fail:

Executionof'/usr/bin/yum-d0-e0-yinstallambari-metrics-monitor'returned1.Error:

Package:

ambari-metrics-monitor-2.1.0-1470.x86_64（Updates-ambari-2.1.0）

Requires:

python-devel

Youcouldtryusing--skip-brokentoworkaroundtheproblem

**Found6pre-existingrpmdbproblem（s）,'yumcheck'outputfollows:

1:

libreoffice-core-4.0.4.2-9.el6.x86_64hasmissingrequiresoflibjawt.so（）（64bit）

1:

libreoffice-core-4.0.4.2-9.el6.x86_64hasmissingrequiresoflibjawt.so（SUNWprivate_1.1）（64bit）

1:

libreoffice-ure-4.0.4.2-9.el6.x86_64hasmissingrequiresofjre>=（'0','1.5.0',None）

2:

postfix-2.6.6-2.2.el6_1.x86_64hasmissingrequiresoflibmysqlclient.so.16（）（64bit）

2:

postfix-2.6.6-2.2.el6_1.x86_64hasmissingrequiresoflibmysqlclient.so.16（libmysqlclient_16）（64bit）

2:

postfix-2.6.6-2.2.el6_1.x86_64hasmissingrequiresofmysql-libs

错误2：

File"/usr/lib/python2.6/site-packages/resource_management/core/sudo.py",line37,inchown

returnos.chown（path,uid,gid）

OSError:

[Errno1]Operationnotpermitted:

'/data0/www/bbs/uc_server/data/avatar/hadoop/hdfs/namenode'

yumsearchpython|greppython-devel

造成这些错误都是安装linux的时候基础包未选择，缺包可以制作cdrom挂载，来安装即可解决！

∙到此即可一步步安装hadoop集群…

Step6:

Test集群使用

1

[whoami@server1~]$sudo-uhivehive

hive>showtables;

OK

Timetaken:

1.36seconds

hive>createtabletest（idint）;

OK

Timetaken:

1.998seconds

hive>insertintotabletestvalues

（1）,

（2）,（3）;

QueryID=hive_20150831184109_0c75d1e8-1003-4ade-bacf-03cd7408082f

Totaljobs=1

LaunchingJob1outof1

Status:

Running（ExecutingonYARNclusterwithAppidapplication_1441014845188_0002）

--------------------------------------------------------------------------------

VERTICESSTATUSTOTALCOMPLETEDRUNNINGPENDINGFAILEDKILLED

--------------------------------------------------------------------------------

Map1..........SUCCEEDED110000

--------------------------------------------------------------------------------

VERTICES:

01/01[==========================>>]100%ELAPSEDTIME:

32.24s

--------------------------------------------------------------------------------

Loadingdatatotabledefault.test

Tabledefault.teststats:

[numFiles=1,numRows=3,totalSize=6,rawDataSize=3]

OK

Timetaken:

41.354seconds

hive>showtables;

OK

test

values__tmp__table__1

Timetaken:

0.112seconds,Fetched:

2row（s）

hive>select*fromtest;

OK

1

2

3

Timetaken:

0.258seconds,Fetched:

3row（s）

hive>selectcount

（1）fromtest;

OK

3

Timetaken:

1.705seconds,Fetched:

1row（s）

相关组件使用一切正常...