Hadoop Cluster 測試 – HDFS

在各Node上,必須先確認其IP與hostname

可下指令修改hostname

sudo hostnamectl set-hostname vm1

再利用指令尋找Node的IP

ip addr

如下圖已修改hostname為vm1,且得知IP為 192.168.56.102

 

將IP位置與命名加入各node的/etc/hosts檔案中

在各個node加入名為hadoop的使用者,並設定密碼

sudo -u root useradd hadoop

sudo -u root passwd hadoop

以hadoop使用者登入主要的node,並建立ssh key

ssh-keygen -t rsa (一路ENTER下去)

再把ssh key送到各個node上

ssh-copy-id hadoop@vm0

ssh-copy-id hadoop@vm1

用hadoop ssh登入vm0,並下載hadoop,解壓,與重新命名binary: http://ftp.tc.edu.tw/pub/Apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

wget http://ftp.tc.edu.tw/pub/Apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

tar -xzf hadoop-3.1.2.tar.gz

mv hadoop-3.1.2 hadoop

調整JAVA_HOME 至 /usr/lib/jvm/jre

vi hadoop/etc/hadoop/hadoop-env.sh

在各Node設定NameNode (~/hadoop/etc/hadoop/core-site.xml)

vi ~/hadoop/etc/hadoop/core-site.xml

同樣的,設定HDFS Path

設定 yarn

vi ~/hadoop/etc/hadoop/mapred-site.xml

vi ~/hadoop/etc/hadoop/yarn-site.xml

設定workers

vi ~/hadoop/etc/hadoop/workers

設定記憶體相關(DEFAULT值給8G RAM用的)

vi ~/hadoop/etc/hadoop/yarn-site.xml

vi ~/hadoop/etc/hadoop/mapred-site.xml

致此,設定完成

將相關的設定檔案COPY至每個Node

scp hadoop-*.tar.gz vm1:/home/hadoop

SSH連進Node

ssh vm1

解壓,重命名,然後離開

tar -xzf hadoop-3.1.2.tar.gz
mv hadoop-2.8.1 hadoop
exit

for node in node1 node2; do
scp ~/hadoop/etc/hadoop/* $node:/home/hadoop/hadoop/etc/hadoop/;
done

格式化HDFS (在vm0上) hadoop/bin/hdfs namenode -format

然後啟動HDFS

hadoop/sbin/start-dfs.sh

可連上網頁觀看 http://vm0:9870

此至,HDFS設定完成

Leave a Reply

Your email address will not be published. Required fields are marked *