Hadoop入门

• 命令脚本

#!/bin/bash
 
if [ $# -lt 1 ] 
then
	echo "No Args Input..."
	exit ;
fi
 
case $1 in 
"start")         
        echo " =================== 启动 hadoop 集群 ===================" 
        echo " --------------- 启动 hdfs ---------------"         
        ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"         
        echo " --------------- 启动 yarn ---------------" 
        ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"         
        echo " --------------- 启动 historyserver ---------------"         
        ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver" 
;;
"stop")         
		echo " =================== 关闭 hadoop 集群 ===================" 
		
        echo " --------------- 关闭 historyserver ---------------"         
        ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"         
        echo " --------------- 关闭 yarn ---------------"  
       ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"         
       echo " --------------- 关闭 hdfs ---------------"         
       ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;; 
*)     
       echo "Input Args Error..."
;; 
esac

• 查看三台服务器进程脚本：jpsall

#!/bin/bash 

for host in hadoop102 hadoop103 hadoop104
do         
    echo =============== $host ===============         
    ssh $host jps  
done

两道面试题

一、常用端口号

hadoop3.x

HDFS NameNode 内部通信端口：8020、9000、9820
HDFS NameNode 对用户的查询接口：9870
Yarn查看任务运行情况的：8088
历史服务器：19888

hadoop2.x

HDFS NameNode 内部通信端口：8020、9000
HDFS NameNode 对用户的查询接口：50070
Yarn查看任务运行情况的：8088
历史服务器：19888

二、常用的配置文件
hadoop3.x

• core-site.xml
• hdfs-site.xml
• yarn-site.xml
• mapred-site.xml
• workers

hadoop2.x

• core-site.xml
• hdfs-site.xml
• yarn-site.xml
• mapred-site.xml
• slaves

集群时间同步

[lln@hadoop102 ~]$ su root
密码：
[root@hadoop102 lln]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@hadoop102 lln]# systemctl start ntpd
[root@hadoop102 lln]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since 二 2022-08-23 17:48:56 CST; 8s ago
  Process: 8298 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 8300 (ntpd)
    Tasks: 1
   CGroup: /system.slice/ntpd.service
           └─8300 /usr/sbin/ntpd -u ntp:ntp -g

8月 23 17:48:56 hadoop102 ntpd[8300]: Listen normally on 2 lo 127.0.0.1 UDP 123
8月 23 17:48:56 hadoop102 ntpd[8300]: Listen normally on 3 ens33 192.168.10.102 UDP 123
8月 23 17:48:56 hadoop102 ntpd[8300]: Listen normally on 4 virbr0 192.168.122.1 UDP 123
8月 23 17:48:56 hadoop102 ntpd[8300]: Listen normally on 5 lo ::1 UDP 123
8月 23 17:48:56 hadoop102 ntpd[8300]: Listen normally on 6 ens33 fe80::fdaa:4495:fcf0:49f UDP 123
8月 23 17:48:56 hadoop102 ntpd[8300]: Listening on routing socket on fd #23 for interface updates
8月 23 17:48:56 hadoop102 ntpd[8300]: 0.0.0.0 c016 06 restart
8月 23 17:48:56 hadoop102 ntpd[8300]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
8月 23 17:48:56 hadoop102 ntpd[8300]: 0.0.0.0 c011 01 freq_not_set
8月 23 17:49:03 hadoop102 ntpd[8300]: 0.0.0.0 c614 04 freq_mode
[root@hadoop102 lln]# systemctl is-enabled ntpd
disabled

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Hosts on local network are less restricted.
restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap

server 127.127.1.0 
fudge 127.127.1.0 stratum 10

*/1 * * * * /usr/sbin/ntpdate hadoop102

[lln@hadoop103 ~]$ sudo date -s "2021-9-11 11:11:11" 
2021年 09月 11日 星期六 11:11:11 CST

[lln@hadoop103 ~]$ date
2021年 09月 11日 星期六 11:11:15 CST

[lln@hadoop103 ~]$ date
2022年 08月 23日 星期二 18:05:09 CST

没有权限

    <!-- 配置 HDFS 网页登录使用的静态用户为 lln-->
    <property>
    	<name>hadoop.http.staticuser.user</name>
    	<value>lln</value>
	</property>

[lln@hadoop103 ~]$ cd /opt/module/hadoop-3.1.3/etc/hadoop/
[lln@hadoop103 hadoop]$ vim core-site.xml
[lln@hadoop103 hadoop]$ xsync core-site.xml 
==================== hadoop102 ====================
sending incremental file list
core-site.xml

sent 671 bytes  received 47 bytes  1,436.00 bytes/sec
total size is 1,258  speedup is 1.75
==================== hadoop103 ====================
sending incremental file list

sent 62 bytes  received 12 bytes  148.00 bytes/sec
total size is 1,258  speedup is 17.00
==================== hadoop104 ====================
sending incremental file list
core-site.xml

sent 671 bytes  received 47 bytes  1,436.00 bytes/sec
total size is 1,258  speedup is 1.75
[lln@hadoop103 hadoop]$ 



[lln@hadoop103 hadoop]$ myhadoop.sh stop
 =================== 关闭 hadoop 集群 ===================
 --------------- 关闭 historyserver ---------------
 --------------- 关闭 yarn ---------------
Stopping nodemanagers
Stopping resourcemanager
 --------------- 关闭 hdfs ---------------
Stopping namenodes on [hadoop102]
Stopping datanodes
Stopping secondary namenodes [hadoop104]
[lln@hadoop103 hadoop]$ myhadoop.sh start
 =================== 启动 hadoop 集群 ===================
 --------------- 启动 hdfs ---------------
Starting namenodes on [hadoop102]
Starting datanodes
Starting secondary namenodes [hadoop104]
 --------------- 启动 yarn ---------------
Starting resourcemanager
Starting nodemanagers
 --------------- 启动 historyserver ---------------
[lln@hadoop103 hadoop]$ 

output3删除成功！！！