Hadoop Windows下安装

lwx_R / 2024-09-20 / 原文

1.下载

winutils：https://github.com/s911415/apache-hadoop-3.1.0-winutils，替换bin文件
hadoop：3.2.0 tar 解压

2.环境变量

hadoop version验证
问题：Error：java_home is incorrectly set
解决：由于jdk在C盘引起：hadoop-env.cmd 加入 set JAVA_HOME=C:\PROGRA~1\Java\jdk1.8.0_112

3.配置文件

core-site.xml

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://localhost:900</value>
	</property>
</configuration>

hadoop-env.sh

export JAVA_HOME=C:\Program Files\Java\jdk1.8.0_112\bin

hdfs-site.xml
在hadoop-3.2.0目录下新键data目录，再新建namenode和datanode目录

<configuration>
	<property>       
	<name>dfs.replication</name>       
	<value>1</value>   
	</property>   
	<property>       
	<name>dfs.namenode.name.dir</name>       
	<value>file:///D:/BigData/Hadoop/hadoop-3.2.0/data/namenode</value>
	</property>   
	<property>       
	<name>dfs.datanode.data.dir</name>     
	<value>file:///D:/BigData/Hadoop/hadoop-3.2.0/data/datanode</value>
	</property>
</configuration>

mapred-site.xml

<configuration>   
	<property>       
	<name>mapreduce.framework.name</name>       
	<value>yarn</value>   
	</property>
</configuration>

yarn-site.xml

<configuration>
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>
</configuration>