2. 从0到1搭建DeltaLake大数据平台 - 配置Spark集群
1. 配置Master
vi $SPARK_HOME/conf/spark-env.sh
export SPARK_MASTER_HOST=192.168.130.126 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_CORES=2 export SPARK_WORKER_MEMORY=2g
启动Master
$SPARK_HOME/sbin/start-master.sh
Master可以启动成功。
2. 配置Worker节点
export SPARK_MASTER_URL=spark://192.168.130.126:7077 export SPARK_WORKER_MEMORY=2g
启动worker 节点
$SPARK_HOME/sbin/start-worker.sh
//提示如下错误
org.apache.spark.SparkException: Invalid master URL: spark://spark:/192.168.130.126:7077
$SPARK_HOME/sbin/start-worker.sh spark://192.168.130.126:7077
//指定Master，提示如下错误
aused by: java.io.IOException: Failed to connect to /192.168.130.126:7077
worker 节点启动失败，提示连接Master失败，可以ping通Master节点，但是telent提示路由失败。
telnet 192.168.130.126 7077
Trying 192.168.130.126...
telnet: connect to address 192.168.130.126: No route to host
//设置防火墙后，可以Telnet，saprk-worker启动成功
sudo firewall-cmd --add-port=7077/tcp --permanent
sudo firewall-cmd --reload
telnet 192.168.130.126 7077
Trying 192.168.130.126...
Connected to 192.168.130.126.
//下面命令可以启动worker成功，但是不带URL参数，还是报同样错误 （暂时用URL参数启动，之后再查找原因）
$SPARK_HOME/sbin/start-worker.sh spark://192.168.130.126:7077
从Master节点，启动多个节点
在192.168.130.126上生成密钥
ssh-keygen -t rsa -b 4096
在192.168.130.126上复制密钥到远程服务器
ssh-copy-id sparkuser@192.168.130.128
ssh-copy-id sparkuser@192.168.130.130
配置 /opt/spark/conf/worker文件，增加下面的主机
192.168.130.128
192.168.130.130