Detailed steps to deploy Hadoop Distributed File System (HDFS) on CentOS:
1. Preparation
Install Java: Make sure that the system has the appropriate JDK version installed and configure the JAVA_HOME environment variable.
Install Hadoop: Download the corresponding version of Hadoop distribution package and unzip it to the specified directory (for example,
/usr/local/hadoop).
2. Environment configuration
- Set environment variables: Edit
/etc/profilefile and add the following environment variables:
export JAVA_HOME=/path/to/your/jdk export PATH=$JAVA_HOME/bin:$PATH export HADOOP_HOME=/path/to/hadoop export PATH=$HADOOP_HOME/bin:$PATH
Replace /path/to/your/jdk and /path/to/hadoop with actual paths. After saving the file, run source /etc/profile to make the configuration take effect.
- SSH password-free login: Configure SSH password-free login for all Hadoop nodes. Use
ssh-keygen -t rsato generate the key pair, and then usessh-copy-id user@nodeXto copy the public key to each node (replaceuserwith username andnodeXwith nodename).
3. Network configuration
Host Name: Ensure that the host name of each node is correctly configured and accessible through the network.
Static IP: Configure a static IP address for each node. Edit the network configuration file (for example
/etc/sysconfig/network-scripts/ifcfg-eth0), and set the static IP, subnet mask, and gateway.Time synchronization: Use NTP service to synchronize the time of all nodes. Install NTP (
yum install ntp) and synchronize the time usingntpdate ntp.aliyun.com(or other NTP server).
4. HDFS configuration
- Core configuration file (core-site.xml): Configure the HDFS default file system. Modify the
$HADOOP_HOME/etc/hadoop/core-site.xmlfile and add the following content:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://namenode_hostname:9000</value>
</property>
</configuration>
Replace namenode_hostname with the hostname of the NameNode node.
- HDFS configuration file (hdfs-site.xml): Configure HDFS data storage path and number of copies, etc. Modify the
$HADOOP_HOME/etc/hadoop/hdfs-site.xmlfile and add the following content:
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/path/to/namenode/data</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/path/to/datanode/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
Replace /path/to/namenode/data and /path/to/datanode/data with the data storage directory of NameNode and DataNode.
5. Format NameNode
Run the following command on the NameNode node to format NameNode:
$HADOOP_HOME/bin/hdfs namenode -format
6. Start HDFS
Run the following command on the NameNode node to start the HDFS cluster:
$HADOOP_HOME/sbin/start-dfs.sh
VII. Verification and Stop
Verification: Use the
jpscommand to check whether the HDFS daemon has been started. Visithttp://namenode_hostname:50070to view the HDFS Web UI.Stop: Execute the following command on the NameNode node to stop the HDFS cluster:
$HADOOP_HOME/sbin/stop-dfs.sh
Note: The above steps are only basic guides, and the actual configuration may vary depending on Hadoop version and cluster size. Be sure to refer to the official Hadoop documentation for more detailed and accurate information. Please modify the path according to actual conditions.
The above is the detailed content of What are the configuration steps for CentOS HDFS. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Difference between centos and ubuntu
Apr 14, 2025 pm 09:09 PM
The key differences between CentOS and Ubuntu are: origin (CentOS originates from Red Hat, for enterprises; Ubuntu originates from Debian, for individuals), package management (CentOS uses yum, focusing on stability; Ubuntu uses apt, for high update frequency), support cycle (CentOS provides 10 years of support, Ubuntu provides 5 years of LTS support), community support (CentOS focuses on stability, Ubuntu provides a wide range of tutorials and documents), uses (CentOS is biased towards servers, Ubuntu is suitable for servers and desktops), other differences include installation simplicity (CentOS is thin)
Centos shutdown command line
Apr 14, 2025 pm 09:12 PM
The CentOS shutdown command is shutdown, and the syntax is shutdown [Options] Time [Information]. Options include: -h Stop the system immediately; -P Turn off the power after shutdown; -r restart; -t Waiting time. Times can be specified as immediate (now), minutes ( minutes), or a specific time (hh:mm). Added information can be displayed in system messages.
Centos stops maintenance 2024
Apr 14, 2025 pm 08:39 PM
CentOS will be shut down in 2024 because its upstream distribution, RHEL 8, has been shut down. This shutdown will affect the CentOS 8 system, preventing it from continuing to receive updates. Users should plan for migration, and recommended options include CentOS Stream, AlmaLinux, and Rocky Linux to keep the system safe and stable.
10 latest tools for web developers
May 07, 2025 pm 04:48 PM
Web development design is a promising career field. However, this industry also faces many challenges. As more businesses and brands turn to the online marketplace, web developers have the opportunity to demonstrate their skills and succeed in their careers. However, as demand for web development continues to grow, the number of developers is also increasing, resulting in increasingly fierce competition. But it’s exciting that if you have the talent and will, you can always find new ways to create unique designs and ideas. As a web developer, you may need to keep looking for new tools and resources. These new tools and resources not only make your job more convenient, but also improve the quality of your work, thus helping you win more business and customers. The trends of web development are constantly changing.
.NET Core Quick Start Tutorial 1. The beginning: Talking about .NET Core
May 07, 2025 pm 04:54 PM
1. The Origin of .NETCore When talking about .NETCore, we must not mention its predecessor .NET. Java was in the limelight at that time, and Microsoft also favored Java. The Java virtual machine on the Windows platform was developed by Microsoft based on JVM standards. It is said to be the best performance Java virtual machine at that time. However, Microsoft has its own little abacus, trying to bundle Java with the Windows platform and add some Windows-specific features. Sun's dissatisfaction with this led to a breakdown of the relationship between the two parties, and Microsoft then launched .NET. .NET has borrowed many features of Java since its inception and gradually surpassed Java in language features and form development. Java in version 1.6
Centos configuration IP address
Apr 14, 2025 pm 09:06 PM
Steps to configure IP address in CentOS: View the current network configuration: ip addr Edit the network configuration file: sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0 Change IP address: Edit IPADDR= Line changes the subnet mask and gateway (optional): Edit NETMASK= and GATEWAY= Lines Restart the network service: sudo systemctl restart network verification IP address: ip addr
Hadoop pseudo-distributed cluster construction
May 07, 2025 pm 04:45 PM
Software preparation I am using a virtual machine with CentOS-6.6, with the host name repo. Refer to the steps to install a Linux virtual machine in Windows, I installed JDK in that virtual machine, refer to the guide to installing JDK in Linux. In addition, the virtual machine is configured with a key-free login itself, and the settings for configuring key-free login between each virtual machine are referenced. The download address of Hadoop installation package is: https://mirrors.aliyun.com/apache/hadoop/common/. I am using hadoop 2.6.5 version. Upload the Hadoop installation package to the server and unzip [root@repo~]#tarzxv
Postman Integrated Application on CentOS
May 19, 2025 pm 08:00 PM
Integrating Postman applications on CentOS can be achieved through a variety of methods. The following are the detailed steps and suggestions: Install Postman by downloading the installation package to download Postman's Linux version installation package: Visit Postman's official website and select the version suitable for Linux to download. Unzip the installation package: Use the following command to unzip the installation package to the specified directory, for example /opt: sudotar-xzfpostman-linux-x64-xx.xx.xx.tar.gz-C/opt Please note that "postman-linux-x64-xx.xx.xx.tar.gz" is replaced by the file name you actually downloaded. Create symbols
