Prerequisites on each nodes before building the cluster:
1. Choosing the supported operating system.
2. Choosing the supported java.
https://wiki.apache.org/hadoop/HadoopJavaVersions
3. Switch off iptables:
Netfilter is a host-based firewall for Linux. It is included as part of the Linux distribution and activated by default. This firewall is controlled by the program called iptable and this should be turned off. iptables applies to IPv4.
Type the following two commands (you must login as the root user):
# /etc/init.d/iptables save
# /etc/init.d/iptables stop
To turn off firewall on reboot:
# chkconfig iptables off
4. Disabling Transparent Hugepage Compaction
The transparent hugepage will automatically use larger pages for dynamically allocated memory and this is not recommended to be enabled for Hadoop.
For RHEL depending upon the version, to disable transparent hugepage compaction, add the following command to /etc/rc.local: (rc.local is the run level script which gets exectued after all the normal services are started.)
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
If reading the above file (defrag/enabled) [always] never means it is enabled
If reading the above file (defrag/enabled) [never] always means it is disabled
5. vm.swappiness Linux Kernel Parameter
Set vm.swappiness to a value 1. We do not want the swappiness on the nodes. Needs to be minimum as possible.
sysctl -w vm.swappiness=1
