Erasure Coding — Slash Storage Costs by 50%​

The biggest storage improvement in Hadoop 3 is Erasure Coding (EC) for HDFS. Previously, the default 3x replication meant storing 200% overhead. With EC (using Reed-Solomon algorithms), you can achieve the same fault tolerance with just 50% overhead — cutting storage costs dramatically for cold or infrequently accessed data.

# Enable erasure coding on a directory
hdfs ec -setPolicy -policy RS-6-3-1024k -path /data/cold-storage
hdfs ec -getPolicy -path /data/cold-storage

Support for More Than 2 NameNodes in HA​

Hadoop 2 supported exactly 2 NameNodes in HA mode. Hadoop 3 supports up to 5 NameNodes, enabling more resilient configurations for large-scale deployments.

YARN Timeline Service v2​

The redesigned YARN Timeline Service v2 offers better scalability using HBase as its backend, replacing the single-writer bottleneck of v1. This makes job history and metrics retrieval much faster on large clusters.

Intra-DataNode Balancer​

A new intra-DataNode disk balancer ensures that data is spread evenly across all disks on a single DataNode, preventing single-disk hotspots that could degrade performance.

hdfs diskbalancer -plan datanode.example.com
hdfs diskbalancer -execute datanode.example.com.plan.json

Java 8 Minimum + Dropped Support for Older Versions​

Hadoop 3 dropped support for Java 7 and requires Java 8 or higher, allowing the codebase to take advantage of modern JVM features.

Architecture Differences​

HDFS co-locates storage and compute. DataNodes store data on local disks, and Hadoop's data locality optimization ensures that MapReduce and Spark tasks run on the same node that holds the data — eliminating network I/O for reads.

Amazon S3 separates storage from compute. Your cluster nodes have no local data; all reads and writes traverse the network. This means data locality is impossible, but it also means your compute cluster can be resized independently of your storage.

Performance​

Latency​

HDFS latency for sequential reads is typically 10-40ms for a local DataNode read. S3 API calls have a base latency of 100-200ms before any data is transferred. For workloads with many small files, this difference is significant.

Cost Model​

With HDFS, you pay for the full node (CPU + memory + disk) even when the cluster is idle. With S3, you pay only for stored bytes when not running jobs, making it dramatically cheaper for intermittent or serverless architectures.

The Hybrid Pattern​

Many organizations adopt a hybrid approach:

• Hot data → HDFS (low latency, high throughput)
• Cold/archival data → S3 (cheap, durable, accessible)
• Hadoop on S3 → Use s3a:// connector with EMRFS or Hadoop 3's S3A committer for correctness

# Access S3 from Hadoop
hdfs dfs -ls s3a://my-bucket/data/
hadoop jar myapp.jar -input s3a://my-bucket/input/ -output s3a://my-bucket/output/

Recommendation​

• On-premises cluster, performance-critical batch jobs → HDFS
• Cloud-native, variable workloads, or data lake → S3 with a cloud Hadoop distribution (EMR, Dataproc, HDInsight)
• Migrating to cloud → Start with S3 for new data, migrate HDFS data gradually

The Core Difference: Memory vs Disk​

MapReduce writes intermediate results to HDFS disk between every Map and Reduce stage. Spark keeps intermediate data in memory (with spill to disk when needed). For iterative algorithms that process the same data repeatedly, this makes Spark orders of magnitude faster.

Performance Benchmark Example​

For a machine learning algorithm that iterates over a dataset 10 times:

When MapReduce Wins​

Despite Spark's performance advantage, MapReduce still makes sense when:

1. Memory is severely constrained — MapReduce handles datasets larger than cluster RAM by spilling everything to disk
2. Long-running, write-once batch jobs — the disk durability of MapReduce is a feature, not a bug
3. Legacy compatibility — existing MapReduce jobs in production don't need to be rewritten if they're working fine

When Spark Wins​

Use Spark when:

1. Iterative ML training — Spark MLlib and graph algorithms benefit enormously from in-memory caching
2. Interactive analytics — Spark's REPL (PySpark, spark-shell) supports exploratory data analysis
3. Streaming — Spark Structured Streaming provides unified batch/streaming APIs
4. SQL workloads — Spark SQL with DataFrames is faster and more expressive than Hive on MapReduce

Running Spark on YARN​

Spark integrates natively with YARN, making it a first-class Hadoop citizen:

# Submit a Spark job to YARN
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-memory 4g \
  --executor-cores 2 \
  myapp.py

# Launch PySpark shell on YARN
pyspark --master yarn --num-executors 5 --executor-memory 2g

Recommendation​

For any new Hadoop workload, start with Spark. Only fall back to MapReduce if you have specific memory constraints or need to maintain a legacy codebase.

What You'll Find Here​

This site is organized into three areas:

Documentation​

Step-by-step guides covering the entire Hadoop stack:

• Getting Started — Install Hadoop, understand HDFS, write your first MapReduce job, and manage resources with YARN
• Advanced Topics — High availability, Kerberos security, performance tuning, and the Hadoop ecosystem

Blog​

Regular articles on:

• Hadoop release highlights and migration guides
• Architecture comparisons (HDFS vs S3, MapReduce vs Spark)
• Real-world deployment patterns and war stories
• Ecosystem tool deep dives

Community​

• Questions? Post on Stack Overflow with the hadoop tag
• Bugs and features: Apache Hadoop JIRA
• Official docs: hadoop.apache.org

A Note on Versions​

This site targets Hadoop 3.x (the current stable series). Where behavior differs significantly from Hadoop 2.x, we'll call it out explicitly.

Happy learning!

What Changed in Hadoop 3.x​

Before upgrading, understand the key differences:

The port changes alone can break existing monitoring, firewall rules, and client configs — plan for those carefully.

Pre-Upgrade Checklist​

Before touching a single config file:

1. Audit all client applications for hardcoded ports (50070, 8020, 50010, etc.)
2. Check Java version — every node must run Java 8 or higher
3. Review deprecated APIs — several mapred and dfs shell commands were removed
4. Back up namenode metadata:

   hdfs dfsadmin -saveNamespace
   cp -r /path/to/namenode/current /backup/namenode-$(date +%Y%m%d)
   

5. Snapshot your HDFS data directories on each DataNode if possible
6. Read the release notes for your specific target version (3.3.x or 3.4.x)

Step 1: Upgrade HDFS Metadata​

The NameNode metadata format must be finalized before DataNodes are upgraded.

1.1 — Put NameNode in safemode and save namespace​

hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode leave

1.2 — Stop all services in order​

stop-yarn.sh
stop-dfs.sh

Stop MapReduce Job History Server last:

mapred --daemon stop historyserver

1.3 — Upgrade NameNode​

Replace the Hadoop binaries on the NameNode host with the 3.x release, then run:

hdfs namenode -upgrade

This writes a new metadata layout while preserving the old layout in a previous/ directory — allowing rollback if needed.

Step 2: Upgrade DataNodes​

With the NameNode upgraded and running, start each DataNode with the new binaries:

hdfs --daemon start datanode

DataNodes are backward-compatible during the rolling upgrade window. You can upgrade them one at a time and keep HDFS serving data throughout.

Monitor upgrade progress:

hdfs dfsadmin -upgradeProgress status

Step 3: Upgrade YARN​

ResourceManager and NodeManagers can be rolled independently in Hadoop 3.x thanks to the work-preserving restart feature.

3.1 — Upgrade ResourceManager​

yarn --daemon stop resourcemanager
# Replace binaries
yarn --daemon start resourcemanager

3.2 — Rolling NodeManager upgrade​

# On each node, one at a time:
yarn --daemon stop nodemanager
# Replace binaries
yarn --daemon start nodemanager

Running containers are preserved across NodeManager restarts (work-preserving upgrade).

Step 4: Update Configuration Files​

Hadoop 3.x uses different default ports. Update core-site.xml, hdfs-site.xml, and any clients pointing to old ports:

Old → New port mappings:

NameNode RPC:       8020  → 9000 (or keep 8020 with explicit config)
NameNode Web UI:    50070 → 9870
Secondary NN:       50090 → 9868
DataNode Web UI:    50075 → 9864
DataNode transfer:  50010 → 9866
DataNode IPC:       50020 → 9867

Update core-site.xml if using the old port:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://namenode-host:9000</value>
</property>

Step 5: Finalize the Upgrade​

Once you've validated that everything is working correctly, finalize the upgrade to reclaim the space used by the previous/ layout backup:

hdfs dfsadmin -finalizeUpgrade

Warning: After finalization, rollback is no longer possible.

Rollback Procedure (if needed)​

If you encounter critical issues before finalization:

stop-dfs.sh
hdfs namenode -rollback
start-dfs.sh

This reverts the NameNode metadata to the Hadoop 2.x layout. DataNodes also need to be rolled back with the 2.x binaries.

Common Upgrade Issues​

Shell Script Changes​

Hadoop 3.x reworked the shell scripts. Commands like hadoop-daemon.sh are deprecated in favor of:

# Old (2.x)
hadoop-daemon.sh start datanode
# New (3.x)
hdfs --daemon start datanode

Classpath Changes​

Third-party tools (Hive, HBase, Spark) that relied on Hadoop's classpath may need updated versions compatible with Hadoop 3.x. Check each ecosystem component's compatibility matrix.

YARN Timeline Service v2​

YARN Timeline Service v2 requires HBase as a backend. If you relied on Timeline Service v1, plan the HBase deployment before enabling v2:

<!-- yarn-site.xml -->
<property>
  <name>yarn.timeline-service.version</name>
  <value>2.0f</value>
</property>

Post-Upgrade Verification​

# Verify HDFS health
hdfs dfsadmin -report
hdfs fsck / -summary

# Check YARN cluster
yarn node -list
yarn application -list

# Run a test job
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 10 100

A successful Pi estimation job confirms that HDFS and YARN are both operational end-to-end.

Summary​

Upgrading Hadoop 2 to 3 is operationally straightforward when done in order. The biggest surprises tend to come from port changes and ecosystem tool compatibility — audit those before you start and the rest is mechanical.

Why S3A?​

Amazon S3 offers virtually unlimited capacity at a fraction of the cost of on-premises HDFS. With the S3A connector (the third and current generation, replacing the older s3:// and s3n:// implementations), Hadoop jobs read and write S3 objects using familiar s3a://bucket/path URIs — no code changes required.

Key advantages of S3A:

• Storage decoupling — scale compute and storage independently
• Durability — S3 provides 99.999999999% (11 nines) object durability
• Cost — typically 70–80% cheaper than equivalent HDFS on-premises storage per GB
• Multi-region availability — replicate data across AWS regions easily

Core Configuration​

Add the following to core-site.xml on all cluster nodes. Never store credentials in config files for production — use IAM roles instead.

<!-- core-site.xml -->
<configuration>
  <!-- S3A implementation class -->
  <property>
    <name>fs.s3a.impl</name>
    <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
  </property>

  <!-- AWS credentials (use IAM roles in production instead) -->
  <property>
    <name>fs.s3a.access.key</name>
    <value>YOUR_ACCESS_KEY</value>
  </property>
  <property>
    <name>fs.s3a.secret.key</name>
    <value>YOUR_SECRET_KEY</value>
  </property>

  <!-- AWS region -->
  <property>
    <name>fs.s3a.endpoint.region</name>
    <value>us-east-1</value>
  </property>
</configuration>

For production on EC2, use an IAM instance role and omit access/secret keys entirely — Hadoop will retrieve credentials from the EC2 metadata service automatically.

Authentication Methods​

S3A supports multiple credential providers, tried in order:

<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    com.amazonaws.auth.profile.ProfileCredentialsProvider
  </value>
</property>

Provider priority (recommended for production):

1. IAM Instance Role (EC2) or IAM Task Role (ECS/EKS)
2. AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
3. AWS credentials file (~/.aws/credentials)
4. Hardcoded keys in core-site.xml (avoid in production)

Performance Tuning​

S3 is not a filesystem — it's an object store. Network latency and request overhead are the main performance factors. These settings dramatically improve throughput:

Parallel Upload (Multipart)​

<!-- Minimum part size for multipart upload (default 5MB) -->
<property>
  <name>fs.s3a.multipart.size</name>
  <value>67108864</value> <!-- 64MB -->
</property>

<!-- Threshold above which multipart upload is used (default 128MB) -->
<property>
  <name>fs.s3a.multipart.threshold</name>
  <value>134217728</value> <!-- 128MB -->
</property>

Connection Pool Size​

<!-- Max connections to S3 per JVM -->
<property>
  <name>fs.s3a.connection.maximum</name>
  <value>100</value>
</property>

<!-- Enable HTTP keep-alive -->
<property>
  <name>fs.s3a.connection.ssl.enabled</name>
  <value>true</value>
</property>

Prefetch and Read-Ahead​

<!-- Prefetch block size for sequential reads -->
<property>
  <name>fs.s3a.readahead.range</name>
  <value>1048576</value> <!-- 1MB -->
</property>

<!-- Enable predictive prefetch (Hadoop 3.3.5+) -->
<property>
  <name>fs.s3a.prefetch.enabled</name>
  <value>true</value>
</property>
<property>
  <name>fs.s3a.prefetch.block.size</name>
  <value>8388608</value> <!-- 8MB -->
</property>

Fast Upload Buffer​

S3A's fast upload mode buffers data in memory or disk before uploading, improving job throughput:

<property>
  <name>fs.s3a.fast.upload</name>
  <value>true</value>
</property>

<!-- Buffer location: disk, array, or bytebuffer -->
<property>
  <name>fs.s3a.fast.upload.buffer</name>
  <value>disk</value>
</property>

Committer: Handling the Rename Problem​

S3 doesn't support atomic directory renames. The classic FileOutputCommitter moves output files from a _temporary/ directory to the final path — on HDFS this is a metadata operation, but on S3 it means copying every byte. For large outputs this is catastrophic.

Use the S3A Magic Committer instead:

<!-- mapred-site.xml -->
<property>
  <name>mapreduce.outputcommitter.factory.scheme.s3a</name>
  <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value>
</property>
<property>
  <name>fs.s3a.committer.name</name>
  <value>magic</value>
</property>
<property>
  <name>fs.s3a.committer.magic.enabled</name>
  <value>true</value>
</property>

The Magic Committer uses S3's multipart upload API to write files directly to their final location during the task — commit just calls CompleteMultipartUpload and is nearly instantaneous.

Working with S3A​

Basic file operations​

# List bucket contents
hadoop fs -ls s3a://my-bucket/data/

# Copy local file to S3
hadoop fs -put localfile.csv s3a://my-bucket/input/

# Run a MapReduce job with S3 input/output
hadoop jar hadoop-mapreduce-examples-*.jar wordcount \
  s3a://my-bucket/input/ \
  s3a://my-bucket/output/wordcount/

# Check S3 usage (note: no block count, S3 is object store)
hadoop fs -du -h s3a://my-bucket/

Per-bucket configuration​

You can configure different credentials or endpoints per bucket using per-bucket properties:

<property>
  <name>fs.s3a.bucket.my-west-bucket.endpoint.region</name>
  <value>us-west-2</value>
</property>
<property>
  <name>fs.s3a.bucket.my-west-bucket.access.key</name>
  <value>WEST_REGION_ACCESS_KEY</value>
</property>

S3-Compatible Object Stores​

S3A works with any S3-compatible object store by overriding the endpoint:

<!-- MinIO example -->
<property>
  <name>fs.s3a.endpoint</name>
  <value>http://minio.internal:9000</value>
</property>
<property>
  <name>fs.s3a.path.style.access</name>
  <value>true</value>
</property>

This enables the same Hadoop code to run against MinIO (on-premises), Ceph RGW, Cloudflare R2, or Backblaze B2 — same config pattern, different endpoint.

Common Issues and Fixes​

Summary​

The S3A connector transforms Hadoop into a hybrid compute engine that can process data wherever it lives. Key takeaways:

• Use IAM roles for authentication — never hardcode credentials
• Enable the Magic Committer to avoid catastrophic rename overhead
• Tune multipart upload size and connection pool for your workload
• S3 strong consistency (available since 2020) eliminates the "read-after-write" problem in modern Hadoop releases
• Per-bucket configuration lets you span multiple regions and credential sets

With S3A, running Hadoop workloads on transient EMR clusters or spot instances — reading and writing directly to S3 — becomes a practical, cost-efficient architecture.

Why DataNodes Are a Security Target​

DataNodes are the workhorses of HDFS — they store actual data blocks and serve reads/writes to clients. In an unsecured cluster:

• Any process that can reach port 9866 (DataNode transfer port) can read or write blocks directly
• There's no per-user access control on who reads which data
• A rogue client can inject corrupt or malicious blocks

Hadoop's security model addresses all of this through Kerberos-based mutual authentication, block access tokens, and optional wire encryption.

Layer 1: Kerberos Authentication​

Kerberos is the foundation of Hadoop security. Every Hadoop service (NameNode, DataNode, ResourceManager, NodeManager) authenticates with a Kerberos principal before communicating.

Prerequisites​

• A running Kerberos KDC (MIT Kerberos or Active Directory)
• DNS properly configured (Kerberos is very sensitive to hostname resolution)
• Synchronized clocks across all nodes (within 5 minutes; use NTP)

Create Service Principals​

For each DataNode host, create a principal:

# On the KDC
kadmin.local -q "addprinc -randkey hdfs/datanode1.example.com@EXAMPLE.COM"
kadmin.local -q "addprinc -randkey host/datanode1.example.com@EXAMPLE.COM"

# Export keytabs
kadmin.local -q "ktadd -k /etc/security/keytabs/hdfs.keytab hdfs/datanode1.example.com@EXAMPLE.COM"

Copy keytabs to each DataNode at /etc/security/keytabs/hdfs.keytab with ownership hdfs:hdfs and mode 400.

Enable Security in hdfs-site.xml​

<!-- hdfs-site.xml -->
<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>

<!-- DataNode SASL RPC authentication -->
<property>
  <name>dfs.datanode.kerberos.principal</name>
  <value>hdfs/_HOST@EXAMPLE.COM</value>
</property>
<property>
  <name>dfs.datanode.keytab.file</name>
  <value>/etc/security/keytabs/hdfs.keytab</value>
</property>

Enable Security in core-site.xml​

<!-- core-site.xml -->
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>
<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>
<property>
  <name>hadoop.rpc.protection</name>
  <value>authentication</value>  <!-- or privacy for encryption -->
</property>

Layer 2: Block Access Tokens​

Block access tokens prevent unauthorized direct block reads/writes even from nodes that have network access to a DataNode. The NameNode issues a short-lived token when a client requests a block location; the DataNode validates the token before serving data.

Enable with:

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>
<property>
  <name>dfs.block.access.token.lifetime</name>
  <value>600</value> <!-- seconds; default 600 -->
</property>

Without block tokens, a client who obtains a block location (host:port + block ID) can read that block without further auth. With tokens, the NameNode effectively gatekeeps all data transfers.

Layer 3: Wire Encryption​

Even with Kerberos, data transferred between DataNodes and clients is in plaintext by default. Enable encryption for data in transit:

RPC Encryption (control plane)​

<!-- core-site.xml -->
<property>
  <name>hadoop.rpc.protection</name>
  <value>privacy</value>  <!-- authentication = auth only; integrity = + checksums; privacy = + encryption -->
</property>

Data Transfer Encryption (data plane)​

<!-- hdfs-site.xml -->
<property>
  <name>dfs.encrypt.data.transfer</name>
  <value>true</value>
</property>
<property>
  <name>dfs.encrypt.data.transfer.algorithm</name>
  <value>rc4</value>  <!-- or 3des for FIPS compliance -->
</property>
<property>
  <name>dfs.encrypt.data.transfer.cipher.suites</name>
  <value>AES/CTR/NoPadding</value>  <!-- hardware-accelerated AES on modern CPUs -->
</property>

AES/CTR with hardware acceleration (AES-NI, available on most modern Intel/AMD CPUs) adds only 5–10% overhead compared to unencrypted transfer.

Layer 4: DataNode SASL on Privileged Ports​

Running the DataNode data transfer on a privileged port (below 1024) proves that the process was started as root and later dropped privileges — adding OS-level verification. This is optional but adds defense in depth.

<property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:1004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:1006</value>
</property>

When using SASL (Hadoop 2.6+) instead of privileged ports, the DataNode proves its identity through Kerberos without needing root-owned ports:

<property>
  <name>dfs.datanode.require.secure.ports</name>
  <value>false</value>
</property>
<property>
  <name>dfs.http.policy</name>
  <value>HTTPS_ONLY</value>
</property>

Layer 5: OS Hardening​

Kerberos secures the Hadoop layer, but the underlying OS must also be locked down:

File Permissions​

# DataNode data directories should be owned by the hdfs user
chown -R hdfs:hadoop /data/hdfs/dn
chmod 700 /data/hdfs/dn

# Keytab files must not be world-readable
chmod 400 /etc/security/keytabs/hdfs.keytab
chown hdfs:hdfs /etc/security/keytabs/hdfs.keytab

Network Restrictions​

Restrict DataNode ports to cluster-internal network ranges using iptables or firewalld:

# Only allow DataNode transfer port from within the cluster subnet
iptables -A INPUT -p tcp --dport 9866 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 9866 -j DROP

Run DataNode as Non-Root​

The DataNode process should run as the hdfs system user, not root:

# /etc/hadoop/hadoop-env.sh
export HDFS_DATANODE_USER=hdfs
export HDFS_DATANODE_SECURE_USER=hdfs

Verifying Security Configuration​

After enabling security, verify that everything works:

# Obtain a Kerberos ticket for the hdfs service user
kinit -kt /etc/security/keytabs/hdfs.keytab hdfs/namenode.example.com@EXAMPLE.COM

# List HDFS root (should succeed)
hdfs dfs -ls /

# Check that unauthenticated access is denied
kdestroy
hdfs dfs -ls /  # Should fail with "No valid credentials"

Run an HDFS health check with auth:

kinit -kt /etc/security/keytabs/hdfs.keytab hdfs/namenode.example.com@EXAMPLE.COM
hdfs dfsadmin -report
hdfs fsck / -summary

Security Audit Checklist​

Summary​

Securing a Hadoop DataNode involves multiple complementary layers:

1. Kerberos — mutual authentication between services and clients
2. Block access tokens — prevent unauthorized direct block access
3. Wire encryption — protect data in transit (RPC + data transfer)
4. Privileged ports or SASL — OS-level service identity verification
5. OS hardening — file permissions, firewall, non-root user

No single layer is sufficient on its own. A properly secured DataNode requires all these working together. For fine-grained row-level and column-level access control beyond what HDFS ACLs provide, look at Apache Ranger as the next step.

Quick Reference: Hadoop ↔ Java Compatibility Matrix​

Rule of thumb for 2025: Run Java 11 for stability. Java 17 with Hadoop 3.4.x is becoming viable. Java 8 is approaching EOL and should be phased out.

Java 8: The Legacy Standard​

Hadoop 3.x requires Java 8 as the minimum. Java 8 was the de facto standard for Hadoop workloads from 2014 through about 2022.

Current status: Oracle Java 8 reached end-of-life for public updates in March 2022. Adoptium (Temurin) and Amazon Corretto still provide free, maintained Java 8 builds, but the ecosystem is moving on.

When to use: Only if your organization has a hard dependency on a third-party library or Hadoop ecosystem component that isn't yet compatible with Java 11+.

Install Temurin 8 (Ubuntu/Debian)​

sudo apt-get install -y wget apt-transport-https
wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | sudo apt-key add -
echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | sudo tee /etc/apt/sources.list.d/adoptium.list
sudo apt-get update
sudo apt-get install -y temurin-8-jdk

Java 11: The Recommended Choice (LTS)​

Java 11, released in 2018, is the most tested Java version with Hadoop 3.x as of 2025. All major ecosystem tools (Spark, Hive, HBase, Flink) support Java 11.

Key changes from Java 8​

• Module system (Project Jigsaw) — --add-opens flags needed for reflective access
• G1GC is now default — better pause times for large heaps (critical for NameNode with hundreds of millions of files)
• javax.* → jakarta.* — does not affect Hadoop itself, but affects some ecosystem tools
• Removed deprecated API — sun.misc.Unsafe usage patterns changed

Required JVM flags for Hadoop on Java 11+​

Hadoop's internal code (and many dependencies) use reflective access that Java 11's module system restricts by default. Add these to hadoop-env.sh:

export HADOOP_OPTS="$HADOOP_OPTS \
  --add-opens=java.base/java.lang=ALL-UNNAMED \
  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
  --add-opens=java.base/java.io=ALL-UNNAMED \
  --add-opens=java.base/java.net=ALL-UNNAMED \
  --add-opens=java.base/java.util=ALL-UNNAMED \
  --add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
  --add-opens=java.base/sun.net.dns=ALL-UNNAMED \
  --add-opens=java.base/sun.net.util=ALL-UNNAMED"

Hadoop 3.3.0 and later automatically adds most of these — check your version before manually adding them.

Java 17: Stricter Modules, Better Performance​

Java 17 (LTS, 2021) tightened the module encapsulation started in Java 11. It fully enforces strong encapsulation of JDK internals that Java 11 only warned about.

Hadoop compatibility​

• Hadoop 3.3.4 and earlier: Not supported — too many internal reflection violations
• Hadoop 3.3.5+: Experimental support
• Hadoop 3.4.0+: Officially supported

What required fixing for Java 17​

The main breaking changes were in sun.misc.Unsafe usage and java.lang.reflect access patterns throughout Hadoop's RPC and serialization code. The community patched these across multiple JIRAs (HADOOP-17975, HADOOP-18079, and others).

GC improvements relevant to Hadoop​

Java 17 includes ZGC and Shenandoah as production-ready collectors (not just experimental as in Java 11). For NameNode workloads with large heaps (32–512GB), ZGC's sub-millisecond pauses can dramatically improve NameNode responsiveness:

# hadoop-env.sh — NameNode with ZGC
export HDFS_NAMENODE_OPTS="-Xms64g -Xmx64g -XX:+UseZGC $HDFS_NAMENODE_OPTS"

Java 21: The Future LTS​

Java 21 (LTS, 2023) is the current long-term support release. It brings virtual threads (Project Loom), pattern matching, and record patterns.

Hadoop compatibility​

Full support for Java 21 is still in progress as of early 2025. Hadoop 3.4.x has preliminary support — test thoroughly before using in production.

Virtual threads are particularly interesting for Hadoop's IPC layer, which handles thousands of concurrent RPC connections. Future Hadoop releases may leverage virtual threads to reduce thread-pool overhead on NameNodes and ResourceManagers.

Setting JAVA_HOME for Hadoop​

Hadoop reads JAVA_HOME from hadoop-env.sh. Always set it explicitly rather than relying on the system default — different users or shell environments may have different defaults:

# /etc/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/temurin-11-jdk-amd64

# Or use java_home helper on macOS:
# export JAVA_HOME=$(/usr/libexec/java_home -v 11)

Verify that every node in the cluster uses the same Java version and JAVA_HOME path to avoid subtle serialization incompatibilities.

Multi-JDK Environments: SDKMAN!​

If you maintain multiple Hadoop environments with different Java requirements, SDKMAN! makes switching trivial:

# Install SDKMAN!
curl -s "https://get.sdkman.io" | bash

# Install multiple JDKs
sdk install java 11.0.22-tem
sdk install java 17.0.10-tem
sdk install java 21.0.2-tem

# Switch for a session
sdk use java 11.0.22-tem

# Set a default
sdk default java 11.0.22-tem

Checking Your Current Setup​

# Check Java version
java -version

# Check what Hadoop thinks JAVA_HOME is
hadoop version

# Check which JVM is actually running the NameNode
ps aux | grep NameNode
# Find the PID, then:
ls -la /proc/<PID>/exe

Summary Recommendations​

The Java version you run affects not just Hadoop but every ecosystem tool layered on top: Spark, Hive, HBase, Kafka. Coordinate your Java version across the entire platform before upgrading, and always test with your actual workloads — JVM GC behavior at scale can differ significantly from simple benchmarks.

What Is a YARN Container?​

A YARN container is a reservation of resources (CPU cores and memory) on a NodeManager. It's the unit in which YARN launches application tasks — MapReduce map/reduce tasks, Spark executors, Tez tasks, and so on all run inside YARN containers.

Each container has:

• Memory (in MB) — hard limit enforced by cgroups or virtual memory check
• vCores (virtual CPU cores) — a logical unit, not necessarily a physical core
• A host (the NodeManager it runs on)
• A priority within the application

Containers are ephemeral: they're created for a task, run to completion, and are released.

How Container Allocation Works​

The flow from application submission to container launch:

Client
  │
  └──► ApplicationMaster (AM) submitted to YARN
         │
         └──► AM registers with ResourceManager
                │
                └──► AM requests containers (resource requests)
                       │
                       └──► ResourceManager scheduler evaluates queue + node availability
                              │
                              └──► ResourceManager issues container allocations
                                     │
                                     └──► AM contacts NodeManager → container launch

The ResourceManager's scheduler is the brain. It tracks available resources on each NodeManager and matches pending container requests against available capacity, obeying queue limits, node labels, and locality preferences.

Resource Request Parameters​

An ApplicationMaster sends resource requests with these key parameters:

Locality preference order​

YARN tries to honor locality in this order:

1. Node-local — container on the same node as the data block
2. Rack-local — container on a node in the same rack
3. Off-rack — any node in the cluster

For MapReduce, HDFS block locality means the map task ideally runs on a node that already has the block locally — avoiding network transfer entirely.

NodeManager Configuration​

Configure how much resource each NodeManager exposes to YARN in yarn-site.xml:

<!-- Total memory available to YARN containers on this node -->
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>49152</value> <!-- 48GB; leave some for OS and system processes -->
</property>

<!-- Total vCores available on this node -->
<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>20</value> <!-- Leave 2-4 for OS -->
</property>

<!-- Minimum container memory allocation -->
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>

<!-- Maximum container memory allocation -->
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>16384</value>
</property>

<!-- Minimum container vCores -->
<property>
  <name>yarn.scheduler.minimum-allocation-vcores</name>
  <value>1</value>
</property>

<!-- Maximum container vCores -->
<property>
  <name>yarn.scheduler.maximum-allocation-vcores</name>
  <value>8</value>
</property>

Important: Container allocations are rounded up to the nearest increment of minimum-allocation-mb. Requesting 1.5GB on a cluster with 1GB minimum gets you 2GB. Size your minimum allocation to match typical workloads.

Memory Enforcement: Virtual vs Physical​

YARN can enforce memory limits in two ways:

Virtual Memory Check (default on)​

YARN monitors the virtual memory (VM size) of each container process. If it exceeds yarn.nodemanager.vmem-pmem-ratio times the allocated physical memory, the container is killed:

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.nodemanager.vmem-pmem-ratio</name>
  <value>2.1</value> <!-- Default; container can use 2.1x its physical memory as virtual memory -->
</property>

This is a common source of spurious container kills on modern Linux systems where JVM virtual memory usage appears large. If you see Container killed due to exceeding virtual memory limits, either increase the ratio or disable the check:

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>

Physical Memory Check​

<property>
  <name>yarn.nodemanager.pmem-check-enabled</name>
  <value>true</value>
</property>

Physical memory enforcement kills containers that exceed their allocated memory in RAM. This prevents one container from starving others on the same node.

cgroups Enforcement (recommended)​

The most reliable enforcement uses Linux cgroups. The NodeManager enforces both CPU and memory limits at the OS level:

<property>
  <name>yarn.nodemanager.container-executor.class</name>
  <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
  <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
  <value>/hadoop-yarn</value>
</property>
<property>
  <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
  <value>true</value>
</property>
<property>
  <name>yarn.nodemanager.resource.cpu.enabled</name>
  <value>true</value>
</property>

Scheduler Types​

Capacity Scheduler (default)​

The Capacity Scheduler divides cluster resources into queues, each with a guaranteed minimum capacity and an optional maximum:

<!-- capacity-scheduler.xml -->
<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>default,production,development</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.production.capacity</name>
  <value>60</value> <!-- 60% of cluster -->
</property>
<property>
  <name>yarn.scheduler.capacity.root.development.capacity</name>
  <value>30</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.capacity</name>
  <value>10</value>
</property>

Queues can borrow unused capacity from sibling queues (elastic sharing) and return it when the owner queue needs it.

Fair Scheduler​

The Fair Scheduler distributes resources so that all running applications get an equal share over time. Good for multi-tenant environments with diverse workload sizes:

<!-- yarn-site.xml -->
<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

Diagnosing Container Issues​

Application stuck waiting for resources​

# Check what resources are available
yarn node -list -all

# Check scheduler queue utilization
yarn queue -status default

# Check ResourceManager logs for why containers aren't being assigned
# Look for: "Application ... is waiting for ..."

Container killed: OOM​

# Check NodeManager logs on the host where the container ran
# Look for: "Container killed due to Physical memory limit"
# Or check YARN application logs:
yarn logs -applicationId application_XXXX_XXXX

Increase the container memory allocation in your job configuration:

# MapReduce
mapred job -Dmapreduce.map.memory.mb=4096 -Dmapreduce.reduce.memory.mb=8192

Container killed: virtual memory​

# Quick fix: disable vmem check in yarn-site.xml
# Better fix: understand JVM memory layout on your OS
# The JVM reserves large amounts of virtual address space for mapped libraries

Sizing Containers for Common Workloads​

MapReduce​

<!-- mapred-site.xml -->
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>2048</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1638m</value> <!-- 80% of container memory for JVM heap -->
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx3276m</value>
</property>

Spark on YARN​

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 8g \
  --executor-cores 4 \
  --num-executors 20 \
  --conf spark.yarn.executor.memoryOverhead=1024 \
  my_spark_app.jar

memoryOverhead covers JVM off-heap memory (direct buffers, native libraries). Set it to at least 384MB or 10% of executor memory, whichever is larger.

Summary​

Mastering YARN container allocation turns resource utilization from a guessing game into a predictable, measurable engineering discipline. Profile your actual workloads, size containers deliberately, and use queue priorities to give critical jobs the resources they need.