What is Apache Hadoop?

Apache Hadoop is an open-source framework that enables distributed storage and processing of large datasets across clusters of computers. Originally developed at Yahoo! based on Google's MapReduce paper, Hadoop has become the foundation of modern big data infrastructure.

Why Hadoop?

Organizations generate enormous volumes of data every day — logs, transactions, sensor readings, social media activity — often reaching petabyte scale. Traditional databases struggle to store and query this data efficiently. Hadoop solves this by:

• Scaling horizontally — add commodity hardware nodes instead of expensive servers
• Processing data where it lives — move computation to data, not the other way around
• Tolerating hardware failures — data is automatically replicated across nodes
• Supporting diverse workloads — batch, SQL, streaming, and machine learning

The Four Core Modules

Module	Purpose
Hadoop Common	Shared utilities and libraries used by other modules
HDFS	Distributed file system for reliable, high-throughput storage
MapReduce	Programming model for parallel processing of large datasets
YARN	Resource manager and job scheduler for the cluster

How Data Flows Through Hadoop

Client
  │
  ▼
HDFS (store raw data across DataNodes)
  │
  ▼
YARN (allocate cluster resources)
  │
  ▼
MapReduce / Spark / Hive (process data)
  │
  ▼
Results stored back to HDFS or exported

Who Uses Hadoop?

Hadoop powers data pipelines at companies of all sizes — from startups running a 5-node cluster to enterprises managing clusters of thousands of machines. Common use cases include:

• Log analysis — parse web server or application logs at scale
• ETL pipelines — transform and load data into data warehouses
• Machine learning — train models on terabytes of training data
• Data archiving — cost-effective long-term storage with queryability

Next Steps

Ready to dive in? Start with Installation & Setup to get a Hadoop cluster running on your machine, then explore HDFS Deep Dive to understand how distributed storage works.