Skip to content

Resilient Distributed Datasets

RDDs are the building blocks of any Spark application.

RDDs Stands for

  1. Resilient: Fault tolerant and is capable of rebuilding data on failure
  2. Distributed: Distributed data among the multiple nodes in a cluster
  3. Dataset: Collection of partitioned data with values

Here are some key points about RDDs and their properties:

  1. Fundamental Data Structure: RDD is the fundamental data structure of Spark, which allows it to efficiently operate on large-scale data across a distributed environment.

  2. Immutability: Once an RDD is created, it cannot be changed. Any transformation applied to an RDD creates a new RDD, leaving the original one untouched.

  3. Resilience: RDDs are fault-tolerant, meaning they can recover from node failures. This resilience is provided through a feature known as lineage, a record of all the transformations applied to the base data.

  4. Lazy Evaluation: RDDs follow a lazy evaluation approach, meaning transformations on RDDs are not executed immediately, but computed only when an action (like count, collect) is performed. This leads to optimized computation.

  5. Partitioning: RDDs are partitioned across nodes in the cluster, allowing for parallel computation on separate portions of the dataset.

  6. In-Memory Computation: RDDs can be stored in the memory of worker nodes, making them readily available for repeated access, and thereby speeding up computations.

  7. Distributed Nature: RDDs can be processed in parallel across a Spark cluster, contributing to the overall speed and scalability of Spark.

  8. Persistence: Users can manually persist an RDD in memory, allowing it to be reused across parallel operations. This is useful for iterative algorithms and fast interactive use.

  9. Operations: Two types of operations can be performed on RDDs - transformations (which create a new RDD) and actions (which return a value to the driver program or write data to an external storage system).

When to Use RDDs (Advantages)#

Despite the general recommendation to use DataFrames/Datasets, RDDs have specific use cases where they are advantageous:

  1. Unstructured Data: RDDs are particularly well-suited for processing unstructured data where there is no predefined schema, such as streams of text, media, or arbitrary bytes. For structured data, DataFrames and Datasets are generally better.

  2. Full Control and Flexibility: If you need fine-grained control over data processing at a very low level and want to optimize the code manually, RDDs provide that flexibility. This means the developer has more control over how data is transformed and distributed.

  3. Type Safety (Compile-Time Errors): RDDs are type-safe. This means that if there's a type mismatch (e.g., trying to add an integer to a string), you will get an error during compile time, before the code even runs. This can help catch errors earlier in the development cycle, unlike DataFrames or SQL queries which might only show errors at runtime

Why You Should NOT Use RDDs (Disadvantages)#

For most modern Spark applications, especially with structured or semi-structured data, RDDs are generally discouraged due to several drawbacks:

  1. No Automatic Optimization by Spark: Spark's powerful Catalyst Optimizer does not perform optimizations automatically for RDD operations. This means the responsibility for writing optimized and efficient code falls entirely on the developer.

  2. Complex and Less Readable Code: Writing RDD code can be complex and less readable compared to DataFrames, Datasets, or SQL. The code often requires explicit handling of data transformations and aggregations, which can be verbose.

  3. Potential for Inefficient Operations: Expensive Shuffling: Without Spark's internal optimizations, RDD operations can lead to inefficient data shuffling. For example, if you perform a reduceByKey (which requires shuffling data across nodes) before a filter operation, Spark will shuffle all the data first, then filter it. If the filter significantly reduces the dataset size, shuffling the larger pre-filtered dataset becomes a very expensive operation. In contrast, DataFrames/Datasets using the "what to" approach allow Spark to rearrange operations (e.g., filter first, then shuffle) to optimize performance, saving significant computational resources.

  4. Developer Burden: Because Spark doesn't optimize RDDs, the developer must have a deep understanding of distributed computing and Spark's internals to write performant RDD code. This makes development harder and slower compared to using higher-level APIs