Scala MapReduce with example

 81 total views,  1 views today

Welcome to Scala series. In this tutorial, you will learn about Scala MapReduce with an example. This post covers how to write basic map reduce program in Scala and please note we will not use Apache Spark code or functionality.

So let us get started.. By the way this is my first post in 2020 🙂

What is MapReduce?

MapReduce is a programming model  of Hadoop. It is of processing technique used widely by the industry to operate on data parallel.There are two main operations/phases map() and reduce().

In map() phase, we convert the input from one form to another. Say for example, you have list of integers and you want the integers to be converted into String. In this case, we can use map() operation to convert list of integers to list of Strings.

In reduce() phase, we will take input from map() operation, combine the input and produce a single output. When I say combine, we can sum the elements or count number of elements etc.

So we combine call it as MapReduce programming model.

Problem statement

  1. Create a list of integers, say to 1 to 100.
  2. Apply map() to multiply each input with 2.
  3. Apply map() to square each input.
  4. Apply filter() to filter only even elements
  5. Apply reduce() to get the sum of all elements.

Scala MapReduce API with example

//MapReduce example

package com.scala

object StaticReference {
  def main(args: Array[String]): Unit = {

    //create List of Integers
    var list = (1 to 100).toList;

    //Multiple each input with 2
    var multiplyMap = list.map(eachElement => eachElement * 2);

    //Square
    var squareMap = multiplyMap.map(eachElement => eachElement ^ 2)

    //Filter only Event Elements
    var filterred = squareMap.filter(eachElement => eachElement % 2 == 0)

    //Apply reduce to get the sum of elements

    var output = filterred.reduce((a, b) => a + b)

    println(output)
  }
}

 

Output

10100

We can write the entire program in single line for better readability and using the functional programming benefits. I am not saying the above way of writing the programme is bad but we can write in better way with functional programming concepts of Scala 🙂 Please feel free to comment if you have different idea !!

//MapReduce Example

package com.scala

object StaticReference {
  def main(args: Array[String]): Unit = {

    //create List of Integers
    var list = (1 to 100).toList;

    var output = list.map(eachElement => eachElement * 2)
      .map(eachElement => eachElement ^ 2)
      .filter(eachElement => eachElement % 2 == 0)
      .reduce((a, b) => a + b)

    println(output)
  }
}

Output

10100

References: 

I hope you like this tutorial and you were able to write a MapReduce example in Scala.

Thanks for reading and please give us a thumb up and comment below!

Tags :

About the Author

Rajasekar

Hey There, My name is Rajasekar and I am the author of this site. I hope you are liking my tutorials and references. Programming and learning new technologies are my passion. The ultimate idea of this site is to share my knowledge(I am still a learner :)) and help you out!. Please spread your words about us (staticreference.com) and give a thumbs up :) Feel free to contact me for any queries!.