Sunday, 15 November 2015

Not getting PiarRDD Functions while using Apache Spark with Scala?

Quick note  (to myself and you) that if you do not import
import org.apache.spark.SparkContext._ 

You will not get the functions which work on PairRDD's in Apache Spark while using Scala.


val domainsUserIds = records1.map(x => (x.getUserId.toString, x.getDomain.toString))
domainsUserIds.reduceByKey - Will not be available unless you explicitly do:

import org.apache.spark.SparkContext._ 
domainsUserIds.reduceByKey (now available)