How do I convert Array[Row] to RDD[Row]

I have a scenario where I want to convert the result of a dataframe which is in the format Array[Row] to RDD[Row]. I have tried using parallelize, but I don't want to use it as it needs to contain entire data in a single system which is not feasible in production box.

val Bid = spark.sql("select Distinct DeviceId, ButtonName  from stb").collect()
val bidrdd = sparkContext.parallelize(Bid)

How do I achieve this? I tried the approach given in this link (How to convert DataFrame to RDD in Scala?), but it didn't work for me.

val bidrdd1 = Bid.map(x => (x(0).toString, x(1).toString)).rdd

It gives an error value rdd is not a member of Array[(String, String)]

1 answer

  • answered 2017-06-17 17:55 Tzach Zohar

    The variable Bid which you've created here is not a DataFrame, it is an Array[Row], that's why you can't use .rdd on it. If you want to get an RDD[Row], simply call .rdd on the DataFrame (without calling collect):

    val rdd = spark.sql("select Distinct DeviceId, ButtonName  from stb").rdd
    

    Your post contains some misconceptions worth noting:

    ... a dataframe which is in the format Array[Row] ...

    Not quite - the Array[Row] is the result of collecting the data from the DataFrame into Driver memory - it's not a DataFrame.

    ... I don't want to use it as it needs to contain entire data in a single system ...

    Note that as soon as you use collect on the DataFrame, you've already collected entire data into a single JVM's memory. So using parallelize is not the issue.