Why kryo register not work in SparkSession?

Here's my code:

public static SparkSession getTestSparkSession(String name) {
    SparkConf conf = new SparkConf()
            .set("spark.master", "local")
            .set("spark.ui.port", "8040")
            .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
            .set("spark.kryo.registrator", "com.test.MyKryoRegistrator");
    return SparkSession.builder()
            .appName(name)
            .config(conf)
            .getOrCreate();
}

But it seems that the register does not work:

  • Classes which I've registered in registrator still not serializable.
  • There're no warning/error logs even I set a wrong registrator class.
  • I set the log level to DEBUG, and found there's only one line log about kryo: DEBUG SparkEnv:58 Using serializer: class org.apache.spark.serializer.KryoSerializer.

Questions:

  • How can I check if the registrator works?
  • I'm running the spark session by mvn test. Does a submit operation is needed?

1 answer

  • answered 2017-12-11 06:30 eliasah

    If you want to verify that Spark has taken into account your configuration, You can just ask Spark to get you the conf for the registrator.

    You just need to get the underlying SparkContext from the SparkSession and call the getConf function as followed :

    sparkContext.getConf.get("spark.kryo.registrator") // (this is in scala)
    

    Otherwise, you can always check Spark UI.

    Now if you which to force registration, you'll need to set spark.kryo.registrationRequired to true. (But then you'll need to deal with registering lots of stuff manually)

    The default case of that parameter is set to false, so Kryo will write unregistered class names along with each object. Writing class names can cause significant performance overhead, so enabling this option can enforce strictly that a user has not omitted classes from registration.

    On the other hand, if it is set to 'true', Kryo will throw an exception if an unregistered class is serialized.

    I advise you to read my other answers about the topic :

    I hope this helps !