Encode & decode protobuf messages

Encoders and Decoders for protobuf Messages can be created in three different ways:

  • Fully Automatic derivation: encoders & decoders will automatically be derived from your models. All the fields in the proto schema must be numbered consecutively starting from one.
  • Semi-automatic derivation (recommended): you have to derive an encoder/decoder for each case class involved in protobuf serialization. You can derive with automatic field numbering, or configure a specific mapping.
  • Hand-crafted encoders/decoders: you have the freedom to compose your own encoders/decoders using existing bricks.

Semi-automatic derivation

Usage

Suppose that we need to work with the following proto3 schema:

message Student {
  int64 id = 1;
  string name = 2;
  //uint32 age = 3; -- Deprecated in favor of birthDate
  string birthDate = 4;
  int32 libraryCard = 7;
  repeated Course courses = 8;
}

message Course {
  string name = 1;
  double price = 2;
  bool external = 3;
}

Note that the Student schema has certainly evolved many times, because the numbering is not sequential, unlike the Course, which is numbered consecutively.

In your application, we want to use these types:

import cats.data.NonEmptyList

case class Course(name: String, price: Double) // we doesn't care about the `external` field
case class StudentId(value: Long) extends AnyVal
case class Student(id: StudentId, name: String, birthDate: String, courses: NonEmptyList[Course])

You can summon encoders and decoders for these models with deriveEncoder and deriveDecoder, respectively :

import io.protoless.generic.semiauto._
import shapeless.{::, HNil, Nat}

// Summon automatic decoder for course (note that we will read a partial representation of the message Course)
implicit val courseDecoder = deriveDecoder[Course]

// Summon a decoder for student with a custom mapping between fields (studentId: 1, name: 2, birthDate: 4, courses: 8)
implicit val studentDecoder = deriveDecoder[Student, Nat._1 :: Nat._2 :: Nat._4 :: Nat._8 :: HNil]

implicit val courseEncoder = deriveEncoder[Course]
implicit val studentEncoder = deriveEncoder[Student, Nat._1 :: Nat._2 :: Nat._4 :: Nat._8 :: HNil] // tips: move the fields definition in a type alias

You’re ready to encode and decode protobuf messages:

val student = Student(StudentId(4815162342L), "Kate", "1977-06-21", NonEmptyList.of(
    Course("airline pilot", 8150),
    Course("US marshall", 4912)
))
// student: Student = Student(StudentId(4815162342),Kate,1977-06-21,NonEmptyList(Course(airline pilot,8150.0), Course(US marshall,4912.0)))

val bytes = studentEncoder.encodeAsBytes(student)
// bytes: Array[Byte] = Array(8, -26, -105, -122, -8, 17, 18, 4, 75, 97, 116, 101, 34, 10, 49, 57, 55, 55, 45, 48, 54, 45, 50, 49, 66, 24, 10, 13, 97, 105, 114, 108, 105, 110, 101, 32, 112, 105, 108, 111, 116, 17, 0, 0, 0, 0, 0, -42, -65, 64, 66, 22, 10, 11, 85, 83, 32, 109, 97, 114, 115, 104, 97, 108, 108, 17, 0, 0, 0, 0, 0, 48, -77, 64)

studentDecoder.decode(bytes)
// res6: io.protoless.messages.Decoder.Result[Student] = Right(Student(StudentId(4815162342),Kate,1977-06-21,NonEmptyList(Course(airline pilot,8150.0), Course(US marshall,4912.0))))

You can also use the syntaxic sugar .as[A] and .asProtobufBytes to replace the explicit call on decoders/encoders:

import io.protoless.syntax._

student.asProtobufBytes
bytes.as[Student]

Literal types

If your project use the typelevel fork of Scala, you can define the field mapping with Literal types.

type StudentMapping = 1 :: 2 :: 4 :: 8 :: HNil

Fully Automatic derivation

Automatically derive the required decoder/encoders, using the automatic field numbering strategy.

This approach requires less code, but only works if your messages are numbered consecutively starting from one.

import io.protoless.generic.auto._

case class Cloud(name: String, nickname: Char)
case class Unicorn(color: String, canFlight: Boolean, locations: Seq[Cloud])

val unicorn = Unicorn("pink", true, Seq(Cloud("Celestial", 'c'), Cloud("Misty Cheeky", 'm')))

val bytes = unicorn.asProtobufBytes

bytes.as[Unicorn]

Hand-crafted encoders/decoders

Hand-crafted encoders and decoders give you a total control on how your objects are encoded and decoded. For each field you have to define how and where to read/write it .

It’s useful if you want to validate or transform protobuf message to fit your model.

message Meteo {
  string city = 1;
  string country = 2;
  int32 temperature = 3; // in °F
  float wind = 4;
  float humidity = 5; // optional
}



case class Location(country: String, city: String)
case class Celcius(temp: Float)

sealed trait Weather
case object Sunny extends Weather
case class Rainy(humidity: Float) extends Weather

case class Meteo(location: Location, temp: Celcius, weather: Weather)



import io.protoless.messages._, io.protoless.syntax._

implicit val meteoDecoder = Decoder.instance[Meteo](input =>
  for {
    city <- input.read[String] // if nothing is specified, index are automatically incremented
    country <- input.read[String].map(_.toUpperCase)
    temp <- input.read[Int].map(f => Celcius((f-32f)/1.8f))
    // we don't care about wind
    weather <- input.read[Option[Float]](5).map {
      case Some(humidity) => Rainy(humidity)
      case None => Sunny
    }
  } yield Meteo(Location(city, country), temp, weather)
)

implicit val meteoEncoder = Encoder.instance[Meteo]{ meteo =>
  output =>
    output.write[String](meteo.location.city)
    output.write[String](meteo.location.country)
    output.write[Int]((meteo.temp.temp * 1.8f - 32).toInt)
    output.write[Option[Float]](meteo.weather match {
      case Rainy(humidity) => Some(humidity)
      case _ => None
    }, 5)
}

val bytes = meteoEncoder.encodeAsBytes(Meteo(Location("France", "Montpellier"), Celcius(38), Sunny))
// bytes: Array[Byte] = Array(10, 11, 77, 111, 110, 116, 112, 101, 108, 108, 105, 101, 114, 18, 6, 70, 114, 97, 110, 99, 101, 24, 36)

bytes.as[Meteo]
// res4: io.protoless.messages.Decoder.Result[Meteo] = Right(Meteo(Location(Montpellier,FRANCE),Celcius(2.2222223),Sunny))