SIP-61 - Unroll Default Arguments for Binary Compatibility

By: Li Haoyi

History

Summary

This SIP proposes an @unroll annotation lets you add additional parameters to method defs,class construtors, or case classes, without breaking binary compatibility. @unroll works by generating “unrolled” or “telescoping” forwarders:

// Original
def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l

// Generated
def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
def foo(s: String, n: Int) = foo(s, n, true, 0)

In contrast to most existing or proposed alternatives that require you to contort your code to become binary compatible (see Major Alternatives), @unroll allows you to write Scala with vanilla defs/classes/case classes, add a single annotation, and your code will maintain binary compatibility as new default parameters and fields are added over time.

@unroll’s only constraints are that:

1. New parameters need to have a default value
2. New parameters can only be added on the right
3. The @unrolled methods must be abstract or final

These are both existing industry-wide standard when dealing with data and schema evolution (e.g. Schema evolution in Avro, Protocol Buffers and Thrift — Martin Kleppmann’s blog), and are also the way the new parameters interact with source compatibility in the Scala language. Thus these constraints should be immediately familiar to any experienced programmers, and would be easy to follow without confusion.

Prior Discussion can be found here

Motivation

Maintaining binary compatibility of Scala libraries as they evolve over time is difficult. Although tools like https://github.com/lightbend/mima help surface issues, actually resolving those issues is a different challenge.

Some kinds of library changes are fundamentally impossible to make compatible, e.g. removing methods or classes. But there is one big class of binary compatibility issues that are “spurious”: adding default parameters to methods, class constructors, or case classes.

Adding a default parameter is source-compatible, but not binary compatible: a user downstream of a library that adds a default parameter does not need to make any changes to their code, but does need to re-compile it. This is “spurious” because there is no fundamental incompatibility here: semantically, a new default parameter is meant to be optional! Old code invoking that method without a new default parameter is exactly the user intent, and works just fine if the downstream code is re-compiled.

Other languages, such as Python, have the same default parameter language feature but face no such compatibility issues with their use. Even Scala codebases compiled from source do not suffer these restrictions: adding a default parameter to the right side of a parameter list is for all intents and purposes backwards compatible in a mono-repo setup. The fact that such addition is binary incompatible is purely an implementation restriction of Scala’s binary artifact format and distribution strategy.

Binary compatibility is generally more important than Source compatibility. When you hit a source compatibility issue, you can always change the source code you are compiling, whether manually or via your build tool. In contrast, when you hit binary compatibility issues, it can come in the form of diamond dependencies that would require re-compiling all of your transitive dependencies, a task that is far more difficult and often impractical.

There are many approaches to resolving these “spurious” binary compatibility issues, but most of them involve either tremendous amounts of boilerplate writing binary-compatibility forwarders, giving up on core language features like Case Classes or Default Parameters, or both. Consider the following code snippet (link) from the com-lihaoyi/mainargs library, which duplicates the parameters of def constructEither no less than five times in order to maintain binary compatibility as the library evolves and more default parameters are added to def constructEither:

  def constructEither(
      args: Seq[String],
      allowPositional: Boolean,
      allowRepeats: Boolean,
      totalWidth: Int,
      printHelpOnExit: Boolean,
      docsOnNewLine: Boolean,
      autoPrintHelpAndExit: Option[(Int, PrintStream)],
      customName: String,
      customDoc: String,
      sorted: Boolean,
  ): Either[String, T] = constructEither(
    args,
    allowPositional,
    allowRepeats,
    totalWidth,
    printHelpOnExit,
    docsOnNewLine,
    autoPrintHelpAndExit,
    customName,
    customDoc,
    sorted,
  )

  def constructEither(
      args: Seq[String],
      allowPositional: Boolean = false,
      allowRepeats: Boolean = false,
      totalWidth: Int = 100,
      printHelpOnExit: Boolean = true,
      docsOnNewLine: Boolean = false,
      autoPrintHelpAndExit: Option[(Int, PrintStream)] = Some((0, System.out)),
      customName: String = null,
      customDoc: String = null,
      sorted: Boolean = true,
      nameMapper: String => Option[String] = Util.kebabCaseNameMapper
  ): Either[String, T] = ???

  /** binary compatibility shim. */
  private[mainargs] def constructEither(
      args: Seq[String],
      allowPositional: Boolean,
      allowRepeats: Boolean,
      totalWidth: Int,
      printHelpOnExit: Boolean,
      docsOnNewLine: Boolean,
      autoPrintHelpAndExit: Option[(Int, PrintStream)],
      customName: String,
      customDoc: String,
      nameMapper: String => Option[String]
  ): Either[String, T] = constructEither(
    args,
    allowPositional,
    allowRepeats,
    totalWidth,
    printHelpOnExit,
    docsOnNewLine,
    autoPrintHelpAndExit,
    customName,
    customDoc,
    sorted = true,
    nameMapper = nameMapper
  )

Apart from being extremely verbose and full of boilerplate, like any boilerplate this is also extremely error-prone. Bugs like com-lihaoyi/mainargs#106 slip through when a mistake is made in that boilerplate. These bugs are impossible to catch using a normal test suite, as they only appear in the presence of version skew. The above code snippet actually does have such a bug, that the test suite did not catch. See if you can spot it!

Sebastien Doraene’s talk Designing Libraries for Source and Binary Compatibility explores some of the challenges, and discusses the workarounds.

Requirements

Backwards Compatibility

Given:

• Two libraries, Upstream and Downstream, where Downstream depends on Upstream

• If we use a newer version of Upstream which contains an added default parameter together with an older version of Downstream compiled against an older version of Upstream before that default parameter was added

• The behavior should be binary compatible and semantically indistinguishable from using a verion of Downstream compiled against the newer version of Upstream

Note: we do not aim for Forwards compatibility. Using an older version of Upstream with a newer version of Downstream compiled against a newer version of Upstream is not a use case we want to support. The vast majority of OSS software does not promise forwards compatibility, including software such as the JVM, so we should just follow suite

All Overrides Are Equivalent

All versions of an @unrolled method def foo should have the same semantics when called with the same parameters. We must be careful to ensure:

1. All our different method overrides point at the same underlying implementation
2. Abstract methods are properly implemented, and no method would fail with an AbstractMethodError when called
3. We properly forward the necessary argument and default parameter values when calling the respective implementation.

Proposed solution

The proposed solution is to provide a scala.annotation.unroll annotation, that can be applied to methods defs, class constructors, or case classes to generate “unrolled” or “telescoping” versions of a method that forward to the primary implementation:

  def constructEither(
      args: Seq[String],
      allowPositional: Boolean = false,
      allowRepeats: Boolean = false,
      totalWidth: Int = 100,
      printHelpOnExit: Boolean = true,
      docsOnNewLine: Boolean = false,
      autoPrintHelpAndExit: Option[(Int, PrintStream)] = Some((0, System.out)),
      customName: String = null,
      customDoc: String = null,
      @unroll sorted: Boolean = true,
      @unroll nameMapper: String => Option[String] = Util.kebabCaseNameMapper
  ): Either[String, T] = ???

This allows the developer to write the minimal amount of code they want to write, and add a single annotation to allow binary compatibility to old versions. In this case, we annotated sorted and nameMapper with @unroll, which generates forwarders that make def constructEither binary compatible with older versions that have fewer parameters, up to a version before sorted or nameMapper was added. Any existing method def, class, or case class can be evolved in this way, by addition of @unroll the first time a default argument is added to their signature after its initial definition.

Unrolling defs

Consider a library that is written as follows:

object Unrolled{
   def foo(s: String, n: Int = 1) = s + n + b + l
}

If over time a new default parameter is added:

object Unrolled{
   def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b + l
}

And another

object Unrolled{
   def foo(s: String, n: Int = 1, b: Boolean = true, l: Long = 0) = s + n + b + l
}

This is a source-compatible change, but not binary-compatible: JVM bytecode compiled against an earlier version of the library would be expecting to call def foo(String, Int), but will fail because the signature is now def foo(String, Int, Boolean) or def foo(String, Int, Boolean, Long). On the JVM this will result in a MethodNotFoundError at runtime, a common experience for anyone who upgrading the versions of their dependencies. Similar concerns are present with Scala.js and Scala-Native, albeit the failure happens at link-time rather than run-time

@unroll is an annotation that can be applied as follows, to the first “additional” default parameter that was added in each published version of the library (in this case, b: Boolean = true and l: Long = 0)

import scala.annotation.unroll

object Unrolled{
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l
}

The @unroll annotation takes def foo and generates synthetic forwarders for the purpose of maintaining binary compatibility for old callers who may be expecting the previous signature. These forwarders do nothing but forward the call to the current implementation, using the given default parameter values:

object Unrolled{
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l

   def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
   def foo(s: String, n: Int) = foo(s, n, true, 0)
}

As a result, old callers who expect def foo(String, Int, Boolean) or def foo(String, Int, Boolean, Long) can continue to work, even as new parameters are added to def foo. The only restriction is that new parameters can only be added on the right, and they must be provided with a default value.

If multiple default parameters are added at once (e.g. b and l below) you can also choose to only @unroll the first default parameter of each batch, to avoid generating unnecessary forwarders:

object Unrolled{
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l

   def foo(s: String, n: Int) = foo(s, n, true, 0)
}

If there are multiple parameter lists (e.g. for curried methods or methods taking implicits) only one parameter list can be unrolled (though it does not need to be the first one). e.g. this works:

object Unrolled{
   def foo(s: String, 
           n: Int = 1, 
           @unroll b: Boolean = true,
           @unroll l: Long = 0)
          (implicit blah: Blah) = s + n + b + l
}

As does this

object Unrolled{
   def foo(blah: Blah)
          (s: String, 
           n: Int = 1, 
           @unroll b: Boolean = true,
           @unroll l: Long = 0) = s + n + b + l
}

@unrolled methods can be defined in objects, classes, or traits. Other cases are shown below.

Unrolling classes

Class constructors and secondary constructors are treated by @unroll just like any other method:

class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0){
   def foo = s + n + b + l
}

Unrolls to:

class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0){
   def foo = s + n + b + l

   def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0)
   def this(s: String, n: Int) = this(s, n, true, 0)
}

Unrolling class secondary constructors

class Unrolled() {
   var foo = ""

   def this(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = {
      this()
      foo = s + n + b + l
   }
}

Unrolls to:

class Unrolled() {
   var foo = ""

   def this(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = {
      this()
      foo = s + n + b + l
   }

   def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0)
   def this(s: String, n: Int) = this(s, n, true, 0)
}

Case Classes

case classes can also be @unrolled. Unlike normal class constructors and method defs, case classes have several generated methods (apply, copy) that need to be kept in sync with their primary constructor. @unroll thus generates forwarders for those methods as well, based on the presence of the @unroll annotation in the primary constructor:

case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true){
  def foo = s + n + b
}

Unrolls to:

case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0L){
   def this(s: String, n: Int) = this(s, n, true, 0L)
   def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0L)
   
   def copy(s: String, n: Int) = copy(s, n, this.b, this.l)
   def copy(s: String, n: Int, b: Boolean) = copy(s, n, b, this.l)
   
   def foo = s + n + b
}
object Unrolled{
   def apply(s: String, n: Int) = apply(s, n, true, 0L)
   def apply(s: String, n: Int, b: Boolean) = apply(s, n, b, 0L)
}

Notes:

1. @unrolled case classes are fully binary and backwards compatible in Scala 3, but not in Scala 2

2. .unapply does not need to be duplicated in Scala 3.x, as its signature def unapply(x: Unrolled): Unrolled does not change when new case class fields are added.

3. Even in Scala 2.x, where def unapply(x: Unrolled): Option[TupleN] is not binary compatible, pattern matching on case classes is already binary compatible to addition of new fields due to Option-less Pattern Matching. Thus, only calls to .tupled or .curried on the case class companion object, or direct calls to .unapply on an unrolled case class in Scala 2.x (shown below) will cause a crash if additional fields were added:

def foo(t: (String, Int)) = println(t)
Unrolled.unapply(unrolled).map(foo)

In Scala 3, @unrolling a case class also needs to generate a fromProduct implementation in the companion object, as shown below:

def fromProduct(p: Product): CaseClass = p.productArity match
  case 2 =>
    CaseClass(
      p.productElement(0).asInstanceOf[...],
      p.productElement(1).asInstanceOf[...],
    )
  case 3 =>
    CaseClass(
      p.productElement(0).asInstanceOf[...],
      p.productElement(1).asInstanceOf[...],
      p.productElement(2).asInstanceOf[...],
    )
  ...

This is not necessary for preserving binary compatibility - the method signature of def fromProduct does not change depending on the number of fields - but it is necessary to preserve semantic compatibility. fromProduct by default does not take into account field default values, and this change is necessary to make it use them when the given p: Product has a smaller productArity than the current CaseClass implementation

Hiding Generated Forwarder Methods

As the generated forwarder methods are intended only for binary compatibility purposes, we should generally hide them: IDEs, downstream compilers, ScalaDoc, etc. should behave as if the generated methods do not exist.

This is done in two different ways:

1. In Scala 2, we generate the methods in a post-pickler phase. This ensures they do not appear in the scala signature, and thus are not exposed to downstream tooling

2. In Scala 3, the generated methods are flagged as Invisible

Limitations

Only the one parameter list of multi-parameter list methods can be @unrolled.

Unrolling multiple parameter lists would generate a number of forwarder methods exponential with regard to the number of parameter lists unrolled, and the generated forwarders may begin to conflict with each other. We can choose to spec this out and implement it later if necessary, but for 99% of use cases @unrolling one parameter list should be enough. Typically, only one parameter list in a method has default arguments, with other parameter lists being implicits or a single callback/blocks, neither of which usually has default values.

Unrolled forwarder methods can collide with manually-defined overrides

This is similar to any other generated methods. We can raise an error to help users debug such scenarios, but such name collisions are inevitably possible given how binary compatibility on the JVM works.

@unrolled case classes are only fully binary compatible in Scala 3

They are almost binary compatible in Scala 2. Direct calls to unapply are binary incompatible, but most common pattern matching of case classes goes through a different code path that is binary compatible. There are also the AbstractFunctionN traits, from which the companion object inherits .curried and .tupled members. Luckily, unapply was made binary compatible in Scala 3, and AbstractFunctionN, .curried, and .tupled were removed

While @unrolled case classes are not fully source compatible

This is due to the fact that pattern matching requires all arguments to be specified. This proposal does not change that. Future improvements related to Pattern Matching on Named Fields may bring improvements here. But as we discussed earlier, binary compatibility is generally more important than source compatibility, and so we do not need to wait for any source compatibility improvements to land before proceeding with these binary compatibility improvements.

Binary and semantic compatibility for macro-derived derive typeclasses is out of scope

This propsosal does not have any opinion on whether or not macro-derivation is be binary/source/semantically compatible. That is up to the individual macro implementations to decide. e.g., uPickle has a very similar rule about adding case class fields, except that field ordering does not matter. Trying to standardize this across all possible macros and all possible typeclasses is out of scope

@unroll generates a quadratic amount of generated bytecode as more default parameters are added

Each forwarder has O(num-params) size, and there are O(num-default-params) forwarders. We do not expect this to be a problem in practice, as the small size of the generated forwarder methods means the constant factor is small, but one could imagine the O(n^2) asymptotic complexity becoming a problem if a method accumulates hundreds of default parameters over time. In such extreme scenarios, some kind of builder pattern (such as those listed in Major Alternatives) may be preferable.

@unroll only supports final methods.

object methods and constructors are naturally final, but class or trait methods that are @unrolled need to be explicitly marked final. It has proved difficult to implement the semantics of @unroll in the presence of downstream overrides, super, etc. where the downstream overrides can be compiled against by different versions of the upstream code. If we can come up with some implementation that works, we can lift this restriction later, but for now I have not managed to do so and so this restriction stays.

Challenges of Non-Final Methods and Overriding

To elaborate a bit on the issues with non-final methods and overriding, consider the following case with four classes, Upstream, Downstream, Main1 and Main2, each of which is compiled against different versions of each other (hence the varying number of parameters for foo):

class Upstream{ // V2
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l
}

class Downstream extends Upstream{ // compiled against Upstream V1
   override def foo(s: String, n: Int = 1) = super.foo(s, n) + s + n
}

object Main1 { // compiled against Upstream V2
   def main(args: Array[String]): Unit = {
      new Downstream().foo("hello", 123, false, 456L)
   }
}

object Main2 { // compiled against Upstream V1
   def main(args: Array[String]): Unit = {
      new Downstream().foo("hello", 123)
   }
}

The challenge here is: how do we make sure that Main1 and Main2, who call new Downstream().foo, correctly pick up the version of def foo that is provided by Downstream?

With the current implementation, the override def foo inside Downstream would only override one of Upstream’s synthetic forwarders, but would not override the actual primary implementation. As a result, we would see Main1 calling the implementation of foo from Upstream, while Main2 calls the implementation of foo from Downstream. So even though both Main1 and Main2 have the same Upstream and Downstream code on the classpath, they end up calling different implementations based on what they were compiled against.

We cannot perform the method search and dispatch within the def foo methods, because it depends on exactly how foo is called: the InvokeVirtual call from Main1 is meant to resolve to Downstream#foo, while the InvokeSpecial call from Downstream#foo’s super.foo is meant to resolve to Upstream#foo. But a method implementation cannot know how it was called, and thus it is impossible for def foo to forward the call to the right place.

Like our treatment of Abstract Methods, this scenario can never happen according to what version combinations are supported by our definition of Backwards Compatibility, but nevertheless is a real concern due to the requirement that All Overrides Are Equivalent.

It may be possible to loosen this restriction to also allow abstract methods that are implemented only once by a final method. See the section about Abstract Methods for details.

Major Alternatives

The major alternatives to @unroll are listed below:

1. data-class
2. SBT Contrabad
3. Structural Data Structures
4. Avoiding language features like case classes or default parameters, as suggested by the Binary Compatibility for Library Authors documentation.

While those alternate approaches do work - data-class and SBT Datatype are used heavily in various open-source projects - I believe they are inferior to the approach that @unroll takes:

Case Class v.s. not-a-Case-Class

The first major difference between @unroll and the above alternatives is that these alternatives all introduce something new: some kind of not-a-case-class class that is to be used when binary compatibility is desired. This not-a-case-class has different syntax from case classes, different semantics, different methods, and so on.

In contrast, @unroll does not introduce any new language-level or library-level constructs. The @unroll annotation is purely a compiler-backend concern for maintaining binary compatibility. At a language level, @unroll allows you to keep using normal method defs, classes and case classes with exactly the same syntax and semantics you have been using all along.

Having people be constantly choosing between case-class and not-a-case-class when designing their data types, is inferior to simply using case classes all the time

Scala Syntax v.s. Java-esque Syntax

The alternatives linked above all build a Java-esque “inner platform” on top of the Scala language, with its own conventions like .withFoo methods.

In contrast, @unroll makes use of the existing Scala language’s default parameters to achieve the same effect.

If we think Scala is nicer to write then Java due to its language features, then @unroll’s approach of leveraging those language features is nicer to use than the alternative’s Java-esque syntax.

Having implementation-level problems - which is what binary compatibility across version skew is - bleed into the syntax and semantics of the language is also inferior to having it be controlled by an annotation. Martin Odersky has said that annotations are intended for things that do not affect typechecking, and @unroll fits the bill perfectly.

Evolving Any Class v.s. Evolving Pre-determined Classes

The alternatives given require that the developer has to decide up front whether their data type needs to be evolved while maintaining binary compatibility.

In contrast, @unroll allows you to evolve any existing class or case class.

In general, trying to decide which classes will need to evolve later on is a difficult task that is easy to get wrong. @unroll totally removes that requirement, allowing you to take any class or case class and evolve it later in a binary compatible way.

Binary Compatibility for Methods and Classes

Lastly, the above alternatives only solve half the problem: how to evolve case classes. This is schema evolution.

Binary compatility is not just a problem for case classes adding new fields: normal class constructors, instance method defs, static method defs, etc. have default parameters added all the time as well.

In contrast, @unroll allows the evolution of defs and normal classes, in addition to case classes, all using the same approach:

1. @unrolling case classes is about schema evolution
2. @unrolling concrete method defs is about API evolution
3. @unrolling abstract method defs is about protocol evolution

All three cases above have analogous best practices in the broader software engineering world: whether you are adding an optional column to a database table, adding an optional flag to a command-line tool, are extending an existing protocol with optional fields that may need handling by both clients and servers implementing that protocol.

@unroll solves all three problems at once - schema evolution, API evolution, and protocol evolution. It does so with the same Scala-level syntax and semantics, with the same requirements and limitations that common schema/API/protocol-evolution best-practices have in the broader software engineering community.

Abstract Methods

Apart from final methods, @unroll also supports purely abstract methods. Consider the following example with a trait Unrolled and an implementation UnrolledObj:

trait Unrolled{ // version 3
  def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String
}

object UnrolledObj extends Unrolled{ // version 3
  def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b
}

This unrolls to:

trait Unrolled{ // version 3
  def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b)
  def foo(s: String, n: Int, b: Boolean): String = foo(s, n)
  def foo(s: String, n: Int): String
}

object UnrolledObj extends Unrolled{ // version 3
  def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l
  def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
  def foo(s: String, n: Int) = foo(s, n, true)
}

Note that both the abstract methods from trait Unrolled and the concrete methods from object UnrolledObj generate forwarders when @unrolled, but the forwarders are generated in opposite directions! Unrolled concrete methods forward from longer parameter lists to shorter parameter lists, while unrolled abstract methods forward from shorter parameter lists to longer parameter lists. For example, we may have a version of object UnrolledObj that was compiled against an earlier version of trait Unrolled:

object UnrolledObj extends Unrolled{ // version 2
  def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b
  def foo(s: String, n: Int) = foo(s, n, true)
}

But further downstream code calling .foo on UnrolledObj may expect any of the following signatures, depending on what version of Unrolled and UnrolledObj it was compiled against:

UnrolledObj.foo(String, Int)
UnrolledObj.foo(String, Int, Boolean)
UnrolledObj.foo(String, Int, Boolean, Long)

Because such downstream code cannot know which version of Unrolled that UnrolledObj was compiled against, we need to ensure all such calls find their way to the correct implementation of def foo, which may be at any of the above signatures. This “double forwarding” strategy ensures that regardless of which version of .foo gets called, it ends up eventually forwarding to the actual implementation of foo, with the correct combination of passed arguments and default arguments

UnrolledObj.foo(String, Int) // forwards to UnrolledObj.foo(String, Int, Boolean) 
UnrolledObj.foo(String, Int, Boolean) // actual implementation 
UnrolledObj.foo(String, Int, Boolean, Long) // forwards to UnrolledObj.foo(String, Int, Boolean)

As is the case for @unrolled methods on traits and classes, @unrolled implementations of an abtract method must be final.

Are Reverse Forwarders Really Necessary?

This “double forwarding” strategy is not strictly necessary to support Backwards Compatibility: the “reverse” forwarders generated for abstract methods are only necessary when a downstream callsite of UnrolledObj.foo is compiled against a newer version of the original trait Unrolled than the object UnrolledObj was, as shown below:

trait Unrolled{ // version 3
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b)
   // generated
   def foo(s: String, n: Int, b: Boolean): String = foo(s, n)
   def foo(s: String, n: Int): String
}

object UnrolledObj extends Unrolled{ // version 2
   def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b
   // generated
   def foo(s: String, n: Int) = foo(s, n, true)
}

// version 3
UnrolledObj.foo("hello", 123, true, 456L)

If we did not have the reverse forwarder from foo(String, Int, Boolean, Long) to foo(String, Int, Boolean), this call would fail at runtime with an AbstractMethodError. It also will get caught by MiMa as a ReversedMissingMethodProblem.

This configuration of version is not allowed given our definition of backwards compatibility: that definition assumes that Unrolled must be of a greater or equal version than UnrolledObj, which itself must be of a greater or equal version than the final call to UnrolledObj.foo. However, the reverse forwarders are needed to fulfill our requirement All Overrides Are Equivalent: looking at trait Unrolled // version 3 and object UnrolledObj // version 2 in isolation, we find that without the reverse forwarders the signature foo(String, Int, Boolean, Long) is defined but not implemented. Such an un-implemented abstract method is something we want to avoid, even if our artifact version constraints mean it should technically never get called.

Minor Alternatives:

@unrollAll

Currently, @unroll generates a forwarder only for the annotated default parameter; if you want to generate multiple forwarders, you need to @unroll each one. In the vast majority of scenarios, we want to unroll every default parameters we add, and in many cases default parameters are added one at a time. In this case, an @unrollAll annotation may be useful, a shorthand for applying @unroll to the annotated default parameter and every parameter to the right of it:

object Unrolled{
   def foo(s: String, n: Int = 1, @unrollAll b: Boolean = true, l: Long = 0) = s + n + b + l
}

object Unrolled{
   def foo(s: String, n: Int = 1, b: Boolean = true, l: Long = 0) = s + n + b + l

   def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
   def foo(s: String, n: Int) = foo(s, n, true, 0)
}

Generating Forwarders For Parameter Type Widening or Result Type Narrowing

While this proposal focuses on generating forwarders for addition of default parameters, you can also imagine similar forwarders being generated if method parameter types are widened or if result types are narrowed:

// Before
def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b + l

// After
def foo(@unrollType[String] s: Object, n: Int = 1, b: Boolean = true) = s.toString + n + b + l

// Generated
def foo(s: Object, n: Int = 1, b: Boolean = true) = s.toString + n + b + l
def foo(s: String, n: Int = 1, b: Boolean = true) = foo(s, n, b)

This would follow the precedence of how Java’s and Scala’s covariant method return type overrides are implemented: when a class overrides a method with a new implementation with a narrower return type, a forwarder method is generated to allow anyone calling the original signature \to be forwarded to the narrower signature.

This is not currently implemented in @unroll, but would be a straightforward addition.

Incremental Forwarders or Direct Forwarders

Given this:

def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l

There are two ways to do the forwarders. First option, which I used in above, is to have each forwarder directly call the primary method:

def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
def foo(s: String, n: Int) = foo(s, n, true, 0)

Second option is to have each forwarder incrementally call the next forwarder, which will eventually end up calling the primary method:

def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0)
def foo(s: String, n: Int) = foo(s, n, true)

The first option results in shorter stack traces, while the second option results in roughly half as much generated bytecode in the method bodies (though it’s still O(n^2)).

In order to allow @unrolling of Abstract Methods, we had to go with the second option. This is because when an abstract method is overriden, it is not necessarily true that the longest override that contains the implementation. Thus we need to forward between the different def foo overrides one at a time until the override containing the implementation is found.

Implementation & Testing

This SIP has a full implementation for Scala {2.12, 2.13, 3} X {JVM, JS, Native} in the following repository, as a compiler plugin:

• https://github.com/com-lihaoyi/unroll

As the @unroll annotation is purely a compile-time construct and does not need to exist at runtime, @unroll can be added to Scala 2.13.x without breaking forwards compatibility.

The linked repo also contains an extensive test suite that uses both MIMA as well as classpath-mangling to validate that it provides both the binary and semantic compatibility benefits claimed in this document. In fact, it has even discovered bugs in the upstream Scala implementation related to binary compatibility, e.g. scala-native/scala-native#3747

I have also opened pull requests to a number of popular OSS Scala libraries, using @unroll as a replacement for manually writing binary compatibility stubs, and the 100s of lines of boilerplate reduction can be seen in the links below:

• https://github.com/com-lihaoyi/mainargs/pull/113/files
• https://github.com/com-lihaoyi/mill/pull/3008/files
• https://github.com/com-lihaoyi/upickle/pull/555/files
• https://github.com/com-lihaoyi/os-lib/pull/254

These pull requests all pass both the test suite as well as the MIMA check-binary-compatibility job, demonstrating that this approach does work in real-world codebases. At time of writing, these are published under the following artifacts and can be used in your own projects already:

• Compiler Plugin: ivy"com.lihaoyi::unroll-plugin:0.1.12"
• Annotation: ivy"com.lihaoyi::unroll-annotation:0.1.12"