@internal APIs can't be stripped properly for TypeScript

[rix0rrr]

For jsii consumers, APIs marked @internal are being erased. An "internal" erased API should have the following effects:

1. The API element cannot be named by a consumer of the library
2. If it's a class or interface, its members are still visible in any of its public subtypes (subinterfaces or subclasses).

For API elements like free functions, methods, enums or non-subclassed types this is easy: they just entirely disappear without a trace. For interfaces and classes that are inherited need to put in additional effort to make the members appear in subtypes.

For example:

/**
 * @internal
 */
class HiddenBase {
  public sayHello();
}

export class PublicSub extends HiddenBase {
  public sayBye();
}

Should appear to public consumers to have been defined as follows:

export class PublicSub {
  public sayHello();
  public sayBye();
}

jsii does this work for its interface model and the code generation for its client languages, but used to rely on TypeScript's default handling of the @internal attribute, combined with the "stripInternals": true compiler setting.

...or so we thought!

Because it turns out that we were always running with "stripInternals": false, and in fact this was enforced by the jsii compiler. If we turn "stripInternals": true things actually break!

The reason is that the behavior of the TypeScript compiler is the following: for all types that are marked @internal, they are replaced with an empty object. That means that if we have a public type that inherits from an @internal type, and "stripInternals": true is turned on, the implementation details of the internal type disappear from the .d.ts file. That means we fail on both counts of what @internal is supposed to mean:

• The API element can still be named 👎
• But its members are gone, and will also not appear in subtypes 👎

This means that any customers trying to use an inherited member of an @internal type will start to see its usage fail if we ever turn on stripInternals: true. Which we did. And it broke.

This was probably the reason why we ended up doing this in the past, but we failed to take any follow-up action to fix it.

The correct solution

As we have seen above, we cannot rely on the behavior of TSC to produce the desired outcome for @internal. But we can re-implemented it in jsii without too much effort.

jsii already has a facility for stripping API elements from the output .d.ts files. We use this for the --strip-depecrated=deprecated_apis.txt feature today, which allows us to remove API elements from the API surface of a new public version of the CDK library. This allowed us to build both CDK v1 and CDK v2 from the same source branch, while making sure that a specific set of APIs would be not available in v2.

For non-inheritable API elements like methods and enums:

• jsii uses its "strip API" feature to remove the API from the .d.ts file (same as tsc would do natively)

For inheritable API elements like classes and interfaces:

• We leave the type in the .d.ts file
• But validate that the type is not accessible from the library entry point

The presence of the type definition in the .d.ts file will make it accessible to the consumer's TypeScript compiler, so that it can read all members and know that they are available on any subtypes.

The absence of the type from the "export path" will make it so that it cannot be named.

Why does this work?

The unit of modularity in TypeScript and JavaScript is the file. To be able name a symbol in TypeScript, there are 2 requirements:

• The symbol must be exported from the file it is in (*)
• The file must be importable by client code.

If one of those requirements isn't met, then the symbol cannot be named. So it might be defined and exported from a .d.ts file... if you cannot import that .d.ts file you still cannot name the type (in the absence of aliases, yada yada, gesticulates wildly, T&Cs apply, good enough for government work)

Technically, the second requirement is outside the domain of jsii because jsii only concerns itself with ONE entry point (lib/index.ts) but in practice JavaScript libraries may have multiple entry points, controlled by whatever is in package.json#exports. In particular, aws-cdk-lib has multiple entry points, one per service submodule: import * as s3 from 'aws-cdk-lib/aws-s3.

But there is good news! We can probably assume that we've set up our exports sanely: as long as every subpath export is also included in the primary set of exports, we can just have jsii look at "the set of exports", and be 99.9% correct on this (perhaps allowing for a tiny bit of false negatives). This is the logical way of setting it up, and we can assume that our users are logical. Right?

(*) Fun fact: there are a couple of places where jsii gets this wrong, and conflates "being exported from a source file" with "being publicly accessible to consumers of the library". Perchance we could fix those as well while we're at it. Or, we might not.

What will it look like?

1. We will have to force-validate "stripInternals": false in tsconfig.json
2. We will add a jsii --strip-internals flag, and probably a { "jsii": { "stripInternals": true } } setting in package.json.