Build Orchestrator API

[Isabel Duan (iisaduan)]

3B of #63875

full description in progress

This PR implements what, in TS<6, was called the SolutionBuilder. In TS7.1, it will now be called the BuildOrchestrator.

class BuildOrchestrator {
    async build(project?: string): Promise<BuildResponse>
    async buildReferences(project: string): Promise<BuildResponse> 
    async clean(project?: string): Promise<CleanBuildResponse> 
    async cleanReferences(project?: string): Promise<CleanBuildResponse>
}

Usage example:

const options = await parseConfigFile("/c/tsconfig.json")
const orchestrator = await api.createBuildOrchestrator(
        { cwd: "/" },
        ["/c/tsconfig.json"],
         options,
);
orchestrator.build()

The outputs are

interface BuildResponse {
    status: number;
    errors?: DiagnosticResponse[];
    statistics: Statistics;
}
interface CleanBuildResponse {
    status: number;
    errors?: DiagnosticResponse[];
    statistics: Statistics;
    filesDeleted?: string[];
}

As of right now (when description still in progress), these may change. It feels useful to report the same things as tsc -b reports, However, it is not formatted in the same way as commandline outputs are (no pretty strings).

notes:

• the API client serves as what used to be the SolutionBuilderHost, so no need to create and pass an extra host
• Watch mode is disabled in the build orchestrator for this PR. The largest additional value of watch mode lies in providing a project file watcher and related reporting, and so the design of that will be for a different PR.

[Copilot]

🟡 Changes recommended

Host routing, configuration refresh, watch lifecycle, and clean-result handling contain blocking correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds the Build Orchestrator API for building and cleaning project graphs through sync and async clients.

Changes:

• Adds selective build, reference-build, clean, and reference-clean operations.
• Exposes orchestration through the API protocol and clients.
• Adds virtual filesystem deletion support and integration tests.

File summaries

File	Description
tsc/internal/project/session.go	Exposes the default library path.
tsc/internal/execute/build/orchestrator.go	Implements reusable build and clean orchestration.
tsc/internal/execute/build/clean_test.go	Tests server-side cleaning behavior.
tsc/internal/api/session.go	Handles Build Orchestrator API requests.
tsc/internal/api/proto.go	Defines orchestration protocol types.
tsc/internal/api/callbackfs.go	Adds delegated file removal.
packages/typescript/src/api/async/api.ts	Adds the asynchronous client API.
packages/typescript/src/api/sync/api.ts	Adds the generated synchronous client API.
packages/typescript/src/api/proto.generated.ts	Adds generated protocol declarations.
packages/typescript/src/api/fs.ts	Exposes the removeFile callback.
packages/typescript/test/async/api.test.ts	Tests asynchronous orchestration.
packages/typescript/test/sync/api.test.ts	Tests synchronous orchestration.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 12
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Isabel Duan (iisaduan)]

the API watch builds will be given an explicit path and lifecycle control in future PR

[Titian Cernicova-Dragomir (dragomirtitian)]

We used to use the incremental program in the API and Andrew Branch (@andrewbranch) suggested the solution builder as an alternative to that. Guessing this PR is an implementation for that in the 7.1 API. I have a few concerns:

1. I don't see a way to get diagnostics at all. Even more specifically, right now we only request diagnostics for specific files and i don't see a way to do that either (we don't want diagnostics for some library declaration files we know are generated so we can't just ask a full program type check).
2. Right now we use getJavaScriptEmit and getDeclarationEmit which will batch the emit into a single IPC call. From what I see the writes will happen one at a time. Without some similar batching I fear performance will be worse
3. Currently once we have the program, we also access the SourceFiles and again I don't see a way to do that with the BuildOrchestrator (Things are a bit more complicated, but we essentially update the original program by replacing all the ts files with their .d.ts output and we do some manipulation of the d.ts files based on type information. We do it this way to make sure the library declaration files don't have to be reparsed. Not sure if there is a better solution here)

Am I missing some piece of functionality that is already there? If there is no way to do these things are there plans to add them in the future?

[Isabel Duan (iisaduan)]

We've been discussing a lot offline about this and related API features. The scope for this PR is to add more diagnostic and status reporting to the build orchestrator, and implement the functions above. I've been working on the watch stuff on the side, and I will separate that out from this for a future PR. Everything else is very much still in discussion/planning, and it's really helpful to hear your concerns and use cases.

For 2, I am not totally sure which part you are asking about, as currently implemented, the emit output isn't returned back through the IPC call.

Since the build orchestrator manages things at the programs/project level, I think because 1 and 3 are operations on files, they would be part of the incremental program like previously. The incremental program that's not related to the build orchestrator is still in discussions. What were you using to get diagnostics per file?

[Titian Cernicova-Dragomir (dragomirtitian)]

The incremental program that's not related to the build orchestrator is still in discussions

So there is a chance we get back the incremental program back ? That is what we were using before.

For 2, I am not totally sure which part you are asking about, as currently implemented, the emit output isn't returned back through the IPC call.

I thought the output might be returned through writes to the vfs, might be mistaken about that.

[Andrew Branch (andrewbranch)]

Titian Cernicova-Dragomir (@dragomirtitian) which incremental-specific Program APIs do you need beyond just the altered construction/emit behavior? Do you use emitBuildInfo(), getSemanticDiagnosticsOfNextAffectedFile(), emitNextAffectedFile(), or anything else?

[Titian Cernicova-Dragomir (dragomirtitian)]

Titian Cernicova-Dragomir (Titian Cernicova-Dragomir (@dragomirtitian)) which incremental-specific Program APIs do you need beyond just the altered construction/emit behavior? Do you use emitBuildInfo(), getSemanticDiagnosticsOfNextAffectedFile(), emitNextAffectedFile(), or anything else?

We didn't explicitly use those. We created the incremental program which picked up the tsbuildinfo via tsBuildInfoFile in compiler options. We then got the program and called emit on it and that wrote out the emitted changed files (which included the tsbuildinfo). We also used the ASTs from the program to preform other checks. So at a simple level this would be our workflow:

options.tsBuildInfoFile = "path to our internal location"
const host = ts.createIncrementalCompilerHost(options, system);

host.getSourceFile = (fileName, languageVersionOrOptions, err) => { 
    /* get the source file by reading the code from memory */
    /* keep a reference to the source file AST in a cache for use in later programs  */
}

const incrementalProgram = typescript.createIncrementalProgram({
    rootNames: rootFilenames,
    options: options,
    projectReferences: project.projectReferences,
    host: host,
});

const program = incrementalProgram.getProgram();
const optionErrors = program.getOptionsDiagnostics();
const syntaxErrors = program.getSyntacticDiagnostics();
const globalErrors = program.getGlobalDiagnostics();

program.getSourceFiles().forEach(s => /* Do some extra checks */);

program.emit(
    undefined,
    (fileName, content) => { /*capture output including tsBuildInfo */ },
);

We also had a cache for declaration file ASTs for installed libraries (like node_modules) for use in a later program that we create based on the outputted declaration files. This avoided reparsing all those declaration files for this second program.

NOTE: The missing incremental build support from the API is the last thing that we are missing to move to the 7.1 API in our build tool.

[Andrew Branch (andrewbranch)]

We also had a cache for declaration file ASTs for installed libraries (like node_modules) for use in a later program that we create based on the outputted declaration files. This avoided reparsing all those declaration files for this second program.

Note that this now happens automatically, if you're strategic about when you release snapshots—if you retain a snapshot with a program that contains those files, and then make another program that loads the same files, the parse cache will be used. Jake has a prototype that would make this kind of thing more explicit, and we could add something like that later in a backward-compatible way, I think: jakebailey@c3b4130

I'm working on a prototype of snapshot-backed incremental programs that I might get you to try out soon and see how it works for you.

[Titian Cernicova-Dragomir (dragomirtitian)]

Note that this now happens automatically, if you're strategic about when you release snapshots—if you retain a snapshot with a program that contains those files, and then make another program that loads the same files, the parse cache will be used. Jake has a prototype that would make this kind of thing more explicit, and we could add something like that later in a backward-compatible way, I think: jakebailey@c3b4130

It did seem like that was the case but I just was not sure if I could rely on that behavior. As long as we can keep those reused ASTs around it doesn't really matter to us what the mechanism is (I was just summarizing the current state)

I'm working on a prototype of snapshot-backed incremental programs that I might get you to try out soon and see how it works for you.

This is great news. Can't wait to try it out.

[Andrew Branch (andrewbranch)]

Titian Cernicova-Dragomir (@dragomirtitian) I have a draft up at #64401; it probably has some bugs, but can you see if that direction meets your needs?

[Andrew Branch (andrewbranch)]

This signature seems a little weird:

• ClientSpawnOptions contains a bunch of irrelevant stuff; only cwd is used
• ParsedCommandLine also contains stuff that will not or cannot be respected, like root files and watch options

We should make the important, non-optional arguments come first, and wrap everything else into a dedicated BuildOrchestratorOptions type, which contains only enough to get the job done, leaving out opportunities for contradictions and unsupported options:

interface BuildOrchestratorOptions {
  buildOptions?: BuildOptions;
  currentDirectory?: string;
}

// ...

class API {
  async createBuildOrchestrator(
    rootProjectNames: readonly string[],
    options?: BuildOrchestratorOptions,
  ): Promise<BuildOrchestrator>
}

This also removes the need for the new diagnostic and the non-fatal creation diagnostics exposed with getErrors(). Either the creation succeeds, or returns an API error that can be caught.

[Isabel Duan (iisaduan)]

Should I make creation fail altogether if a watch option is passed in?

[Andrew Branch (andrewbranch)]

In my example above, it's impossible to pass watch options in.

[Andrew Branch (andrewbranch)]

This should be a Disposable like Snapshot so you can use it with using, or manually call .dispose() and clear out its server resources.