GitRoot

  1# Modifying the API in the SDK Plugin
  2
  3The last feature for version 0.4 is about adding `mv` (move) and `cp` (copy) functions to the SDK plugin. I'm taking advantage of this development to explain how the plumbing works under the hood.
  4
  5Right now, the SDK plugin is undocumented because there are a lot of breaking changes coming. Every time I develop a new plugin, I discover better ways to do things, and the SDK needs to adapt.
  6
  7For example, when I added `apex_code` (a plugin that takes code as a string and returns syntax-highlighted HTML), I wondered: "Is this a standalone plugin?". Not really, because there's no file to manipulate, neither in input nor output. Instead, it manages blocks inside an existing file. The SDK wasn't originally built with that use case in mind.
  8
  9Anyway, I think explaining how the SDK works through a concrete example will be helpful, if not for you, at least for me.
 10
 11## What is the SDK's job?
 12
 13The SDK exists to pass data from GitRoot (the host) to the plugin (the guest). As you know, GitRoot uses WebAssembly (WASM) to manage plugins. Unfortunately, WASM is still a bit low-level to implement: between the host and the guest, only integers can travel.
 14
 15Wait, only integers?
 16
 17Yes. So, how do we send a string? The host can store any kind of data in the guest's memory. GitRoot stores a string, retrieves the pointer (the memory address) and the size, and sends those two integers to the guest. The guest then retrieves the string from its own memory. It's a complex dance just for a simple string. The SDK is here to hide all that mess.
 18
 19When you build a plugin, you just implement an interface. The SDK handles the rest: you receive objects and strings, and you send back data without ever worrying about the underlying WASM pointers.
 20
 21## What does a concrete implementation look like?
 22
 23When I add a new function, I always start with the host side. All functions are located in [runtime.go](../app/server/plugin/runtime.go).
 24
 25We need to register the function in [wazero](https://github.com/wazero/wazero), our WASM runtime. I add two versions because I haven't found a clean way to bridge the difference between AssemblyScript and Go/Rust yet. Hopefully, I'll figure that out one day.
 26
 27```go
 28r.wazRun.
 29    NewFunctionBuilder().WithFunc(r.MoveFileBuilder(false)).Export("moveFile").
 30    NewFunctionBuilder().WithFunc(r.MoveFileBuilder(true)).Export("moveFileAS")
 31```
 32
 33Then I implement the builder:
 34
 35```go
 36func (r *runtime) MoveFileBuilder(forAS bool) interface{} {
 37	if !forAS {
 38		return func(_ context.Context, m api.Module, ptr, size, ptr2, size2 uint32) {
 39			// from ptr/size to string or object
 40			r.moveFile(fromPath, toPath)
 41		}
 42	} else {
 43		return func(_ context.Context, m api.Module, ptr, ptr2 uint32) {
 44			// from ptr to string or object
 45			r.moveFile(fromPath, toPath)
 46		}
 47	}
 48}
 49```
 50
 51> Yeah, I know. As a Gopher, seeing `interface{}` as a return type and all these uint32 feels wrong. But hey, we're not here for a code review, we're here to add an API. If you have a cleaner way, feel free to submit a graft!
 52
 53We want to allow any plugin to move a file. Let's define the internal `moveFile` function:
 54
 55```go
 56func (r *runtime) moveFile(fromPath, toPath string) error {
 57	//need to do something...
 58}
 59```
 60
 61Actually, there's a catch, GitRoot uses three internal filesystems: one for the git worktree, one for web content, and one for the cache. Even if they are abstracted under one root, the code often expects a specific FS. I'll probably clean this up later, but for now:
 62
 63```go
 64func (r *runtime) moveFile(fromFs, fromPath, toFs, toPath string) error {
 65	//need to do something...
 66}
 67```
 68
 69Now your plugin can call `moveFile("cache", "artifacts/myBin", "web", "releases/myBin-v1.0")` to publish an artifact, or `moveFile("web", "index.html", "web", "old/index.html")` to back up a file.
 70
 71I won't bore you with the implementation logic here. If you're curious, check the [code](../app/server/plugin/runtime.go).
 72
 73Here is how the final builder looks, converting pointers to strings:
 74
 75```go
 76func (r *runtime) MoveFileBuilder(forAS bool) interface{} {
 77	if !forAS {
 78		return func(_ context.Context, m api.Module, fromFsPtr, fromFsSize, fromPathPtr, fromGlobPathSize, toFsPtr, toFsSize, toPathPtr, toPathSize uint32) uint64 {
 79			fromFs, err := r.readString(m, "MoveFile fromFs", fromFsPtr, fromFsSize)
 80			if err != nil {
 81				return r.sendEmptyOrError(m, err)
 82			}
 83			fromPath, err := r.readString(m, "MoveFile fromGlobPath", fromPathPtr, fromGlobPathSize)
 84			if err != nil {
 85				return r.sendEmptyOrError(m, err)
 86			}
 87			toFs, err := r.readString(m, "MoveFile toFs", toFsPtr, toFsSize)
 88			if err != nil {
 89				return r.sendEmptyOrError(m, err)
 90			}
 91			toPath, err := r.readString(m, "MoveFile toPath", toPathPtr, toPathSize)
 92			if err != nil {
 93				return r.sendEmptyOrError(m, err)
 94			}
 95			return r.sendEmptyOrError(m, r.moveFile(fromFs, fromPath, toFs, toPath))
 96		}
 97	} else {
 98		return func(_ context.Context, m api.Module, fromFsPtr, fromPathPtr, toFsPtr, toPathPtr uint32) uint64 {
 99			fromFs, err := r.readASString(m, "MoveFileAS fromFs", fromFsPtr)
100			if err != nil {
101				return r.sendEmptyOrError(m, err)
102			}
103			fromPath, err := r.readASString(m, "MoveFileAS fromGlobPath", fromPathPtr)
104			if err != nil {
105				return r.sendEmptyOrError(m, err)
106			}
107			toFs, err := r.readASString(m, "MoveFileAS toFs", toFsPtr)
108			if err != nil {
109				return r.sendEmptyOrError(m, err)
110			}
111			toPath, err := r.readASString(m, "MoveFileAS toPath", toPathPtr)
112			if err != nil {
113				return r.sendEmptyOrError(m, err)
114			}
115			return r.sendEmptyOrError(m, r.moveFile(fromFs, fromPath, toFs, toPath))
116		}
117	}
118}
119```
120
121When I need to pass complex objects, I use `json.Marshal` in GitRoot and `json.Unmarshal` in the SDK. It works, but don't get too attached to it, I might change the serialization method in the future.
122
123## Inside the SDK
124
125Now that the host is ready, we need to implement the other side: the SDK itself. I usually start with Go, then move to Rust and AssemblyScript. Sometimes I forget the others... shame on me! I'm planning to automate this with some boilerplate generation later.
126
127In the [server](../app/libs/golang/plugin/model/server.go) interface:
128
129```go
130type Server interface {
131    //...
132    MoveFile(fromFs, fromPath, toFs, toPath string) error
133    //...
134}
135```
136
137To make it more "user-friendly", I use a struct to represent the filesystem in [model/fs.go](../app/libs/golang/plugin/model/fs.go):
138
139```go
140func (g *GrFs) MoveFile(fromPath string, toPath string) error {
141    return g.server.MoveFile(g.base, fromPath, g.base, toPath)
142}
143
144func (g *GrFs) MoveFileToFs(fromPath string, toFs *GrFs, toPath string) error {
145    return g.server.MoveFile(g.base, fromPath, toFs.base, toPath)
146}
147```
148
149The actual WASM call happens in [plugin/server.go](../app/libs/golang/plugin/server.go), where we transform strings back into pointers:
150
151```go
152// MoveFile implements [model.ServerNeeded].
153func (s *pServer) MoveFile(fromFs model.FsBase, fromPath string, toFs model.FsBase, toPath string) error {
154	ptr, size := stringToPtr(string(fromFs))
155	ptr2, size2 := stringToPtr(fromPath)
156	ptr3, size3 := stringToPtr(string(toFs))
157	ptr4, size4 := stringToPtr(toPath)
158	resPtrSize := _moveFile(ptr, size, ptr2, size2, ptr3, size3, ptr4, size4)
159	runtime.KeepAlive(fromFs)
160	runtime.KeepAlive(fromPath)
161	runtime.KeepAlive(toFs)
162	runtime.KeepAlive(toPath)
163	return ptrSizeToError(resPtrSize)
164}
165```
166
167And finally, the link to the host via `go:wasmimport` in [plugin/imports.go](../app/libs/golang/plugin/imports.go):
168
169```go
170//go:wasmimport gitroot moveFile
171func _moveFile(fromFsPtr, fromFsSize, fromPathPtr, fromPathSize, toFsPtr, toFsSize, toPathPtr, toPathSize uint32) (ptrSize uint64)
172```
173
174## How to use it?
175
176Now, any plugin can use this. Here's a simple Go plugin that triggers a build whenever a `.go` file is modified and moves the binary to the web folder.
177
178```go
179package main
180
181import (
182	"fmt"
183	"path/filepath"
184	"strings"
185
186	gitroot "gitroot.dev/libs/golang/plugin"
187	"gitroot.dev/libs/golang/plugin/model"
188)
189
190var defaultRun = []model.PluginRun{{
191	Path:   "**/*.go",
192	When:   model.PluginRunWhenAll,
193	Branch: []string{"main"},
194	Write: model.PluginWrite{
195		Web: []model.PluginWriteRight{{
196			Path: "releases/main",
197			Can:  model.PluginWriteRightCanAll,
198		}},
199	},
200}}
201
202type Plugin struct {
203	server model.Server
204    needToBuild bool
205}
206
207func (p *Plugin) Init(repoName string, confHasChanged bool, serializedConf string) error {
208    p.needToBuild = false
209    return nil
210}
211
212func (p *Plugin) StartCommit(commit model.Commit) error                                  { return nil }
213
214func (p *Plugin) AddFile(file model.File) error                                          {
215	p.needToBuild = true
216    return nil
217}
218
219func (p *Plugin) ModFile(file model.File) error                                          {
220	p.needToBuild = true
221    return nil
222}
223
224func (p *Plugin) DelFile(file model.File) error                                          {
225	p.needToBuild = true
226    return nil
227}
228
229func (p *Plugin) EndCommit(commit model.Commit) error                                    { return nil }
230
231func (p *Plugin) Finish() error                                                          {
232    if p.needToBuild {
233        res, _ := p.server.Exec(model.Exec{
234            Cmds: []model.Cmd{
235                {Cmd: "go", Args: []string{"build"}},
236            },
237            Artifacts: []string{"main"},
238        })
239        return p.server.Cache().MoveFileToFs(res.Artifacts[0], p.server.Webcontent(), filepath.Join("releases", "main"))
240	}
241    return nil
242}
243
244func Build(server model.Server) model.Plugin {
245	return &Plugin{
246		server: server,
247	}
248}
249
250//go:wasmexport install
251func main() {
252	gitroot.Register(defaultRun, Build)
253}
254```
255
256## Is that all?
257
258Nope. I still need to update the [Rust](../app/libs/rust/plugin/src/fs.rs) and [AssemblyScript](../app/libs/ts/plugin/fs.ts) SDKs and, of course, add [tests](../app/libs/golang/plugin/test/fakeserver.go). But this should give you a good idea of how the plumbing works.
259
260If anything is unclear, [let me know](../contact.md)!