Netlify Deployment Manager Notebook

Netlify Deployment Manager Notebook

To recap I am building my blog using Observable as the Content Management System (CMS) interface to a statically deployed site (Jamstack). The main motivation for building yet-another-static-site-generator is that Observable is a web-based literate programming notebook environment. So the unique angle of this jamstack is that the content interface is programmable and forkable (like this) which gives me unlimited creative freedom and extension points as a technical content author.

Even the deployment toolchain is hosted as a notebook that can be forked and customized. This article describes some of the features so far for the deployment notebook.

Netlify Deployment Manager

So I just got partial deployment working nicely so I thought now would be a good time to summarize the deployment features so far.

Some of my frustrations with existing CMSs are

  1. Content changes either take a long time to propagate, or the overall page is slow, depending on the cache settings.

  2. Deployment can take a long time.

Instant CDN cache preloading and invalidation

Netlify solves the cache problems with a smart cache. Caches are not cold because the content is actively pushed to the CDN on deploy, and, the old CDN state is invalidated on deploy. So some hard problems are solved just by using Netlify. Thus the website is super fast without the drawback of stale caches.

Faster Deployment with Delta Sync

The other issue is that static sites tend to be slow to deploy due to an O(n) deployment complexity. Again, thanks to Netlify functionality we can send just the content that changes in deployment. Furthermore, thanks to the CMS data model we can model the dependencies across pages so we only need to regenerate the pages that change too.

Netlify offers a deployment API, so we can deploy content directly from a notebook (see the deployStaticFile call).

Tag-based dependencies

File record metadata is stored in Firestore which plays well with Observable. Each record includes a tags array. When an article is updated, we do a reverse query for pages that depend on file tags using the "contains-array-any" query operator. Examples of content that do this are the index.html and the rss.xml against any files tagged "article". When an article is deployed, the page indexes are deployed too.

Parallel Materialization

To improve deploy speed, each notebook contains a serverside cell used to render the page preview of the page. The process of deployment is materializing of the preview link into Netlify. As the data exchange is a URL, we are pretty flexible about how content is expressed. The content doesn't even need to be hosted on Observable, for instance, the URL could be external (e.g. for materializing 3rd party libraries)

The other useful thing about using a URL as the content representation, and using serverside cells to generate the content, is that we can parallelize materialization jsut by reading the links in parrallel.

The most awesome thing about building on Observable is that this deployment toolchain is hosted within Observable notebooks too. The Netlify Deployment Manager contains all the Oauth code and API calls used to implement the deployStaticFile library cell. You can see how it works and change it whenever you want!

Next steps

The next job is to fix the authentication so it's easier for other Observable users to fork my blog examples and deploy their content on their Netlify accounts. We have not reached a usable service yet but it is getting closer!

-- Tom2