Recent Articles

Sun Jan 24 2021 19:49:49 GMT+0000 (Coordinated Universal Time)

Making the Trash Joyfull, Marie Kondo style

We believe in Marie Kondo decluttering maxim that every household object should spark joy. We recently turned around our most disliked object, the trash can, to something that the kids fight over who will empty it. How? here is the story…

To make being at home pleasanter, we pondered which object in the house brings us the least joy? Worse than even the toilet brush, the trash can we find torturous. We hate it to the point of avoiding it 'til trash is spilling onto the floor.

Furthermore, the bags often burst, getting gross liquid everywhere. Yuck, the trash sucks! How could we possibly turn it into something fun? We googled around for top of the range trash cans and found a wonderful concept...

Here is a can that: 1. opens itself, so you do not need to touch it ever. 2. Bags itself when full. 3. Replaces the bag fully automatically! 4. Seals the outgoing full bags.

So this is a new robotic product which I am deeply suspicious of (I have a PhD in robots), but, we bought it 4 months ago, it's still going strong, and it genuinely brings us joy. We love showing guests the rebagging cycle. It’s a showpiece and a point of pride now! Wow!

The kids love activating the rebagging cycle, and will happily take the bagged trash to the front door. The can is small, the bags are airtight sealed and don’t leak.

There are negatives, the bags are small. Though, it makes them easy to take out and stops them from breaking, so we do not mind the more frequent trips. Overall it's a really great purchase that has improved our lives. Marie Kondo is right!

I did not think we could make the trash fun and joyful but it is possible. It’s kinda expensive but it’s well worth it. The product we bought was a Xiaomi Townew T1 Self-Sealing and Self-Changing Trash Can (commercial link)


Mon Jan 11 2021 12:33:12 GMT+0000 (Coordinated Universal Time)

The Internet is a Market for Lemons

Do you get anxious when installing/authorizing software? That feeling keeps you to ‘the beaten path’ which exacerbates inequality and amplifies monopolistic power. Here I explain the underlying economic mechanisms that have turned the modern internet into a battlefield...

Internet software distribution has a huge design flaw, you cannot see what you are buying. You cannot see how your data is processed. You cannot verify the software does what it says it does. In many cases it does not.

For example, your internet service provider, that you pay money to provide you with internet, probably also sells your surfing data, in a free market, to anybody. Nobody would willingly choose that package!

Similarly, apps in the app store advertise doing one thing, whilst hoovering up data to sell to data brokers. If given a choice, you would pick the app that does not do that, but you can’t. You cannot observe how software is going to behave post purchase.

The software market is stuffed with software that advertises doing one thing, but behind your back, also does several other things that are against your best interests. This is why we are anxious. The internet, on aggregate, is actively hostile. Why has this happened?

George Akerlof won a Nobel prize for observing that when providers are more informed than the buyers a “market-for-lemons” forms. The market malfunctions by rewarding sleazy and deceptive practices. This is where we are today on the internet.

When buyers cannot assess the quality of a product directly, they use different buying signals. For software, this is often brand reputation. Established companies are incentivised to play by the rules, as their protection of the brand itself becomes worthwhile.

But this encourages winner-takes-all market dynamics. So we end up with just a handful of household brand technology companies (the FAANGs) whose individual opinions dominate the global narrative. This is not a healthy market for diversity.

We need to amplify the smaller good guys whilst avoiding the bad guys. Remember: this whole mess is because software buyers cannot assess the quality of the software they are purchasing. My investment hypothesis is if we fix this, we fix the market.

So my goal is to improve software service observability. Imagine if end users could view the source code in a continuously publicly auditable system. It would take one motivated technical user to inoculate all the non-technical users against hostile service providers

Serverside open source. We have a demo. Checkout https://observablehq.com/@tomlarkworthy/serverless-cells on the Observable platform. Follow the journey on twitter https://twitter.com/tomlarkworthy. Let’s deescalate the end user/service provider relation.


Fri Dec 11 2020 22:41:18 GMT+0000 (Coordinated Universal Time)

Netlify Deployment Manager Notebook

To recap I am building my blog using Observable as the Content Management System (CMS) interface to a statically deployed site (Jamstack). The main motivation for building yet-another-static-site-generator is that Observable is a web-based literate programming notebook environment. So the unique angle of this jamstack is that the content interface is programmable and forkable (like this) which gives me unlimited creative freedom and extension points as a technical content author.

Even the deployment toolchain is hosted as a notebook that can be forked and customized. This article describes some of the features so far for the deployment notebook.

Netlify Deployment Manager

So I just got partial deployment working nicely so I thought now would be a good time to summarize the deployment features so far.

Some of my frustrations with existing CMSs are

  1. Content changes either take a long time to propagate, or the overall page is slow, depending on the cache settings.

  2. Deployment can take a long time.

Instant CDN cache preloading and invalidation

Netlify solves the cache problems with a smart cache. Caches are not cold because the content is actively pushed to the CDN on deploy, and, the old CDN state is invalidated on deploy. So some hard problems are solved just by using Netlify. Thus the website is super fast without the drawback of stale caches.

Faster Deployment with Delta Sync

The other issue is that static sites tend to be slow to deploy due to an O(n) deployment complexity. Again, thanks to Netlify functionality we can send just the content that changes in deployment. Furthermore, thanks to the CMS data model we can model the dependencies across pages so we only need to regenerate the pages that change too.

Netlify offers a deployment API, so we can deploy content directly from a notebook (see the deployStaticFile call).

Tag-based dependencies

File record metadata is stored in Firestore which plays well with Observable. Each record includes a tags array. When an article is updated, we do a reverse query for pages that depend on file tags using the "contains-array-any" query operator. Examples of content that do this are the index.html and the rss.xml against any files tagged "article". When an article is deployed, the page indexes are deployed too.

Parallel Materialization

To improve deploy speed, each notebook contains a serverside cell used to render the page preview of the page. The process of deployment is materializing of the preview link into Netlify. As the data exchange is a URL, we are pretty flexible about how content is expressed. The content doesn't even need to be hosted on Observable, for instance, the URL could be external (e.g. for materializing 3rd party libraries)

The other useful thing about using a URL as the content representation, and using serverside cells to generate the content, is that we can parallelize materialization jsut by reading the links in parrallel.

The most awesome thing about building on Observable is that this deployment toolchain is hosted within Observable notebooks too. The Netlify Deployment Manager contains all the Oauth code and API calls used to implement the deployStaticFile library cell. You can see how it works and change it whenever you want!

Next steps

The next job is to fix the authentication so it's easier for other Observable users to fork my blog examples and deploy their content on their Netlify accounts. We have not reached a usable service yet but it is getting closer!


Tue Dec 08 2020 20:29:35 GMT+0000 (Coordinated Universal Time)

RSS Feed added

An RSS feed is an XML file describing what new articles have appeared in a blog. They used to be popular for notifying readers of new content, but that use-case has dwindled in recent years. However, they are still very useful for notifying other computers of changes, enabling a blog to become the hub for personal media automation.

I have now added an RSS feed to the site (here). The RSS feed, like the other pages of the site, is served statically. When a new article is written, the RSS.xml needs to be update too. This requires new technology for the Observable jamstack.

I drew inspiration from Fred Wilson's blog. He writes a ton but the site is quite minimal. He organizes articles by tags, allowing topics to have dedicated lists while allowing a single article to be a member of many lists. Article tags seem enough to build an RSS feed if we can search over articles using them.

Also, to display an RSS item we need a title and description and a few other metadata fields. So on top of tags, support for arbitrary fields was added. The Observable netlify-deploy library now allows previously deployed static files (the atom of static site deploys) to be queried by tags.

So the content to deploy the (RSS.xml) is reactively updated based on the result of a realtime article query. I have granted anonymous read access to the backing Firestore for my blog so those realtime queries can be viewed by anybody.

Tag query support is possible with Firestore indexes using the "array-contains" query semantic. Firestore continues to works very well as the backing store for the Observable jamstack CMS thanks to its realtime and web-based operation.


Tue Dec 08 2020 19:29:10 GMT+0000 (Coordinated Universal Time)

Static site generation in Observable

This post was authored in Observable at @tomlarkworthy/blog-first-post. I love programming in Observable. I have always felt limited by the expressivity of CRMs like WordPress and Contentful. I want to blog using code. I want to use Observable as an interface to a static site.

Write with Code

With Observable I can generate static prose programatically:

                              #                              
                             ###                             
                            ##  #                            
                           ## ####                           
                          ##  #   #                          
                         ## #### ###                         
                        ##  #    #  #                        
                       ## ####  ######                       
                      ##  #   ###     #                      
                     ## #### ##  #   ###                     
                    ##  #    # #### ##  #                    
                   ## ####  ## #    # ####                   
                  ##  #   ###  ##  ## #   #                  
                 ## #### ##  ### ###  ## ###                 
                ##  #    # ###   #  ###  #  #                
               ## ####  ## #  # #####  #######               
              ##  #   ###  #### #    ###      #              
             ## #### ##  ###    ##  ##  #    ###             
            ##  #    # ###  #  ## ### ####  ##  #            
           ## ####  ## #  ######  #   #   ### ####           
          ##  #   ###  ####     #### ### ##   #   #          

And this is generated and embedded into a pure HTML site.

Animate with Code

I can also embed Observable cells for dynamic content (kudos Lionel Radisson). Find more of his great code here

So now I have a kick-ass static site that's super easy to update! I don't need to run a CLI command or do a PR to update it. All features can be done in the browser, including the build chain. The whole thing is entirely in Observable. Furthermore, it's all backed by CDN and is super fast, there are no compromises on the output, exactly because it's self authored.

Tech Used

I used a serverside cell called preview to dynamically serve the page. You can see that preview at the following link:

https://endpointservice.web.app/notebooks/@tomlarkworthy/blog-first-post/deployments/preview

By default, the preview page renders every visit. This is somewhat slow, taking around 2-3 seconds, but it means published changes are reflected quickly. However, it is a horrible URL and too slow for production.

I give the page a nice URL using Netlify. To make the production page fast, I max the shared cache settings in the serverside cell when a production X-Version header is present. Thus, so we lean heavily on the integrated CDN.

On the Netlify end, I set up the page to redirect to the serverside cell URL and add a custom X-Version header. When the production page is updated, the version header is bumped, so the upstream cache is invalidated.

Stay tuned

The personal webpage is a work in progress. Meta tags are missing, the RSS feed doesn't work and it doesn't support more than one page yet! But I will add to this over the next few weeks and hopefully get it to a state where anybody can create a page easily. For now, follow along on Observable RSS feed icon or Twitter.

Check


Tue Dec 08 2020 19:28:50 GMT+0000 (Coordinated Universal Time)

A Zero Install Forkable Jamstack

This blog doesn't require tools to be installed.

Its trivial to write and update content from any computer.

  • Everything required to write content or customize the deployment engine is web hosted.
  • Content is written in Observable notebooks (e.g. this post, an earlier one or the navbars).
  • The deployment toolchain is also hosted in an Observable Notebook (e.g. Netlify deploy).
  • Observable is designed for literate programming. Markdown or HTML or roll your own DSL.

This blog is fast and does not require Javascript

The usual Jamstack advantages apply.

  • Compiled to static assets deployed to a CDN.
  • Exploits Netlify' instant cache invalidation so production updates are fast.
  • Scaleable and secure.

Google Page Speed test

This blog engine is Programmable, Open Source and Forkable.

Because the engine is programmed in Observable:

  • Content is written within a web hosted IDE. You generate content programatically.
  • Content pages, the deployment pipeline are executed in the browser, in cells viewers can look at.
  • All pages can be forked and reprogrammed, allow blog developers to customize their blog engine without installing tooling.