{🌩} Coding Up A Storm

Prodzilla: From Zero to Prod with Rust and Shuttle

Wed, 21 Feb 2024 00:00:00 +0000

I’ve been working on Prodzilla, a modern synthetic monitoring tool built in Rust. I wanted to share how it’s different to existing tools, why I’ve built it in Rust and how you can host it for free on Shuttle, and what I hope to achieve in the long-term.

What-zilla?

Synthetic monitoring is a staple component in the testing and observability of SaaS products - probing services in production to ensure they’re up and working as expected. Many synthetic monitoring tools exist, which generally shallowly check connectivity to an endpoint - they rarely support the testing of real system behavior.

As a synthetic monitoring tool Prodzilla is unique in a couple of key ways. Firstly, it focuses on testing complex user flows in the way a real user would, by allowing chained requests, user authentication, and use of response body values in subsequent requests.

Secondly, it optimizes for human readability, allowing the creation and verification of these complex flows with just yaml, rather than requiring written code. This yaml file is all that’s needed to have a multi-step user flow scheduled to be tested every 60 seconds, assert that the system behavior is as expected, and notify you if it’s not. I hope to make this more human readable over time, both in yaml and via UI.

stories:
  - name: get-ip-info
    steps:
      - name: get-ip
        url: https://api.ipify.org/?format=json
        http_method: GET
      - name: get-location
        url: https://ipinfo.io/$/geo
        http_method: GET
        expectations:
          - field: Body
            operation: Contains 
            value: "Australia"
    schedule:
      initial_delay: 10
      interval: 60
    alerts:
      - url: https://my.site/notify

As for why I’ve built this: We should all be testing more in production. Once an engineering faux-pas, testing in production is now more of a sign of sophistication than a sin. I wrote more about this in another article, ‘Everyone’s Talking About Shifting Left - Here’s Why I’m Shifting Right’.

Prodzilla is a synthetic monitoring tool right now. Over time I want to deliver a range of tools designed to help everyone test in production. More on that below.

Why Rust?

Using Rust for Prodzilla was a very deliberate decision - for the following reasons.

Performance and Predictability

The primary reason for choosing Rust was that I wanted to be proud of Prodzilla performance-wise. A synthetic checker is a nice, compartmentalized project, it shouldn’t need significant overhead. It should be able to run on virtually anything, and it should be cheap. Rust is of course known for its low-memory and high-speed. What really won me over is this oft-cited table from a 2017 paper that describes the relative energy efficiency of programming languages:

This is a contrived measurement, but it’s a function of execution time and memory consumption - and Rust is just behind C!

Yet despite such strength in performance, Rust behaves like a high level language for general http client / server functionality. In some ways it feels like getting a performance boost “for free”.

Given I was building a monitoring tool, and also one that I’m going to release publicly and tie my name to, I also wanted it to be maximally predictable and reliable itself. Generally this is all referenced under the umbrella of “memory safety”, but I have continually found the compiler is protecting me from doing dumb things - for example catching iterator invalidation, or ensuring that Prodzilla’s execution state can be shared across threads.

Community

Alongside the technical reasons though, I’ve been following and learning Rust for a few years, and I really like the Rust community. It’s hard to ignore Rust repeatedly topping the StackOverflow developer survey in languages developers want to use more, or spicy articles from Discord about the benefits of porting a service from Go to Rust.

Moreover, I especially like where Rust is right now in the web space. It really feels like there’s a lot of smart people working on the next generation of web development tools - it feels like the place to be. There are a range of great open-source web dev tools that are just reaching critical levels of maturity. Axum, which I used to build Prodzilla, feels ready for out of the box web dev, and is crazy-performant, as I write about later. More recently available is Loco, a Rails-like framework for building web applications in Rust that’s picking up steam. And in dev-tooling and hosting there’s Shuttle, a 1-line hosting solution for Rust backends.

I saw all of this happening and wanted to contribute.

It’s not just open-source tools either; it feels like there is real industry traction here too, with Microsoft looking for Rust engineers to rewrite parts of Office 365, and Meta endorsing Rust internally for backend engineering.

The Learning Curve

Before Prodzilla, I’d read ‘The Book’ a couple of times, and had made my way through Rustlings, but hadn’t yet built a serious project in Rust.

Faster Than Expected

Knowing the performance and safety benefits of using Rust, I expected to find it much slower than languages like Kotlin, C#, or Golang to build a real project in. Surprisingly, across the board I found that I was generally moving fast. Some small syntactic sugar goes a long way - coming from Golang and its verbose error handling, using ? for error propagation feels great.

Fighting the Borrow Checker - and Calling in an Ally

A sentiment I’ve heard over the years is of developers ‘fighting the borrow checker’. ‘Fighting’ seems too strong a word for my experience. But maybe that’s because - one thing that I haven’t heard enough people mention enough - ChatGPT is pretty good at fighting back!

On the occasions when I’ve been stuck, I’ve copy-pasted the offending code into ChatGPT 4, along with the error I’ve been getting, and asking for help. And every single time I’ve received thoughtful suggestions for how to navigate around or correctly achieve the desired outcome for cases totally specific to my situation. A specific example that came up recently whilst trying to return some probe results through the Prodzilla API. The values were nested in a hashmap, in a read write lock, in application state. I tried to access them like so:

state.probe_results.read().unwrap().last().cloned()

But got back the error temporary value dropped while borrowed. Given the complexity of the multiple wrapping types, not getting an instant answer from Google, and that I was still learning Rust, I dropped the offending code into ChatGPT and it corrected it with an explanation right away:

let lock = state.probe_results.read().unwrap();
lock.last().cloned()

I now know that in a situation like this we need to explicitly assign the variable so that the lock lives for the lifetime or scope of that variable - in this case for the function block - otherwise we’re creating a situation where the lock is closed before the result can be returned.

Fighting the Borrow Checker is a Good Thing

It’s kind of cool, looking at those two lines, I have a much better intrinsic mental sense (still not a perfect sense - definitely still learning!) of how long the memory lives, without allocating anything myself.

Given all of this, I’m glad the borrow checker’s been fighting me. I’ve had to be more thoughtful about what memory should live where, and for how long, I’ve had AI help, and as you’ll see in the next section the performance benefits are real!

The Outcome - Prodzilla

After a couple of months spending time on Prodzilla after work, I’m proud to say that Prodzilla currently supports the following features:

Probing individual endpoints and Stories: chained requests to multiple endpoints, emulating real user flows
Verifying that response bodies, headers, and status codes meet expectations using operations such as Equals, Contains, etc.
Passing variables from one step of a Story to another, Github Actions-style, e.g. ${{steps.authenticate.response.body.token}}
Automated notifications for probe or story failures via webhooks
Manually triggering a probe or story via a /trigger endpoint
Retrieving a history of previous behavior for all probes and stories
OpenTelemetry integration - trace IDs for all probes
Configuring all of this via yaml - without writing code!

Small Footprint, Lightning Fast

Whilst running, Prodzilla consumes about 9MB of memory. Note that currently probe and story history is stored in memory rather than in a database, so this can inflate when calling sites with large response bodies. In the future I’ll introduce persistent storage.

When testing with 10 multi-step probes being triggered every second (more than I expect anyone is reasonably going to be probing), memory barely moved from that 9MB mark, and at max consumed around 1.5% of my CPU, a crusty old i7-8550U @ 1.80GHz.

The size of the production binary is 7.8MB.

As someone who has worked as a backend engineer for their whole career, across Golang, Kotlin, C# - these numbers are pretty crazy. An empty spring boot application - a point of reference as a very common backend framework - is going to run with somewhere between 100-200MB of RAM. To have a web server and a synthetic monitoring agent running under 10MB - this feels like magic, but also a step in the right direction.

I’d like to spend some time benchmarking Prodzilla properly, and especially comparing it to some of the existing tools, such as Grafana’s synthetic monitoring agent - hopefully a story for another article.

Hosting

When building Prodzilla, I had a couple of hosting problems to overcome:

How can I host a Rust app cheaply for my own use?
How can my users cheaply host a Rust app that I’ve built?

One of the hardest parts of encouraging usage of open-source tools is the burden users have of hosting it themselves. To get a real backend service up and running, connected to a database, and exposed at some nice url - despite being the backbone of every production application ever - is not a streamlined process. When building and hosting any proof-of-concept or side project, these tasks are a huge time-sink.

Enter Shuttle

Trying to work out how to overcome this, somewhere along the way I found Shuttle, which is an open-source platform that allows free hosting of Rust applications (using AWS behind the scenes). What surprised me about Shuttle was that without manually provisioning anything in AWS or GCP, I could deploy an application for free with just slight tweaks to my application code. I ported Prodzilla to deploy on Shuttle in about 5 minutes. Here’s the diff.

The core of the change is essentially adding an attribute and a return type to the main function:

#[shuttle_runtime::main]
async fn main() -> shuttle_axum::ShuttleAxum {

I’ve also started experimenting with integrating a database to store probe and story results in Prodzilla. Again, all the local setup is just gone, and with slight tweaks Prodzilla runs locally with a containerized database, or can be deployed to the cloud with a provisioned database:

#[shuttle_runtime::main]
async fn main(#[shuttle_shared_db::Postgres] pool: PgPool) -> shuttle_axum::ShuttleAxum {

This is kind of bizarre for me to acknowledge, but with Rust and Shuttle I could get an app with a database up and running at some real url, and into prod faster than in any other language. It feels like the future. This is what backend development should feel like. Products like Vercel have made frontend development and hosting considerably easier - but on the backend I feel like we’re used to kind of wrestling through cloud platforms, hence why Shuttle is such a nice surprise.

I came into Prodzilla thinking that I’d get some awesome benefits from Rust, but would be slowed down by the language and hosting maturity. I was definitely wrong!

Vision and Goals

My ultimate goal with Prodzilla is to help fix production observability and testing, in the process helping everyone write more reliable software.

I don’t think that observability and quality testing should be as divergent as they are, with different parts of the development lifecycle, different codebases, different methodologies. By allowing Prodzilla to easily run in CI/CD pipelines and against different environments it can help tackle that.

I don’t think that system behavior documentation, both internal and customer-facing, should go out of date as quickly as it does. By leveraging Prodzilla’s history of observed behavior, building a feedback loop between documentation and observed behavior, with a bit of LLM magic, I think we can fix that.

I think that we should test in production more, even for cases where it feels too hard. Creating test users alongside production data, supporting canary releasing, flagging test requests to avoid calling specific downstreams - these are all fairly opaque and difficult things at the moment that I think we can make easier.

I know these are big goals - maybe too ambitious, but at least that’s the direction Prodzilla is heading in.

Conclusion

All up, I’ve been working on a new synthetic monitoring tool, and I’m glad I decided to build it in Rust. I’ve found that between the language itself and tools like Shuttle it’s a joy to write really performant backend services very quickly!

More than anything else I’m looking forward to feedback from everyone that tries out Prodzilla, and especially from anyone that would love to use it but needs just that one more specific feature.

Feel free to get in touch at any of the below:

Everyone's Talking About Shifting Left - Here's Why I'm Shifting Right

Tue, 07 Nov 2023 00:00:00 +0000

Every so often in software engineering, new methodologies and frameworks emerge with the promise of increasing the reliability and delivery velocity of software. These “Grand New Ways Of Doing Things” have a period of time during which they are applied universally, touted by consultants and Engineering Managers alike, to solve all the current problems. Previous “Grand New Ways Of Doing Things” such as Agile, Test Driven Development, even Microservices, despite starting as cure-alls, have settled into the toolkit of sometimes-useful strategies, contingent upon context.

The latest buzzword that’s probably injecting a few too many technical items on your product roadmaps is “Shifting Left,” a concept that champions the idea of integrating more testing, development, and planning stages earlier in the software development lifecycle. The principle is simple: the sooner you catch and fix a bug, the less it costs and the faster you can move. This approach encourages developers to run more tests locally, integrate earlier, and generally, speed up the feedback loop on the development process.

However, despite its popularity, I remain skeptical about the effectiveness of Shifting Left, or at least the way that it’s generally applied. Here’s why:

The Downsides of Shifting Left

Running services locally requires high dependence on mocks / fakes. To run things locally or in early environments, teams often rely on mocks or fakes to stub out responses from downstream services. These require significant time to create and maintain, obviously don’t operate the same way as their real counterparts in production, and risk becoming outdated over time as the real service is updated.

The Production Paradox. No matter how sophisticated the local or pre-production environments are, they never fully emulate the complexities of production. Here’s a list of some things that are likely to differ in production vs other environments:

HTTP/ GRPC vs HTTPS / GRPCS, self-signed certs vs TLS, etc
Connections to vendors and third-parties
Access issues, networking and permission-wise - what the services and the engineers have access to
Number and availability of each service - everything pre-prod is usually scaled down for cost
The test data in pre-prod often looks vastly different to real use
The patterns of use in each environment

This single gap is the leading cause of overconfidence in the stability and performance of software, and hence production incidents.

Dev Efficiency. It’s 2024. Software engineering is changing. The fastest delivering engineers on any team will at times have multiple pull requests open, relying on automated tests run somewhere in GitHub or GitLab land to check their work for them. Despite knowing how to do it in specific situations, I don’t believe great engineers regularly spin up a local environment, as it can be incredibly time-consuming, detracts from dev time, and this work can be rightfully offloaded to the cloud VM farm.

Embracing the Fear: Shifting Right

Shifting Right, on the other hand, is about embracing the once-taboo practice of testing in production. “Testing in Production” used to be synonymous with recklessness. Today, I see it as a sign of sophistication, and I see a lot of potential in this domain for enhancing software reliability and delivery speed.

The Case for Shifting Right

Authentic Validation. Testing in prod offers a level of assurance unmatched by any pre-prod environment. When you observe your software operating successfully in the real world, you know it’s genuinely ready.

Shifting Right unifies testing strategy. In the current standard development process, each of blackbox and end-to-end tests, synthetic monitoring, and observability occupies their own slice of a development cycle and has own codebase. Yet all of these things are extremely related! Convergence can not only simplify the testing process under one cohesive plan, but allows for less dev time spent on each individual step, and high levels of code reuse.

Cost Efficiency: One thing engineering management everywhere is talking about more than shifting left is reducing cloud costs! Most dev teams are operating with 4-5 environments in various states of disrepair, but get almost all of their value from 1-2 of these (hint: one of these is prod!). Maintaining multiple dev or testing environments is a costly affair. By focusing on production, we can reduce unnecessary expenditure on cloud resources.

Introducing Prodzilla

It’s from this perspective that I’m developing Prodzilla.

Right now Prodzilla is an open-source, low-code, synthetic monitoring tool, with a focus on easily testing complex user flows not traditionally tested in prod. The intention is to grow it into a framework that provides everything you need to test in production, and surface the discovered behaviour in useful outputs such as internal or external docs and alerts.

If you like the idea, please give us a star on our Github!

Conclusion

All this is to say, as with all things, the correct approach is likely balanced. If your engineers aren’t getting any feedback about the quality or behaviour of their code until the end of some pipeline, then yes, please shift left. But don’t fret just because your team isn’t running anything locally - consider this might actually be a good thing! Shifting Left on its own is not a silver bullet, and the losses associated with an over-reliance on simulated environments are significant.

Shifting Right offers a pragmatic, reality-grounded approach that aligns testing with actual user experiences. I hope Prodzilla can be one part of this journey, aiming to make testing in production a feasible, efficient, and integral part of the software development lifecycle.

How I'm Getting Free Synthetic Monitoring

Tue, 07 Nov 2023 00:00:00 +0000

I’m using a couple of open-source and Rusty tools to get free synthetic monitoring for my side projects. It’s really easy - maybe it will help you!

Why Synthetic Monitoring?

Synthetic monitoring is a proactive approach to web application monitoring, whereby we fire real requests at real endpoints to simulate user interactions. It allows for consistent testing of website availability and behaviour, and is particularly useful for identifying and resolving issues before they impact real users!

The Tools

Prodzilla is a new open-source synthetic monitoring application built in Rust. At present it allows defining probes that will call defined endpoints on a defined schedule, and assert that the responses meet expectations. When they don’t, Prodzilla currently has the ability to alert by sending a webhook to a given URL. No need for any special setup to use it, so long as you have Rust installed. Full disclosure: I built it!

Shuttle is a great, quick, hosting service for Rust applications. Importantly it’s also currently free, and guarantees a forever free tier. Shuttle makes it extremely easy to deploy a live Rust application fast. For this tutorial you’ll need to create an account at shuttle.rs and then install the Shuttle CLI with:

cargo install cargo-shuttle

The Process

Configuring Prodzilla

Prodzilla has a specific branch ready to launch with Shuttle, so clone that. If you’re curious about the structural differences between this and a standard Rust application, the specific changes required to migrate are outlined in ‘Migrating to Shuttle’.

To get your local prodzilla ready to deploy, we’ll need to customise a couple of things.

Firstly, you’ll want to customise your prodzilla.yml. You likely want something like the below, substituting in the site you want to probe. This will probe a site is up and returning a HTTP 200 OK every 5 minutes (300 seconds), starting from 5 seconds after Prodzilla starts up.

probes:
  - name: Your Probe
    url: https://yourwebsite.com/endpoint
    http_method: GET
    expectations:
      - field: StatusCode
        operation: Equals 
        value: "200"
    schedule:
      initial_delay: 5
      interval: 300

The other fields are mostly self-explanatory, but you can also include expectations about Body, and can also use the operations Contains or IsOneOf (using pipe separated values).

If you need to add auth, Prodzilla currently supports auth using Bearer tokens in headers, though you would need a long-lived token with the current version of Prodzilla.

probes:
  - name: Your Probe
    ...
    headers:
        Authorization: Bearer <token>

If you’d like some system to receive a signal when one of your probes fails, you can add a webhook receiver URL to your probe as well:

probes:
  - name: Your Probe
    ...
    alerts:
      - url: https://alertme.site/webhook

Deploying to Shuttle

The last thing we need to do in code before deploying is to set the Shuttle project name, in Shuttle.toml. Choose a unique name:

name = "my-prodzilla"

And now we just need to send it! Which we can do by executing the following commands. Note that shuttle will complain if you haven’t committed in git, so do that or bypass it by adding the –allow-dirty flag.

cargo shuttle project start
cargo shuttle project deploy

And voila! If you navigate to https://{name}.shuttleapp.rs you should see a Roar! And if you navigate to https://{name}.shuttleapp.rs/probe_results you should see a json output of the results of your probes, which are stored in memory.

Get in Touch

I hope that’s been helpful - would love to hear any feedback. Happy roaring!

X / Twitter: @codingupastorm.

LinkedIn.

prodzilla.io.

A Foray into Neuro-stuff

Tue, 07 Nov 2023 00:00:00 +0000

I haven’t posted in 3 years. I’m pursuing some private study / learning into Neuroscience generally, and trying to find some interesting, solvable problems I can commit to contributing to, bringing my previous experience in general software engineering.

I’m deliberately avoiding academic research - something like pursuing a PhD. I’ll post about why one day.

What I want to do is share what I’m learning and looking into with some cadence. I could note this privately, but I think there are some benefits to it being public.

I’m loosely committed to continue to work and post by talking about it publicly
Writing about things forces one to have clarity of thought, so I get a chance to refine my own thinking
Assuming I do “get somewhere” with this, it could provide a useful roadmap for myself or others on how I got here. (I’d love to elaborate also on what “get somewhere” means - again I’ll post why one day.)

The format will evolve as I go.

Here’s what I’m thinking about now

Current wider ideas:

For the most part, academic code is poorly readable, not highly reusable, and slow.
There is starting to be a heap of data available online, which, so far as I can tell, noone is using cohesively. See OpenNeuro
There are a heap of documented problems available online, usually in the form of PhD breifings. See The Florey Institute for an example.

Specific Learnings:

It seems like the most available and useful format for gathering information from living brains is the MRI. An MRI can retrieve different data based on running MRI sequences.
Tractography is a specific imaging technique based on a set of particular MRI sequences, the DWI (Diffusion Weighted Image)
I think this is a good starting resource for tractography, which I got to by following a more recent article on tractography, which was part of a PhD brief: Hot topics in diffusion tractography

Executing Rust code from C#

Mon, 04 May 2020 00:00:00 +0000

Right at the front of “The Rust Book” there is a section about calling Rust code from other languages. The examples use Python, Ruby and Javascript and show how using Rust for expensive standalone processes can save time.

I’ve spent a lot of time working with C# in my career, so naturally I was curious how much faster the example Rust code would be than the C# equivalent.

The full repo is available here if you want to try it for yourself.

Here’s the Rust code:

pub extern fn process() {
    let handles: Vec<_> = (0..10).map(|_| {
        thread::spawn(|| {
            let mut x = 0;
            for _ in 0..5_000_000 {
                x += 1
            }
            x
        })
    }).collect();

    for h in handles {
        println!("Rust thread finished with count={}",
        h.join().map_err(|_| "Could not join a thread!").unwrap());
    }
}

This code is doing something very trivial: it’s starting 10 threads, counting to 5,000,000 on each of them, and then announcing when it’s complete.

Here is roughly the C# equivalent:

    private static void ProcessCSharp()
    {
        const int threadCount = 10;
        Task[] tasks = new Task[threadCount];

        for (int i = 0; i < threadCount; i++)
        {
            tasks[i] = Task.Factory.StartNew(() =>
            {
                int count = 0;
                for (int j = 0; j < 5_000_000; j++)
                {
                    count += 1;
                }
                Console.WriteLine("C# thread finished with count={0}", count);
            });
        }

        Task.WaitAll(tasks);
    }

We’ll see how these compare in terms of run-time soon.

Building a DLL From a Rust Project

To be able to call Rust methods from C#, we need to put it into a format that C# can understand. In this case we’re going to use DLL (Dynamic Linked Library) files.

To generate a DLL from the Rust project, we have to firstly ensure that the external-facing methods are inside lib.rs and given the visibility pub extern. They also need to be marked with the #[no_mangle] attribute, whch prevents Rust from internally messing with the API names during optimisation.

Lastly, we need to add these lines to Cargo.toml before the [dependencies] section:

[lib]
name="RustLibrary"
crate-type = ["dylib"]

These lines tell Rust and Cargo that we want a DLL named RustLibrary.dll when we build the project.

When all this is done, all we have to do is build our Rust project (with the Release flag because we want it to be as fast as possible):

cargo build --release

And voila! Our DLL, target/release/RustLibrary.dll has been created for us!

Calling a DLL From A C# Project

Now that we have a DLL, we can interact with it by adding RustLibrary.dll to a Visual Studio project, ensuring that it gets copied to the output folder on build, and importing the DLL inside our C# code.

Here’s how that looks:

    [DllImport("RustLibrary.dll", EntryPoint = "process")]
    private static extern void ProcessInRust();

This is where the magic happens - if you now call ProcessInRust() anywhere in your C# project, what happens behind the scenes is your .NET process calls into the Rust code we created earlier!

Is it Faster? Yes.

Here’s the extremely basic code I wrote to test the speed of the ProcessInRust() and ProcessCSharp() methods above.

Stopwatch csharpStopwatch = new Stopwatch();
Stopwatch rustStopwatch = new Stopwatch();

csharpStopwatch.Start();
ProcessCSharp();
csharpStopwatch.Stop();

rustStopwatch.Start();
ProcessInRust();
rustStopwatch.Stop();

Console.WriteLine();
Console.WriteLine("Execution time in C#: " + csharpStopwatch.Elapsed);
Console.WriteLine("Execution time in Rust: " + rustStopwatch.Elapsed);

When I run this on my computer in the C# project:

dotnet run -- --release

This is the output:

C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
C# thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000
Rust thread finished with count=5000000

Execution time in C#: 00:00:00.0387920
Execution time in Rust: 00:00:00.0133602

The results varied but this is about the closest C# got to Rust in my runs. According to this, Rust is around 3 times faster at summing to 5,000,000 on 10 threads. That’s not as big a difference as I was expecting, though I understand that calling into an external library itself is accounting for some of the Rust method’s execution time.

I’m excited to try out some more complicated scenarios and see what the difference is like.

Testing as a Selfish Endeavour

Mon, 13 Jan 2020 00:00:00 +0000

At the start of my career I avoided writing tests wherever possible. I could start my application, and see that it worked for me. Why write tests for something I already know works? I get nothing out of it! This mindset is really okay for your own projects. Where you’re the boss, if you want to work like this and noone else is dependent on it - fine. No worries.

I think about tests radically differently now though. Selfishly, I get a lot out of tests. They save me a lot of time, but they also help me save face. That’s on top of all of the usability and financial benefits for the other developers on the project and the companies I work for.

How do I know this works?

Whenever you introduce new functionality to a repository, someone else is likely going to want to use it. And when they do, there is a chance it doesn’t work for them. Guaranteed. So many issues arise when running code on another computer. From different environments and system variables to missing packages, something will come up. If you push new code without tests demonstrating at least a success case, then it just looks like you committed broken code. That sucks! Especially because it might be a configuration issue on their end. If you committed your new code with a passing test case, you can point to it. If it didn’t work for someone else, maybe it’s a problem on their end. Tell them to go look at the test. What are they doing differently?

By writing that test, you are automating away blame.

Moreover, if I’m a code reviewer for a repository, just looking at the code alone I have absolutely no idea if this thing does what it’s meant to. I would have to read some description of the expected behaviour, pull the code, run the application and then manually test it. At scale, when iteratively developing software, with a team of full-time developers, this is unfeasible. A test does all of these things for you. It describes the things that need to happen, runs the software, and then confirms that it did what was required. Be kind to your code reviewers, and decrease the chances of your pull request being rejected - write tests!

Vanquishing bugs forever

A cool way to think about testing when fixing a bug is this:

When you add a test to your automated test suite that checks for a specific bug, and then you fix that bug, you are vanquishing that bug from the face of the Earth. It will never appear again. Noone else will be able to reintroduce that bug because your regression tests will recognise it forever.

Look how fast this is

Profile before you start making optimisations to existing code. The amount of times I have assumed that a certain part of the code was slow, micro-optimised it like crazy and found that I shaved off almost nothing… is too many. Write tests first, know for certain what the bottleneck is, and then fix it. Save your own time.

And the best part, the most self-boosting thing of all: There is absolutely nothing better than posting a screenshot of a test that used to run in 30 seconds and now runs in under 1 second thanks to your work. Staple that screenshot to the top of your pull request.

Conclusion

I think developers mostly consider testing to be something that is done for someone else. To meet contribution guidelines or appease reviewers. In reality, it serves the developer him/herself more than anyone. It’s a slightly narcissistic idea, but it definitely helps me write better code!

The CREATE2 OpCode and DApp Onboarding in Ethereum

Thu, 17 Jan 2019 00:00:00 +0000

The Ethereum network’s next major upgrade is the Constantinople upgrade (was meant to be January 16th but has since been delayed). The upgrade introduces several new features, one of which seems unexciting on the surface but enables a range of possibilities for scalability and user onboarding. This feature is the introduction of a new opcode for the Ethereum Virtual Machine: CREATE2. This article will briefly outline what CREATE2 does and how it could drastically improve the adoption cycle for decentralised applications.

What is CREATE2?

The important thing about CREATE2 is it allows DApp (Decentralized Application) developers to generate contract addresses without having to actually deploy a contract. Previously there was no way of “reserving” a contract address without deploying it. We’ll discuss why this is a problem for adoption later.

The actual CREATE2 opcode behaves virtually identically to the current CREATE opcode with 1 slight change. Both attempt to deploy some EVM bytecode as a new contract. However, whilst the contract address that a CREATE call deploys to is dependent solely on the sender and nonce, the new contract address for CREATE2 is dependent on extra input data.

In simple terms, you can think of it as allowing developers a level of “control” over the new contract address generated.

The Onboarding Process Before CREATE2

Think of some DApp that we’re trying to build and to market to the general public. At some point in the process of users interacting with our DApp, we likely want to give them some on-chain reward; maybe ether, tokens, or some non-fungible token. To do this, of course, the users need their own address, despite this application being their first interaction with the Ethereum blockchain.

There are a couple of options here.

We could maintain a list of private keys on some centralised server behind the scenes of the DApp. This would allow us to cheaply distribute new addresses to all of our users. However this is a significant security burden for us, and really takes the “D” out of “DApp”.

Another option is to create a wallet-like contract for every new user. Initially, our DApp will have full rights to all of the operations on this contract. Users can still receive everything they need, and whenever they do create their own Ethereum address, we can easily transfer full ownership over to them, removing the rights of our DApp. In theory, this works great. In practice, it’s very expensive. Contract deployments cost gas, and as the developers of this DApp we have to continually fund the creation of new contracts — even for cases where the users never come back. Sunk cost.

If only we could know about these contract addresses without having to spend the gas to create them!

Enter CREATE2

With CREATE2 our DApps can now know a contract’s address before it’s created. In the case above, we can easily generate wallet contract addresses for all of our users. Off-chain. For free.

We can send all of the tokens and in-game items that we need to to this address, and when the user is ready to commit and claim their new property, we can require that they send a small amount of ether to the contract address. Which will allow our DApp to go and create the contract for free, taking some of the funds to cover gas costs!

Concluding

This is just one possible brainstormed workflow, but hopefully gives you an idea of the kinds of things we can start doing when we are able to reserve contract addresses for currently unidentified users.

Let me know if you have any thoughts, or if this high-level explanation can be improved in any way.

Find me on Twitter: https://twitter.com/codingupastorm

The Importance of Modularity in a Blockchain Platform

Wed, 16 Jan 2019 00:00:00 +0000

Modularity refers to a system’s ability to be broken down into components which can be separated and recombined. The best software development tools are often highly modular, and allow the reuse of “assemblies” or “modules”. This principle is at the core of development frameworks like Java, C# and Go.

Modularity promotes innovation because new ideas become trivial to implement. When creating a new web app, the majority of the code running is really inside of externally-developed packages; whether they be database connections, page rendering engines, input validators etc. As such, it’s extremely easy for developers to spin up a prototype of a new website idea.

Enter: Crypto

However, if you were to go and look at the most prized crypto codebase of all, the Bitcoin Core repository, you’d notice that it’s extremely inflexible. Unmodular code is generally characterised by long files and classes, and individual implementations of classes often depend on other concrete implementations. For an example see Bitcoin Core’s validation.cpp. This isn’t just the case with Bitcoin Core. Most of the biggest blockchain node repositories — another example is Go Ethereum- are absolutely not built with flexibility in mind.

Now it’s important to note that in the case of Bitcoin, modularity isn’t a priority. If you believe you’re building the singular “currency” of the future, why make it easier for others to build with your code? Bitcoin Core is highly integrated, but it has stood the test of time and it works. This is not a criticism of Bitcoin Core’s codebase.

Experimenting with Blockchains

With blockchain being cited as the answer to everything, many projects are starting to experiment with different node implementations. Especially in the case of sensitive data and private chains, or cases where external data is core to a chain’s operation, projects are finding that a smart contract based approach does not allow them enough flexibility. Unfortunately until recently this has meant that in order to innovate, developers are pulling apart integrated nodes like Bitcoin’s, or starting new nodes from scratch.

Both of these avenues are painful, time-consuming, and error-prone (insecure).

This space is really going to thrive when a blockchain network with a completely new set of features can be spun up clicks fingers like that.

Stratis — A Modular Platform

The most exciting thing about Stratis for me is we’re thinking about building blockchains radically differently. Different consensus algorithms, smart contract executors, wallets, and two-way peg implementations are all features in the one codebase.

Currently you can run — all from the one codebase:

A Bitcoin (PoW) node
A Stratis (PoS) node
A Cirrus (PoA + smart contract) node.

Moreover, because of how flexible the node is, I’m confident we could build out full node integrations for Bitcoin Cash, Gold, Private, Doge, Litecoin, etc. in *days *each. Down the road we could extend this to support even more chain types easily, after some foray into the account model and other features.

When building a new website in .NET Core, you can preload a template and adjust the components to your needs rapidly, punching out a new prototype web app in under a day. We hope that with Stratis developers will be able to do the same but for their own blockchain networks.

IFullNode node = new FullNodeBuilder()

    .UseNodeSettings(nodeSettings)

    .UseBlockStore()

    .UseMempool()

    .AddRPC()

    .AddSmartContracts()

    .UseCLRExecutor()

    .UseApi()

    .Build();

Isn’t it beautiful? Our Full Node repo is here.

In Conclusion

Modularity == Innovation.

I don’t think anyone is as far ahead as we are when it comes to a strong composable approach to building blockchain networks.

To find out more follow us on Twitter: https://twitter.com/stratisplatform

Or if you just want to wave: https://twitter.com/codingupastorm

The Strangest Smart Contract Bug You Haven’t Heard About

Mon, 14 Jan 2019 00:00:00 +0000

Note: I have no intention of discrediting the project or anyone involved with it. The beautiful part about crypto and smart contract development is all of the code is on display. I’m just commentating on a monumental screwup and a very interesting situation.

Legolas Exchange (LGO) raised ~$30M USD early in 2018 and deployed an ERC-20 token on Ethereum. The token has a peculiar feature built into the code that is supposed to reward hodlers (sic). For every 6 month period in the first 2 years, all of the accounts that haven’t moved their funds yet should receive an additional 5% in tokens as a reward for their loyalty. The reward has the potential to be higher than 5% — increasing as others abandon their rewards.

Sounds great right?

Unfortunately the immutable contract was coded such that the extra 20% of tokens, that should have been used to pay out the bonuses, is virtually guaranteed to be lost.

I say “virtually guaranteed” because there is a way to retrieve them — it just requires ungodly levels of cooperation. More on this soon.

The Bug

There are several parts of the contract that I think should not have made it through any serious audit, such as the perverse use of percentages to assign token amounts.

Nevertheless, the problematic bug that led to this article can be found on line 326 of the contract source:

// calculate the bonus for one holded LGO
uint256 bonusByLgo = (BONUS_AMOUNT / 4)/unspentAmounts[_bonusDate];

I’m not going to go deep into this — you can easily check what I’m saying for yourself. All you need to know is that this variable, bonusByLgo is expecting to be assigned a percentage. Something like 0.05. If you’re a developer (from any background, not just smart contract development!) you know how absurd this is. A uint256 can only be assigned 0, 1, or any other whole integer.

So when the bonus percentage to dish out is calculated, the user claiming is always going to receive a 0% reward due to natural rounding-down.

Besides this mistake — the bigger mistake is this: Any testing of the claimBonus method would have found this problem. $30 million US Dollars were put into a contract where the core feature was completely untested.

How They “Resolved” It

To their credit, the LGO team started paying out the bonuses manually, from their own reserve, which was also 20% of tokens. You can see these transactions happening over a long period of time — if you want to do in-depth chain analysis you can find one LGO-owned address dispensing a round of bonuses here.

Potentially holders of LGO will be reading this and have no idea something was wrong, but this mistake cost the company ~$7M USD at the worst, if we were to value the tokens at LGO’s highest historical market cap.

A Really Weird Way Out

So as I mentioned when discussing the bug, the variable bonusByLgo will almost always evaluate to 0. Except in one case. When (BONUS_AMOUNT/4) is bigger than unspentAmounts[_bonusDate] . When this happens, the remaining holders will receive one bonus token for every LGO token they hold — a 100% bonus!!

What can trigger this condition? Only if 95% of the holders move their tokens. Only when 5% of the tokens are still eligible for bonuses, they would all receive a 100% bonus at each of the remaining bonus dates.

The really crazy part is this! If the community came together and all decided to move their tokens — completely the opposite of what the contract intended — they could potentially deliver LGO back at least some of their tokens and save LGO money. You could even imagine a situation where the LGO team or another rogue developer builds a smart contract that allows the bonus to be claimed and distributed amongst the remaining holders — though good luck getting the required >95% participation.

The Lesson

Be skeptical of any token contracts with overly complex functionality. “Smart” smart contract developers should be staying away from any such gimmickry, and when it is absolutely needed, it should be tested and audited to extremes.

For more on interesting aspects of smart contracts find me on Twitter: https://twitter.com/codingupastorm

The First Token Smart Contract Executing in Native C# / .NET on Stratis

Sat, 20 Oct 2018 00:00:00 +0000

Note: Hundreds of you have been messaging me that X other platform has smart contracts in C#. I can say with certainty that Stratis is the first to be executing C# natively. Other platforms (e.g. NEO) may be compiling C# syntax to custom machine code but we’re definitely the first to be executing actual .NET code on the Common Language Runtime, .NET’s virtual machine. (I should know, I previously built EthSharp which compiled C# for the EVM!)

Over 12 months ago, slightly frustrated with the “young” state of solidity and Ethereum smart contract development environments, I started craving bringing my favourite development environment (C# / .NET / Visual Studio) to blockchain development. Over the course of a couple of months I made progress on a project called EthSharp(see it on github). It was starting to functionally generate EVM bytecode from C# syntax but I ultimately discontinued it because I felt that compiling to a non-native bytecode negates the possibility of leveraging an established framework’s ecosystem and may carry a range of security risks. Note to self: Write an article about this!

The smart contracts team at Stratis has since been building something much more exciting, a blockchain with the ability to execute .NET code on-chain. Smart contract platforms so far have generally introduced or repackaged a custom virtual machine like the EVM, whilst we’ve gone for a very different approach :)

The protocol isn’t in production yet — it’s scheduled to be live by the end of the year. You can find out more about the stacked roadmap here.

However, what’s cool right now is that we have a first draft of a functional token contract (think ERC-20) written completely in C#, and executing on-chain. Thanks to Rowan de Haas for putting this one together. Find the most up-to-date source in our node repository on github, or for now just marvel at this little preview:

We’re expecting to formalise the standard in the coming weeks, so yes it will have a catchy little name like “SRC-3” soon too.

I’m hoping there is someone else out there as excited about this as I am. If I find time I’d love to go into the pros and cons of our smart contract execution approach.

In the meantime, catch me on Twitter: @codingupastorm

Or join us on Discord! Find me in the #smart_contracts channel.