Show HN: Structured Cooperation – A new way of building distributed apps & POC
Show HN (score: 10)Description
I wanted to share something I've been working on for the past couple of months, which may be interesting to developers interacting with distributed architectures (e.g., microservices).
I'm a backend developer, and in my 9-5 job last year, we started building a distributed app - by that, I mean two or more services communicating via some sort of messaging system, like Kafka. This was my first foray into distributed systems. Having been exposed to structured concurrency by Nathan J. Smith's beautiful article on the subject (https://vorpus.org/blog/notes-on-structured-concurrency-or-g...), I started noticing the similarities between the challenges of this message-based communication, and that of concurrent programming, and GOTO-based programming before that - actions at a distance, non-trivial tracing of failures, synchronization issues, etc. I started suspecting that if the symptoms were similar, maybe the root cause, and therefore the solution, could be as well.
This led me to design something I'm calling "structured cooperation", which is basically what you get when you apply the principles of structured concurrency to distributed systems. It's something like a "protocol", in the sense that it's basically a set of rules, and not tied to any particular language or framework. As it turns out, obeying those rules has some pretty powerful consequences, including:
- Pretty much eliminates race conditions caused by eventual consistency
- Allows you to recover something resembling distributed exceptions - stack traces and the equivalent of stack unwinding, but across service boundaries
- Makes it much easier to reason about the system as a whole
I put together three articles that explain:
1) what structured cooperation is (https://developer.porn/posts/introducing-structured-cooperat...),
2) one way you could implement it (https://developer.porn/posts/implementing-structured-coopera...), and
3) why it works (https://developer.porn/posts/framing-structured-cooperation/).
I also put together a heavily documented POC implementation in Kotlin, called Scoop (linked in the title). I guess you could call it an orchestration library, similar to e.g. Temporal (https://temporal.io/), although I want to stress that it's just a POC, and not meant for production use.
I was hoping to bounce this idea off the community and see what people think. If it turns out to be a useful way of doing things, I'd try and drive the implementation of something similar in existing libraries (e.g. the aforementioned Temporal, Axon (https://www.axoniq.io/products/axon-framework), etc. - let me know if you know of others where this would make sense). As I mention in the articles, due to the heterogeneous nature of the technological landscape, I'm not sure it's a good idea to actually try to build a library, in the same way as it wouldn't make sense to do a "structured concurrency library", since there are many ways that "concurrency" is implemented. Rather, I tried to build something like a "reference implementation" that other people can use as a stepping stone to build their own implementations.
Above and beyond that, I think that this has educational value as well, and I did my best to make everything as understandable as possible. Some things I think are interesting:
- Implementation of distributed coroutines on top of Postgres
- Has both reactive and blocking implementation, so can be used as a learning resource for people new to reactive
- I documented various interesting issues that arise when you use Postgres as an MQ (see, in particular, https://github.com/gabrielshanahan/scoop/blob/09db323bf6c8a7... and https://github.com/gabrielshanahan/scoop/blob/09db323bf6c8a7...)
Let me know what you think.
More from Show
Show HN: I built a dream interpreter in JavaScript, no AI, no server, just logic
Show HN: I built a dream interpreter in JavaScript, no AI, no server, just logic I built a browser-only dream interpreter that maps symbols to meaning using a custom keyword engine. No AI, no tracking, no backend it’s all deterministic logic, built with vanilla JavaScript. The idea was to see if emotional and symbolic interpretations could still feel personal without using machine learning. Dreams are parsed against a map of 300+ symbolic meanings. Everything runs locally in your browser and saves via localStorage. You can revisit old dreams, reflect on interpretations, or just explore what the system sees in your subconscious. Happy to answer questions, would love feedback!
Show HN: Integrated System for Enhancing VIC Output
Show HN: Integrated System for Enhancing VIC Output ISEVIC stands for Integrated System for Enhancing VIC output and is a cartridge for the C64 that uses the Tang Nano 20K FPGA to monitor the bus and recreate the C64 video for HDMI output. While a tremendous effort has gone into handling all the clever C64 tricks, this should be considered a 1.0 release as there are still some demos and games that exhibit bugs. Everything needed to build one is on the github page.
No other tools from this source yet.