On more than one occasion, people have told me that trading venues are built around coordination because we—as an industry—don’t know how to design modern systems for the cloud. Coordination enables autonomous and loosely coupled machines to jointly determine how to control basic behaviors, including the order of access to shared memory – such as that found in the internal state of a trading venue. Coordination is expensive, as system components cannot proceed independently when changing any shared state. Typical coordination approaches include approaches such as ACID compliant databases, distributed consensus, and distributed ledgers.

To answer the question posed in the title: we can’t build resilient systems without coordination using current technologies and still comply with the business and (sometimes) legal requirements. I will touch on a few aspects as to why this is the case from the following viewpoints:

• The legal framework under which some trading models operate;
• How trading models work on several common venue types;
• A look at the current state of the art research around avoiding coordination;
• A look at how coordination is done on the cloud with many real-world trading systems across multiple asset classes.

Legal Framework

A large body of rules and laws are in place that regulated trading venues must comply with to qualify for and retain their trading licenses. Each trading jurisdiction has its own specific set of rules. At times, these rules apply strict limits on what is allowed. Here’s one example from SEC Rule 15c3-5:

Specifically, the risk management controls and supervisory procedures would be required to be reasonably designed to:

• prevent the entry of orders that exceed appropriate pre-set credit or capital thresholds, or that appear to be erroneous;
• prevent the entry of orders unless there has been compliance with all regulatory requirements that must be satisfied on a pre-order entry basis; and
• prevent the entry of orders that the broker-dealer or customer is restricted from trading, restrict market access technology and systems to authorized persons, and assure appropriate surveillance personnel receive immediate post-trade execution reports.

Often, very similar requirements exist at the business level in unregulated asset classes (for example, Crypto) as well. An architecture that relies on coordination can guarantee that the entry of orders that exceed appropriate pre-set credit or capital limits is impossible under normal operating conditions. Uncoordinated architectures cannot meet these conditions with such strong guarantees.

Common Trading Models

Continuous Trading

Continuous Trading Models, like a central limit order book, continuously accept both buy and sell orders. When a participant submits a new order, it is placed into a memory-based order book. The system then tries to find a buyer or seller to match with. If it finds a match, a trade is generated. To decide which orders get matched first, these systems use a set of rules, often prioritizing order matching firstly on price and then on the time the order was placed.

Being the first to submit an order at a specific price point has clear advantages in a trading system of this type. This underscores the importance of the order in which orders are accepted into the order book. As these matches occur, financial transactions are created. These transactions can range from transferring ownership, creating future contract agreements, buying debt securities, or swapping one asset for another. Clearly, all parties to these transactions have a very strong interest in the correctness of the transaction details. Moreover, regulators will at times request details on the exact sequence of events that led to the trading outcomes observed.

This requirement for strictly ordered processing in Continuous Trading Models makes them ill-suited for systems without a coordinated architecture.

Periodic Auction

Periodic auctions are more prevalent in European trading venues. Unlike the continuous matching of buy and sell orders, these auctions gather orders in short, discrete intervals or batches. Orders accumulate until a set time is reached when the matching occurs. The length of these auction periods is often randomized to prevent strategic manipulation. The frequency of the periodic auctions tends to vary from multiple times a second to every few seconds.

In these trading models, the exact time an order reaches the order book might be less critical, but whether an order is placed within a specific auction period’s order book can be vital. Orders that arrive after one particular auction period has ended are either placed into the next periodic auction or are rejected. Like continuous trading models, periodic auctions must depend on a certain level of coordination to function correctly.

Negotiations

Negotiation-based trading models, such as RFQ (request for quote), are built upon a strict state machine, with permitted interactions determined by the current state of the negotiation. Clearly, a negotiation could not be safely operated without coordination.

Auctions

Some trading models that are modeled as auctions make use of a much longer, fixed-duration auction. In these models, counterparts will submit bids for a duration, and only once the auction end time is reached is the outcome decided. The auction is often decided using a mathematical or rule-based model, which typically has aims more aligned with fairness than the first-come, first-served continuous trading model.

Coordination Avoidance Research

One key research area that revolves around the removal of coordination while retaining consistency in distributed systems relates to the CALM theorem. The CALM theorem, or Consistency As Logical Monotonicity, shows that distributed systems can achieve consistency if the system is monotonic. A program is monotonic if its outcomes do not change when new information becomes available.

I find the monotonic concept a bit complex to understand without examples. Let’s consider a system with an internal state representing an order book. A system that exclusively performs queries on its internal state such as “Have any orders been submitted to the order book?” is monotonic since it will stay consistent across nodes, regardless of how many reads are performed as soon as the value has flipped to true. Either some orders have been submitted, or none have. A non-monotonic system would perform queries such as “what is the average of all bids”, or “list the orders in the order book in the order submitted”. Different nodes, when continuously queried, would provide inconsistent results for these queries without coordination.

The AWS Dynamo paper describes a key concept in that system designers can elect to solve conflicts on write or read. In traditional systems, conflicts are solved on write, while Dynamo moves this conflict resolution to the application layer during the read. This enables the service to operate with very high availability, and if configured to allow it, will still accept operations during network partitions. In the paper, the authors discuss modeling a shopping cart as a Dynamo hosted object. All add and removal operations are written as insertions to the object, and each change creates a new version. Should a network partition happen, then multiple versions of an object may be written. Upon read, multiple versions of the shopping cart object (which could be a forked graph, not necessarily a linear history) must be consumed by the shopping cart system and the final results computed. Removals can be lost during this computation.

This shopping cart design shares traits with Conflict-Free Replicated Data Types (CRDTs), which embody the CALM principles. CRDTs are a rare example in real-world systems where coordination is minimized while maintaining a level of consistency. Although Dynamo, doesn’t implement CRDT technology, this shopping cart design resembles a two-phase set CRDT in certain aspects. The behavior of resolving conflicts in the read model shows a common usage pattern of these approaches: they follow monotonic behavior initially, until a critical juncture in an operation, at which point the system switches to a partially or fully coordinated mode. This switch reduces errors, but doesn’t eliminate them entirely.

And this brings us to the core problem with CRDTs. They allow writes and updates even during severe failure conditions, but reading their state is not guaranteed to be consistent. When you read the state of a CRDT, replicas are not required to coordinate, and as a result, you can get inconsistent results across replicas if they have not yet fully converged. Let’s consider the CRDT shopping cart again. If you had, for example, added a chocolate bar and a television into your cart. Then, you changed your mind and removed the television and continued with the checkout. Because multiple node replicas hold the data, and they do not coordinate, then it is entirely possible that the removal operation is missing from at least one replica. As a result, the television could remain in the shopping cart, and you’d be surprised to find a television in your basket during checkout. This is known as the Early Read problem.

There seem to be only two ways to solve Early Reads. The first is to coordinate the data across replicas via some explicit protocol, and the second is to include the number of expected state updates for a consistent read and have the system wait until this number of state updates has been reached. Both of these go against the idea of building systems coordination-free, in my view.

It should be clear that the types of queries required for the trading models described earlier are not monotonic, and cannot safely fit into models such as CRDTs. As a result, the CALM theorem shows we cannot safely build consistent distributed systems when building trading systems without coordination.

Coordination in Practice

A trading venue will have several areas where highly contended and strictly consistent shared state is required.

We’ve already discussed two of them: credit limits and order books. We can build high-performance, consistent trading venues by pushing this shared state into replicated state machines and then using some form of coordination to access this state safely.

In cloud-based trading venues, I’ve come across two technical approaches to provide this coordination:

• Rely on an external sequencer of some kind to apply a coordinated order of commands to submit to the replicated state machines. A concrete example of this includes Apache Kafka 00The higher and less predictable latency tends to limit the asset classes that this design approach can be used with. – the Kafka broker acts as the coordinator, applying order across multiple inputs. These venues tend to have a higher, less predictable latency but can more easily be sharded via multiple partitions.
• Apply distributed consensus, with the shared state residing within a state machine in a container running Raft or similar. A concrete example of this includes the Aeron Cluster, which can achieve low latency (sub-millisecond) and reasonably high degrees of throughput 00 Approximately 625 billion debit and credit card network transactions were performed globally in 2022. A single three-node Aeron Cluster on the cloud could coordinate 625 billion commands in under 4 days. (millions of commands per second)
  on the cloud. Although this is an application-level concern, sharding is also possible (and often done) with Aeron Cluster.

In a discussion held in 2008, staff from several cloud vendors had several variations of this to say about consistency mechanisms (or, in other words, coordination):

The first principle of successful scalability is to batter the consistency mechanisms down to a minimum, move them off the critical path, hide them in a rarely visited corner of the system, and then make it as hard as possible for application developers to get permission to use them

While this probably remains true for the scale of systems within cloud vendors, the high performance offered by Aeron Cluster removes coordination as a key source of scalability bottleneck in trading system development. We can build systems centered around coordination, with coordination in the critical path, and have the application developers directly build business logic within the coordinated container.

Aeron Cluster is exciting in the financial trading world. It solves throughput and resilience concerns on the cloud using coordination and offers trading system designers the strong guarantees needed.