Over the past several years we have integrated with dozens of exchanges (35 at last count). Along the way we have seen, experienced, and suffered many different things related to crypto exchange APIs. In the spirit of collaborating with the broader crypto community, and with the (slightly selfish) hope of making our future integrations easier, we thought we would share some thoughts on what an ideal trading API should look like.
The reason for this proposal is the wide variety of API implementations out there, where most APIs are repeating the same mistakes. Mistakes that complicate their usage in an institutional environment. What we propose are a few simple changes to existing API models to build what we think is a robust and institutional grade API for crypto trading.
Flavours of APIs
There are a number of flavours of APIs available for automated trading across the crypto exchange universe. From a technological point of view there APIs fall into:
1. REST: all market data and trading is over REST endpoints
2. Websocket: all market data and trading is over Websockets
3. FIX: Pure FIX implementations are rare however these styles of trading APIs are gaining momentum in the crypto space. (Deribit have a pure FIX implementation which also handles account updates)
4. Combinations
- Hybrid, REST for private endpoints, Websocket for market data
- Hybrid-2.0, REST for private endpoints, Websocket for market data and private data (order updates, fills etc.)
- FIX + combination of above
Pure REST
With this style of API, typically all interactions are poll based. Needless to say, this is incredibly limited, primarily by arbitrary rate limits (which are required to scale.) This style of API does not sit well with a full scale institutional grade trading system. The requirement to poll for updates (whether market data or our own executions) is simply debilitating.
Pure Websocket
There are several good examples of pure websocket apis coins.ph (pro) has a very well defined API for pure websocket trading, as does HitBTC. Bitfinex also have a good (on paper) v2, but the current beta implementation suffers from instability. For exchanges looking to implement a good websocket based API, a good model to follow is the coins API.
Hybrid
There are numerous examples of exchanges which implement a hybrid trading API. Usually either only market data or market data and some private data are available over a websocket, whilst trading (the process of creating an order or cancelling) is via REST endpoints. Okex, Kucoin, Binance etc are examples of this type of trading API.
FIX
These styles of trading APIs are gaining momentum in the crypto space. More and more new exchanges offer this (and existing exchanges are also starting to support.) This style of API is great for institutional trading, however there are some limitations. The primary being FIX as a standard cannot address all the requirements we have in crypto (for example instantiate transactions to withdraw coins.) Other missing elements are critical information such as account balances — there simply is no matching equivalent in FIX.
One could argue that these could be implemented via the new position maintenance messages in the latest FIX protocol standards — but this feels like a hack. Until the protocol is extended to support additional messages required for this, we will find these APIs being a hybrid of FIX for trading and market data and REST (or Websocket) for other account related activity. It’s this requirement to implement lots of different protocols to accomplish full trading functionality which leads us to say that FIX works only in limited cases (trading on some specific exchanges like LMAX, LXDX, or Blockfills.)
What is lacking with Crypto APIs?
So far what has been discussed should not come as a surprise if one spent some time analysing the landscape. When we built our trading system, the first phase (and it continues to be one of the largest drains of our resources) is the connectivity to a large number of exchanges. As our models can trade across any number of exchanges and we can have multiple models trading across a given exchange, the critical functionality we require is robust order and execution handling.
Robust order handling means we need to be aware of the state of our orders at all times, and robust execution handling means we need to know all our executions (and be able to allocate these executions to the models that generated the orders.) This last step is critical to allow us to manage risk at the model level.
Client Order ID
The first gap we see in the majority of APIs (even of exchanges claiming to be institutional grade) is that very few support the concept of a client order id. A client order id is our internal id that we allocate to the order in our order management system. This key allows us to identify the model that generated the particular order, and this then becomes the key to allocate any executions on that order to that model.
Where the APIs fail is that we cannot set this client order id. As a result, we make a REST call to create an order, and the response gives us the exchange order id — but for some reason, if the HTTP request fails (for example there is a service provider such as Cloudflare in the way — there could be several points of failure) we have no way to identify orders that we detect that are live on the exchange if somehow the order made it through to the matching engine. Worse still, sometimes these orders then trade and we receive executions — which we cannot allocate (difficult to tell which model traded if we have multiple models trading!)
With hybrid APIs this problem is exacerbated as the order updates pushed out via the websocket does not have the client order id, only the exchange order id and as a result, we have to buffer everything till the REST request returns to understand what we received on the websocket. We think the simple addition of a client order id (for create, change, cancel operations, whether it be Websocket or REST) and the subsequent inclusion of that field in the output (order updates and executions) vastly improves the robustness of the API — as we can handle all messages from the exchange and deal purely with the client order id that we set. We can even distinguish manual orders on the exchange specific UI from automated orders and handle them appropriately.
Every Execution on an Order
Some of the older generation of APIs did not provide fill endpoints, one polled the order status to detect fills on any orders and compute partial fills (synthetically) by looking at the executed quantity on an order. This doesn’t really work well, and later when we need to reconcile the exchange view of our trading with ours, the process is complicated as individual trades are not listed — often an aggregation at the order level.
The recent generation of APIs provide fill endpoints, but have introduced fairly limiting constraints such as the order id must be provided to obtain fills (for example Okex.) This is a debilitating requirement as a typical institutional grade trading system will be generating many orders across many models, and having to poll each individual order for fills is a waste of time. The endpoint which returns our fills can filter by some criteria such as symbol and perhaps last trade id, and using these criteria it should return all our partial fills across all our orders.
Batch Operations
As a crypto market maker, we are often required to post a deep order book on both the bid and offer. As a result, having the ability to send a batch of orders (via websocket, see for example Bitfinex) or via REST (see for example Okex) reduces load on our order processing and one would imagine on the gateways of the exchanges. Typically we would like to see a batch create and batch cancel endpoint such that we can in a single operation move our orders without having to make individual requests for each order action. For example, if we are required to post three bids and three offers, every time the price moves, we will send six cancels and six creates. If the API has batch support, this would be two requests, and if the API supported order replace — one request with six changes. The load on exchanges would reduce drastically.
Cancel on Disconnect and an Option to Cancel all on Demand
For any connected API (websocket etc.), this is a feature that is very useful in general. When this is enabled, if we disconnect from the session, any open orders should be cancelled. This affords us some protection in the event that we are out of the market and will prevent any orders from being lifted while we are away. Ideally if there is a private channel to which we are connected, it should support this feature.
So What Does an Ideal API Look Like?
From our experience (unless it’s a very specific niche case where FIX is relevant) the most ideal API is a Hybrid API. We think this has the least implementation overhead for exchanges and the simplest model for us to handle. However for this to work, we need the following:
- REST for create / update / cancel
- Websocket for updates and executions
- The REST endpoints for create order must take a client order id (and ideally be batched)
- There should be a REST endpoint for order modification, this adds complexity to matching engines, and could be obviated by having batch endpoints.
- The REST endpoints for cancel should be batched
- All order updates and individual fills should be sent over a private websocket channel, and these updates should contain the client order id. Without the client order id, any allocation we do will rely purely on the exchange order id (which is only available on the response to the create operation.) Hence this introduces a race on our side which can be removed by exposing the client order id in the updates.
- Last but not least, good documentation and a test-net
Full websocket implementation is also great to work with, however to implement a full solution there perhaps is technically more challenging and as a result, we would refrain from making this model the most ideal type of API.
As you can see, these are not major requirements or controversial, just a few simple guidelines, which if satisfied, should result in a fairly robust API for institutional traders such as ourselves. What is surprising and controversial is how few APIs in crypto have this.