• Work all day and sleep all night. You can live out your roughneck dreams as a lumberjack. Available for Android. Download Now!
  • Our very first game "The Shaft" is available exclusively on Android. Try this fun and addictive game for free. Download Now!
  • The Shaft is now available in an ad-free version. With no more annoying adverts or delays between plays. Buy Now!
  • Bangy Dots - The exciting new puzzle game by Maple Scot development. Colourful and addictive match-3 game with a difference. Try it today. Download Now!
  • They say you can't buy happiness. We say, they're wrong! Happiness is now just a click away. Enjoy relaxing imagery on your smarphone or tablet any time to drift away into a space of pure delight. Buy Now!
  • Prismatoids, geometric shapes made fun. Prismatoids is a new and exciting puzzle game from Maple Scot development with three addictive game modes to unlock, each with their own challange and strategies. Be sure not to miss it. Fruit Miner mode is our favourite! Download Now!
Joomla Vorlage by Jackets

Introducing Hudson Trading Platform

Category: Financial Systems Published: Wednesday, 21 November 2018 Written by Troy Peterson

Introducing Hudson

Hudson is a new cross-asset financial trading platform from MapleScot Development. Designed to be fast, business agnostic, and extremely extensible, It represents a new paradigm in trading applications for banks, fund managers, brokers, or any other players in the industry on both buy and sell sides. The key difference between Hudson and existing platforms is it's feature and behavior based model that allows clients to develop new implementations with little or no coding. It is based on an architecture known as an Entity Component System (ECS) framework which is a model that has been used in developing video games for a number of years. This is the first application of the ECS model to financial trading applications, however it fits perfectly well and brings with it many advantages that have been enjoyed by video game developers since the inception of the ECS architecture. In essence, Hudson, is best described, a not a trading platform, but a financial software engine upon which reliable, high performance trading platforms can be built.

On this platform we are currently building a full featured Repo trading system. Ultimately Hudson will be a full cross-asset solution for enterprises wishing to unify their systems and licensing costs into one easily supportable package.

A demo system is available on the cloud at hudson.maple.scot. This is a demo release that is updated occasionally. It is usually a few features behind the current progress.

The Entity Component System Architecture

The ECS architecture has been used in the games industry since at least 1998 and virtually all of today's triple-A rated titles were developed with an engine that implements some variation of the basic ECS concepts. The design arose out of the pressures of the game industry to develop extremely complicated new games on very short timelines and budgets. It allows people with a number of different roles and backgrounds to work on their own areas of expertise and connect the pieces together without a lot of inter-dependencies, encouraging a very agile approach to development.

As the name implies there are three major concepts in an ECS implementation:

  • Entities: Represent any thing in the application. They are very abstract and are simply containers of data with a reference identifier. Entities have any number of components attached to them.
  • Components: Represent discreet pieces of data about an entity. This could be considered as a feature that is present on an entity. Each component has zero or more fields that describe it.
  • Systems: Represent the behaviors and interaction between entities. Systems handle the business logic of the application but are small and discreet units which are like building blocks that are combined together to provide complete functionality.

Since the ECS architecture was born out of video games, a clear game example best illustrates how these concepts work before we apply them to a financial trading application. Suppose we are developing a simple tennis game. The first step is to consider the entities:

  • Player 1
  • Player 2
  • The Ball
  • The Net
  • The playing court

Components represent the features (or properties) of an entity and hold data describing those features. Lets consider the components in terms of the overall features required and data that each of those features needs:

  • Size: width, height, depth
  • Shape: shapeType (sphere, rectangle, etc)
  • Mass: grams
  • Position: x, y, z (cartesian coordinates in 3d space)
  • Velocity: x, y, z (the speed in each plane, representing a vector)
  • Score: points
  • PlayerType: Human or Computer

Not every entity will have every feature. A key concept in an ECS is that Entities are abstract and can have any combination of properties. Properties can even be added and removed at any time. An entity isn't intrinsically a ball or a player by any definition in the actual programming coded, it is just given a name or identifier and a set of components that describe it. Thus the ball entity will probably have attached to it; Size, Shape, Mass, Position, and Velocity. The remaining components are not features of a ball and therefore are not added to the ball entity.

The last part to consider for our hypothetical game is the systems which represent the gameplay and link everything together. The systems are best thought of as behaviours which work together to make the game. Systems work on the components and as input take any entity which happens to have components matching it's filter criteria. Here are the basic systems:

  • Gravity: For any entity which has mass and Velocity -> increases the vertical velocity by the 9.8m/s for each second of game play.
  • Collision: For any entity which has a Shape, Size, Position, and Velocity -> Detects if any two objects have hit each other. If any objects have collided, inverts the velocity of the lighter object to change it's direction
  • Movement: For any entity which has a position and a velocity -> for each frame of the game, changes the position of the object by a small amount based on it's velocity in each direction.
  • Input: For any entities which have velocity and PlayerType of "Human" -> detects what buttons are pressed on the keyboard and adjusts the entities velocity appropriately
  • ArtificialIntelligence: For any entities that have velocity and PlayerType of "Computer" -> Analyses the current situation and Determines what the best direction to move. Adjusts the entity's velocity appropriately
  • ScoreKeeper: if the ball entity goes out of bounds, adds a point to the Score component of the player that last touched it.

Given those 6 different Systems it is easy to see that some of them require specialist knowledge (like physics or AI) and other are much more generic. However, none of them directly rely on each other, only on the components that have been defined. This means that the development of the Systems can be prioritized and assigned to the most suitable team members who can operate in parallel without waiting on eachother. The systems can be tested in isolation and plugged in to the overall game when they are ready.

This approach is also extremely resilient to new or changing requirements or enhancements as required. For example, during development the team member that is writing the ScoreKeeper realizes that he needs to keep track of the last player that touched the ball. A new Component can be defined called LastTouched which can be added to the player that last touched the ball. During each exchange of the rally this component is removed from the entity that it's assigned to and assigned to the new entity. Adding this new component doesn't impact any of the other developers.

Late in the game development one of the game designers decides that the game would be more fun if there were some obstacles and adds a requirement for birds to fly across the tennis court. Without touching any of the existing code only a single new system needs to be created:

  • BirdSpawner: Randomly creates bird entities with a shape, a position on one side of the court and a velocity that causes it to fly across the court.

Since the newly created entities have components that already exist and systems are already in place that deal with those components, once the new System is enabled then the existing Movement system will cause them to fly across the court and the existing Collision system will cause the ball to bounce off of the birds. Since no mass was added to the birds they will not be affected by the Gravity system.

It's easy to see how this system can optimize development on new games. In addition to the inherent separation of concerns and ease of making changes it also lends itself to re-use. Most of those Systems are generic and apply to many different styles of games. If you have written a good Gravity system or Movement system once they can be re-used in hundreds of games without any change. Often an entirely new game can be created by simply choosing the right pre-made systems and configuring them appropriately. In the above example, the 'rules' for playing tennis are really encoded in the ScoreKeeper system - there is a good chance that all of the other systems already exist and only the new ScoreKeeper needs to be created. Graphical tools can allow you to build new games by virtually just drag-and-drop. Many modern games take many hundreds of hours to play though and run at over a hundred frames per second with millions of entities.

How does this apply to a trading platform?

There is a lot of different data inside a trading platform, but ultimately this data is all about describing entities. Some entities in a trading system would be:

  • Books
  • Users
  • Trades
  • CashFlows
  • Securities
  • Currencies
  • Accounts
  • etc

The list of possible entities is endless. A limitation of traditional systems is that you need to cleanly define your entities and model them in a strict fashion. Everything relies on a solid definition of a model (schema) and thus your business logic is very close and specific to the entity you are modelling. You end up with specialist code for each type of entity and if there is a change in requirements or a new type of entity to consider the impacts span the whole system.

However, if entities are just collections of features (components) and you focus on what those features are, you begin to rapidly see similarities between them which leads to opportunities to create small flexible pieces that can be infinitely recombined in endless ways. Lets look at a simple case of a user in the system. Here are some possible components that a user entity might have:

  • Name: shortName, fullName
  • Login: id
  • AuditInformation: createdDate, createdBy, lastAccessedDate, modifiedDate, modifiedBy
  • Permissions: role, authorizations
  • Books: bookList

If an entity is just a collection of components then many of those components can be applied to other entities too. AuditInformation could be attached to any trade or bit of static data in the system. Name could be used on a book, account, security, or anything that requires a name. These are all composable paces that do not require any additional code to use once they are defined. Any arbitrary data can be applied to any entity. This means that you can define new entities at any time and create new types of entities (or trade products) that have never been conceived of just by considering what features you want it to have and applying those features to it. With a solid library of pre-existing components to choose from any subject matter expert can define and create new types of data for the system to manage without any involvement of developers.

Systems represent your business logic and processes. However, if broken down into base elements then it's simple configuration of filters and settings to create or alter functionality. For example, given the following components:

  • Interest Rate: rate
  • Principal: quantity
  • Term: days
  • CompoundingDetails: frequency, daysPerYear

A CompoundInterestSystem or SimpleInterestSystem could add a calculated FuturePrincipal component to the entity. By simply activating the appropriate system and configuring it's input filter to select the entities you wish it to act on, all entities that have the appropriate features will have the behaviour of Compound or Simple interest applied to them. Systems can also be responsible for sending data out of the system (for example, to downstream settlement or risk systems) or loading data into the system by reading from a source message (like a message in XML on an MessageQueue) and creating entities. Building up complicated systems becomes a matter of having a solid library of well defined and clearly scoped systems, then a subject matter expert who knows the business can define new business flows by simply choosing and configuring the appropriate systems. New systems only need to be created by developers where functionality gaps exist.

Here's an extremely high-level example of how an STP flow works:

  • A new trade comes in via XML on a message queue
  • A gateway system listening on the message queue reads the XML message and creates an entity in the platform. For each section of the XML that matches a known component, that component is added to the entity and any unrecognised data is put into a general purpose unknownData component on the entity.
  • The gateway system adds a WorkflowStatus component to the entity with a value of New
  • business logic systems are triggered by their input filters matching the properties of the new trade. A number of systems run to validate the data, and enrich it.
  • Systems derive additional details, perform calculations, make calls to external systems for enrichments like risk calculations, etc. Any validation systems that find problems add a new Errors component on to the entity.
  • When all applicable systems have been completed the WorkflowStatus component on the entity is updated to a new state indicating that it has been validated and enriched.
  • The entity is persisted to the database and the WorkflowStatus is updated to indicate that the trade is now complete.
  • Systems with filters configured to handle entities with a "complete" WorkflowStatus trigger to send the trade to downstream systems, update positions (which are simply other entities), send updates to user's blotters, etc.

For realtime trade entry the process is different but the same logic and code applies:

  • A user opens a new trade ticket in their GUI application. The server create's an empty entity.
  • The user selects what type of trade they are entering in. Based on their selection a pre-defined set of components are added. WorkflowStatus is set on the entity to indicate that the trade is unsubmitted
  • As the user enters data in their ticket screen the components for the trade are updated on the back-end. As soon as enough data is available for an enrichment or validation system to act it immediately performs it's task.
  • This could result in validation messages being set which will show up in the trader's gui or additional fields on the ticket being defaulted / populated in realtime.
  • For fields like Security, the users will enter part of a security identifier in the form. When they first enter a few characters an InstrumentSearchQuery component is created on the server and updated as they type. A System on the backend, FindInstrument, receives the InstrumentSearchQuery and adds a InstrumentMatches component to the entity for the possible matches, which are immediately reflected in the users's screen. When a user selects an instrument from the list of matches the InstrumentSearchQuery and InstrumentMatches components are removed and the InstrumentReference component is added.
  • A similar process happens for books, accounts, etc.
  • By the time the trader is finished entering the ticket, most of the validations and enrichments have already been done and reflected in real time. When the user submits the ticket the WorkflowStatus is updated to submitted and any systems that are not able to act in real-time during entry apply.
  • The trade is persisted and further systems bradcast the result or update positions as required.

How Hudson Works

The Hudson back-end is written in standard Java, currently targeting Java 8. It takes full advantage of all of the new features of Java 8 and is highly multi-threaded. The entity engine is built around the Hazelcast DataGrid, an enterprise grade in-memory cache and data processing solution. For persistence virtually any database can be used, however the current release targets only Oracle. We intend to support NoSQL databases in the future alongside traditional RDBMS databases.

Hudson was designed with reliability, supportability, and performance as top concerns with no compromises made in any area. The core data grid servers which host the cache are designed for seamless failover. All data is cached across multiple instances. We recommend at least three servers running the core datagrid for reliability. Data is organized in the cache following "Data-oriented Design" principals which encourage data locality among entities that are related.

Systems are deployed in Java host containers. It is possible to deploy a Hudson system in many different configurations with large monolithic services or in a microservice approach with each system being an independent process. All servers are designed to run in a saleable deployment and automatically adapt to either locality based or round-robin work distribution for horizontal scaling. Running instances can be configured for role-specific operation (i.e. Trade Server, Position Server, Static Data Server, etc) or as singular instances which each incorporate all of the services into one running instance.

Components are created as Java objects that are made available across the grid. They can also be defined dynamically at runtime, though this functionality is best reserved for testing.

Hudson supports rolling releases, meaning that the system never has to be completely taken offline for an update. With Multiple daemons and system containers running it is possible to bring them down individually and upgrade them without any interruption. If new components are added or changed during a release they will immediately be reflected across the datagrid.

Internally, for performance, Hudson makes extensive use of code generation. That is, at runtime, new classes are created and loaded into the running java instance to allow optimization of mappings. This is used for creating serializers and deserializers for communication with the UI, to create optimized serializers at runtime for network communication and cache storage, and for defining filters from filter strings which are compiled at runtime and loaded as Java predicates.

Hudson is designed to run either on dedicated Linux servers in house or on cloud based solutions. It can be deployed across a cluster of servers easily using Docker and Kubernetes or directly to the hardware. Our sample instance at hudson.maple.scot runs on Amazon AWS.

Entities in Hudson

Hudson follows the ECS architecture described above very closely. However there some differences in order to adapt it to our requirements. Entities are simply identifiers which are mapped to lists of components. An entity also has a domain (described below) but stores no data in itself. All data is in the components and each component is cached and persisted independently with a link to it's owner entity.

Domains

Entities in a typical game ECS are not segregated in any way, however for purposes of data locality and supportability we do separate entities into different data domains which the entities belong to. Any entity an belong to any domain but must be in only one at any time. It is not possible to directly move an entity from one domain to another because they are treated as discreet containers but entites may be copied from one domain to another. Domains can be configured with different parameters as appropriate for the entities that they hold. For example, cache eviction rules and persistence rules vary between the domains. These, like everything in Hudson, can be configured. Domain cache storage can be linked to specific instances (i.e. StaticDataDaemon, caching only static data), or spread across multiple generic daemons, which is the standard approach. Here are some typical domains:

  • System: Storage of runtime data and sessions. Entities in here show the health state of the overall system and are used to track the current status of users and operations. The system domain is configered not to persist any data to the database
  • StaticData: All entities related to static data. Entities are persisted, and versioned with full audit history.
  • StaticDataBooking: An un-persisted temporary holding area for StaticData entities that are being mutated.
  • Insturments: A persisted store of Instrument (security) data. Because of the importance of intsrument data it gets it's own domain.
  • MarketData: A persisted domain for storing of data that is expected to change frequently (i.e. prices).
  • Trades: Trades and cashflows that are part of trades. Entities in the Trades domaine are persisted, and versioned with full audit history.
  • TradeBooking: An un-persisted temporary holding area for trade entities that are being mutated.
  • Positions: Position ladders. Some entities in this domain are persisted (SOD / EOD position snapshots) while intra-day positions are not persisted because they can be re-calculated.

Typically, for data that can be edited online in the GUI or via incoming messages there is a concept of a "Booking" area (StaticDataBooking, TradeBooking). The purpose of this is to hold a current version of an entity while it is being changed without persisting every change to the database. For example, if the user creates a new ticket in the UI their ticket screen will appear, at the same time a new entity will be created in the TradeBooking domain. As changes are made in the UI they are immediately reflected on this entity. Systems run to enrich the entity in this domain. When the users submits a trade then it is moved from the booking domain to the Trades domain where it is persisted. This approach has numerous benefits. For one, a trader may close a trade without losing it. They could start to create a trade ticket, close it, but then re-open it and continue where they left off. Although this area is not persisted, the trade will be available for editing as long as at least one instance of the back-end daemon servers are running (we recommend at least three or more for reliability). Since failover is automatic, if one of the daemon servers fails the data will available from another instance and will still be available to the user, although performance may noticeably reduce in this case due to latency.

Another advantage of the "Booking" domain approach is that if a user's front end is accidentally closed their work is not lost. It is also possible for support teams to see the trade booking data in realtime as it is entered and assist if needed.

Families

Another important concept in Hudson is that of families. A family is a grouping of entities based on similar attributes. It is defined by a filter that qualifies whether any given entity belongs to the family or not. This is how the systems in Hudson subscribe to entities. When a system comes online it submits a family definition to the daemon along with the events that it responds to (i.e. New Entity, Updated Entity, Deleted Entity, etc). The deamon returns all entities matching this criteria. At any time it is possible to define a family and query the system for all entities belonging to that family. The family is the key means by which entities are distinguished and identified as being, for example, a book, or a user, or a trade. The family represents a definition on what qualifies as a trade and thus when a request for that family is sent to the server it returns all trades back. Although families are not end-user facing concepts, it is important to understand how they work for anyone administering they system.

Workflows

Although the standard ECS approach is very powerful and effective, it does present a few issues with a typical trading application. Systems may be interdependent on each-other. It is possible that a system is unable to run until it's prerequisites (defined in it's family filter) have been satisfied by another family. This is somewhat indeterminate since all qualifying families run in parallel it can be difficult to understand the relationship between them and it may require defining a lot of status flag components to track which systems have run and which have not. To resolve this we have introduced the concept of a workflow. A workflow is really just a System itself that has a defined set of child systems. A workflow definition indicates, via a family definition, what condition starts it to run and it will then execute it's child systems in the defined order. Child systems can be run both sequentially or in parallel or combinations of both. I.e. it is possible for a workflow to define a sequence of systems and then a set that run in parallel at some point and more sequential ones after that. This looks like a directed graph. As part of the workflow it possible to define what happens if any of the child systems encounter problems. A complete status of the workflow processing shows not only what systems ran as part of the workflow, but also how long they took to execute. This workflow status is attached to the entity to provide a detailed flow of operations for support purposes.

Workflows are defined via configuration which is currently in the form of configuration files, however a UI will be developed for creating and editing workflows with a graphical flow-chart like view. These can be loaded into the system immediately (in development) and then exported for loading as part of the release process on a production system.

The Hudson User Interface

The user interface for Hudson is based on HTML5. It is a 'thin' client approach with no persistence on the user side, all data is stored entirely on the server. Communication with the server is via a combination of secure REST webservices and realtime Websocket connections. The UI itself operates directly on entities in Hudson and works in many ways just like other systems. Therefore it can subscribe to events based on a family definition to receieve immediate realtime updates of anything. Though the details of how this works is hidden from the end user, it allows for some unique advantages. For example, a ueser's authorities can be changed on the fly and reflected immediately. All aspects of the user's profile are data that is provided by the server, this even includes the menu options that they see. It is possible to change the rights or profile of a user on the server and immediately that they see in the front end will be updated. This could include new menu options appearing. This functionality also allows a unique administrative capability where all administrators can be setup as read-only users with the ability to execute a tracked and audited "break-glass" process to temporarily elevate their rights when needed to perform tasks.

We also have the ability to broadcast notifications to users from the server. Administrators can send messages to every currently logged in user or to specific teams to notify them of production issues or expected service interruptions.

Another feature in Hudson that is not commonly found in web applications is the ability to move windows around and dock them in the client area in any way that they desire. By default screens are inserted into a new tab, but it is possible drag the tab out and place it in a split screen view so two screens can be viewed simultaneously. It is even possible to "pop-out" tabs into floating windows that can be moved to different monitors and then returned back into the main application.

One drawback to the use of HTML5 is that current web browsers do not allow windows to hide the navigation bar anymore. This is a security feature so that users can always see that they are at a trusted web address and not being hijacked. However, in a trusted internal environment on a productivity focused application like Hudson this bar reduces valuable real-estate on the screen. So, as an alternative to traditional browsers, Hudson can instead be run inside of an Electron container. Many popular modern apps are developed this way. By placing Hudson inside an Electron container it looks and behaves like a traditional desktop application and can be deployed directly to users desktops following any corporate release procedure.

Current Status

Hudson is under active development. The back end server functionality for the engine is complete with all of the above features. At the moment the business logic for building a functional Repo trading system is in progress. The UI development is still somewhat in it's infancy compared to that of the server but it's roadmap is clear and it is moving forward consistently. The design of Hudson presents a lot of advantages over traditional systems and it can be had at a much lower total cost. The key objective is to be able to quickly adapt to the changing business environment and market as new regulations come in or market norms evolve as well as provide the ability to consolidate a large number of different systems and different business lines under a single well supported, product to reduce cost, complexity, and risk. Agility and adaptability is fundamental to the product, without sacrificing reliability and performance. We can provide analysis and consulting services and operate under many potential models of engagement for development, rollout, and ongoing support.

The Name

Hudson is named after the 16th century explorer Henry Hudson who explored much of the east coast of Canada and the United States. There are many landmarks named after him, including Hudson Bay in northern Canada and the Hudson River in New York.

Hits: 140