Data Acquisition - Redback x IBM Cloud
It all begins with an idea.
Redback Racing Blog 2022
Redback Cloud is the system that connects onboard data logger, weather station and simulators together to form one cohesive motorsport cloud platform. Redback Racing team chose to go with a microservices architecture and typescript for both the frontend and backend by using React.js and Node.js.
This microservices architecture where the backend is built with lots of small servers for independent features was chosen because:
It simplifies the New Member Onboarding process on a rapidly growing Redback software team. Instead of a monolithic application with one huge codebase, each feature team works with a small codebase for their own microservice and UI which is much easier to manage.
It allows for features to be scaled independently based on the demand for them through a microservices architecture. For instance, we can provide more processing power to the data searching service for faster searches and more network bandwidth to the live streaming service for less latency.
The Data Acquisition team chose typescript for the programming language on both the backend and UI for the following reasons:
The client libraries for all the microservices can be shared among the frontend and backend meaning less duplicated code written in different languages.
The same data structures used in the client libraries can also be used in the servers. This means that all data passed between the client and server is type-checked at compile time. Any bugs in the data structures passed to the server will be picked up in the automated code pipelines and will be blocked from going into production.
Redback Cloud is deployed as a containerised application in Kubernetes, a powerful tool for orchestrating all the servers which make up a microservices system. Kubernetes is a free and open source, thus it can be run both at the track for engineers to access locally, and in the cloud for engineers working remotely. Our system allows for autoscaling, as it has been designed to support multiple replicas of each of our microservices running at the same time. Kubernetes will automatically start up more instances of a service if it is in high demand and load balance the traffic across them. This makes Redback Cloud a very robust application capable of handling massive amounts of traffic.
The DAQ team is currently working with a distributed systems thesis student and academic to achieve better scalability and fault tolerance for all subsystems in Redback Cloud. These qualities will be necessary to achieve our goal of providing a motorsport SaaS platform for other teams in the future.
Built on IBM Cloud
DAQ chooses IBM Cloud as their public cloud provider for building, deploying and managing all Redback Cloud infrastructure. Partnering with IBM Cloud as a top-tier Red sponsor with the team, Redback Cloud Engineers leverage IBM’s vast range of managed service offerings to improve the fault tolerance, reliability and scalability of Redback Cloud.
Deploying the microservices stack to two separate fully managed Kubernetes clusters on IBM Kubernetes Service, allows DAQ to ensure 100% system up-time by staging and testing new changes on a development cluster, then deploying to a stable production cluster with nodes in multiple IBM Cloud availability zones for maximum fault tolerance.
A wide range of managed service offerings from IaaS (infrastructure-as-a-service) to PaaS (platform-as-a-service) are leveraged to abstract away the provisioning and management of compute, networking and storage resources that Redback Cloud relies on, allowing DAQ team members to spend less time on managing systems and more time on coding new features. To power it’s systems, Redback Cloud makes use of:
IBM Cloud Databases for Postgres for fully managed Postgres Databases
IBM Cloud Object Storage for fully managed s3 compatible simple storage
IBM Cloud Databases for Elasticsearch for fully managed Elasticsearch Databases
IBM Kubernetes Service for fully managed Kubernetes
IBM VPC Infrastructure for faster virtual server instance provisioning, auto scaling and higher network bandwidth
to name a few.
Track Builder
The Redback Cloud track builder is where the team can graphically build track maps and create matrices that translate GPS, Assetto Corsa and lap simulator coordinates onto the map image. We also use the track builder to make tracks for the lap simulator engine itself. All tracks made with a defined racing line are available on the point mass simulator track selection page.
Garage
The Redback Cloud Garage is where we manage all of our team’s vehicles and the data sources used for those vehicles. Cars can be created and each of the data sources can be configured for them. For example, the pipeline of modules in the data logging firmware and settings for each of those modules can be defined.
A fundamental part of track days - car setup sheets are designed and maintained in the Redback Cloud Garage. Currently, the team uses spreadsheets to record the setup of the car for each track session, which becomes tedious to fill and search for. Bringing the setup sheets onto Redback Cloud will allow for the automatic pairing of a setup sheet with each sensor log file, saving us time. Furthermore, the setup sheets will be readily accessible when analysing log files, eliminating the need for unnecessary searching for information within different systems.
We will first be able to use the UI to define fields representing all of the adjustable components on the current year’s car, such as rear wing angle or throttle map. Once this is done, we will be able to populate the generated form with the car setup and store customised presets in the garage. For example, the best setup for each dynamic event at comp can be stored in the garage. Making the car setup sheet user-defined allows us to customise them year to year as the car design changes without changing the code.
Track Days
The Redback Cloud Track Days feature is for planning track days and cataloguing data at them. Firstly we can create track day plans which include the track and a list of car setup sheets we plan to run through on the day. These setup sheets can be built based on the preset sheets stored in the garage for the particular track.
During track days we can control the data logger on the vehicles and driving simulator to catalogue the data logged on the car with the setup sheets from the track day plan on the fly. This is done by selecting a setup from the track day plan and then sending a message to the car to tell the data logger to start logging with a reference to the setup sheet. In addition to the car setup sheet, we also catalogue the data by the driver, the date and time, information from the weather station, notes and more.
After each session on the track, we can then download all of the log files of the car into the cloud over the WebSocket connection. The reason we do this and do not log the live telemetry data on the server is because the live data is downsampled and will be lost if the network connection goes down.
Once all the data is downloaded from the car it is sent to our data storage service where it is processed, catalogued and stored in a way that it can be searched later.
Live Telemetry
The Redback Cloud Live Telemetry provides visualisation of live data from the vehicles and driving simulator on a variety of widgets. The user interface allows us to create custom tabs each with configurable dashboards made up of a series of widgets including:
Charts
G-Force Target
Gyro Dials
Steering Position
Track Map
Video
Lap Times
Live Values (Colour coded based on value ranges to highlight important readings)
The charts widget was built in-house using D3 and Canvas, which will allow us to implement custom features in the future such as:
Viewing lap by lap charts
Overlaying different laps (such as current and best lap)
Interactive markers etc.
In 2020, live telemetry was the primary focus of Redback Cloud as it allows the engineers to make sure all systems are operating correctly and allows them to react to any problems immediately. In prior years, data logging and analysis have been done through the MOTEC ECU and i2 systems but the team had no solution for live telemetry. In 2021, we aim to expand the functionality of our live telemetry to make remote participation in track days a reality and grow the amount of engagement in track days within the team.
To do this, we have developed a video streaming system based on WebRTC to provide a low latency onboard visual to engineers following the data remotely. We also plan to extend the radio communications at the track to the cloud and web browser which will allow the engineers working remotely to listen into the pit wall and driver. We will also be able to communicate through the browser to the pit wall and other engineers working remotely using this technology.
Data Analysis
The Redback Cloud Data Analysis feature allows users to search through the database of catalogued data from track days and the simulators. The advanced search feature allows very specific search criteria to find track days or log files that meet specific requirements. Possible examples of such requirements are:
The track temperature being in a certain range
The driver being person X
The car was set up with a cambre angles Y and Z etc.
We can then choose to view the data from the selected log file on the same dashboards as the live telemetry feature or compare data from multiple log files.
Future developments for the data analysis system include adding comments threads to each log file so notes can be kept in one location. This concept can be further developed by adding timestamp markers to the charts which can be linked to comments, allowing for users to tag others in the team, and so on.
Insights
Redback Cloud Insights provide a series of dashboards to provide engineers with more context in how the team and car are performing such as:
Summaries of track days
Performance of cars from year to year
Insights source data from all the other Redback Cloud features and organise them into infographics.