Asset Management and Streaming Services
With a specific goal in mind, Octopus Media, then OctopusTV, tasked me with designing and developing a system that could achieve the following:
- Manage the storage of assets and their associated metadata
- Deliver content in a variety of ways from downloading to streaming
- Support for uploading large assets
- Manage authentication and authorisation on a variety of levels from individual to company
- Include a payment model where a user or company could by space and/ or processing time
The end result was a WCF based solution consisting of several services, where each service could operate in one of two modes - primary/ secondary or peer to peer.
Services
- The Registrar - this was the connection point for clients and services. All services would register their various end-points here allowing the client to choose which service and what end point (usually HTTP or TCP) to use.
- The Asset manager - used to upload/ download and manage the metadata of assets. This service had a custom TCP based chunking channel, allowing users to resume aborted uploads and downloads.
- Job manager - used to manage long running jobs for users
- Job running - this was a peer services that could be run on multiple machines and used to run jobs such as transcoding videos or audio.
Primary/ secondary
Each service of this type would operate with two instances, ideally on two separate machines - one, the primary, would be the single point of contact for that specific function. If that service was un-contactable - due to an error or through software, or hardware the secondary would take up the responsibility of the first. Over a configurable period of time or operation, the primary would push its state to the secondary.
Peer-to-peer
The goal of the peer-to-peer service types were to provide scalable processing for operations such as transcoding, each service was capable of determining how busy the parent machine was which was used in the service selection process for the client - part of a global load balancer for each service type.
