Why Crystal? The Need for Open SDS
SDS is changing the way storage is managed. Lead SDS systems such as IOFlow has enabled innovation in the control plane than was not feasible on closed storage systems. However, the path to an open SDS is strewn with impediments: Lack of transparency, poor programmability and poor extensibility.
Lack of transparency
Transparency is a very important requirement. To be a true open architecture, a SDS system should have the ability to inspect and change the state of storage in a transparent manner. This means that system should hide the details of the storage stack and provide simple interfaces to inspect or change the functionality of the system. The appeal here is to open SDS for reflection and provide extensible introspection and interception hooks. Opening the underlying system implementation to reflection may not be a trivial task and require a significant effort. But it is a necessary thing to prevent the addition of new features from causing substantial changes in the storage architecture every time. We were able to incorporate reflection into Crystal by tapping into the standard interception layer of OpenStack Swift.
Poor programmability
Although control plane programmable abstractions are powerful in some SDS systems, rich data plane programmability is also fundamental to lower the barrier to deploying new services. And ideally, it should allow the execution of arbitrary computations and transformations on workflows, which is not possible in most of the existing SDS systems. At the control plane, the SDS system should provide a flexible meta-programming model, in order to open the system to new cross-cutting concerns not anticipated in the initial design. To better understand this, the popular IFTTT (If This Then That) service1 constitutes a perfect analogy to the “ideal” model for storage administrators. This service allows users to express small rule-based programs, called “recipes” using triggers and actions. Triggers are the “this” part of a recipe. They are the events (like being tagged in Facebook) that trigger the action. Finally, actions are the “that” part of a recipe that must be performed when the condition is met. Following this analogy, a simple compression policy could be specified as:
TRIGGER: compressibility of an object is > 50% ACTION: compress RECIPE: IF compressibility is > 50% THEN compress
IFTTT is becoming very popular and its creator announced in 2015 that IFTTT users were “cooking” about 20 million “recipes” every day. The major reason behind the enormous popularity of IFTTT is its extensibility and reusability. A simple model like IFTTT would be ideal for open SDS, as it would provide a simple means for administrators to write portable policies. For this reason, Crystal includes a meta-programming DSL in the control plane that has been inspired in IFTTT.
Poor extensibility
Extensibility at all levels of the architecture is also a fundamental principle. Like in IFTTT, an open SDS system pursues automation of the system management based on reusable “recipes”, that is, policies in SDS parlance. Consequently, the equivalent elements to triggers, actions and recipes should be ready for extension and reusability in an open SDS model. Extensibility at this level is not possible in today’s SDS systems. Crystal is trying to fill this gap. In practice, this includes opening two main hooks: Inspection hooks (triggers) and interception hooks (actions). As result, the IFTTT-like DSL language will be able to seamlessly grow with the creation and sharing of new triggers and actions over the storage system. It is not hard to imagine an external system developer implementing a bandwidth differentiation “action” along with a bandwidth inspection “trigger”, and based on that, a storage administrator writing a policy. Such a policy may be later shared with other storage administrators as a simple recipe to throttle IO bandwidth. The result will be the transparent extension of the underlying storage system.