Made for Administrators, Designed for Developers
Next, we demonstrate the benefits of Crystal’s design by extending the system with data management storage policies and distributed control of IO bandwidth. These guidelines may serve others to further extend Crystal with new SDS services.
P1: FOR TENANT T1 WHEN OBJECT_TYPE=DOCS DO SET COMPRESSION ON PROXY, SET ENCRYPTION P2: FOR CONTAINER C1 WHEN GETS_SEC > 5 DO SET CACHING
Data plane (Interception): To enable such policies, we first need to develop the filters at the data plane. In Crystal this can be done either using the native or isolated execution environments.
public class FilterName implements IStorlet{
@Override
public void invoke (ArrayList<StorletInputStream> iStream, ArrayList<StorletOutputStream> oStream,
Map<String, String> parameters, StorletLogger logger) throws StorletException {
//Develop filter logic here
}
}
The code snippet shows how to develop a filter for our isolated execution environment. A system developer only needs to create a class that implements an interface (IStorlet), providing the actual data transformations on the object request streams (iStream, oStream) inside the invoke method. To wit, we implemented the compression (gzip engine) and encryption (AES-256) filters using storlets, whereas the caching filter exploits SSD drives at proxys via our native execution environment. Then, once these filters were developed, we installed them via the Crystal filter framework API to enable their automatic enforcement.
Repository: https://github.com/Crystal-SDS/filter-samples
Data plane (Introspection): To achieve dynamic policy enforcement requires from monitoring information. Thus, we instructed our introspection middleware to inject monitoring information of object requests (e.g., PUTs/GETs per second of a tenant) at individual queues of the MOM broker. Via the Crystal API (see Table 1), we also deployed several workload metrics (one per metric and target granularity) that aggregate such monitoring information to be published to controllers. Also, our filter framework middleware is already capable of triggering filters based on object metadata, such as object size (OBJECT_SIZE) and type (OBJECT_TYPE).
Control Plane: Finally, we registered both the filters and the workload metrics at the metadata store with intuitive keywords (e.g., CACHING, GET_SEC_TENANT) using the Crystal registry calls. To achieve P1, we also registered the keyword DOCS, which contains the file extensions of common documents (e.g, .pdf, .doc). At this point, we can use such keywords in our DSL to design new storage policies.