PureApplication plugin transforms: From the application model to the topology model

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by John Hawkins on 27 August 2013 (7977 visits)

This post was written by John Hawkins, a Principal Consultant at Icon Solutions.

---

I’ve always found the complexity of the plugin model somewhat overbearing, so I figured it would be good for me (and hopefully others) to explain the aspect that comes up time-and-again: how does the model that the user draws on the canvas actually get translated into the underlying topology model for deployment to the hardware?

The plugins that you can see on the canvas are logically split into three main elements: the model, the packages and the transformers:

• The model contains what literally appears on the canvas of the PureApplication Pattern: the modelling artefacts, such as the icons that you want to represent your plugin and the other elements it can interact with.
  
• The packages contain the deployment and runtime functional code of the plugin, such as product installation and lifecycle management scripts.
  
• The transformers convert the application model (as drawn on the canvas) to the topology model (as deployed to the hardware) including the target location of packages.

There are three core sub-types that can be defined in a plugin:

• Components contain product and application-specific configuration; examples are the EAR application component and the DB2 component.
  
• Links represent a relationship between components (they are shown on the canvas as the blue lines literally joining components together).
  
• Policies are the real “secret sauce” of application patterns; they can contain levels of complexity impenetrable to all but the most dedicated system administrators! At present, not many policies have been implemented; the most obvious (and arguably one of the most complex), is the scalability policy. This policy allows the user a choice of four different load metrics to control the dynamic scaling of WAS; it’s a really powerful policy and hopefully one that IBM will continue to invest in, to broaden its reach to other middleware products.

When you draw the application diagram on the canvas, a JSON document (the application model) is being created under the covers; it is visible in the Virtual Application Builder under the Source tab. Assuming you’ve created a coherent model, containing artefacts linked together correctly, you should be able to deploy it, which is the procedure I’m interested in for the purposes of this blog.

As previously stated, the PureApplication System doesn’t deploy the application model directly because there is no information relevant to physical manifestation. For example, what follows is a snippet of the Web Application sample that comes with a PureApplication System:

"model": {
                  "name": "Sample Web application only", 
                  "nodes": [
                           {
                                    "attributes": {
                                             "archive": "artifacts/webappSampleWeb.war", 
                                             "ctx_root": "webapp", 
                                             "ignoreFailedIfix": true
                                    }, 
                                    "id": "Web Application", 
                                    "type": "WAR", 
                                    "groups": {
                                    }
                           }, 

This snippet shows how the Web Application component on the canvas (the node type of type “WAR” in the model) has been configured: the user has asked for the webappSampleWeb.war file to be deployed with a specific web context root of webapp.

This is where you begin to realise that the PureApplication System is a framework for the plugins it contains: the transformation of the application model to the topology model. It is the topology model that dictates the creation and configuration of Virtual Machines. PureApplication System doesn’t understand how to create the topology model itself; this is delegated to each individual plugin that has been placed on the canvas. The plugins are invoked in order: components first (such as WAS and DB2), followed by the links between plugins and policies.

The diagram below shows the logical flow and also introduces a new element of the puzzle – Kernel Services. Kernel Services is the orchestration engine of PureApplication Systems. For the purposes of this blog I shall just show what it does in terms of the transformers and not discuss what else it does in PureApplication systems (there is, in fact, another element – The Storehouse – which we will ignore for now).

Each plugin must declare a transformer, whose role it is to convert the application model into the topology model for deployment (1 & 2). Each transformer is given access to its element of the application model, including any user-specified attributes (e.g. EAR file name) and, in the case of a component, the output from the transformer is likely to include Virtual Machine requirements – OS, memory, etc. – as well as software packages and application elements.

We’ve already seen how the application model represented the WAR component; what follows below is just a small part of the topology model document for an instance of the “Sample Web Application Only” application pattern, that IBM supplies with a PureApplication System:

"vm-templates": [{
                  "parts": [{
                           "part": "https:\/\/127.0.0.1:9444\/storehouse\/admin\/plugins\/was\/2.0.0.5\/parts\/was.tgz",
                           "parms": {
                                    "installDir": "\/opt",
                                    "binaryFile": "was\/was-8005-linux32-20121115-product.tgz"
                           }
                  },
                                    ..
                                    ..
                                    ..
                  {
                           "part": "https:\/\/127.0.0.1:9444\/storehouse\/admin\/plugins\/wasfile\/2.0.0.5\/parts\/wasfile.scripts.tgz"
                  },
                                    ..
                                    ..
                                    ..
              "roles": [{                       "plugin": "was\/2.0.0.5",                       "parms": {                                "logLevel": "$$1",                                "ARCHIVE": "$$2",                                "USERID": "virtuser",                                "KEYSTORES_PASSWORD": "<xor>HjswCyUWbW8nHCw8Ng==",                                "restrictedDirectories": "\/root",                                "HTTPS_PORT": 9443,                                "RESTART_INTERVAL": 24,                                "extraLogDirectories": "\/var\/log\/clientlogs\/",                                "HTTP_PORT": 9080,                                "ctx_root": "webapp",                                "hugePageSize": 2048,                                "ignoreFailedIfix": "true",                                "PASSWORD": "$$3"                       },                       "depends": [                                ..                                ..                                ..                                ..                       {                                "parms": {                                         "extract_path": "\/webapp",                                         "external_archive": "https:\/\/127.0.0.1:9444\/storehouse\/user\/deployments\/d-280703eb-ebac-4ee7-b33c-36648f57576b\/artifacts\/page.zip"                                },                                "type": "WASFILE"                       }],                       "type": "WAS",                       "external-uri": [{                                "ENDPOINT": "http:\/\/{SERVER}:9080\/webapp\/"                       }],                       "name": "WAS",                       "requires": {                                "memory": 512,                                "cpu": 1,                                "disk": 300                       },                       "quorum": 1              },

(For any given deployment of an application instance, you can see the complete topology document – plus all the other JSON documents – in the storehouse browser. The Storehouse browser is located under the “System” tab in the PureApplication Systems web administration interface. The documents are held under User à Deployments à <deployment_id>.)

In the example above we can see that there is only one Virtual Machine being created (as there is only one vm-template element) and that there are a number of “parts” being deployed to that single VM (I’ve edited out many more parts that were irrelevant to this discussion!).

We can also observe that the “WAS” role has been put onto the VM; it was added by the WAR component transformer whose operation we have been examining. You will note that although the WAR file name was clearly visible in the original application model, it has now been substituted for the identifier “$$2”, which is a reference to another JSON file under the same deployed application tree: the parameter.json file. This contains the following entry, amongst many others:

"$$2":{"value":"https:\/\/127.0.0.1:9444\/storehouse\/user\/deployments\/d-280703eb-ebac-4ee7-b33c-36648f57576b\/artifacts\/webappSampleWeb.war","role":"Web_Application-was.WAS","id":"WAS.ARCHIVE"}

So, that’s enough about component transformers: we’ve now got an idea of the level of complexity contained therein!

Link transformers are slightly different; they are driven by the outputs of the transformers of the components that they are connecting, i.e. the components’ topology models. They will then augment one or both of the component topology models, to create the deployable manifestation of the connection between the components (it is unlikely more VMs would be created, although this is certainly not prohibited). For example, in the case of the WAR-to-DB2 link, the link transformer manipulates the output of the WAR transformer – the topology model that the transformer created – and adds a WAS resource adapter installation and accompanying configuration packages into the WAS VM role.

Policy transformers are different again; policies may require their own independent VM in order to host a centralized policy management service. Alternatively, they may just need to configure some code on their dependent component.

In terms of implementation, the transformers are OSGI components which implement the com.ibm.maestro.model.transform.TopologyProvider interface, either in Java code or by providing a template: both implementations have access to the same components of the model, and are called at the same time in the deployment lifecycle. I won’t discuss those here – perhaps in another blog !

In Summary: there is a proliferation of new and re-used terms around PureApplication System, and some of them don’t relate obviously to the problem statement. I’ve discussed just a tiny bit of the system here: the relationship between what the user draws on the canvas (the application model) and how the deployed plugins turn this into deployable entities (the topology model).