How to simulate the impact of a Compute Node failure on running VMs in PureApplication?

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Jonathan Deberdt on 16 January 2017 (5924 visits)

When designing and building patterns on PureApplication, most clients need to demonstrate the impact of hardware failure of a Compute Node on VMs running software. For example, what happens when the Compute Node hosting the primary DB2 VM fails? Deploying a pattern instance and pulling out a Compute Node from a PureApplication System in the data center is usually not an option (as suggested in this thread on the dw Answers forum). Instead you could try and simulate this by:

  1. Explicitly killing the process(es) on the VM.
  2. Issuing a “shutdown -h” from the shell within a VM.
  3. Issuing a “Power off” from the PureApplication UI for the VM.

The issue with (1) is that this is very specific to the actual software. When performing (2), the OS actually performs a graceful shutdown (so again not realistic). Option (3) looks like the best option, however we found that the command to “Power off” a VM actually send a signal to the OS to perform a graceful shutdown as well.

We will describe a mechanism here to perform a more “realistic” and abrupt “Power off” of the VM here. We can achieve this by making sure that this signal that is sent to the VM simply does not arrive or does not perform a graceful shutdown. Under the covers, VMWare Tools is installed on every VM in PureApplication (on Intel). VMWare Tools is running a set of services within the OS of the VM, which can be used to receive the call from VMWare to perform a graceful shutdown of the OS. In order to prevent this for testing purposes, we can simply stop the VMWare Tools services:

Note:    Please keep in mind that performing this simulation of an abrupt shutdown of a VM may lead to data corruption of the filesystem(s) within the VM!

-bash-4.1# /etc/vmware-tools/services.sh stop
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   VM communication interface socket family:               [  OK  ]
   VM communication interface:                             [  OK  ]

With the VMWare Tools services stopped, issuing a “Power off” from the PureApplication UI for the VM will effectively be an immediate shutdown. You can validate this by examining the file messages in /var/log after the “Power off”, it should show a message that recovery is required on a readonly filesystem:

Jan  5 08:11:29 pure-9-3-172-232 kernel: dracut: Scanning devices sda2  for LVM logical volumes vg_root/LogVol00 vg_root/LogVol01 
Jan  5 08:11:29 pure-9-3-172-232 kernel: dracut: inactive '/dev/vg_root/LogVol01' [2.00 GiB] inherit
Jan  5 08:11:29 pure-9-3-172-232 kernel: dracut: inactive '/dev/vg_root/LogVol00' [9.75 GiB] inherit
Jan  5 08:11:29 pure-9-3-172-232 kernel: EXT4-fs (dm-1): INFO: recovery required on readonly filesystem
Jan  5 08:11:29 pure-9-3-172-232 kernel: EXT4-fs (dm-1): write access will be enabled during recovery
Jan  5 08:11:29 pure-9-3-172-232 kernel: EXT4-fs (dm-1): recovery complete
Jan  5 08:11:29 pure-9-3-172-232 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: 
Jan  5 08:11:29 pure-9-3-172-232 kernel: dracut: Mounted root filesystem /dev/mapper/vg_root-LogVol00

Another approach here would be to simply uninstall the plugin from VMWare Tools that facilitates the call from VMWare to the OS for a graceful shutdown. The name of this module is vmware-tools-plugins-powerOps, and you can simply uninstall it using the following command:

yum remove vmware-tools-plugins-powerOps

The advantage of this approach is that you will have the remaining VMWare Tools modules still in place, which in general optimise performance of the OS running inside the VM. Should you wish to re-install the module vmware-tools-plugin-powerOps, you can simply download it from here. Note that you must ensure to download the correct version, i.e. it should match the OS and VMWare ESXi version used. For PureApplication System 2.2.2, ESXi 6.0.0 is installed and you can find the vmware-tools-plugin-powerOps for RHEL6 here. Older versions of PureApplication System use ESX 5.1.0 update 3, you can find the vmware-tools-plugin-powerOps for RHEL6 here.

Note: You can determine the version of ESX by performing a REST GET call to https://<PSM>/admin/resources/hypervisors/. Here you will find the ESX version installed on each Compute Node:

software_version": "VMware ESXi 6.0.0 build-3620759",

You can find more information about the VMWare Tools Operating System Specific Packages here.

DB2 1.2.4.2 pattern type now also available from IBM Fix Central

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 9 January 2017 (6074 visits)

PureApplication System and Bluemix Local System 2.2.2.x come with the DB2 with BLU Acceleration Pattern version 1.2.4.2, as documented in System maintenance and integrated components in PureApplication System Version 2.2.2.1. Most of you are probably familiar with this pattern, which enables fast and easy deployment of DB2 10.5 and is also used by a number of other patterns. Typically IBM installs the corresponding pattern type that contains the DB2 pattern. However since today, this pattern type can now also be downloaded directly from IBM Fix Central here. Please refer to this previous blog post for more details on how to install this pattern. This post also includes a handy link that takes you straight to all versions of the DB2 with BLU Acceleration Pattern available from IBM Fix Central.

For reference, this version of the pattern type delivers a fix for APAR IT16492, which is also mentioned in System maintenance and integrated components in PureApplication System Version 2.2.2.1. What’s perhaps even more important is that the DB2 software component no longer always install the TSM client binaries. With previous versions of the pattern type, this was always done. However with this version, the TSM client binaries are only installed when the tsmclient System Plug-In has been fully configured (under Catalog > System Plug-Ins).

Deployment of RHEL 7.1 VM using GPFS client requires Red Hat Satellite Server integration

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 6 January 2017 (5864 visits)

I was working with a client this week where we discovered that some pattern deployments using a GPFS client policy failed. Upon investigation, it turns out that the IBM OS Image for Red Hat linux Systems 3.0.4.0 (RHEL 7.1) does not contain a particular package required by the GPFS client (kernel-devel-3.10.0-229.14.1.el7.x86_64). As we did not yet have Red Hat Satellite Server integration in place, the installation of this package fails which in turn makes the GPFS client setup and pattern installation fail. See below for a snippet from /opt/IBM/maestro/agent/usr/servers/<servername>/logs/install/trace.log:

Install GFPS Prereqs
There are 1 parameters given
Install GFPS Prereqs
Check rpm dependencies ...
Check rpm kernel-devel-3.10.0-229.14.1.el7.x86_64 ...
Install rpm kernel-devel-3.10.0-229.14.1.el7.x86_64 ...
Could not install rpm package kernel-devel-3.10.0-229.14.1.el7.x86_64 via yum. The Red Hat shared service is likely not running.

The 3.0.6.0 version of the OS image from IBM (which is based on RHEL 7.2) does not have this issue, so that is probably the most straightforward solution. Of course it is strongly recommended to have Red Hat Satellite Server integration in place, but sometimes it can take a little longer to do so. Finally we also performed some tests with RHEL 6.x OS images from IBM, those all worked fine. The table below summarises our findings.

Virtual ImageRHEL versionRequires Red Hat Satellite Server for GPFS client policy
IBM OS Image for Red Hat Linux Systems 3.0.4.07.1Yes
IBM OS Image for Red Hat Linux Systems 3.0.6.07.2No
IBM OS Image for Red Hat Linux Systems 2.1.5.06.7No
IBM OS Image for Red Hat Linux Systems 2.1.7.06.8No

How do I setup new or renew Red Hat Registration files with Red Hat Satellite Server?

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 4 January 2017 (5899 visits)

I just came across this excellent blog post here from my colleague Tim Elwood. He describes how to setup new or renew Red Hat Registration files for deployment of RedHat Satellite Server. Some of this was already documented in sections 7.2.2 and 7.2.3 of this Redbook, but hist blog post is more recent and contains detailed instructions in particular on how to register your entitlements with Red Hat.

Cognos BI v11 – how to turn your installation into a success

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Tom Bal on 21 October 2016 (8143 visits)

Read this blog post if you wanted to install the latest Cognos BI v11 pattern on any IBM PureApplication environment – including IBM Bluemix Local System – and ran into an issue deploying it.

During the installation of the pattern, the required installation binaries are repackaged and uploaded into the IBM PureApplication’s storehouse. At deployment time, these files are extracted, stored on an NFS server which is part of the pattern, pulled over to the different virtual machines and executed respectively. Because of a difference in the anticipated file name and the actual one, this installation fails after the initialisation of the NFS server.

The rest of this article assumes you have read the Cognos BI v11 installation guide available here, and still have the installation binaries and the extracted installation files available on the system from which you installed it previously, or want to install it.

To address this installation issue, it’s sufficient to change the name from:

ca_srv_lnxi38664_11.0.4.16092813.tar.gz

into:

ca_srv_linuxi38664_11.0.4.16092813.tar.gz

if you have tried this upload before, the wrong installation files are already in your storehouse. These are to be removed prior to rerunning the upload with the changed file name.

To delete items from the storehouse, you could use the curl command or a browser plugin such as Poster.

Sample syntax for curl:

curl -i -u : -k -X DELETE https:///storehouse/admin/files/cognos/bi/11.0.4/

Sample screenshot with Poster where I remove a test entry:

You can identify the files to be removed by going to the storehouse in the browser, opening the admin, files, cognos, bi, 11.0.4 section. When you have deleted the file(-s), fresh the browser window (F5).

To be absolutely sure the installer overwrites already existing parts, from the installation directory into the PureApplication environment, look for an installPatternType.py and (optionally) change this line:

-----------------------------------------------------------------------------------------------------------
 Install the BI Pattern Type
 -----------------------------------------------------------------------------------------------------------
 print 'Installing pattern type "%s". Please wait…' % BIPatternFileTgz
 print
 deployer.patterntypes.create(BIPatternFileTgz)

into:

deployer.patterntypes.create(BIPatternFileTgz,True)

After you have removed those files, changed the installPatternType.py and have changed the file name, relaunch the installation program. Some 20 minutes later, the Cognos BI v11 pattern is ready to use.

Finally, create the database according to the steps described in the installation guide (see above) and deploy the pattern.

As always, if you run into an issue with an IBM pattern, please do not hesitate to open a PMR with IBM Support.

IBM® WebSphere® Application Server Patterns for PureApplication 2.2.2.0 support WebSphere 9.0

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 13 September 2016 (6634 visits)

Amongst other things, IBM shipped WebSphere 9.0 patterns with PureApplication 2.2.2.0 firmware. This is great news for clients who are keen to start working with the latest version of WebSphere, but I wanted to highlight something that has changed in terms of compatibility.

Since PureApplication 2.2.0.0, the WebSphere and Liberty pattern types are more restricted in terms of the minimum firmware version of PureApplication that they support. The good news is that it is easy to tell what minimum firmware version is required, it basically needs to match the version of the pattern types. You can find an overview of all pattern type versions here, including the minimum supported firmware, list of included APARs and links to download the pattern types themselves. An overview of what is new in each of the different versions of the patterns can be found in the WebSphere Application Server Patterns Knowledge Center here.

So in conclusion, you must upgrade to PureApplication 2.2.2.0 or higher in order to start using the latest pattern types that support WebSphere 9.0. Of course the latest version of these pattern types continue to support WebSphere 8.0 and 8.5.5.

How to use your own OS image from VMWare on PureApplication

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 28 July 2016 (6793 visits)

I recently worked with a client who was had to bring their own Windows OS image from VMWare onto PureApplication System. They exported the OS image as an .ova file from VMWare, but could not get it to work. There is some confusion in terms of the process that should be used, so I put together the different options to import an .ova file into PureApplication below. The key message here is that from PureApplication 2.2.1.0 onwards, option (3) is the only supported and recommended approach today.

  1. Export .ova from VMWare and import as virtual image
    Even though you can import the actual .ova as a new virtual image, the imported virtual image will not have the  IBM Activation Engine installed. As a result, you will not be able to use this virtual image for your pattern deployments.
  2. Use IBM Image Construction and Composition Toolkit (ICCT) to import .ova from VMWare
    ICCT has been around for many years and can directly import a virtual machine as a virtual image and install this in PureApplication. However the use of the IBM OS Toolkit has been the preferred mechanism for some time now. From PureApplication 2.2.1.0 onwards, ICCT is no longer supported.
  3. Export .ova from VMWare, import as Virtual Appliance and use IBM OS Toolkit to create virtual image
    This is the recommended and supported approach. The use of the IBM OS Toolkit is required in order for the IBM Activation Engine to be installed in the image. It will also help ensure that the OS meets the requirements for the IBM Activation Engine. You can find more details about the IBM OS Toolkit in the Knowledge Centre for for Linux and Windows.​ The diagram below illustrates the process.
    Finally a word of warning. You should always download the IBM OS Pattern Kit from the links in the PureApplication console, as shown below. An old version of the IBM OS Pattern Kit is also available here on ibm.com, but you should avoid using that. However you will need the metadata OVA file file from that link!

Finally a word of warning. You should always download the IBM OS Pattern Kit from the links in the PureApplication console, as shown below. An old version of the IBM OS Pattern Kit is also available here on ibm.com, but you should avoid using that. However you will need the metadata OVA file file from that link!

Syslog integration with IBM PureApplication System

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 22 July 2016 (5148 visits)

Since its inception, IBM PureApplication System has provided the ability to send audit events to an external system. This was done through scp, so the external system had to be running running an ssh daemon. One of the drawbacks of this solution was that audit logs were not immediately sent to the remote server. Events were sent in batches, which means that some information could be “in transit” for some period of time. A number of clients preferred an approach whereby audit information would be sent synchronously to an external system, typically a Security Information and Event Management (SIEM) solution such as IBM QRadar.

IBM PureApplication System 2.1.2.0 and higher now support this through syslog integration. This allows audit logs as well as other logs to be sent to an external system running a syslog daemon. Syslog can process logs in a synchronous fashion, whereas at the same time it is a widely accepted standard for doing so.

The above has been documented in the IBM Knowledge Center here, although confusingly the topic is referred to as Configuring logging settings for IBM QRadar. Keep in mind that IBM PureApplication System uses syslog and therefore supports a wide variety of SIEM solutions including (but limited to) IBM QRadar. The diagram below shows what the configuration looks like. While logged on to the IBM PureApplication System console, go to System > System settings and expand Log Management. There you can specify the FQDN or the IP address of your external syslog server and optionally apply a configurable maximum retention policy for all log files on IBM PureApplication System. You can select one or more sets of log fies, as shown below.

Known limitations for the IBM MQ 8.0.0.4 pattern

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 4 July 2016 (5244 visits)

In this recent blog post, we referred to an excellent article on how to deploy a highly-available IBM MQ and IIB environment on PureApplication. We though it would be helpful to point out some of the known limitations in the MQ 8.0.0.4 pattern.

One of the most relevant limitations is that on PureApplication 2.1.2.0 and higher, the MQ 8.0.0.4 pattern always installs MQ 8.0.0.2 (even if you chose MQ 8.0.0.4). This is a known problem, APAR IT15252 has been provided by IBM to address this (and a fix is available from IBM Fix Central).

IBM also published a Readme for IBM MQ V8.0 and its maintenance, which highlights a number of additional limitations and best practices for the MQ 8.0.0.4 pattern on PureApplication. We highly recommend going through these as this could save valuable troubleshooting time.

Deploying a highly-available IBM MQ and IIB environment on PureApplication

Originally posted on IBM Developer blog “Exploring PureApplication System, Software Service and more”  by Hendrik van Run on 23 June 2016 (5933 visits)

I recently came across the following article from my colleague Dave Arnold in Australia. He has built a set of patterns to implement a highly-available MQ and IIB environment on PureApplication. Dave used the MQ 8.0 and IIB 10.0 software components from IBM as a starting point and added customisations on top. This is a great example of what you can do with PureApplication patterns, please refer to “Need a Highly Available Messaging and Integration Bus in 30 minutes at the press of a button?” on the IBM Messaging forum for more details.

Design a site like this with WordPress.com
Get started