Software Evaluation Based On Consumability

Abstract concept helps in practice for evaluation of monitoring software

Introduction

The term Consumability describes the user experience with a technical product over its entire lifecycle. It can be distilled into six subtopics:

Evaluation
Acquisition
Commissioning
Operation and Use
Troubleshooting
Maintenance

The concept of consumability was conceived by IBM for product development, but the idea never grew beyond that construct. But, it can be very helpful in the evaluation of a product. In the following, we want to show just how helpful it can be using the example of a small business evaluating network monitoring software. To do this we must first be clear about the importance of the software to be evaluated. Only so we can put the effort in relation to the benefits.

Example: Monitoring in mid-sized companies

Many large organizations have specialized teams in charge of individual areas of IT, such as virtualization systems or pure hardware administration. In medium-sized companies, however, the IT teams are smaller - sometimes even a single administrator - and are expected to be a jack of all trades. The central task of a monitoring solution in this case is to relieve the administrator or the IT team. For this purpose, the software needs to bring the widest possible options for monitoring the entire IT infrastructure, and ideally have a well-documented API to integrate non-standard applications and devices. It must be easy to implement and to use, and priced in the context of what a small business may invest. The market offers many monitoring solutions with different sales and licensing models, which have a major impact on single aspects of consumability:

Traditional Software License

You buy a software license from a dealer or directly from the manufacturer, and download the solution through the Internet. There is an ongoing annual maintenance fee, and you receive support and new versions/updates.

Open Source

Open-source solutions are available for free, with nothing required but the download. For most open-source tools, developer communities provide numerous add-ons that you can also use for free (mostly).

Hardware Appliance

The same sales model as the traditional software license but, there is also a specifically configured hardware component.

SAAS (Software As A Service)

Many monitoring solutions are offered as a service: You don't buy software anymore, but rent the monitoring you need. A service is installed in your network that collects the monitoring data and sends it to the provider's server. There the data is analyzed and the results published, usually through a web portal.

The market is still dominated by the traditional licensing model, so for the purposes of this article we will treat it as the standard, and compare the advantages and disadvantages of other models using that standard.

1. Evaluation

Any product evaluation needs to be taken seriously and requires extensive research, installation and testing. It is critical to ensure a product meets a number of key standards before the evaluation process begins, otherwise one can become caught in a loop of wasting weeks or even months evaluating products that were flawed or ill-fitting from the start. Here are several key questions that need to be answered prior to an evaluation:

• Is a trial version available and can be installed without major efforts?

• Is technical information in form of feature lists, screenshots, manuals, a knowledge base etc. freely available?

• Is technical assistance by the manufacturer or competent partners available?

• Are licensing and pricing understandable and transparent?

2. Acquisition

Even the seemingly simple purchase of a software can be unexpectedly complicated under certain circumstances. Licensing often forms the first hurdle. Many solutions are offered, which are made up of various components, for which numerous modules and add-ons are available. It is difficult to determine just what will actually be necessary in terms of additional software, and often can't be determined without consulting the vendor or a third-party - who often have their own revenue in mind. Price also plays an important role here. The more tools and modules offered, the higher the expected price. Many manufacturers reveal prices only on request. You send a request list and get an offer - with little ability to control the response.

Watch out for simple licensing models, a limited number of modules, fair pricing, and above all a lot of transparency: The more information available, the easier it is to purchase the software - with less hidden costs.

Acquisition is usually not a single action. Most commercial solutions require regular maintenance purchases to be eligible for updates and bug fixes. Networks grow, new devices and technologies are added, and you may need additional modules or upgrades. Consider possible future purchases when evaluating a software.

Appliance Solutions

Here the issue is hardware: In addition to all aspects already mentioned, the delivery of a physical appliance is added. In some cases, there may be taxes to take into account, transport accidents, delivery times or other delivery problems. There are also warranty issues or updates and upgrades that require significantly greater effort than a simple software download.

Open Source

With open-source solutions, the acquisition consists only of a download, license fees are not payable. A big plus, but which is mostly leveled by the next point, the commissioning.

SAAS

The acquisition of SaaS monitoring solutions is usually simple: The initial investment for the license that you may not even use completely, does not apply. Instead, you pay monthly service charges, adjusted to your actual requirements. The SaaS model saves on upfront costs, but will eventually exceed traditional software licenses in the long run.

3. Commissioning

The third phase in the introduction of a monitoring software is the most challenging. Ideally, a test installation implemented during the evaluation phase can simply be adopted and reused. But even then it often takes many steps until the software does everything it should. Here, open-source programs usually lose the advantage they have gained with the acquisition, by requiring enormous efforts for implementation and configuration. Classically licensed monitoring solutions range from simple, highly automated installation and configuration routines to complex software suites, whose start-up is only possible through complex projects.

Agentless vs. Agent Based

Many monitoring solutions are agent based, meaning they require the installation of a software agent on each device to be monitored. These systems are independent of standard protocols such as SNMP or WMI and can generate comprehensive data on the device itself. However, agent-based systems require high expenditure in commissioning and maintenance, and require labor-intensive work. When it comes to agent-based systems, automation is not the rule, and even when there are automated aspects, it still involves a lot of time consuming setup.

Agentless systems obtain their data through protocols such as SNMP, WMI, Flow, etc. Strictly speaking, they don't really work agentless, but use existing agents on the monitored devices. These must be configured accordingly, but this is usually significantly less expensive than setting up the native agents which can also generate additional load on the device to be monitored.

Appliance Solutions

Many appliance solution providers claim to ease the costs of commissioning significantly: The software is pre-installed on hardware or as a „Virtual Appliance", but with the operating system supplied, so you save the installation process. This takes away a little bit from the scare of open-source solutions, since even installation can be very expensive here. However, if the installation is so complex that the manufacturer undertakes it by appliance, you can expect corresponding complexity in configuration and operation.

For commercial appliances, it is mostly about meeting hardware requirements: The hardware is optimally matched to the requirements of the software, so that performance issues due to inadequate or incorrectly adjusted hardware can be avoided. For this purpose, the hardware used is usually correspondingly powerful and therefore expensive.

SAAS

Again the provider saves you the installation of a central software instance, you only need to set up the local service - which is generally quick and easy - and define the necessary approvals for the firewall.

Whether SaaS, hardware, or virtual appliance: the lion's share of work comes at the start, namely the configuration and set up of your custom monitoring scenarios. Here it is important that the software offers automation to relieve you of as much work as possible. That includes functions such as auto-discovery, templates for popular devices and applications, easily generated dashboards and maps, etc. Also, a well-documented API should be available, with templates for easy connecting of non-standard equipment and applications. As many administrators have only limited experience with monitoring, it is very helpful if the solution offers best practice-based suggestions for settings and thresholds. Also important in this context are aspects such as data storage, alarm, publication of the monitoring results or reporting. Many solutions require additional tools such as SQL databases, external reporting tools, mail servers or modules for alarming and data publication. Basically, the more the solution includes, the easier and faster the commissioning will be.

4. Operation and Use

Even after commissioning, usability of the software plays a big role. If monitoring software isn't easy to use, it may not be used at all. In the worst case, a solution with poor usability will be ignored by everybody and end up as a bad investment. A modern network is a constantly evolving organism: New devices and applications are being added, which in turn requires the support of new technologies, old devices are eliminated or replaced, and employees come and go. All this requires a constant adjustment of the monitoring solution. While a costly commissioning with assistance of the manufacturer or a service provider may still be done within a project, a complex and difficult-to-use monitoring solution in everyday life will be less and less cared for, and ultimately take over only partial aspects of the originally envisaged tasks.

Similar to the implementation, features like an automated network scan, pre-defined device templates or an API are also important for the operation of a software. In addition, the presentation of the data is of great importance: Are the monitoring results being presented in a clear and easily understandable manner? Contemporary graphs and charts provide a faster and better overview than tables. A modern web interface provides attractive presentation possibilities, allowing access at any time, from anywhere. Is historical data being stored and how are the access options? Some solutions save monitoring data in RAW format, i.e. in the original Monitoring interval, thus allowing precise research.

Open Source

As with the installation, the ongoing operation of open-source solutions is not simple. Usually these tools require profound Linux programming skills and require high efforts even for small adjustments. Although qualified service providers or employees can create very powerful and comprehensive monitoring solutions with open-source tools, the cost of this is usually much higher than for comparable commercial solutions. Additionally, open-source tools become "competency islands," where one highly skilled employee can serve as a massive point of failure for an entire system. A company can't afford to suffer a monitoring failure because its best Linux programmer left for a new job.

SAAS

Thanks to the outsourcing of the actual software maintenance to the manufacturer, most SaaS offerings are distinguished by a relatively good operability. You don't need to worry about how data is stored, published or presented - the vendor will take care of that along with alerting. Instead, you have little to no access to the central authority, which may prove to be a problem especially if individual enhancements are needed in order to incorporate your own applications or non-standard equipment in your monitoring. Also, a reliable and stable Internet connection is mandatory, otherwise there is a deterioration in the monitoring.

5. Troubleshooting

Monitoring solutions must face the challenge that IT infrastructures are extremely heterogeneous systems. Often, there is obsolete equipment in addition to the latest hardware, old, but still needed software solutions that have to be operated on new servers, or advanced security tools working with outdated equipment. No monitoring software vendor can take into account all possible configurations. Therefore, the experience of the manufacturer and the maturity of the product are key quality characteristics especially with monitoring software. The more customer feedback that has gone into the development of the monitoring solution, the longer it has been adapted and developed and the further it is spread, the greater the chance that you will get a tool that is sophisticated enough to monitor most of your IT in a stable and comprehensive manner, without confronting you with many problems.

Nevertheless: Almost every complex software can and will eventually need support. This may come in the form of questions about configuration or operation, requests for missing features, or reports of bugs in the software. Even during the evaluation of the software, it is important that you pay attention to comprehensive and readily available support offerings. Are manuals and other online support offerings available? Does the manufacturer offer competent e-mail or telephone support? Is a professional network of partners available, if you encounter problems that are not going to get readily at hand? Ask. Test the manufacturer's support during the evaluation process. That's the only way to get a reliable picture of its quality.

Appliance Solutions

Hardware appliances have specific demands on troubleshooting: As a general rule you cannot or must not fix hardware issues yourself. This means time-consuming and tedious backand-forth with the manufacture, or costly on-site service.

Open Source

While you can work with the manufacturer on larger problems with commercial software, for open-source tools there is only the community, or perhaps a few consultants, making reliable support difficult to obtain. But even in daily operation of open-source monitoring solutions, comprehensive scripting and programming skills are required. Although you will find comprehensive forums on the net for most open-source solutions, it is not always guaranteed that you find helpful answers there.

SAAS

First of all, the SaaS model offers an enormous advantage in terms of troubleshooting: You do not have to worry about problems in the central instance, since your provider is required to operate it for you. The unseen downside, however, is that you can't proactively solve problems yourself, even small ones.

6. Maintenance

Monitoring software must be continually developed: New technologies, new equipment and new applications create new requirements in continuously evolving networks. Experience shows that new versions of a monitoring software bring new bugs - and again, new versions to fix the bugs. In practice, this means regular updates that need to be implemented. It's a bonus if the software only requires an update of the central instance, and then updates all agents, probes and polling engines, etc. automatically. It's even better if the software regularly looks for available updates itself, and you only need to confirm the installation. The import or activation of new features in the form of modules or add-ons as well as upgrades to larger licenses should not require too much effort. It is important to note that many vendors are as invested in generating headline-grabbing features for their core products than they are in improved maintainability and stability.

Appliance Solutions

Hardware appliances in particular require a number of questions up front. Does the hardware still meet the requirements after an upgrade or the purchase of an additional module? Or does it have to be adjusted or replaced, and how costly or time-consuming is it? What maintenance does the hardware require? Does the manufacturer provide required firmware updates? Ask these questions in advance to make sure that no surprises are waiting for you.

Open Source

Similar to commissioning and operation, most open-source tools also have high demands for maintenance. Especially when the system operates agent-based, manual updates can be very costly. Also, new features in the form of plug-ins or modules can place high demands on both your programming skills as well as your work time.

SAAS

Solutions hosted by the manufacturer or service provider require considerably less maintenance, because you do not have to worry about the central instance of the software. Nevertheless, you should inform yourself of what is required for maintenance activities on the ground. Under some circumstances it makes sense to evaluate which options provides the solution with respect to possible extensions: Especially the binding of individual devices and applications can be difficult or even impossible in SaaS solutions, because you usually do not have access to the central instance of the software.

Conclusion

Consumability provides you with a comprehensive checklist that can help you in the evaluation of new products as well as in the appraisal of existing solutions. The main focus for the introduction of new software solutions is on features and price. Implementation and usability are usually also queried, but not as thoroughly in most cases. Troubleshooting and maintenance are often neglected and, until you realize how much effort the evaluation itself requires, you are usually already stuck. Take your time at the start of the evaluation and make a checklist based on the individual aspects of Consumability. You might find a list especially for the product to be evaluated, presented by an industry insider. That way, you benefit from expertise and possibly save yourself a lot of trouble and work.